VMware ESXi Storage Server/SAN

[CyberTron]

Hey guys, i'm in the process of researching a project I'd like to complete in the next few weeks, and I'm having a bit of a hard time finding all the info I need.

In short, I'd like to build an ESXi server for the purpose of testing SQL clusters, 2008 server clusters, AD, etc, etc. Basically a learning lab. I can see running 8 machines at the same time at any given point. I'd also like the option to sell this setup to a client down the road if I find myself not using it enough. So it needs to be semi-enterprise class. So all of this will be rackmount. Please understand that I'm not looking to build a $10k server, but something middle of the road.

My two basic options are to create a single box with maxed out cpu/memory/HDD(raid,etc), and the other is to create a box with all HDD's, plus a decent ITX mobo to serve as an iSCSI SAN. (if that would even work).

I'm leaning toward a SAN + ESXi server though because down the road it could work out to be a HA ESX server config for a client.


For the ESXi server, I'll get some kind of quad core cpu, 16gb ram, etc, etc.
For the SAN side of things, I'm thinking 3x Supermicro sata/sas enclosure with a total of 15 drive bays. This san will also be my storage box (currently a 4TB 2U box). So I'm thinking 6x 1TB or 2TB in RAID5 or 6. I can handle all of those details.

The details I'm not so familiar with are how to configure the VMFS LUN/Array. Fiber Channel is out of the question, it's just way to expensive, and SAS is up there, but 4x 147GB would be doable. or possibly 4x SSD. Both of these would be in RAID10 (?) to gain the most performance out of them. The questions I have are all about performance for these 4 or so drives in whatever type of raid will get me the best performance.

Can anyone suggest what would be the best way to build this myself? I understand ESX(i) has very strict hardware requirements, but I plan on following the HCL as well as others experience.

 

[Emulex]

run esxi free on the san - get the lefthand VSA 60 day trial - it will continue to work as a SAN only AFTER 60 days - create/serve iscsi - its a vmware appliance.

it's by far the best choice you'll find for free. lefthand can use ram for caching and runs on a linux like o/s - so no windows overhead like starwind free trial. 98% of the ram is used for caching.

Sata really sucks for vmfs - 12 7200rpm drives in raid 1+0 is ALOT slower than 12 450gb SAS 15K drives in raid-5 . The huge cache on lefthand vsa helps this realization.

ram is going to kill you - you need lots of it. i run out with 32gb all the time - 8 servers.

 

[CyberTron]

run esxi free on the san - get the lefthand VSA 60 day trial - it will continue to work as a SAN only AFTER 60 days - create/serve iscsi - its a vmware appliance.

it's by far the best choice you'll find for free. lefthand can use ram for caching and runs on a linux like o/s - so no windows overhead like starwind free trial. 98% of the ram is used for caching.

Sata really sucks for vmfs - 12 7200rpm drives in raid 1+0 is ALOT slower than 12 450gb SAS 15K drives in raid-5 . The huge cache on lefthand vsa helps this realization.

ram is going to kill you - you need lots of it. i run out with 32gb all the time - 8 servers.

From your sig... I love me some Corrado's as well... sweet cars indeed.

I'm rethinking this whole project because I'm seeing that going production quality equipment is going to kill me and be way more expensive than I had intended.

Instead I'm thinking more along the lines of a single machine with an array for VM's, and an array for storage of my personal stuff. This whole thing has stemmed from me wanting to ditch my 3rd box, and upgrade/replace my 2nd box. So if I'm going with 1 box, I dont need lefthand VSA I assume. Now I just have to make sure the hardware is going to work with ESXi... picky picky stuff.

Can you foresee an issue running 8 test machines (ie: nothing really running on them but testing cluster failover, etc, etc) and someone watching a BR DVD streaming from the other array?

 

[pjkenned]

Honestly, just buy a real server motherboard for ESXi. I stopped trying to buy cheaper Asus/ Gigabyte motherboards a while ago in favor of Intel and Supermicro boards.

 

[CyberTron]

Honestly, just buy a real server motherboard for ESXi. I stopped trying to buy cheaper Asus/ Gigabyte motherboards a while ago in favor of Intel and Supermicro boards.

You are actually the man I've been wanting to get in touch with. I saw you posts from [H] and really like your website. Great information. I actually thought about the gigabyte mobo/config you setup as my SAN (running linux + iSCSI or something like that) and then maybe a xeon for the ESXi parts, but yeah, I completely agree. My first server for a client was a built PC with desktop parts, and I regretted it. I'll browse your site for serve board suggestions, but I'm trying hard to keep the noise down, so those chassis wont work for me I dont think. I currently have 9 disks running in my room and my hope is to load everything onto an array of 6x2TB and then 4x(whatever works best for VM's).

It sounds like everyone is suggesting SAS... so if that just wont work for me (heat, cost, etc) then I can always go with a SATA/SAS backplane and have the option of easily upgrading in the future. I really wish I could find a quiet 2U or 3U box though, that I could host at a DC pretty easily.

Any comment on the heat/noise that your systems put out?

*edit* btw, what is the deal with WHS... I really want to use enterprise stuff.. why do people seem to like it so much? (I have a technet sub, so licensing is no issue for me)

 

[pjkenned]

Thanks for the kind words. You can always PM/ e-mail/ contact form me whenever. Here are some thoughts:
1. I'm working on doing a lot more motherboard reviews. Next 7 days should be the Supermicro X8ST3-F which has 14 SATA ports (8 are SAS on a super compatible LSI 1068e)... and you can run a HP SAS Expander off the onboard LSI. If you saw my new JBOD/ SAS Expander enclosure article about 4 pictures down I show the modified PCMIG board. It is next to my noise solution, a fan controller. If you don't want the RPC-4220 to be loud, you basically need one. I replaced all the fans in my original WHS (in a RPC-470) with S-Flex's, but the fan controller set to 70-75% is much less expensive. Either way 4U or any server chassis is going to be loud.
2. I've used about 5 different types of SAS backplanes over the past 18 months. They are all pretty loud.
3. If you don't need a ton of storage, you can use 2.5" 36GB SAS drives. They are loud, but really cheap for lots of spindles.
4. WHS is just super compatible (basically if hardware works with XP, it works with WHS), super easy to use (windows UI + enhancements), has media center integration, software (NTFS file system) duplication, deduplication for files, and awesome backup software. Not the best for ESXi though.
5. In a 2U you basically want 2.5" SAS. 3U there are things like the RPC-3216. I've never used it, and it is probably loud, but may be worth a look.
6. If you don't want hotswap, you could just get a huge tower, and throw 4-in-3 mounts in the 5.25" bays. Using 120mm fans things would stay quiet.
7. Heat: lots, but I have 40 drives running, 3 CPU's, 3 motherboards, tons of add-in cards, 3 PSUs and etc all running. CPU's aren't that bad, especially the Clarkdale and Lynnfield CPUs.
8. Get IPMI 2.0 with KVM over IP. I'm testing a Intel S3420GPLC motherboard right now and the BMC doesn't have this standard like the Supermicro boards. It is driving me nuts.

 

[CyberTron]

Thanks for the kind words. You can always PM/ e-mail/ contact form me whenever. Here are some thoughts:
1. I'm working on doing a lot more motherboard reviews. Next 7 days should be the Supermicro X8ST3-F which has 14 SATA ports (8 are SAS on a super compatible LSI 1068e)... and you can run a HP SAS Expander off the onboard LSI. If you saw my new JBOD/ SAS Expander enclosure article about 4 pictures down I show the modified PCMIG board. It is next to my noise solution, a fan controller. If you don't want the RPC-4220 to be loud, you basically need one. I replaced all the fans in my original WHS (in a RPC-470) with S-Flex's, but the fan controller set to 70-75% is much less expensive. Either way 4U or any server chassis is going to be loud.
2. I've used about 5 different types of SAS backplanes over the past 18 months. They are all pretty loud.
3. If you don't need a ton of storage, you can use 2.5" 36GB SAS drives. They are loud, but really cheap for lots of spindles.
4. WHS is just super compatible (basically if hardware works with XP, it works with WHS), super easy to use (windows UI + enhancements), has media center integration, software (NTFS file system) duplication, deduplication for files, and awesome backup software. Not the best for ESXi though.
5. In a 2U you basically want 2.5" SAS. 3U there are things like the RPC-3216. I've never used it, and it is probably loud, but may be worth a look.
6. If you don't want hotswap, you could just get a huge tower, and throw 4-in-3 mounts in the 5.25" bays. Using 120mm fans things would stay quiet.
7. Heat: lots, but I have 40 drives running, 3 CPU's, 3 motherboards, tons of add-in cards, 3 PSUs and etc all running. CPU's aren't that bad, especially the Clarkdale and Lynnfield CPUs.
8. Get IPMI 2.0 with KVM over IP. I'm testing a Intel S3420GPLC motherboard right now and the BMC doesn't have this standard like the Supermicro boards. It is driving me nuts.

Yeah, I was going to PM you or comment on your blog, but I just got busy/lazy. But I'm ready to start buying stuff for this and other projects, so I hope to start moving forward this week.

As for your PCMIG board, that's a great idea.. so all it does is power that PCIe port? And one thing I'm still trying to understand is, with a JBOD, you can then select disks to create multiple (or a single) array(s)? I've never understood what job was... if it was seen as a single drive, or how it worked.

So by plugging your drives into that PCMIG, you have only 1 motherboard/OS install for that whole configuration? I've read your post, but you've got a lot of stuff going on, and I didnt follow all of it. I understand the concept of an external bay of drives, with a SAS cable in between... so from your pictures and comments, it looks like you have 2 machines running all those disks?

A Fan controller is one of my next items to purchase actually, that will be very useful. As for these SAS 36GB drives.. I'm seeing a couple for around $50ea on ebay, is that what you are referring to? I'm thinking I'll run 2-3 drives in RAID0 since nothing will need to be critical and I'm just looking for performance. But is there a difference from 2.5" drives to 3.5"? They are showing up around the same price. I'd like to get a SAS/SATA cage so that I can interchange them easily as I purchase more sas or sata.

Can you give me some more detail about the IPMI 2.0 stuff? I've see boards with it, but a lot of times it's an add-on or something. What kind of KVM would I need for this? I'm currently using an 8 port Avocent w/ PS/2 (k/m) and VGA. I have learned my lesson in the server mobo department, so I'll definitely go with that, but I did like your gigabyte board as the SAN controller, but like you'd I'd want the hardware to be doing something other than powering a raid card, etc. So my box-of-disks will likely be my sabNZBd box.

So now I just have to decide if I use Xeon or i3-5-7 for the VMware server.

 

[pjkenned]

That's exactly what the PCMIG board does. It powers the HP SAS Expander. All of the drives in the second enclosure, the RPC-4220, (well I now dedicate 4 to the test machine) go through the PCMIG board + HP SAS Expander to the Areca 1680LP in the main system (RPC-4020). The Core i7 920 running Windows 2008 Server R2 + that Areca 1680LP manages all of the disks. The second enclosure (PCMIG + HP SAS Expander + RPC-4220) has no internal raid functions, so the raid controller just sees the disks there as a bunch of disks connected. So basically, one machine with 30+ drives connected, then I'm using spare real estate for a second test system.

The SAS Drives $50 is pretty expensive TBH. I think I sold 8 of them (used for over a year) for $20 each and I bought them for about $25 each. 2.5" drives tend to have lower sequential read/write speeds but better random performance because of the rotating platter geometries.

I'm really really liking the X34xx based chips. I have a X3440 and now a X3460. The X3430 and X3440 are at great price points for what they offer. The nice thing with the LGA 1366 based chips though (my main server uses one) is that you get 6 memory channels.

On the IPMI 2.0, that's a next weekend writeup See the Supermicro X8SIL-F review for a mini IPMI 2.0 writeup and some screenshots. Basically, you have a web interface that runs from a browser after you navigate to the server's management IP. From there you can do all of the power control, add network iso's as mounted CD/DVD/ Floppy media, and etc. Also you have a JAVA app that gives you the KVM over IP features.

Hope that at least helped a bit.

 

[CyberTron]

Reading about IPMI 2.0.... I'm going to ditch my KVM and get all my new machines with this stuff. I cant stand all these cables.

One thing I'm interested in knowing... the Norco 4220 and 4020... do either of those support SGPIO? I currently have a server that does not, and it would be really nice to have this feature. I know the 3ware web app has a check box to identify each drive, but it doesnt work.

 

[Tristor]

I'd recommend you look into either OpenFiler or Nexenta. Both should be capable of running on a Mini-ITX or MicroATX type solution as long as it has at least 4GB of RAM (8GB preferred). Nexenta is based around an OpenSolaris kernel with Debian userland on top of it running ZFS as the backend which gets you block level dedup and Z-RAID, as well as disk pooling. It also natively supports iSCSI.

OpenFiler is built around rPath Linux and supports multiple filesystems, iSCSI, CIFS/SMB, and has Kerberos/Active Directory integration support. Additionally OpenFiler has a very nice WebUI. However, OpenFiler does NOT support deduplication.

There is a third distro called OpenDedup which supports iSCSI and deduplication which is based around Linux with a customized filesystem called SDFS, there are Debian packages available. Unfortunately OpenDedup has not yet released a 1.0 stable release, and is also using a new filesystem that has not been fully vetted, although they claim it is production stable.

I've personally used all 3 in testing, and am currently working for a client on making a decision about building their next SAN. They are looking at building a DIY multi-array SAN with around 300TB of capacity. Nexenta is looking to be the best way to go, as it has superb hardware support and ZFS is quite frankly the best filesystem available for serious storage applications.

Do some reading, but I think you will find the above 3 solutions are the best free solutions available, and they have the ability to scale down as well as scale up extremely well, which means you can start small in a lab environment but still have a decent setup which is saleable later for someone who wishes to expand by adding additional disk arrays/controllers.

I would recommend you consider still building around a server board/platform (EATX or larger) which supports enough expansion slots that controllers for externally attached arrays could be added a later date by your future client.

Additionally it may of be interest to know that if you want to maintain ESX HCL compatibility that only certain SAN software providers and drive controllers/mobos are supported. Nexenta as a SAN provider is supported on the HCL, AFAIK.

 

[CyberTron]

I'd recommend you look into either OpenFiler or Nexenta. Both should be capable of running on a Mini-ITX or MicroATX type solution as long as it has at least 4GB of RAM (8GB preferred). Nexenta is based around an OpenSolaris kernel with Debian userland on top of it running ZFS as the backend which gets you block level dedup and Z-RAID, as well as disk pooling. It also natively supports iSCSI.

OpenFiler is built around rPath Linux and supports multiple filesystems, iSCSI, CIFS/SMB, and has Kerberos/Active Directory integration support. Additionally OpenFiler has a very nice WebUI. However, OpenFiler does NOT support deduplication.

There is a third distro called OpenDedup which supports iSCSI and deduplication which is based around Linux with a customized filesystem called SDFS, there are Debian packages available. Unfortunately OpenDedup has not yet released a 1.0 stable release, and is also using a new filesystem that has not been fully vetted, although they claim it is production stable.

I've personally used all 3 in testing, and am currently working for a client on making a decision about building their next SAN. They are looking at building a DIY multi-array SAN with around 300TB of capacity. Nexenta is looking to be the best way to go, as it has superb hardware support and ZFS is quite frankly the best filesystem available for serious storage applications.

Do some reading, but I think you will find the above 3 solutions are the best free solutions available, and they have the ability to scale down as well as scale up extremely well, which means you can start small in a lab environment but still have a decent setup which is saleable later for someone who wishes to expand by adding additional disk arrays/controllers.

I would recommend you consider still building around a server board/platform (EATX or larger) which supports enough expansion slots that controllers for externally attached arrays could be added a later date by your future client.

Additionally it may of be interest to know that if you want to maintain ESX HCL compatibility that only certain SAN software providers and drive controllers/mobos are supported. Nexenta as a SAN provider is supported on the HCL, AFAIK.

I've had mixed results with zfs... that was what I initially used for my 4TB server that I'm now using hardware raid5 on. I was really annoyed by the writing bursts that would take place. I'd get like 70MB/s and then 30MB/s, and so on and so forth. I guess I need to build a setup that will handle either situation and if Nexenta doesnt work for me, I can always start over. If I use Nexenta, I can forgo the $$$ raid controller though right?

 

[CyberTron]

Hope that at least helped a bit.

I also noticed you have 10 Giga nics on your server system... I'm curious, how would one maximize the speed of all those drives? are they all in 1 big array? and if so, do you access, say.. a windows share over a balanced nic configuration?

 

[Emulex]

10 gbe would be so nice. just run mpio over two links instead of 4 or 6 iscsi gigabit.

The only suck part is the 1GB of ram loss per port per 10gb nic (usually dual port). that bites. 2GB down the drain.

 

[pjkenned]

I also noticed you have 10 Giga nics on your server system... I'm curious, how would one maximize the speed of all those drives? are they all in 1 big array? and if so, do you access, say.. a windows share over a balanced nic configuration?

Actually this may sound dumb, but every VM I run, has its own NIC. Some have 2-3. Also 10 is pretty conservative since I often have two more. I've just found that making sure each VM has its own NIC simplifies my life. At <$50/ port, it is a fairly inexpensive luxury.

The big negative is that it requires a bit of organization to remember which NIC (or NICs) are paired to a given VM. Another side effect is that you end up using bigger switches because aside from the main data NICs you also have one 10/100 Realtek NIC for IPMI 2.0 and another NIC for Areca out of band management.

And Emulex... 10 gigabit NICs are too expensive right now especially since you need to terminate them on another end. Corresponding switches are fairly expensive also. It is one of those things that will happen in due time though.

 

[classy]

Well 10K is really light if you want to go with a SAN and another server. I would skip the san and purchase a very good 2u server. For several reasons, one its for lab work. Two you want the option of selling it maybe down the road. Three if you want to do HA ESX you are going to need two hosts and a san. And the cost for iscsi software ain't cheap. There are some decent free ones, but for your project I would buy one good single box. 5K should buy you one seriously good server. I would just check the vmware compatability list, but 4 supports a lot of the newer boxes. You got Dell, HP, and IBM is what I would choose from.

 

[CyberTron]

Actually this may sound dumb, but every VM I run, has its own NIC. Some have 2-3. Also 10 is pretty conservative since I often have two more. I've just found that making sure each VM has its own NIC simplifies my life. At <$50/ port, it is a fairly inexpensive luxury.

The big negative is that it requires a bit of organization to remember which NIC (or NICs) are paired to a given VM. Another side effect is that you end up using bigger switches because aside from the main data NICs you also have one 10/100 Realtek NIC for IPMI 2.0 and another NIC for Areca out of band management.

And Emulex... 10 gigabit NICs are too expensive right now especially since you need to terminate them on another end. Corresponding switches are fairly expensive also. It is one of those things that will happen in due time though.

Sorry for not scouring your website for this answer, but what NICs do you run? I have a 64bit intel pro GT card, but will obviously need something new for this config.

 

[pjkenned]

BTW Add EON ZFS Storage to Tristor's list of you go that route. I have been pretty happy with it thus far.

In the main box I use the two onboard Intel NICs + 2x Intel Pro/1000 PT Quads + (sometimes) the Pro GT 32-bit PCI cards when I need extras. I have stopped using anything other than Intel NICs at this point in any of my machines.

 

[yinan]

Actually this may sound dumb, but every VM I run, has its own NIC. Some have 2-3. Also 10 is pretty conservative since I often have two more. I've just found that making sure each VM has its own NIC simplifies my life. At <$50/ port, it is a fairly inexpensive luxury.

The big negative is that it requires a bit of organization to remember which NIC (or NICs) are paired to a given VM. Another side effect is that you end up using bigger switches because aside from the main data NICs you also have one 10/100 Realtek NIC for IPMI 2.0 and another NIC for Areca out of band management.

If you are planning on any sort of redundancy this is really dumb and shows that you do not know much about how ESX/i load balances NICs. ESX uses a round robin methodology to balance traffic when a VM boots. If you want to have 10 nics assigned to 10 different VMs then just put all the physical NICs in one vSwitch. This will allow you to have redundancy if a link fails. In your scenario you have zero redundancy.

 

[CyberTron]

If you are planning on any sort of redundancy this is really dumb and shows that you do not know much about how ESX/i load balances NICs. ESX uses a round robin methodology to balance traffic when a VM boots. If you want to have 10 nics assigned to 10 different VMs then just put all the physical NICs in one vSwitch. This will allow you to have redundancy if a link fails. In your scenario you have zero redundancy.

"this is really dumb and shows that you do not know much about"

Comments like these really irritate me. Maybe I'm the only one that interprets this as contempt, but I really wish some people could learn to be more uplifting and less brash

 

[pjkenned]

If you are planning on any sort of redundancy this is really dumb and shows that you do not know much about how ESX/i load balances NICs. ESX uses a round robin methodology to balance traffic when a VM boots. If you want to have 10 nics assigned to 10 different VMs then just put all the physical NICs in one vSwitch. This will allow you to have redundancy if a link fails. In your scenario you have zero redundancy.

Hi,

#1 I'm not using ESX/ ESXi on this box.
#2 I have different physical servers, in different locations for redundancy. The one in question is just my home server, not something I would put into production for a business.
#3 The VMs are mostly just test clients so they sit around 0&#37; utilization most of the time, but I need them to go through external switches. I specifically do not want them going through a single vSwitch. Ever tried using Hyper-V R2 virtual networks @ 95-100% CPU load when you have some VM's using legacy NIC emulation? It is a big weakness of the platform actually.
#4 You have a funny sig despite it being in caps.

 

[child of wonder]

If you are planning on any sort of redundancy this is really dumb and shows that you do not know much about how ESX/i load balances NICs. ESX uses a round robin methodology to balance traffic when a VM boots. If you want to have 10 nics assigned to 10 different VMs then just put all the physical NICs in one vSwitch. This will allow you to have redundancy if a link fails. In your scenario you have zero redundancy.

In ESX/ESXi the network load balancing is performed on outgoing traffic only and it is not round robin. It is based on originating MAC, an IP hash, or virtual port ID.

Inbound redundancy has to be configured on the physical switch.

And assigning one NIC per VM is huge overkill. The whole point of virtualizing is to increase utilization of your physical resources, including NICs.

 

[Emulex]

round robin is most definitely not easy nor default for ESX/i 4.0u1 - it is quite complex and its not very fun at all to setup. which is why i can't wait for more affordable 10gbe product/switches.

HP virtual connect ethernet for bladesystem is quite a brilliant idea.

 

[SillyBit]

Here is an example of a custom SAN build for use with VMware/ESXi. It looks like a pretty nice build for the price performance. I run a similar setup at home. But you could definitely use a build like this for a small environment or lightweight production environment. I like that they use several Supermicro components where it counts...

Custom SAN Tutorial:
http://www.itechstorm.com/build-a-custom-san-hardware

 

[Emulex]

i've found low end san/vsa is many times slower than DAS especially if you can afford say lsi cachecade with hyper-v or esxi

results show HOT caching for work (database) can bring RE4 (say 8 of them in raid-10) up to 15K SAS performance and reliability - or bring 10K SAS beyond 15K SAS performance and reliability.

appserver running on sqlserver running on DAS (barebones) is over 10 times faster than app server -> lan -> sql server -> lan -> storage (all vm'd) - even if LAN is virtualized. why? latency. it's a ah heck. not everyone writes the most efficient stored procedures.

hyper-v3 even supports vmotion-like with DAS slick