ESXi Lab Build

[brshoemak]

1. What YOUR PC will be used for. That means what types of tasks you'll be performing. - ESXi Lab. Total of (3) systems:

(2) matching systems with the ability to take at least 32GB of memory but more would be better, 32GB DDR3, running ESXi so no local storage needed (USB drive will contain OS), quad port gigabit NIC card, would prefer a quad core processor

(1) "SAN" which will be a case/PSU (I have one in mind for ~$100), a RAID controller that will support 8 drives, a low end mini-ITX system, and lets say (3) 2TB WD Red drives in a RAID5 or RAID6,

2. What YOUR budget is. A price range is acceptable as long as it's not more than a 20% spread. Hopefully under $2000-2400 all in (lower is always better though)

3. What country YOU will be buying YOUR parts from. USA

4. IF you're buying parts OUTSIDE the US, please post a link to the vendor you'll be buying from. N/A

5. IF YOU have a brand preference. Intel for CPU. Obviously it will have to Don't really care for

6. If YOU intend on using any of YOUR current parts, and if so, what those parts are. Nothing used.

7. IF YOU plan on overclocking or run the system at default speeds. Default speeds

8. What resolution, not monitor size, will you be using? N/A

9. WHEN do you plan to build it? Before Jan 31st, 2013

A pre-built system from one of the big players would probably be too expensive. I thought about Supermicro for the two hosts but I was concerned about cost premium for Xeon CPUs and supporting MB. Even though they're going to be 'servers' I don't really need server class reliability.

TIA

 

[MAW1082]

I would just make sure that whatever parts you buy are compatible with esxi out of the box. It's not trivial to add drivers.

Also, I would make sure if you are going to go with a RAID, that you make sure it's compatible with esxi in regards to management. You don't want to have to go into the RAID card hardware BIOS to check the health of your array.

 

[tynopik]

isn't esxi limited to 32gb anyways?

 

[brshoemak]

Yeah I will make sure that all the hardware is on the HCL. Good reminder for the We use Dell servers at work and they have a custom ESXi image that supports higher visibility into different functions (BBU, RAID status, drive status, etc.) I need to look closer at that functionality for white-box components.

I actually did forget that ESXi (free version) is limited to 32GB. At least that keeps the cost down a bit. I really shouldn't need THAT much anyway, but in virtualization more memory is always better (along with I/O performance).

Any more specific ideas on hardware or at least certain components that you would recommend for RAID controller, etc. ?

 

[imagoon]

isn't esxi limited to 32gb anyways?

No. ESXi (5.1) is limited to 2TB RAM / Host. The free edition is artificially limited to 32GB. Depending on the lab, if he intends to pay for it that restriction can be lifted for less than $1000.

 

[imagoon]

Yeah I will make sure that all the hardware is on the HCL. Good reminder for the We use Dell servers at work and they have a custom ESXi image that supports higher visibility into different functions (BBU, RAID status, drive status, etc.) I need to look closer at that functionality for white-box components.

I actually did forget that ESXi (free version) is limited to 32GB. At least that keeps the cost down a bit. I really shouldn't need THAT much anyway, but in virtualization more memory is always better (along with I/O performance).

Any more specific ideas on hardware or at least certain components that you would recommend for RAID controller, etc. ?

My home lab hosts:

i3750k
ASRock Pro4M
32GB Corsair Ram
PERC 6i

To get a machine that supports more than 32GB of RAM is going to drive the price up. Most of the motherboards with that support jumped up to the $300+ range and needed the Xeon or SandyBridge-E processors. IE Big $. I boot ESXi off a memory stick. Remember that if you intend to actually learn VMWare, you need a vCenter server and some license keys. I *cough* borrowed keys for my home lab but I only do training on it. VMWare does intend to offer training labs online at some point. I signed up but the invites still have not gone out.

The Intel consumer CPU's are all limited to 32GB of RAM at the moment. Except the SB-E and Xeons.

 

[Chapbass]

My home lab hosts:

i3750k
ASRock Pro4M
32GB Corsair Ram
PERC 6i

To get a machine that supports more than 32GB of RAM is going to drive the price up. Most of the motherboards with that support jumped up to the $300+ range and needed the Xeon or SandyBridge-E processors. IE Big $. I boot ESXi off a memory stick. Remember that if you intend to actually learn VMWare, you need a vCenter server and some license keys. I *cough* borrowed keys for my home lab but I only do training on it. VMWare does intend to offer training labs online at some point. I signed up but the invites still have not gone out.

The Intel consumer CPU's are all limited to 32GB of RAM at the moment. Except the SB-E and Xeons.

I'll +1 this post. those ASRock MB's are good for esxi boxes.

Current lab uses 2 asrock pro3 MB's, they've been working great with it. A proc like the 3470 will add in vt-d if you need that (whereas the k series chips wont do it). Going to the i7 for hyper threading could be neat if you need that kind of cpu power, but its not required at all (I used the 3470 in my hosts).

 

[mfenn]

Remember that if you intend to actually learn VMWare, you need a vCenter server and some license keys. I *cough* borrowed keys for my home lab but I only do training on it.

Excellent points. OP, can we assume that you have the licensing side of things covered?

 

[brshoemak]

Excellent points. OP, can we assume that you have the licensing side of things covered?

I do not have the licensing side of things covered right now. I will have to rock the free version for while so I lose out on some of the features that make virtualization attractive, but it will allow me to work with the storage and networking side of things. Certain features like HA and vMotion are pretty straightforward so I'm not too worried about that side of things. I'm kind of going on the path of working on my Cisco certs and if that goes well I'll find a way to invest in a full featured license of vSphere. I am good from a OS license perspective, though.

My experience with ESXi is fairly limited at this point, but I had to learn quick as a situation arose where Dell sold a client a bunch of hardware, and they decided that I would set it up - without asking me. Currently, their setup is a pair of Dell R620s (dual QC Xeons/64GB memory), dual-port SAS HBAs, running ESXi 5.1 off redundant SD cards, connected redundantly to a Dell MD3220 utilizing a 4TB RAID10 setup. It's running vSphere Essentials Plus in order to get HA (but no Fault Tolerance in Essentials, apparently it isn't that important to them). By the way, it's running beautifully according to best practices.

Anyway.......I'm not looking to replicate their environment (nor can I financially) but I'd like to have a decent framework setup so I can work on the more advanced topics when the time comes.

 

[mfenn]

Well, if you're not going to be doing anything that requires a vSphere server, then I don't see much point in getting two machines. You certainly don't need it from a capacity point of view. You'd have to do some slightly underhanded things to even get two free ESXi licenses (it's technically limited to one per person, but the way they enforce it is one per VMware account).

Also, have you given any thought to the type of switch that you will use? Since you're going for Cisco cert., you'd find it very useful to divert the funds that you were planning to spend on a 2nd hypervisor and instead get a used Cisco switch off Ebay.

 

[brshoemak]

Good point on the multiple hosts. I was looking to get a matched pair of hosts, but I guess the hardware itself doesn't really matter as long as there is enough storage to handle the VMs

As for the Cisco gear, I have a couple 2950/60s (EMI) and two 2611XMs (which are running 12.4) I also have access to a literal pile of switches if I need them. I should be good there from that perspective.

Maybe for now, I'll just get a decent machine to run ESXi (free) and get some local storage on a hardware RAID controller. Then shift that hardware array to another box to use as an iSCSI target for the datastores. Maybe look at adding another host later.

Any thoughts on a RAID controller that will handle at least 8 drives and do RAID1/5/10/50 ? Dynamic expansion would also be nice. I'll probably end up using WD Red drives for storage.

 

[Viper GTS]

Remember that you can virtualize ESXi on ESXi... So if you only build one machine (with as much RAM as allowed, but even 32 is stupidly small). Were it my money I'd rather spend the $560 for an essentials license and then take advantage of the unlimited RAM.

If you decide you want to do a VCP do ESXi on ESXi with trial versions of everything (including vCenter). You can do shared storage this way as well, there's no reason you couldn't do a full VCP prep one one physical machine. If you were really careful with your RAM allotments you could probably even do it in 32 GB.

In spite of my feelings about VMware having crippled the free version with vRAM limitations the $560 entry price is very reasonable and I don't regret buying it.

For storage I would be looking for OEM branded parts with major manufacturer guts - Think Dell PERC and the like. They're widely supported, and usually dirt cheap on ebay. I'd stick with RAID 10 if you can afford enough disks to make usable capacity, RAID 5 with consumer SATA drives is not a recipe for great performance. For myself I bought 24 1 TB 7200 RPM drives. My core ESXi storage is an 8 drive RAID 10 set and it's decent for home use.

Viper GTS

 

[mfenn]

Viper GTS makes a good point, though I would probably go a slightly different direction on the storage. In a "real-life" virtualization setup, you're not like to be playing with direct-attached SAS storage. I say "real-life" because obviously your workplace is a special case (only two hosts, no existing storage infrastructure).

I'd definitely build a separate storage box and throw FreeNAS on it so that you can play with NFS and iSCSI datastores "for real."

Especially since you have access to a bunch of real networking gear. You'd be able to set up pretty much any arbitrary "enterprise-looking" network topology with what you have. Separate storage VLANs, jumbo versus normal frames, LACP versus multipath-IO, etc. etc.

As an added bonus, it lets you avoid dropping ~$300 on a RAID card.

 

[brshoemak]

@Viper

I made the mistake of looking at Essentials Plus which is extremely expensive - but I could probably handle $500 for plain Essentials.

For hardware RAID controller I would probably look at a PERC6, it should handle everything pretty well and it will be cheap (although the BBU will probably be close to shot). As far as RAID level I wouldn't do RAID5 - decent reads, abysmal writes. RAID10 would be ideal but also a little salty for the wallet. I'd like to look at RAID50 + hotspare. I'll probably use 2TB drives, but good god the rebuild time on that would probably be a day or more. You said you have 24 1TB drives, what case are you running those in? I'm sure you don't have 24 in one chassis, but what enclosure are you using for the 8 drive RAID10?

@mfenn

I agree that direct attached SAS is a limited setup but they'll never ever (ever) go beyond two hosts - the level of overkill on their setup is tremendous. Dell calls the MD3220 a 'SAN' but it's more like a DAS device - their MD3220i is iSCSI-based.

I know the merits of FreeNAS and I've used it before, but I'm still stuck in the enterprise mindset where everything is hardware based, which can be limiting but also gives me peace of mind. I'll look into it, but I don't know enough about RAID-Z yet to give it a green light - more research is required.

Regardless of the setup, I'll have to start with a MB, CPU and memory. If I am looking for a solid setup that will handle over 32GB of memory, I'll want a quad core CPU (single socket is fine), and 32GB of memory - what items would you recommend?
I have a lot to think about and I appreciate all the insights.

 

[thecoolnessrune]

I can say that RAID-Z is awesome. I run RAID-Z2 on a Supermicro-based Nexenta Storage array that delivers CIFS shares, NFS shares (for ESXi ISO images), iSCSI (for testing), and and SRP over Infiniband for the host node that I also run off of a Supermicro-based box.

The performance is great as long as you have a processor that can keep up (a dual core decent clocked Intel processor can keep up just fine as long as you don't turn on dedup and don't get crazy with compression). I am the opposite and am running full array compression and dedup but I'm running it off of an E5-2650 (8 2Ghz cores). With 2 guests hitting the storage array I barely break 30% load, so there is plenty of headroom left. (Anandtech spoke of almost saturating one of their processors so I planned worse-case scenario, but it really ended up being overkill).

Also, if you intent to have over 32GB of memory, then you have to move to the Intel Xeon or AMD Opteron platform. Intel Core processors ceiling at 32GB of memory. You could then probably get around 64GB in the box before having to move to ECC modules.

 

[mfenn]

I agree that direct attached SAS is a limited setup but they'll never ever (ever) go beyond two hosts - the level of overkill on their setup is tremendous. Dell calls the MD3220 a 'SAN' but it's more like a DAS device - their MD3220i is iSCSI-based.

I wasn't really referring to the needs of you workplace, but your professional development needs. IMHO it's necessary to know at least the basics of iSCSI or FibreChannel to be a good VMware sysadmin.

I am very familiar with the MD32xx and the MD3000. I've build a 600 TB GPFS system out of MD3000 stacks. It was cheap, but damn that those SAS cross-links were a pain in the ass to cable! Now that their iSCSI and FC versions are cheap (especially iSCSI), I'd never do it again.

Also, if you intent to have over 32GB of memory, then you have to move to the Intel Xeon or AMD Opteron platform. Intel Core processors ceiling at 32GB of memory. You could then probably get around 64GB in the box before having to move to ECC modules.

Well, a Socket 2011 Core i7 could handle more than 64GB, but at that point, you're talking Xeon money anyway. Opterons are probably your best bet for getting a lot of memory on the cheap. Not that I really see the need for it though.

OP, why do you want to have more than 32GB of RAM in the box? 32GB is a ton of VMs if you are realistic about memory allocation.

 

[Viper GTS]

You can virtualize the shared storage on the same machine. You're not likely to do FC at home for cheap but iscsi and nfs are easily provided with lots of free options.

Long term having a dedicated storage appliance is of course nice but if the short term budget doesn't allow it its easily worked around. Just do a datastore migration when you bring up the new appliance.

Viper GTS

 

[Viper GTS]

I wasn't really referring to the needs of you workplace, but your professional development needs. IMHO it's necessary to know at least the basics of iSCSI or FibreChannel to be a good VMware sysadmin.

I am very familiar with the MD32xx and the MD3000. I've build a 600 TB GPFS system out of MD3000 stacks. It was cheap, but damn that those SAS cross-links were a pain in the ass to cable! Now that their iSCSI and FC versions are cheap (especially iSCSI), I'd never do it again.



Well, a Socket 2011 Core i7 could handle more than 64GB, but at that point, you're talking Xeon money anyway. Opterons are probably your best bet for getting a lot of memory on the cheap. Not that I really see the need for it though.

OP, why do you want to have more than 32GB of RAM in the box? 32GB is a ton of VMs if you are realistic about memory allocation.

The annoying thing about esxi 5 is not only will it not let you power on any vms if you physically have more than 32 gb installed it also cuts you off based on actual vram use. So you don't even have the normal over subscription headroom. You can do a reasonable amount in 32 GB but multiple esxi hosts, vcenter, a storage appliance vm, AND guests to run on those hosts and you are going to hate life with 32 GB.

Viper GTS

 

[ebberz]

If it's just for training why not do what I did for my VCP:-

VM Workstation 8
One ESX server (with nested VMs)
One ESXi server (with nested VMs)
One Openfiler for shared storage

i7 860
16GB RAM
Gigabyte P55 board

I was running 5 servers at any one time and able to do vmotion and storage vmotion with no issue.

I'd like a full lab at some point with a couple of DCs, a SQL box, Exchange server etc so my next build will probably be a 64GB all in one machine. Possibly running a 6 core unless Haswell is up to the task. Although I am aware RAM and HDD IOPS are the main limiting factors.

 

[brshoemak]

I wasn't really referring to the needs of you workplace, but your professional development needs. IMHO it's necessary to know at least the basics of iSCSI or FibreChannel to be a good VMware sysadmin.

I should have clarified. I meant that their workplace only used a SAS-based SAN and (as you said) and that I need a solution that leverages an actual networked-based storage solution like iSCSI or Fibre Channel. Any idea where I should look for inexpensive Fibre Channel switches?

I am very familiar with the MD32xx and the MD3000. I've build a 600 TB GPFS system out of MD3000 stacks. It was cheap, but damn that those SAS cross-links were a pain in the ass to cable! Now that their iSCSI and FC versions are cheap (especially iSCSI), I'd never do it again.

I can’t imagine trying to cable that many shelves, I’m sure beyond the actual connections, labeling them and doing all the documentation was a treat.

OP, why do you want to have more than 32GB of RAM in the box? 32GB is a ton of VMs if you are realistic about memory allocation.

I guess I really don’t need that much. Most of the VMs would be a mix of *nix and Windows VMs that won’t be doing anything intensive. Are there consumer boards that will do 32GB or will I still need to look at server level hardware (or high-end consumer gear that would hit the same price range)? I guess one reason I’m so memory conscious is that I’m currently running a old-as-hell Dell SC440 server at home that is maxed out with 8GB. I find myself creating and deleting VMs (currently just through Virtualbox) constantly due to memory constraints. It’s not really fair to compare though, as it’s also handling the actual OS and associated overhead alongside the VMs.

 

[imagoon]

The annoying thing about esxi 5 is not only will it not let you power on any vms if you physically have more than 32 gb installed it also cuts you off based on actual vram use. So you don't even have the normal over subscription headroom. You can do a reasonable amount in 32 GB but multiple esxi hosts, vcenter, a storage appliance vm, AND guests to run on those hosts and you are going to hate life with 32 GB.

Viper GTS

I think VMWare's thought at the moment is: "By the time you can afford the hardware that will get you above the 32GB limit / host, you can pony up the for the Essentials license."

I would guess that it will change just like the free edition has changed in the past when hardware had grown.

At the moment you need to move to much more expensive hardware to even get more than 32GB on a board. This is due to the CPU's and not the boards. You simply can't plug 4 x 16GB chips in to board with a 2500k / 3750k as the CPU itself just ignores the RAM above 32GB anyway.

Besides, what is your alternative? Hyper-V standalone? At least as of 2008R2 it was limited to 32GB and 4 processors only also unless you joined it to the $4500 VM manager.

No idea if 2012 standalone is different yet.

There is also no vRam limit in 5.1 free. It was done away with. Oversubscribe as much as you want.

 

[imagoon]

Are there consumer boards that will do 32GB or will I still need to look at server level hardware? I guess one reason I’m so memory conscious is that I’m currently running a server at home (Dell SC440 – old as hell) that is maxed out with 8GB.

See above:
http://forums.anandtech.com/showpost.php?p=34406486&postcount=6


Here is what I am running at the moment to mess around:

32GB gives me a lot of area to mess around with. If I get the machine doing anything intensive, it is normally disk i/o the floors on me.

 

[mfenn]

I should have clarified. I meant that their workplace only used a SAS-based SAN and (as you said) and that I need a solution that leverages an actual networked-based storage solution like iSCSI or Fibre Channel. Any idea where I should look for inexpensive Fibre Channel switches?

You can find old 2Gb stuff on Ebay for pretty cheap, as in a few hundred bucks for a switch. HBAs are a little harder because most 2Gb ones are PCI-X, but PCIe ones are available for less than $100. Obviously try to get switches and HBAs that come with transceivers. I would just buy the fiber itself new from Monoprice, it's not that expensive and certainly not worth troubleshooting.

Obviously, you can also go the iSCSI route for next to nothing since you already have some switches. Unlike FC, there are plenty of good open source software iSCSI targets and initiators out there.

I can’t imagine trying to cable that many shelves, I’m sure beyond the actual connections, labeling them and doing all the documentation was a treat.

Yeaah it was a treat, let me tell you. :awe:

I guess I really don’t need that much. Most of the VMs would be a mix of *nix and Windows VMs that won’t be doing anything intensive. Are there consumer boards that will do 32GB or will I still need to look at server level hardware (or high-end consumer gear that would hit the same price range)? I guess one reason I’m so memory conscious is that I’m currently running a old-as-hell Dell SC440 server at home that is maxed out with 8GB. I find myself creating and deleting VMs (currently just through Virtualbox) constantly due to memory constraints. It’s not really fair to compare though, as it’s also handling the actual OS and associated overhead alongside the VMs.

Any run of the mill 4-slot H77 or Z77 board will do 32GB (4x8GB) no problem for ~$420. That's very cheap compared to just a processor for Socket 2011.

32GB gives me a lot of area to mess around with. If I get the machine doing anything intensive, it is normally disk i/o the floors on me.

:thumbsup: Good point.