Hardware recommendations for gaming VMs

[Hajime]

Hi,

I should probably explain my current configuration before asking for advice.

Current, racked:
Dual Xeon E5-2690s, 256gb RAM, ZFS 4TB SSD storage/ZFS 32TB spinning rust.
Runs Proxmox for 2x virtualized gaming machines (6 cores allocated), ZFS raid arrays, etc. ZFS is allocated 92GB of RAM for speed for the gaming VMs, and the gaming VMs have 32gb apiece.

I also have quite a few more machines on that rack serving various duties with everything being on a 10gbe switch and hitting the main server for storage (Please note I'm on 1gbit and run some hosts for personal use.)

The problem is the 10gbe switch is loud. Actually, the whole rack is loud, and it sucks quite a bit of power and generates a good bit of heat. Also, my environments aren't exactly small, moving to anything less than 128GB on my main server is not possible. I've been given wifely approval of a solution that is less offensive to the ears. I've sorted out two things to replace the current setup. Additionally, the 2690s are lacking raw clock speed plus modern IPC improvements -- I can't run pcsx2 at 4k/full settings on them for some of the stuff I want to stream to TV when I could with the other options, as one example.

My research has me down to one of the two following:

2x 6700ks, 32gb of RAM (16gb maybe with nvme?), etc.
Xeon D-1567 or maybe a D-1541, 128GB of RAM to replace the current dual E5-2690 server and other servers. Reduce the amount of RAM ZFS is allotted, moving the virtualized gaming machines to the above new machines should be sufficient to fit it in this server. I'd probably have to move some of my stuff to AWS and Azure for this as well.

However, something I'm tempted to do with this option is go with dual Denvertons instead of the Xeon D due to the latest news. Toying with the idea of Avoton as well, considering two C2750 setups are the same price as a single D-1541 setup. Plex leaves me worried about the C2750 though.

Or

I7-6950x@4ghz, 128GB of Ram, dual M.2s, etc. as a drop-in replacement for the dual E5-2690s, reconfiguring ZFS to move down to 32GB of RAM (since the nvme M.2's would be passed through to the gaming VMs, I think I can also reduce the RAM down to 16gb apiece)
Move some stuff to AWS or Azure. Downside, however, is gaming VMs would move from 6 cores to 4 cores, and this would leave 2 6950x @ 4ghz cores to replace 4 2690 cores for ZFS and assorted VMs...but it's probably close in raw performance.
This would be watercooled with a Kelvin S36

Currently leaning towards the latter option because of hardware lust and being able to move to Xeons when the latest gen becomes cheap, but I am interested to hear thoughts as well as other options? Also leaning towards it because Proxmox makes the gaming VMs ridiculously stable and reliable.

 

[mnewsham]

For raw performance having two 6700k rigs for gaming is probably the best solution.

The 6950x solution would work, but you run into a small performance hit because of virtualization.

Also the 6700k's can be clocked ~500-700MHz higher than the 6950x. Most 6700k's can hit 4.4GHz without issue on all cores, and 4.6GHz+ isn't unheard of with decent cooling.

 

[Hajime]

Perf hit is only 1%-3% in worst case scenarios in testing on raw hardware vs. virtualized for my use cases. Should be largely negligible.

Noted on the 6700ks. It's a big pro on the 6700k route that the hardware would be near silent to silent on the server side. Gonna have to read up more on the C2750's I think -- the Xeon D route ends up costing more than the 6950x as it is with the 1567, and the 1541 isn't that much cheaper than the 6950x.

 

[mnewsham]

Perf hit is only 1%-3% in worst case scenarios in testing on raw hardware vs. virtualized for my use cases. Should be largely negligible.

Yeah I know, i've done it myself, still 1-3% can be the difference between 60fps solid and dips into the 50s. I found dual booting to be an easier solution for my needs than running a VM from my *nix distro. Though obviously this depends on your particular needs.

the Xeon D route ends up costing more than the 6950x as it is with the 1567, and the 1541 isn't that much cheaper than the 6950x.

If you're not doing any production work (read actual work for your job), then you might consider looking at some ES's on ebay, if it's just doing local pfsense, etc for your home network and other small VMs or w/e, then it shouldn't matter too much if you're running an engineering sample.

I only say this because ES's and OEM xeons can be gotten off ebay for a fraction of their retail MSRP.
For example a E5-2658 V3 ES can be had for $170-200. 12 core at 2GHz. At that point you just need the motherboard.


Now if you're doing some real work (that you expect to make money from), I would suggest avoiding ES's as they technically don't belong to you since an ES is and always will be the sole property of intel and is not meant for resale.

 

[Hajime]

Going back to dual Xeons puts me in the same spot. If it was simply something that could be solved with $$$, I'd buy retail Xeons. Money's not really an issue, sound is a big one followed by heat.

 

[mnewsham]

Going back to dual Xeons puts me in the same spot. If it was simply something that could be solved with $$$, I'd buy retail Xeons. Money's not really an issue, sound is a big one followed by heat.

I think you misunderstood, I was saying get the two 6700k's AND a e5-2658 V3 ES off ebay.

That's two quad core with hyperthreading at 4GHz+ AND a 12 core 24 thread at 2GHz for ~$800 in CPU costs. Cheaper than a single 6950x by a large margin.

Two Z170 motherboards and a C612 motherboard will run you another ~$550.

16GB RAM each for the two 6700k/Z170's, 64-128GB for the Xeon/C612 depending what you'll be doing with it.

Unless you REALLY need a lot of server power, I simply can't see why you'd need more than this. A 2658v3 has more than enough power for any sort of home network type needs, unless you're running VMs for no reason besides wanting to run more VMs.



I'd love to know what you're doing that is actually going to need 48+ threads.

 

[Hajime]

Masters thesis on big data. It's why aws and azure are critical, because I can get rid of much of my existing hardware, but I need a certain minimum amount of on premises perf and RAM.

 

[mnewsham]

Masters thesis on big data.

if that's the case you're never really going to avoid all the noise. I'd advise looking at building a sound dampened server room or similar for your house as it will probably be the easiest solution. If it were me i'd keep your current dual xeon, place it somewhere more secluded and out of the way, then build two 6700k rigs for gaming that I would take care to build with low noise fans and the like.

If you need 48+ threads, you're gonna have to dissipate the heat, and even though the newer xeons are better, you're still dumping 200w+ of heat into your case with dual xeons.

Xeon-D might save you a bit on the power/noise front, but then you're paying 5-6x as much for the exact same performance you'd get from a $200 ES from ebay.

Is that $1000 extra per CPU really worth saving ~50-100w of dissipation? For amazon or google running 500+ of them, yeah that energy savings can really add up. For you as a home user doing big data?

Sorry, I just don't see the value.

 

[Hajime]

Hah. Tried that route before, was denied. Server room is right out for me.

 

[dave_the_nerd]

If you're doing a thesis on big data, and not actually trying to virtualize multiple gaming systems, that's kinda what AWS is for.