New high power compute workstation - comments?

[bobsmith1492]

Howdy all,

I'm putting together a high-end compute workstation for a buddy (EM simulation for electronics) - he needs both GPGPU and multi cores, depending on the simulation mode.

1. Any recommendations during assembly/test?
2. Any idea on tools I can use to stress the CUDA GPU for testing (under Windows)?
So far it looks like the best bet is a folding@home stress test that can do CUDA.

Thanks!!

These are the parts:

MB: SUPERMICRO MBD-X10DRI
GPU: Tesla K40 + Quadro K620
RAM: SAMSUNG 8GB DDR4 2133 (M393A1G40DB0-CPB) x 8 = 64GB
CPUs: Xeon E5-2660v3 x2
Coolers: Noctua NH-D9L 92mm x2
Case: Cooler Master HAF 932
Drives: Samsung 840 EVO 500GB, 1TB HDD
PSU: SeaSonic SS-1050XP3 1050W

Win 8.1 Pro

 

[mfenn]

It looks to me like the MBD-X10DRI uses the narrow ILM instead of the square one most common on desktop boards. So the NH-D9L won't fit. The narrow ILM really limits your choices to fairly loud server style heatsinks. Does he really need to have a dual-socket box sitting under his desk? If so, I think it makes sense to go for a solution with more thermal engineering behind it like a Dell Precision.

 

[Burpo]

Better mobo would be the ASUS Z10PE-D8 WS
http://www.newegg.com/Product/Product.aspx?Item=N82E16813132268

 

[vailr]

What exact "EM simulation for electronics" software? If a pair of normal PC video cards (such as 2x GTX Titan's) would suffice, that combo would be much less expensive than a $3,200 Tesla K40 + $160 Quadro K620.

Also: 128 Gb may be possible with these:
http://www.anandtech.com/show/9188/gskill-announces-16gb-unbuffered-modules-128gb-kit-at-ddr42800

 

[bobsmith1492]

Mfenn - thanks a ton for the heads-up! It seems unnecessary to have one CPU socket with two mechanical mounting configurations. I wouldn't have noticed until I got the parts except I found a site that showed the two sockets side-by-side. The manufacturer didn't mention the narrow ILM anywhere that I could find.

Anyway, I found the SUPERMICRO SNK-P0050AP4 which should do the trick and looks like it will perform better than the Dynatron part.

Burpo - better in what sense?

Vailr - I looked at the GTX Titans, but three things steered me away.
1. It sounds like their double precision performance isn't as good (must be a different core variant)
2. They use significantly more power and would likely run hotter, so their lifespan may be shorter / less reliable
3. The K40 uses ECC RAM, which, while slower, will avoid computational errors - I've never known my gaming cards to not end up with graphical artifacts which I believe are probably RAM errors, and my buddy can't tolerate that in his simulations

Thanks for the input everyone!

 

[vailr]

>>What exact "EM simulation for electronics" software?<<

I'm no expert on workstation graphics cards, but AFAIK: the GPU chips are interchangeable between workstation and normal PC video cards. Only the firmware defines the difference between the two. I would also tend to doubt that RAM errors are the root cause of any "graphical artifacts" that you speak of. More likely a driver issue, would be my guess.
Re: "their double precision performance isn't as good". Is that feature really a requirement, or would you get identical results using a less expensive solution?

 

[mfenn]

>>What exact "EM simulation for electronics" software?<<

I'm no expert on workstation graphics cards, but AFAIK: the GPU chips are interchangeable between workstation and normal PC video cards. Only the firmware defines the difference between the two. I would also tend to doubt that RAM errors are the root cause of any "graphical artifacts" that you speak of. More likely a driver issue, would be my guess.
Re: "their double precision performance isn't as good". Is that feature really a requirement, or would you get identical results using a less expensive solution?

NVIDIA uses ECC support as a market segmentation lever even though the GPUs are indeed the same. Graphical artifacts can definitely be caused by bit flips, but you typically don't notice because it's unlikely for the bit flip to affect more than one pixel for one frame. With the amount of DRAM on GPUs, ECC is a real consideration for long running simulations, and you do want ECC for work that is going to be published.

As for single versus double precision, it really depends on the code in question. Some will require high-performance double precision math, others couldn't care less.

 

[mfenn]

Anyway, I found the SUPERMICRO SNK-P0050AP4 which should do the trick and looks like it will perform better than the Dynatron part.

Yes, it looks a lot better, but it still has a pretty loud fan.

Burpo - better in what sense?

Well, it is square ILM for one. It also has workstation creature comforts like sound and more USB 3.0 ports.

You do need an SSI EEB case for it, but ditching the HAF 932 is no big loss since the 932 is an outdated case design anyway. I would check out the Phanteks Enthoo Pro instead.

 

[bobsmith1492]

Well it's all done! I did some stress testing too. Pretty fun to run 35 Prime95 workers, RTHDRIBL, and fahbench all at once.

Got 47.3ns/day in fahbench, while everything else was running so it looks like it's going full speed. https://foldingforum.org/viewtopic.php?f=38&t=24225 It's right on par with a GTX Titan.

Couldn't find any Tesla K40 temperature monitoring tools, but the CPUs only got up to 47C.

Thanks for the input everyone!

Pictures:

 

[vailr]

I'm still curious: what exact "EM simulation for electronics" software is being scheduled for use with this hardware?
Thanks.

Edit:
In case you missed it, there's a firmware update for the Samsung 840 EVO SSD
http://www.pcper.com/news/Storage/S...840-EVO-Firmware-Released-Downloads-Throttled

 

[mfenn]

Couldn't find any Tesla K40 temperature monitoring tools, but the CPUs only got up to 47C.

There's a command line tool installed at

C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi.exe

which will report the GPU temperature along with information about ECC errors, memory usage, compute processes, etc.

 

[bobsmith1492]

The EM software is CST: https://www.cst.com/

 

[mfenn]

The EM software is CST: https://www.cst.com/

In my experience, CST is one of the better packages when it comes to GPGPU support, so the Tesla gets a :thumbsup: from me.

 

[vailr]

Hardware Recommendation
https://www.cst.com/Products/CSTS2/Hardware-Recommendation
Our hardware experts are available to review your desired hardware configuration and provide you with feedback and suggestions to improve the configuration.

https://www.cst.com/Products/CSTS2/TechnicalSpecification
Minimum Hardware Requirements

• Intel® Xeon® based PC, 4 GB RAM (8 GB recommended), DVD drive, at least 30 GB of free hard disc space (60 GB recommended).
• Fully OpenGL compliant graphic card
• Windows 7, Windows 2008 Server R2, Windows 8, Windows 2012 Server, Windows 8.1, and Windows 2012 Server R2
• RedHat Enterprise Linux (RHEL) 5.x und 6.x. and SUSE Linux Enterprise Server (SLES) 11.x. Some features may not be available, however.
• Hardware recommendation depends on your application. If in doubt, please contact your local sales office for further information. For further details, please see our hardware recommendations.

 

[bobsmith1492]

My buddy ran a CST simulation he had set up and it took 20 minutes compared to all day on his old desktop.