4GPU RTX2080Ti setup

[jkanters]

I am going for research purposes to setup a Deep Learning machine.

I have planned a four GPU machine with 4xRTX2080Ti cards. I need PCIe16 connection to the motherboard which should be equipped with the AMD Ryzen Treadripper which have enough memory lanes. However our PC vendor tells us that the motherboard I chosen ( ASUS ROG ZENITH EXTREME X399) there is likely not space enough to ensure proper ventilation. Any one with experience or recommendations how to build a 4xGPU machine wir air cooling

 

[UsandThem]

There are no RTX 2060ti cards, but to get four cards in one PC, you will need to make them all reference blower type cards so they exhaust heat outside of the PC, and you need a case that can accommodate 3 intake fans (120mm or 140mm) at a minimum, and it wouldn't hurt if you could install another fan on the bottom of the case. Of course you will want a 140mm exhaust fans in the back and top as well.

 

[jkanters]

There are no RTX 2060ti cards,.

Sorry Typo. I mean RTX2080. Changed it.

 

[LTC8K6]

I assume he meant 2080ti cards?

 

[UsandThem]

There are several cases you can choose from, but just two off the top of my head that would work for you:

http://www.coolermaster.com/case/mid-tower/mastercase-h500/

http://www.coolermaster.com/case/mid-tower/mastercase-mc500p/

They both come with solid front panels which you would not want to use at all, but they also include mesh front panels which will allow good airflow. Both of them also support 3 x 140mm front fans (with the H500 also supporting 2 x 200mm in the front).

 

[jkanters]

Acoording to our computershop the problem is not the case but the motherboard. Since the cards is 50 mm wide there is not room enough for ventilation when 4 cards is stacked.

 

[UsandThem]

Acoording to our computershop the problem is not the case but the motherboard. Since the cards is 50 mm wide there is not room enough for ventilation when 4 cards is stacked.

In that case, you'll have to carefully measure the video card dimensions with what you have on the board. You won't need too much of a gap for blower style cards, as they will pull in the cool air being put into the case. You might need to use some high-performance, high-RPM fans.

 

[LTC8K6]

https://www.gamepc.com/shop/custom?rig=prowess-gp-300&a=print

One with 4X 2080ti cards.

 

[StefanR5R]

I am going for research purposes to setup a Deep Learning machine.

I have planned a four GPU machine [...]

Do you simply need 4 GPUs, and that's it?
Or does the performance of your application depend on whether or not the GPUs are

• attached through a single PCIe root,
• connected through NVLink?

 

[DigDog]

You could get a mining case; the cards have cable leads to the pci slot.

 

[StefanR5R]

Riser cables in mining rigs are only 1 lane wide. The OP's application probably calls for a much wider connection. Wider PCIe extender cables exist, but I have no experience with their reliability.

If no NVLink connection is required, than another alternative for improved air flow is to mount every other card onto a PCIe extender card. The computer case needs to be wide enough then, or the side panel be omitted.

 

[aigomorla]

thermaltake pci-e extenders have been known to work even extended 7 times on itself, unless linus somehow got a super cherry ones from Tt.

I would second digdog's recommendation. Go Mining case, if you ened that many gpu's and use 16x pci-e extension cables.

 

[thecoolnessrune]

What's your topology here? If you're doing deep learning leveraging all GPUs, then you probably shouldn't be using Threadripper. It's NUMA separation creates Infinity Fabric congestion when all cards start trying to communicate with each other. Single Root PCIe complex is most desirable. Single Root designs are still available in the older v3 and v4 Xeons.

If you're leveraging NVLink to minimize the impact from the Threadripper root complexes, note that NVIDIA artificially limits the implementation of NVLink via physical design. NVLink bridges for GeForce are:

1. Incompatible with other card bridges like Quadro.
2. Only built in 3 slot and 4 slot variety (2 cards consume a minimum of 5 slots, 4 cards consume 10 slots). In the densest form, you could do card, heatsink, slot, card, heatsink, card, heatsink, slot, card, heatsink. That would give 2 different NVLink pools. This is because GeForce cards only have one NVLink connector, and cannot run in peer-to-peer mode. You'll also notice that in this design, your slots would not line up when using the NVLink bridges with your motherboard.
3. More than a single NVLink domain on GeForce cards is not supported.

Basically, NVIDIA has capped it so that if you want to use NVLink for more than 2 cards, you need the Quadro cards. If you don't want to use NVLink, then you're at the mercy of your PCIe channels, and while dual die threadripper is acceptable, things would be better with an older single root Xeon design if buying today, or rating for Rome based EPYC 2 to fix the PCIe root complex problem on their existing architecture.