Question 1600W Titanium suitable for 3x4090 (AI/ML)

[Steelbom]

I'm looking at picking up 3x4090's for ML/AI usage.
Do you think a 1600W Titanium PSU would be suitable?

I'm thinking of the Dark Power 13 1600W because it has two 16 pin connectors.
Apparently it can handle brief spikes to double its rated capacity.

I'm also thinking I could underclock them a little if needed?

Cheers,
SB

 

[jpiniero]

You do have the proper outlet for it, right? A standard US outlet isn't going to cut it.

 

[Steelbom]

You do have the proper outlet for it, right? A standard US outlet isn't going to cut it.

Yep, I do.

 

[Ajay]

Your case supports 2 PSUs - I’d use that option for running 3xRTX 4900s + board power. I’m guessing your ML application is low bandwidth - otherwise you don’t have enough PCIe lanes.

 

[Steelbom]

Your case supports 2 PSUs - I’d use that option for running 3xRTX 4900s + board power. I’m guessing your ML application is low bandwidth - otherwise you don’t have enough PCIe lanes.

Yeah, this Aorus motherboard doesn't have enough. I'm thinking of upgrading to the MSI X670E Godlike which can do PCIE-5 at 8/8/4. I'm assuming PCIE-5 x8 isn't going to limit a 4090 (like PCIE-4 x16) and I suspect a small impact from the PCIE-5 x4 -- let me know if you think otherwise.

I'd like to use that option -- it'd certainly make the power situation easier. However, the third card will cover some connectors (maybe not so important) on the bottom of the motherboard.

 

[MrTeal]

Yeah, this Aorus motherboard doesn't have enough. I'm thinking of upgrading to the MSI X670E Godlike which can do PCIE-5 at 8/8/4. I'm assuming PCIE-5 x8 isn't going to limit a 4090 (like PCIE-4 x16) and I suspect a small impact from the PCIE-5 x4 -- let me know if you think otherwise.

I'd like to use that option -- it'd certainly make the power situation easier. However, the third card will cover some connectors (maybe not so important) on the bottom of the motherboard.

It's not PCIe 5, because the 4090 doesn't support Gen 5. That's probably less of a bottleneck for you application than it would be for gaming though. What motherboard would you be using for these though, and would you be watercooling them? Modern consumer motherboards really aren't laid out well for triple GPUs anymore, you might be better off with a server or HEDT platform.

 

[Ajay]

It's not PCIe 5, because the 4090 doesn't support Gen 5. That's probably less of a bottleneck for you application than it would be fort gaming though. What motherboard would you be using for these though, and would you be watercooling them? Modern consumer motherboards really aren't laid out well for triple GPUs anymore, you might be better off with a server or HEDT platform.

Yeah, the 'reason' for dropping NVLink on the 4090 was because more data could be pushed over a PCIe 5.0 slot between cards. But, then it turned out that the 4090 only supported PCIe 4.0. Yet, if you buy a workstation class card, you get NVLink back - lolz! NV knows how to make money.

 

[Steelbom]

It's not PCIe 5, because the 4090 doesn't support Gen 5. That's probably less of a bottleneck for you application than it would be for gaming though. What motherboard would you be using for these though, and would you be watercooling them? Modern consumer motherboards really aren't laid out well for triple GPUs anymore, you might be better off with a server or HEDT platform.

Well, that's good to know. So on the PCIE-5 x4 slot, the 4090 will run at PCIE-4 x4 and not x8 speeds, right?

The motherboard is MSI X670E Godlike. It'll fit the three GPUs, but it covers some bottom connectors on the motherboard. Nothing too important though.

The thing is, I

Yeah, the 'reason' for dropping NVLink on the 4090 was because more data could be pushed over a PCIe 5.0 slot between cards. But, then it turned out that the 4090 only supported PCIe 4.0. Yet, if you buy a workstation class card, you get NVLink back - lolz! NV knows how to make money.

It's a bit annoying. I could buy HEDT w/ 4x4090's, but it's like 2.5-3x more expensive than to upgrade to to motherboard+3x4090's myself.

 

[MrTeal]

Well, that's good to know. So on the PCIE-5 x4 slot, the 4090 will run at PCIE-4 x4 and not x8 speeds, right?

The motherboard is MSI X670E Godlike. It'll fit the three GPUs, but it covers some bottom connectors on the motherboard. Nothing too important though.

The thing is, I

It's a bit annoying. I could buy HEDT w/ 4x4090's, but it's like 2.5-3x more expensive than to upgrade to to motherboard+3x4090's myself.

Correct, it'll run four lanes at the fastest common speed, so Gen4 x4. That'd be the same bandwidth as PCIe2 x16, or the brown bar in the graph below.

NVIDIA GeForce RTX 4090 PCI-Express Scaling

The new NVIDIA GeForce RTX 4090 is a graphics card powerhouse, but what happens when you run it on a PCI-Express 4.0 x8 bus? In our mini-review we've also tested various PCI-Express 3.0, 2.0 and 1.1 configs to get a feel for how FPS scales with bandwidth.

www.techpowerup.com

Compute is often less dependent on link speed though, so you might not see that performance drop.

The reason I ask about watercooling and motherboard is just physical spacing. Server and HEDT can have a full set of 7 x16 slots, which gives you flexibility to put a card in slot 1, slot 4, and slot 7. Even then 4090 FEs would all be touching, but at least they fit minus maybe some surgery on the bracket. Most consumer ones like the Godlike have the primary GPU slot as slot 2, and then you have the other two x16s at slot 5 and slot 7. The 4090 FE is triple slot and AIB ones can be chunkier, so you'll need to check you can physically fit your GPUs on your motherboard, and that you have clearance at the bottom of the case for a 3 or 4 slot card in the bottom slot. Watercooling helps because then all the cards can basically be made single slot.

 

[Steelbom]

Correct, it'll run four lanes at the fastest common speed, so Gen4 x4. That'd be the same bandwidth as PCIe2 x16, or the brown bar in the graph below.

NVIDIA GeForce RTX 4090 PCI-Express Scaling

The new NVIDIA GeForce RTX 4090 is a graphics card powerhouse, but what happens when you run it on a PCI-Express 4.0 x8 bus? In our mini-review we've also tested various PCI-Express 3.0, 2.0 and 1.1 configs to get a feel for how FPS scales with bandwidth.

www.techpowerup.com

Compute is often less dependent on link speed though, so you might not see that performance drop.

The reason I ask about watercooling and motherboard is just physical spacing. Server and HEDT can have a full set of 7 x16 slots, which gives you flexibility to put a card in slot 1, slot 4, and slot 7. Even then 4090 FEs would all be touching, but at least they fit minus maybe some surgery on the bracket. Most consumer ones like the Godlike have the primary GPU slot as slot 2, and then you have the other two x16s at slot 5 and slot 7. The 4090 FE is triple slot and AIB ones can be chunkier, so you'll need to check you can physically fit your GPUs on your motherboard, and that you have clearance at the bottom of the case for a 3 or 4 slot card in the bottom slot. Watercooling helps because then all the cards can basically be made single slot.

Thanks! I've decided to shy away from the watercooling. It'll be challenging to fit it all and I'd need to mount one on the bottom which isn't ideal (so I've read).
I'm now looking at the Godlike + 3x 4090 MSI Ventus 3x OC. Just going to upgrade motherboard and add a single 4090 for now along with my 3090 Ti and 3060. Then I can see if I'll be able to add the other 4090 in slot #2 without it covering slot #3. I feel like I will since the Ventus is quite thin (60mm) for a 4090. It is a 450W card, which may make it easier to run three as well.