Question Can the MSI X670E Godlike fit 3x 2-slot GPUs?

[Steelbom]

G'day,

Page 34 of the manual shows an example of this: https://download.msi.com/archive/mnu_exe/mb/MEGX670EGODLIKE.pdf

However, looking at the motherboard diagram on page 31, it appears a two-slot card would cover a bunch of connections.
Are any of those needed?

This is the GPU I'm thinking of: https://www.msi.com/Graphics-Card/GeForce-RTX-4090-SUPRIM-LIQUID-X-24G

Cheers,
SB

 

[Tech Junky]

Looking at 32 it would seem to work but looking at the headers under the bottom slot you'll need to be tidy with cabling. Those are all thin cables though and should have some flex for routing under the card. I would try to bring them from the backside to the header so you don't have to try to wrap them along the bottom to the front of the case.

 

[Steelbom]

Looking at 32 it would seem to work but looking at the headers under the bottom slot you'll need to be tidy with cabling. Those are all thin cables though and should have some flex for routing under the card. I would try to bring them from the backside to the header so you don't have to try to wrap them along the bottom to the front of the case.

Thanks! I'm thinking I'll try using a smaller 4090 (Ventus 3X) and use a riser cable for the third slot.

 

[Tech Junky]

Since you're looking at $5k of GPU costs anyway something is tickling my brain about more HP units other than consumer grade options. Something more along the lines of the https://www.amd.com/en/graphics/workstations

Might be more efficient at running the workload with less space consumed and less heat generated.

 

[Steelbom]

Since you're looking at $5k of GPU costs anyway something is tickling my brain about more HP units other than consumer grade options. Something more along the lines of the https://www.amd.com/en/graphics/workstations

Might be more efficient at running the workload with less space consumed and less heat generated.

Ah, it's for CUDA so needs to be NVIDIA. I actually always used to prefer AMD GPUs until I needed CUDA.

I did consider HEDT but it's way more expensive than just upgrading my existing system.

 

[gdansk]

Why buy a $1200 motherboard that will still be sub-optimal? If you're concerned about "cheaper" run only 2 GPUs for now and avoid wasting money on an overpriced motherboard that still doesn't really handle it properly (PCIE4.0 or PCIE5.0 x16/x16/x16).
Sure it's cheaper to upgrade your current system but you'll end up with a limited system if a workload ever requires communication between cards. 3 cards should be reserved for W790 or Threadripper Pro.

 

[Tech Junky]

it's for CUDA so needs to be NVIDIA.

That's a PITA and there's really nothing better in terms of CUDA from them right now. The next release of ADA though should bump up the processing benchmarks though. I don't keep up with GPUs though as much as CPUs. I typically spend more time on my laptop than anything I would be able to slot a GPU into as that system is headless and runs 24/7. For the laptop though I normally shoot for the xx60 model price v performance sweet spot. Though I did play around with mining and dealt with NV CUDA with that for a bit and it just wasn't worth the added thermals / electrical costs vs the value of the coins.

Another thing you're going to run into if you're pegging these things is horrible temps and loud fans. I could get the temps to be moderate w/ 70% fan profile but still the entire case was heat soaked by the space hearers inside. I would suggest mounting them in a bench style setup that's open air if you plan to get the most performance out of them.

 

[Steelbom]

Why buy a $1200 motherboard that will still be sub-optimal? If you're concerned about "cheaper" run only 2 GPUs for now and avoid wasting money on an overpriced motherboard that still doesn't really handle it properly (PCIE4.0 or PCIE5.0 x16/x16/x16).
Sure it's cheaper to upgrade your current system but you'll end up with a limited system if a workload ever requires communication between cards. 3 cards should be reserved for W790 or Threadripper Pro.

Yeah the motherboard is heavily overpriced -- especially in Australia it's closer to $1500 USD.
I do plan on using accelerate / deepspeed to spread the load across cards but I think it should be fast enough. I won't be doing any training, just inference, etc.

For now, I've got a single 4090 coming so I'll be able to test 3090 Ti + 4090 + 3060. If I see the need, I'll upgrade the other two cards to the 4090.

In terms of cost, buying 3 4090's plus the new motherboard and PSU work out to be about one third of a new HEDT system which is definitely overkill for me.

That's a PITA and there's really nothing better in terms of CUDA from them right now. The next release of ADA though should bump up the processing benchmarks though. I don't keep up with GPUs though as much as CPUs. I typically spend more time on my laptop than anything I would be able to slot a GPU into as that system is headless and runs 24/7. For the laptop though I normally shoot for the xx60 model price v performance sweet spot. Though I did play around with mining and dealt with NV CUDA with that for a bit and it just wasn't worth the added thermals / electrical costs vs the value of the coins.

Another thing you're going to run into if you're pegging these things is horrible temps and loud fans. I could get the temps to be moderate w/ 70% fan profile but still the entire case was heat soaked by the space hearers inside. I would suggest mounting them in a bench style setup that's open air if you plan to get the most performance out of them.

For my use case it shouldn't be too bad at all. The Ventus I'm looking at is 450W (like my 3090 Ti) and for inference the workload is much lighter -- I just may need that 72GB of memory to load larger 30-40B models.

 

[Steelbom]

Thought I'd post an update with my current set up.

It's a very tight fit on the Godlike to get 3 GPUs in there. I've managed (kind of) to fit the 60mm 3-slot 4090 Ventus in the 1st PCI-E slot (3.5 slot card covered the 2nd PCI-E slot). The 3060 in the 2nd slot actually brushes up against the 4090 and stops two of its fans from spinning. I managed to shove a little 5mm piece of rubber between the two to give it space. GG if it comes out.

The 3090 Ti fits fine on the left but you'd need a case that has that kind of room for mounting GPUs in odd positions.

I think this setup would work well with 4090 watercooled (2-slot) in PCIE #1 and then another 2-slot card (40mm) in PCI-E #2 and then cable riser for #3.

I may retire the 3090 Ti (bit of a waste) and 3060 for two 2-slot watercooled 4090's in Slot #1 and #2 sometime down the line.

 

[mikegrok]

While my motherboard

H13SSL-NT | Motherboards | Products | Supermicro

can take 3 16x GPUs, that is usually not the ideal way to build out a box like this. See:

Molecule AIR XL

The Molecule Air XL GPU rendering workstation is an 8 way air-cooled Nvidia 4090 workstation for virtual production with the AMD Threadripper Pro at it's heart.

www.renderboxes.com

Especially with PCIe5, having long distances between the socket and the slot is a problem. And the problem is specifically with cross talk between the lines to different sockets. To get around that problem, they put MCIO (multi channel in out) ports on the motherboard, and run those to slot adapters that also take a power supply connector.

You may need multiple power supplies, which can be joined together with special atx cables. Each power supply may need to be connected to a different electrical circuit at your house, ie you may need to hire an electrician for an afternoon.

Here is an example of a board that can take 2 GPUs in slots, then an additional 10 via the MCIO connections.

H13SSH | Motherboards | Products | Supermicro

 

[Steelbom]

While my motherboard

H13SSL-NT | Motherboards | Products | Supermicro

can take 3 16x GPUs, that is usually not the ideal way to build out a box like this. See:

Molecule AIR XL

The Molecule Air XL GPU rendering workstation is an 8 way air-cooled Nvidia 4090 workstation for virtual production with the AMD Threadripper Pro at it's heart.

www.renderboxes.com

Especially with PCIe5, having long distances between the socket and the slot is a problem. And the problem is specifically with cross talk between the lines to different sockets. To get around that problem, they put MCIO (multi channel in out) ports on the motherboard, and run those to slot adapters that also take a power supply connector.

You may need multiple power supplies, which can be joined together with special atx cables. Each power supply may need to be connected to a different electrical circuit at your house, ie you may need to hire an electrician for an afternoon.

Here is an example of a board that can take 2 GPUs in slots, then an additional 10 via the MCIO connections.

H13SSH | Motherboards | Products | Supermicro

Thanks for the info! I was trying to do the most that I can without upgrading to Epyc/Threadripper.

In the end, I took out the 3060 which was blocking the 4090 from spinning its fans. So it's just a dual GPU board now (4090/3090 Ti).

In the future, I'll probably look at building something like you suggested.

Cheers!

 

[mikegrok]

By the way the epyc 8004 series is coming out, it is low speed (half the single threaded of ryzen), but is sort of a mini epyc with 2/3 the performance for 1/3 the price. If you need fast CPU stay with ryzen, but if you need more fast gpu (it has 96 pcie lanes) , then that may be an option in the future.

 

[mikegrok]

You can get cards off of eBay etc. Which operate like below. They are usually about $20-$40 per card and $20-$40 for the cable

In case you wanted to add another GPU to your existing box.

Unlike epyc, ryzen does not support bifurcation, so you can't further split up your pcie lanes.

 

[Steelbom]

By the way the epyc 8004 series is coming out, it is low speed (half the single threaded of ryzen), but is sort of a mini epyc with 2/3 the performance for 1/3 the price. If you need fast CPU stay with ryzen, but if you need more fast gpu (it has 96 pcie lanes) , then that may be an option in the future.

Ahhh that's pretty cool. Yeah, I think I'll stick with Ryzen for now.

You can get cards off of eBay etc. Which operate like below. They are usually about $20-$40 per card and $20-$40 for the cable

In case you wanted to add another GPU to your existing box.

Unlike epyc, ryzen does not support bifurcation, so you can't further split up your pcie lanes.

Good to know, thanks!

 

[Shmee]

Are we certain Ryzen doesn't support bifurcation? I am not certain, keep in mind Ryzen is a broad term...we could be talking an AM4 system, an AM5 system, or even one of the Ryzen TR platforms. So narrowing down to a chipset, say X570 on AM4, is it possible a some motherboards support bifurcation? Maybe with or without a BIOS mod. This can also depend on what CPU is used, possibly. (Zen 3 Ryzen CPU would be desired.)

I understand OP is on AM5, I could have sworn that those had bifurcation. I could be wrong of course.

I mention this as I have seen topics with people modding their BIOS to enable bifurcation on X570 boards, such as what I did myself on my X99 board, (though this is an Intel board). Anyway, you may want to look further into this if you do want to split lanes on either AM4 or AM5, it could be very situational depending on board, CPU, and firmware used.

EDIT: I came across this post:

https://www.reddit.com/r/Amd/comments/ycyn2i

which may shed additional light on this. Of course, a problem with the consumer platforms is that they are rather lane limited anyway.

 

[aigomorla]

you don't have enough lanes to run 3 videocards unless one is though usb 3.2 / thunderbolt on a external.

You need a HEDT.
Intel is the cheapest at the moment with pci-e 5.0.
3435x

https://www.newegg.com/intel-xeon-w5-3435x-lga-4677/p/N82E16819118452?Item=N82E16819118452

2455X

https://www.newegg.com/intel-xeon-w5-2455x-lga-4677/p/N82E16819118460

The complete board and cpu will net around 2400 / 1900 dollars. You will need 8 sticks of ddr5 ram tho on the top link, unless you decide to go on the cpu to a 2400 series which i would say do a HARD pass on, as you lose PCI-E lanes on a 3400 vs 2400.
However if your looking for the cheapest to support 3 video cards its the 2455X.

AMD TRPro's (Stormpeak) are just right around the corner, but no one has pricing on them yet, or if you can get the lower end cpu's without going though a SI.

The upper end TRPro and Xeon's expect to pay 5-6k just on the cpu, which isn't really worth it, unless you do heavy content creation / professional stuff that isn't quite yet on the level as full enterprise.

I understand OP is on AM5, I could have sworn that those had bifurcation. I could be wrong of course.

I don't think many will because only the first slot is a physical 16x.
Unless they decided to Bifur the second 8x slot into 2 x 4x.

EDIT:
PCIE bifurcation on ASRock Lightning PG X670E
PCIE bifurcation on ASRock X670E Taichi

So i guess the X670E's have them.

Typically i don't think the first slot which is always the dedicated gpu slot, would ever have bifur without bios mod on a consumer grade board that is non HEDT.

There might be some other enterprise class X670 that MIGHT have it as well, like ones from ASRack, or Supermicro.
But im not too familiar with that line of boards.

 

[Tech Junky]

PCIE bifurcation on ASRock Lightning PG X670E

I just built this one and am familiar with the specs.

1st x16 is PCIE 5 x16
2nd is PCIE 4 x4
3rd is PCIE 4 x1

I went with this one though because it doesn't auto slash the top slot like all others do when you put something into the 2nd slot. This is good for when you do want to bifurcate a full x16 electrical into x4's for storage use.

Can't speak to the Taichi though.... but, for the PG it's a finicky B to say the least. It took a couple of tries find the right UEFI version to get it to do a soft reboot w/o hanging and requiring a button press on the case. 1.28 works vs anything after it within the past couple of weeks. AMD / OEM UEFI is a mess though if you've seen any of the posts around the net. Nothing like Intel based systems that tend to just work.

As to the bifurcation thing... most just hide it under storage raid vs using proper naming.

 

[Tech Junky]

@aigomorla

Not to mention the bifurcation thing but how they dice up the lanes in AMD land can be like dodging land mines. Compared to Intel DMI vs AMD implementation it's kind of dumb. Putting Gen4 drives on a x4 uplink is a bit of bad engineering. The PG has 2 additional sockets under the chipset G3x2 G4x2 but, if you add up the bandwidth it's a bottleneck to plan for which most won't. At least on Intel w/ G4x8 DMI you can put 2 drives on w/o much hesitation of bottleneck issues but, when I was playing with the idea of using M2's for Raid 10 the GB Aero G ended up being my choice w/ 5 M2 sockets. It worked fine but, I found more capacity per dollar with U3 NVME instead. The U3 was the demise of my ADL setup though uncovering an issue with the MOBO that I thought was just a quirk for the past couple of years with a OTA Tuner card that didn't like it at all.

AMD is not for the novice unless you get a prebuilt IMO. There is so much research to do in comparison to Intel if you don't want to blow your foot off and pull your hair out while trying to do the first boot. If you have specific plans you really need to put in the time to find the right recipe to make things work. There really isn't a perfect mix though in the consumer realm price point. To get there either step up to Epyc / TR unlocks the shackles of lanes being diced up oddly. I started looking into Epyc when someone else mentioned it for something else. The option isn't too bad w/ the 9000 series CPU/board ~$3000

https://www.shopblt.com/cgi-bin/sho...0HT46P.shtml&order_id=522476033&sitem=B10HT46 $2263
https://www.newegg.com/asrock-rack-genoad8ud-2t-x550-amd-epyc-9004-series/p/N82E16813140098 - $800

This unlocks 128 G5 lanes and the freedom to do what you want where you want how you want. Might need some extensions and creativity for more than single slot cards but, the bandwidth is there to do it. It would be really nice if there were a board option like this for the Ryzen side of the house instead of the general market putting 10 x M2 sockets and 3 x16 slots that are gimped. It gets to be a real PITA when you want to take advantage of certain tech options and not be able to find a reasonable option that doesn't cost 4 figures or more to implement.