Can't boot with more than 16 NVMe cards

[Ty7]

I have a SuperMicro X10DRX MOBO and I'm trying to utilize 6 Amfeltec Squid PCIe carrier boards that have 4 Samsung 960 PRO M.2 NVMe cards each. So far, I haven't been able to boot the machine with more than 16 NVMe cards. It doesn't matter if I use 4,5, or 6 carrier boards as long as I don't use more than 16 NVMe cards at a time. Depending on which slots I use, the machine will either beep once and fail to power on or fail during boot with the error: "DXE-OOB Data Sync-Up."

This MOBO has 10 x 8 lane slots and each carrier utilizes 8 lanes so there should be plenty of lanes available. The output of lspci -vv prints a width of 8 for the LnkCap and LnkSta for each of these carrier boards.

I don't think it's a power issue because I still wasn't able to add a 17th NVMe card even after removing 3 other non-carrier cards (video, etc.).

The bios is up-to-date and all hardware is new.

Any suggestions?

 

[VirtualLarry]

Do those carrier boards require PCI-E lane bifurcation support? What CPU platform is that? I'm not certain that there are enough PCI-E Root Ports to go around.

 

[UsandThem]

That might a question to SuperMicro's tech support as there aren't probably many people who try and use 16 NVMe cards on that board, let alone more.

There was another guy here who couldn't get more than 20 SATA drives to work with his board, and just like your issue, there just won't be many people who can tell you what the motherboard's limitations are with unique setups.

 

[Ty7]

Do those carrier boards require PCI-E lane bifurcation support? What CPU platform is that? I'm not certain that there are enough PCI-E Root Ports to go around.

Do those carrier boards require PCI-E lane bifurcation support?
- Each NVMe card uses 4 lanes but the carrier boards use a switch to enable a fully loaded carrier board with 4 NVMe cards to only require 8 lanes.

What CPU platform is that?
- The chipset is C612 and the processors are Xeon E-2643

 

[Micrornd]

That board has a huge amount of variables in the bios that can interact and can/will make troubleshooting complicated.
Just a couple things you may or may not have tried -
Are there any clues in the Bios System Event Log?
Have you tried enabling/disabling "WHEA support" in the bios?
Have you tried enabling/disabling "Above 4G Decoding" in the bios?
Have you tried enabling/disabling any unneeded OROMs in the bios?
Have you tried enabling/disabling the "Interrupt 19 Capture" in the bios?
Have you made sure the "IIO1 and IIO2" settings in the bios are set to "Auto" and "Gen3"?
Have you tried enabling/disabling the "System Management Bus to PCI slot" jumpers?
Have you contacted SM ?
It may just be a straight forward arbitrary bios restriction that is hard coded in the bios that they can change or remove.
Or they may be able to tell you which bio settings or combination of settings is causing it.