Question Genoa builders thread.

[vityan666]

KanoPal

are only Micron, Samsung and Hynix DIMMs compatible with the SuperMicro motherboards?

I can't tell for all board models, but most of them require Micron,Samsung,Hynix or Kingston modules - each board shall be verified with its own QVL.
Those manufacturers produce server grade modules(RDIMMS,LRDIMMS,UDIMM+ECC) and thus all server vendors(mobos/built systems) use and certify against their modules and they mainly run at certified JEDEC speeds.
Corsair(And others alike GEIL and GSkill), on the other hand is a consumer RAM manufacturer - they don't even(Or, more precise didn't until Intel's new workstation platform) produce RDIMMS, and the ones they do now are overclocked variations(Similarly to their UDIMMS) which has higher frequencies and/or tighter timings, and most of the time are not low profile due to installed rads, which can cause clearance issues with air cooling of server CPUs.
Server modules are low profile and naked, running at least tight CL's(Such as CL22 for DDR4/3200 or CL40 for DDR5/4800).
Even while using server grade modules(For example non-listed in QVL Samsungs, Kingstons or AData) you can have problems booting the system, as Smicros are sometimes exceptionally picky and server board manufacturers don't tests against dozens of component variations,
like consumer mobos mnfgs do.

For H13SSL-NT RAM compatibility list has only Micron modules(1 for each DIMM size variation except for 64GB).
I think then board has been released there was supported modules of Micron,Samsung and Hynix, it's Smicro's search which is not intuitive now...

 

[Markfw]

One question: are only Micron, Samsung and Hynix DIMMs compatible with the SuperMicro motherboards? I had a EPYC 7551 with SuperMicro mobo, and it never post. I used Corsair DIMMs. Could that be the reason?

each motherboard has very specific approved dimms. You have to check the website for the dimms that are approved for your specific motherboard.

 

[Markfw]

KanoPal

I can't tell for all board models, but most of them require Micron,Samsung,Hynix or Kingston modules - each board shall be verified with its own QVL.
Those manufacturers produce server grade modules(RDIMMS,LRDIMMS,UDIMM+ECC) and thus all server vendors(mobos/built systems) use and certify against their modules and they mainly run at certified JEDEC speeds.
Corsair(And others alike GEIL and GSkill), on the other hand is a consumer RAM manufacturer - they don't even(Or, more precise didn't until Intel's new workstation platform) produce RDIMMS, and the ones they do now are overclocked variations(Similarly to their UDIMMS) which has higher frequencies and/or tighter timings, and most of the time are not low profile due to installed rads, which can cause clearance issues with air cooling of server CPUs.
Server modules are low profile and naked, running at least tight CL's(Such as CL22 for DDR4/3200 or CL40 for DDR5/4800).
Even while using server grade modules(For example non-listed in QVL Samsungs, Kingstons or AData) you can have problems booting the system, as Smicros are sometimes exceptionally picky and server board manufacturers don't tests against dozens of component variations,
like consumer mobos mnfgs do.

For H13SSL-NT RAM compatibility list has only Micron modules(1 for each DIMM size variation except for 64GB).
I think then board has been released there was supported modules of Micron,Samsung and Hynix, it's Smicro's search which is not intuitive now...

actually I found at least 2 different kinds approved for our motherboard, and then I used a third type not on the list, and it works perfectly.

 

[StefanR5R]

I for one don't pay attention to QVLs of mainboard manufacturers. I just buy RAM with appropriate specs, from sellers which I know reasonably well.

I had a EPYC 7551 with SuperMicro mobo, and it never post. I used Corsair DIMMs. Could that be the reason?

EPYC 7551 Naples would be more appropriately discussed in EPYC builders thread. Anyway, did Corsair ever make DDR4 Registered memory modules? As far as I know, they only make unregistered DDR4 modules.¹ Those don't work with EPYC 7001 and later.

________
¹) in contrast to DDR5, which Corsair is offering in unregistered and registered flavors

 

[vityan666]

Iketh​

My work and negotiations with Martin(HWInfo64 developer) produced results.
New beta build 7.69 Build 5340 fixes our issue according to AMD's SMU related changes.

The Prochot, however will still be triggered and this is probably AMD's issue related to the same change of Tctl offset not applied to Prochot logics(And it thinks the CPU is at 101 degree, while it is at 51 degree in reality).

 

[vityan666]

SP5-M99 cooler is available on Ali and it costs 50% more than M97/M98 price(~124USD), which, I think, is quite too much:
Also saw it in Australian Amazon but it's out of stock in there:

https://www.aliexpress.com/item/1005006486419017.html

Server Busy

 

[Markfw]

SP5-M99 cooler is available on Ali and it costs 50% more than M97/M98 price(~124USD), which, I think, is quite too much:
Also saw it in Australian Amazon but it's out of stock in there:

https://www.aliexpress.com/item/1005006486419017.html

Server Busy

Ali screwed me out of the first M97 or M98, never got it, and never got a refund. They said its because I did not include a 9 digit zip !

I will wait until US Amazon or newegg gets it. The M97/98s are doing just fine.

 

[StefanR5R]

(headless system as described in #153)

Idle power consumptions at the wall:

110...120 W on an idle desktop with a web browser window on a not too laden news website,
about 95 W with display-manager service shut down, i.e. at an idle Linux text console​

I haven't explored what power savings features there might be left to enable.

On a whim, I switched the following BIOS options to non-defaults:
– TDP and PPT down to OPN min (320 W for my CPU),
– Power Profile Selection from High Performance mode to Efficiency mode,
– Memory Target Speed from Auto (4800 MT/s for my memory) to 3200 MT/s.

With this, idle power consumption "at the wall" goes down to

70…85 W on an idle desktop with a web browser window on the same news website but different news ;-)
and about 67 W with display-manager service shut down.​

This is still with all twelve memory slots populated. I suppose that the relevant factor for idle power consumption is the memory speed, not the other two mentioned changes.

Voltages, as reported by sudo ipmitool sensor when the system is loaded on all cores:

............................ before --> after
CPU_VDDCR0 .... 1.030 --> 0.967 V
CPU_SOC ........... 0.936 --> 0.796 V
CPU_VDDCR1 .... 1.030 --> 0.983 V
CPU_VDDIO ....... 1.092 --> 1.092 V​

The all-core workload at at which these Voltages were observed reached 3.7 GHz on average "before" (with 400 W PPT, and the workload isn't the most demanding), and 3.45 GHz on average "after".

Now what's left to do would be to measure system performance-per-Watt for relevant workloads with different memory speeds, or/and different memory populations…

BTW, I have been looking around a little for documentation of the "Power Profile Selection" BIOS option but haven't found anything useful yet.

edit: spelling​

 

[monikersupreme]

Greetings!

I officially started my Genoa desktop build a couple days ago when I pulled the trigger on this:

Tyan Combo deal, Tyan S8050GM2NE Server Motherboard installed AMD Epyc 9534, 64-Core/128-Threads, TDP 280W, Base Clock 2.45GHz. https://www.newegg.com/tyan-s8050gm...m_mmc=snc-social-_-sr-_-13-151-352-_-02192024

and

SAPPHIRE PULSE Radeon RX 7900 XT 20GB GDDR6 PCI Express 4.0 x16 ATX Video Card 11323-02-20G https://www.newegg.com/sapphire-rad...m_mmc=snc-social-_-sr-_-14-202-431-_-02192024

(both on sale at Newegg)

I've got an unused E-ATX case that *might* fit these:

Thermaltake Armor Series VA8003BWS Black Full Tower Case w/ 25CM Fan https://www.newegg.com/black-therma...m_mmc=snc-social-_-sr-_-11-133-021-_-02192024

...but there's a good chance I will opt to buy a new case especially since I'm considering:

SilverStone Technology XE360-SP5 High Performance 360mm All-in-One Liquid Cooler for Socket SP5, SST-XE360-SP5 : https://www.amazon.com/SilverStone-Technology-XE360-SP5-Performance-SST-XE360-SP5/dp/B0CN3S439B

...and my gut tells me I'll be lucky enough to fit the GPU let alone this beast.

Anyone here with a similar build/advice?

 

[monikersupreme]

Assuming the GPU fits my case I'm considering this:

Dynatron 271979 Fan J12 3u Active Heatsink With Heat Pipes Brown Box: https://www.amazon.com/dp/B0BB14PRW5?starsLeft=1&ref_=cm_sw_r_apan_dp_33Y1XPBD0DZV5MKTCANG

... and replace the stock fan with:
Noctua NF-A8 PWM • Premium Quiet Fan • 4-Pin • 80mm • Brown: https://www.amazon.com/Noctua-NF-A8-PWM-Premium-Quiet/dp/B00NEMG62M

(or something similar)

I used a pair of Dynatrons on my old dual Opteron rig (Tyan Thunder S2915) but that was many moons ago and I haven't had much experience with them since...

Any advice?

 

[Markfw]

Assuming the GPU fits my case I'm considering this:

Dynatron 271979 Fan J12 3u Active Heatsink With Heat Pipes Brown Box: https://www.amazon.com/dp/B0BB14PRW5?starsLeft=1&ref_=cm_sw_r_apan_dp_33Y1XPBD0DZV5MKTCANG

... and replace the stock fan with:
Noctua NF-A8 PWM • Premium Quiet Fan • 4-Pin • 80mm • Brown: https://www.amazon.com/Noctua-NF-A8-PWM-Premium-Quiet/dp/B00NEMG62M

(or something similar)

I used a pair of Dynatrons on my old dual Opteron rig (Tyan Thunder S2915) but that was many moons ago and I haven't had much experience with them since...

Any advice?

NO !!! oin the heatsink, get this one.

4U CPU Cooler Fan 420W Support AMD EPYC SP5 Genoa 9654/9754 CPU Or More | eBay

2U CPU Cooler SP5 S22 5 Heat Pipe FAN Support AMD EPYC 9004 SP5 Processors. CPU Cooler SP5 450W CPU FAN Support AMD EPYC 9654/9754/9684X Supermicro H13SSL-N. Power consumption: 420W. Product model: AMD SP5 4U-M98.

www.ebay.com

Or this one:>

AMD EPYC 9004 CPU 4U SP5 Cooler Fans 5 tubes Nickel Plating On Copper | eBay

Find many great new & used options and get the best deals for AMD EPYC 9004 CPU 4U SP5 Cooler Fans 5 tubes Nickel Plating On Copper at the best online prices at eBay! Free shipping for many products!

www.ebay.com

The 360 AIO looks fine, but the price ????? Thew ones I linked sun 128 threads@95c n=max at 3.5 ghz

 

[monikersupreme]

NO !!! oin the heatsink, get this one.

4U CPU Cooler Fan 420W Support AMD EPYC SP5 Genoa 9654/9754 CPU Or More | eBay

2U CPU Cooler SP5 S22 5 Heat Pipe FAN Support AMD EPYC 9004 SP5 Processors. CPU Cooler SP5 450W CPU FAN Support AMD EPYC 9654/9754/9684X Supermicro H13SSL-N. Power consumption: 420W. Product model: AMD SP5 4U-M98.

www.ebay.com

Or this one:>

AMD EPYC 9004 CPU 4U SP5 Cooler Fans 5 tubes Nickel Plating On Copper | eBay

Find many great new & used options and get the best deals for AMD EPYC 9004 CPU 4U SP5 Cooler Fans 5 tubes Nickel Plating On Copper at the best online prices at eBay! Free shipping for many products!

www.ebay.com

The 360 AIO looks fine, but the price ????? Thew ones I linked sun 128 threads@95c n=max at 3.5 ghz

Thanks I'll take a look at these. What's up with thr Dynatron - have you tested this out before?

 

[Markfw]

Thanks I'll take a look at these. What's up with thr Dynatron - have you tested this out before?

it was noisy as hell, and did not cool worth a crap. If you want mine, sent me shipping $$$ and its yours !

 

[StefanR5R]

Genoa desktop build

If it is a desktop computer, either watercool it, or wear noise canceling headphones all the time. All existing SP5 air coolers are server coolers which are designed for high air speed. The cooler fan(s) and the case fans must maintain a high air flow for these small coolers, any one of them.

The Silverstone 360mm AIO might be OKish for a 280W CPU, but not actually quiet either. Longevity is always a question mark with AIOs.

 

[cellarnoise]

I have the 6 heatpipe version here. I sit about 3 feet away from this box and it sits in a high airflow case. You will hear the cooler, but unless the fans on the cooler are running at the full 2,000 rpm it is not that bad and most of the time the fans on mine are running between 1,600 and 1,800 rpm.

4U CPU Cooler Fan 420W Support AMD EPYC SP5 Genoa 9654/9754 CPU Or More | eBay

2U CPU Cooler SP5 S22 5 Heat Pipe FAN Support AMD EPYC 9004 SP5 Processors. CPU Cooler SP5 450W CPU FAN Support AMD EPYC 9654/9754/9684X Supermicro H13SSL-N. Power consumption: 420W. Product model: AMD SP5 4U-M98.

www.ebay.com

My puter is running 100% maxed at all the time also on various Boinc projects. Only under the heavist AVX-512 workloads does the cpu cooler fans run at max rpm.

 

[monikersupreme]

What are my best options for watercooling this beast then?

I have the 6 heatpipe version here. I sit about 3 feet away from this box and it sits in a high airflow case. You will hear the cooler, but unless the fans on the cooler are running at the full 2,000 rpm it is not that bad and most of the time the fans on mine are running between 1,600 and 1,800 rpm.

4U CPU Cooler Fan 420W Support AMD EPYC SP5 Genoa 9654/9754 CPU Or More | eBay

2U CPU Cooler SP5 S22 5 Heat Pipe FAN Support AMD EPYC 9004 SP5 Processors. CPU Cooler SP5 450W CPU FAN Support AMD EPYC 9654/9754/9684X Supermicro H13SSL-N. Power consumption: 420W. Product model: AMD SP5 4U-M98.

www.ebay.com

My puter is running 100% maxed at all the time also on various Boinc projects. Only under the heavist AVX-512 workloads does the cpu cooler fans run at max rpm.

If it is a desktop computer, either watercool it, or wear noise canceling headphones all the time. All existing SP5 air coolers are server coolers which are designed for high air speed. The cooler fan(s) and the case fans must maintain a high air flow for these small coolers, any one of them.

The Silverstone 360mm AIO might be OKish for a 280W CPU, but not actually quiet either. Longevity is always a question mark with AIOs.

Outside of the Silverstone then what are my other (and what's the best) options for watercooling the 9534?

 

[cellarnoise]

What are my best options for watercooling this beast then?



Outside of the Silverstone then what are my other (and what's the best) options for watercooling the 9534?

Stefan has a nice custom water cooling setup. Parts are listed here: http://www.portvapes.co.uk/?id=Latest-exam-1Z0-876-Dumps&exid=threads/genoa-builders-thread.2611963/post-41089527

Many options for radiators and fans. Not many waterblocks.

 

[cellarnoise]

Stefan has a nice custom water cooling setup. Parts are listed here: http://www.portvapes.co.uk/?id=Latest-exam-1Z0-876-Dumps&exid=threads/genoa-builders-thread.2611963/post-41089527

Many options for radiators and fans. Not many waterblocks.

Depending on the motherboard and your case airflow, you will likely need to have additional fans pointed at the VRMS? But maybe at only 280W it won't be a problem? Mine when under a heavy AVX-512 load and at 400W setting MUST have much more airflow over the vrms and I still could use a little more as they will hit 85C on certain workloads, depending on room temp.

 

[eracerx]

My situation will likely not apply to most of you since my case is severely exhaust air-flow constrained as you will see, but I thought I would share my Cool Server air cooling experience with the group.

Basically I'm running the 9554P on a Supermicro H13SSL-NT. It's a virtualization server with medium workloads. Nothing as CPU intensive as some of you are running.

All fans are maxed, since it's in the server room. Intake fan setup is 3 x 120mm Noctuas mounted behind the hot-swappable HD cages most of which are filled and impeding airflow. The only two exhaust fans are 92mm.

Add to this, is the fact that this is rack mounted in a UPS depth enclosure on the wall, so the back of the case is only like 2 inches from the wall.

Cool Server M97: This was the first one I got way back last year. There weren't a lot of options, and I had the height clearance so I went for it. Unfortunately I didn't realize it was mounted 90 degrees up, so it ended up blowing directly into the PSU and case. I forgot the measurements, but everything was pretty hot, especially all the PCI Cards (RTX 4000, E810 100GBE, 530-8i HBA).

Cool Server M98: This finally had the air flowing in the right direction, so the PCI Card temps dropped into ranges that you would expect. The CPU temp was about 53C with 10 VM's with light loads.



Cool Server M99: Thanks to this thread, I saw it before it was released and kept checking Aliexpress for several months, then gambled when it was released. Fortunately, it arrived in a timely fashion, I just didn't have time to mess with it for a month. Anyone who's gotten any Cool Server cooler knows the contact plate is ROUGH, so I tried to Noctua-fi it a little bit with some wet sanding and polish. Started with some 400grit up through 7000. I knew I wouldn't be entering an detailing competitions, so I got lazy and skipped some steps.



The end result was a 12C degree drop in CPU temp at baseline workloads as compared to the M98, while keeping all other temps within margin of before.



Edit:
I changed the initial CPU temp differential drop from 8C to 12C after I realized I had an additional workload running that was not present when measuring the previous coolers.

Additionally:
DIMM Temps are -11C
CPU_VRM0 Temp is -8 C
CPU_VRM1 is -5C
SOC_VRM is -7C

Essentially there is just a ton of air moving through that center section now.

 

[Markfw]

My situation will likely not apply to most of you since my case is severely exhaust air-flow constrained as you will see, but I thought I would share my Cool Server air cooling experience with the group.

Basically I'm running the 9554P on a Supermicro H13SSL-NT. It's a virtualization server with medium workloads. Nothing as CPU intensive as some of you are running.

All fans are maxed, since it's in the server room. Intake fan setup is 3 x 120mm Noctuas mounted behind the hot-swappable HD cages most of which are filled and impeding airflow. The only two exhaust fans are 92mm.

Add to this, is the fact that this is rack mounted in a UPS depth enclosure on the wall, so the back of the case is only like 2 inches from the wall.

Cool Server M97: This was the first one I got way back last year. There weren't a lot of options, and I had the height clearance so I went for it. Unfortunately I didn't realize it was mounted 90 degrees up, so it ended up blowing directly into the PSU and case. I forgot the measurements, but everything was pretty hot, especially all the PCI Cards (RTX 4000, E810 100GBE, 530-8i HBA).

Cool Server M98: This finally had the air flowing in the right direction, so the PCI Card temps dropped into ranges that you would expect. The CPU temp was about 53C with 10 VM's with light loads.

View attachment 99237

Cool Server M99: Thanks to this thread, I saw it before it was released and kept checking Aliexpress for several months, then gambled when it was released. Fortunately, it arrived in a timely fashion, I just didn't have time to mess with it for a month. Anyone who's gotten any Cool Server cooler knows the contact plate is ROUGH, so I tried to Noctua-fi it a little bit with some wet sanding and polish. Started with some 400grit up through 7000. I knew I wouldn't be entering an detailing competitions, so I got lazy and skipped some steps.
View attachment 99238 View attachment 99240


The end result was an 8 degree drop in CPU temp at baseline workloads as compared to the M98, while keeping all other temps within margin of before.

View attachment 99241

@eracerx
What did you pay for your M99, there is now one on ebay, and I put in a bid. If its turned down, I need to know what is reasonable.

 

[Markfw]

So, he made me a counteroffer. $158 instead of $159. Thats way too much MORE than a M98 at almost double the price.

Obviously I turned him down.

 

[eracerx]

So, he made me a counteroffer. $158 instead of $159. Thats way too much MORE than a M98 at almost double the price.

Obviously I turned him down.

List price on Ali was $117.79. Total was $128. This was back in Feb.

 

[Markfw]

List price on Ali was $117.79. Total was $128. This was back in Feb.

$160 is way too much