Question Genoa builders thread.

[Markfw]

Anyone have a link to purchase the Cooler Server 4U-M99?

Believe me, if I had one, I would link immediately.

OK, new news, I have no memory yet, but my $300 motherboard sure looks brand new/sealed. Its a $700 motherboard. Now sure who messed up.

Pic with CPU only installed. I used a magnifying glass on the pins first, and they looked OK, and the protective covers were on.

Then case/PSU (1000 watt) and hsf.

Memory won't be here until next Tuesday ! damn....

 

[Markfw]

Genoa number 5 done. Note, and feedback from other requested. I plugged the USB 3 cord into the motherboard and attempted to use those ports. I had nothing but issues. So, I totally power down, and plugged everything into the USB ports in the back, and then it booted ! I don't have time for other troubleshooting, as its working perfectly on primegrid, so in 8-9 days I may mess with it more.

 

[vityan666]

Long time I didn't came here.
So I've reached 1 month+1 weak of stable run and then got another uncorrectable ECC error. Then I screwed a heatsink a little more(Memtests passed again). Afterwards I had week only of stable runtime.
I've discovered new BIOS 1.6 for my H13SSL-NT, and it's a little strange for me.
Why everything is similar to previous BIOS's the Tctl shown in HWInfo64 and temp shown in Memtest86(The same value) is 98-117 degrees of celcius instead of 45-60 with previous BIOS's.
tctl in Linux(sensors command) is somehow fine for both BIOSs(1.5 and 1.6).
Then it reaches the bad mark of 101(Which is probably 50-55) and up HTC and Prochot are triggered.

All CCDs and IOD die have the same temps as before(Normal ones), IPMI is fine as well. Working with SM support right now on this issue and they are not able to reproduce it for now.
I suspect possibly bad heatsink mount overall.
@Markfw , I must ask, then you install the M97's you leave the pre-applied thermal compound on it or replace it with your own(Noctua/Arctic)?
::HEADACHE::

Markfw
So, Coolserver released extra modifications of their SP5 cooler, and now have larger M99, Just like the famous M99 heavy duty sniper barret
Even the middle M98 is nowhere to be found... Still didn't expect from therm to make more coolers for SP5(Judging from other sockets product lines).

Genoa number 5 done

Congrats
I still struggle with my 1-st

Silverstone's AIO is available on eBay, but I highly doubt their quality(Especially the possible Alumi/Copper mixing), and reviews about their other AIO systems with pumps dying withing half year for some ppl(But it can be rare cases - I dunno, ppl w/o problems don't tend to write about experience).

Silverstone XE360-SP5 AMD SP5 SOCKET CPU AIO Water Cooling Kit | eBay

Water pump integrated within the radiator. 3 x 120mm radiator-optimized cooling fans. Nickel plated copper water block with large-sized micro-channel copper base plate. Aluminum alloy cavity pump strengthens the overall structure.

www.ebay.com

 

[Markfw]

I have some Genoa with the original paste ( the lastest one, and others with aftermarket. Depends on me replacing the HSF's with the better ones. Not much different. They have been running 24/7 @ 100%. no errors. Did you use the approve supermicro memory ?

 

[vityan666]

Markfw

Did you use the approve supermicro memory ?

Yep, The Micron's 2R 32GB modules and their Smicro rebranded versions(So a mix of original Micron's and Smicro's rebranded version of the same module).

They have been running 24/7 @ 100%. no errors.

So original compound is not highly degrading within first 6-8 months.
Maybe my problems have to do with running for few months with SP5 screws not fully tighten(Without too much force, ofc) and I have unbalanced thermal contact in some area...

 

[Markfw]

Markfw

Yep, The Micron's 2R 32GB modules and their Smicro rebranded versions(So a mix of original Micron's and Smicro's rebranded version of the same module).


So original compound is not highly degrading within first 6-8 months.
Maybe my problems have to do with running for few months with SP5 screws not fully tighten(Without too much force, ofc) and I have unbalanced thermal contact in some area...

I used the approved torque, same as with threadrippers.

 

[vityan666]

Markfw

I used the approved torque, same as with threadrippers.

I have only standard screwdrivers, unfortunately, for manual screwing w/o torque control, like used for consumer CPU coolers installation(And even then first installing I didn't have full T20 screwdriver and took a bit for it from a friend - It was short, so it was extremely hard for me to screw the screws just near the radiator). Still AMD's recommendation is 12.5-15.0 kgf/cm, which is not a small precise range. My main task was to not apply too much force, to not not break it physically. And its scary then dealing with CPU this complex,large and pricey...

 

[Trotador22]

Even the middle M98 is nowhere to be found...

bay

 

[Iketh]

Long time I didn't came here.
So I've reached 1 month+1 weak of stable run and then got another uncorrectable ECC error. Then I screwed a heatsink a little more(Memtests passed again). Afterwards I had week only of stable runtime.
I've discovered new BIOS 1.6 for my H13SSL-NT, and it's a little strange for me.
Why everything is similar to previous BIOS's the Tctl shown in HWInfo64 and temp shown in Memtest86(The same value) is 98-117 degrees of celcius instead of 45-60 with previous BIOS's.
tctl in Linux(sensors command) is somehow fine for both BIOSs(1.5 and 1.6).
Then it reaches the bad mark of 101(Which is probably 50-55) and up HTC and Prochot are triggered.

All CCDs and IOD die have the same temps as before(Normal ones), IPMI is fine as well. Working with SM support right now on this issue and they are not able to reproduce it for now.
I suspect possibly bad heatsink mount overall.
@Markfw , I must ask, then you install the M97's you leave the pre-applied thermal compound on it or replace it with your own(Noctua/Arctic)?
::HEADACHE::

Markfw
So, Coolserver released extra modifications of their SP5 cooler, and now have larger M99, Just like the famous M99 heavy duty sniper barret
Even the middle M98 is nowhere to be found... Still didn't expect from therm to make more coolers for SP5(Judging from other sockets product lines).


Congrats
I still struggle with my 1-st

Silverstone's AIO is available on eBay, but I highly doubt their quality(Especially the possible Alumi/Copper mixing), and reviews about their other AIO systems with pumps dying withing half year for some ppl(But it can be rare cases - I dunno, ppl w/o problems don't tend to write about experience).

Silverstone XE360-SP5 AMD SP5 SOCKET CPU AIO Water Cooling Kit | eBay

Water pump integrated within the radiator. 3 x 120mm radiator-optimized cooling fans. Nickel plated copper water block with large-sized micro-channel copper base plate. Aluminum alloy cavity pump strengthens the overall structure.

www.ebay.com

Apparently we all have the 100C+ tctl/tdie. That's good info though - happening with 1.6 and windows only. You haven't done anything wrong to your cpu or bad mount.

 

[vityan666]

Trotador22

bay

Thanks, while many of links point to M97(You can know it by '5 pipes' in description), there are indeed 2 sellers with M98 cooler.
No signs of M99 through, which has Fans rotated 90 degree(I.e going across the long side of the socket) allowing for horizontal airflow direct front intake to rear exhaust, which is nice(With M97 its bottom to top).

Iketh

Apparently we all have the 100C+ tctl/tdie.

Interesting.... So it's not only me .
Do you have HTC and Prochot triggered as well then it crosses the 101 degree border? I feel that system is indeed slower and it trips Prochot a lot more and for a long periods of time, especially during/after gaming than with older BIOS's.
From other side older BIOS's had Prochot triggering too sometimes without any temp being high, but HTC was not tripped and Prochot Ext was, so it was due to some other unknown component. With 1.6 the main Prochot+HTC is triggered.
Memtest86 speed does not seem to be affected(Judging by the time spend on pass 1 and all 4 passes w/o Hammer).

That's good info though - happening with 1.6 and windows only.

Yep, Linux shows it normally. However, it's not only Windows - Memtest86(Bootable) has it's temps detection messed as well as I've said already...

You haven't done anything wrong to your cpu or bad mount.

I hope so, or we both have bad mounts
Smicro can't reproduce it, so possibly there are configurations of H13SSL-NT+Epyc Genoa+BIOS 1.6 with normal Tctl detection, or they didn't test with Memtest86 yet?

Just for sake of finding similarities in our configs:
1) Do you have H13SSL-N or H13SSL-NT or another Smicro Board?
2) Do you have regular or frequency optimized(Model with F in the end, and higher base freq) Epyc Genoa CPU?
3) Workstation/Desktop chassis or rack(If rack then Smicro's or custom)?
4) Which CPU cooler(Dynatron/Smicro OEM/CoolServer)?
5) CPU cooler screwing: Did you used torque limited screwdriver or manual? Did you screwed 1-6 in one pass(As, seems like the recommended way)? I've used 'Car wheels strategy' - multiple times 1-6, half-full rotation each time to reach even pressure - But maybe 1-2 being first+instnt screw reccomendation is to screw IOD die harder than CCD's... I'm not sure at this point. I've ordered 1.2-3.0Nm Variable Torque Wera screwdriver from Amazon just to have it for future, while I still quite trust my manual screwing sensitivity, and SP5 heatsing mounting spring system to provide a contra against possible excessive input torque(Like Corsair's AIO and many others do), we can never know...

If we only could know if @Markfw 's systems exhibit similar behaviour...

 

[Markfw]

If we only could know if @Markfw 's systems exhibit similar behaviour...

Not sure exactly what behaviour... I can say that at 100% load I am at 76c temp. Sometimes I have seen it as high as 87, once at 93.

 

[vityan666]

Markfw

Not sure exactly what behavior...

With BIOS 1.6, booting into Memtest86 ISO/USB, even without starting test(Just go into System Information), so the system is actually idle - It will show 95-100 degree Celcius temp. Will show 115-117 after load.
In earlier BIOSs(<= 1.5) it was fine(40-50 in my case).
Also I suspect that real throttling will happen reducing real life performance in this case, but not sure(As I judge this part only by feeling - Like few seconds slower starting of specific game, More stutters in it against almost no-stuterring, etc).

 

[StefanR5R]

I've still got BIOS v1.1a. The two quirks which I observed so far are still not mentioned in v1.6's release notes. Given the regressions which you are seeing, I certainly won't update my BIOS anytime soon.

 

[Markfw]

1.1a and no, never seen temps over 93 for any reason. I just checked another box. bios 1.0 ! The most recent purchase. And I don't update unless there is a reason to. All 5 Genoa boxes have been perfectly stable at 100% load their entire lifetime.

 

[vityan666]

StefanR5R

Given the regressions which you are seeing, I certainly won't update my BIOS anytime soon.

I needed the Inception vulnerability protection(In the firmware, due to Windows which unfortunately doesn't have normal real-time firmware updates like Linux, and WU can get late by months/years) so 1.5+ was needed for me anyway. Also one of changes was fix to sometimes incorrect DIMM slot reported in the SEL on errors(I think this is one of the changes in 1.6).

For the sake of lols, this is screenshot of Sensors page from BMC 01.01.08, It's so narrow, that you have horizontal scrolling in the bottom of it, so you can scroll to field by field... Thankfully Excel export is working






Markfw

1.1a and no, never seen temps over 93 for any reason.

1.1a - It was my first BIOS. Stable one.
Sincerely, while I didn't expect super stability from an AMD platform here(Because I know enough of it from consumer parts - Out of 15 BIOS versions for ASUS mobo I had only 3 for my R9 3900X/X570, which I carefully selected after reading zillion of reviews/reddit threads about each, with many others having critical issues), but expected it to be better in server platform, but seems like AGESA circo continues - each version is complete gamble. Waiting for AMD's announced plan to gradually replace AGESA with their own open source firmware, which will probably be a jump in quality similar to the fglrx->oss radeon drivers. BMC bugs are Smicro's only fault...

 

[Iketh]

Trotador22

Thanks, while many of links point to M97(You can know it by '5 pipes' in description), there are indeed 2 sellers with M98 cooler.
No signs of M99 through, which has Fans rotated 90 degree(I.e going across the long side of the socket) allowing for horizontal airflow direct front intake to rear exhaust, which is nice(With M97 its bottom to top).

Iketh

Interesting.... So it's not only me .
Do you have HTC and Prochot triggered as well then it crosses the 101 degree border? I feel that system is indeed slower and it trips Prochot a lot more and for a long periods of time, especially during/after gaming than with older BIOS's.
From other side older BIOS's had Prochot triggering too sometimes without any temp being high, but HTC was not tripped and Prochot Ext was, so it was due to some other unknown component. With 1.6 the main Prochot+HTC is triggered.
Memtest86 speed does not seem to be affected(Judging by the time spend on pass 1 and all 4 passes w/o Hammer).


Yep, Linux shows it normally. However, it's not only Windows - Memtest86(Bootable) has it's temps detection messed as well as I've said already...


I hope so, or we both have bad mounts
Smicro can't reproduce it, so possibly there are configurations of H13SSL-NT+Epyc Genoa+BIOS 1.6 with normal Tctl detection, or they didn't test with Memtest86 yet?

Just for sake of finding similarities in our configs:
1) Do you have H13SSL-N or H13SSL-NT or another Smicro Board?
2) Do you have regular or frequency optimized(Model with F in the end, and higher base freq) Epyc Genoa CPU?
3) Workstation/Desktop chassis or rack(If rack then Smicro's or custom)?
4) Which CPU cooler(Dynatron/Smicro OEM/CoolServer)?
5) CPU cooler screwing: Did you used torque limited screwdriver or manual? Did you screwed 1-6 in one pass(As, seems like the recommended way)? I've used 'Car wheels strategy' - multiple times 1-6, half-full rotation each time to reach even pressure - But maybe 1-2 being first+instnt screw reccomendation is to screw IOD die harder than CCD's... I'm not sure at this point. I've ordered 1.2-3.0Nm Variable Torque Wera screwdriver from Amazon just to have it for future, while I still quite trust my manual screwing sensitivity, and SP5 heatsing mounting spring system to provide a contra against possible excessive input torque(Like Corsair's AIO and many others do), we can never know...

If we only could know if @Markfw 's systems exhibit similar behaviour...

Here's a screenshot to answer some of your questions. I remounted the cpu in the socket and heatsink 4 times 4 different ways. Nothing changed. I've confirmed nothing is throttling and this is probably just a reporting error.

I'm running H13SSL-N + 9654 in desktop chassis using Silverstone water cooling.

It's not just you and I. Many are reporting the error online. Probably all 1.6+Windows has the error, but most don't notice since performance is unaffected.

 

[vityan666]

Iketh

It's not just you and I. Many are reporting the error online.

Ok, I just didn't managed to find anything by Googling, unless it's on closed forums.

Silverstone water cooling.

It's the mentioned previously XE360-SP5? I'm little skeptical about its quality right now. Give us, pls, some feedback after 1y of runtime of this one

Here's a screenshot to answer some of your questions. I remounted the cpu in the socket and heatsink 4 times 4 different ways.

Yep, Similar to my case with this BIOS, but yours have slightly higher tops(121 vs 118), probably due to slightly higher minimal temps(42 vs 34) and also HTC and Prochot shown on.
The only bad thing in my case is high peak 'CPU Die/Avg' value, which (Unlike the Tcl) I suspect is real (51.2-72.6 on yours and 65.9-97 on my) which can indicate of real overheating in my case...
CCDs peak near 73 for both of us.

 

[Iketh]

Iketh

Ok, I just didn't managed to find anything by Googling, unless it's on closed forums.


It's the mentioned previously XE360-SP5? I'm little skeptical about its quality right now. Give us, pls, some feedback after 1y of runtime of this one

Yep, Similar to my case with this BIOS, but yours have slightly higher tops(121 vs 118), probably due to slightly higher minimal temps(42 vs 34) and also HTC and Prochot shown on.
The only bad thing in my case is high peak 'CPU Die/Avg' value, which (Unlike the Tcl) I suspect is real (51.2-72.6 on yours and 65.9-97 on my) which can indicate of real overheating in my case...
CCDs peak near 73 for both of us.

If you have a HWiNFO debug file, attach it to this thread for the author. It could be HWiNFO problem.

https://www.hwinfo.com/forum/threads/possible-incorrect-prochot-readout.8924/

 

[vityan666]

Iketh

If you have a HWiNFO debug file

I tried to generate one a week ago, however in my case it doesn't export anything for the 'Sensors' category, so I can only export main computer information, and without Sensors its useless for our issue.
My case can be a little more complicated, oooor may-be even somehow related, because I'm running an Hyper-V host, so not all hardware is directly accessed. Do you run Hyper-V host too by the way?

I contacted Passmark yesterday, so maybe they will be able to find something on their side.

 

[vityan666]

Iketh
Meantime I've ordered M98 Air and Silverstone's AIO.
Passmark got back to me with the following reply:

We made some changes to apply an offset when certain bits get set after the BIOS update.

And a testing 10.6 Build 3004 build. I'm, however not in mtc window now, so can't shutdown yet. It will probably take until components arrival or another crash(Tfu-tfu-tfu), or centralized Windows Update deployment.
You can refer to this, however, hoping it can guide HWInfo64 developer in the correct direction...

Markfw
I've asked the seller(AliEx), I've ordered a M98 from about M99 and they say, they don't have a stock of it yet.
So, I guess it will be available for purchase within 1-2 months...

 

[Markfw]

Iketh
Meantime I've ordered M98 Air and Silverstone's AIO.
Passmark got back to me with the following reply:

And a testing 10.6 Build 3004 build. I'm, however not in mtc window now, so can't shutdown yet. It will probably take until components arrival or another crash(Tfu-tfu-tfu), or centralized Windows Update deployment.
You can refer to this, however, hoping it can guide HWInfo64 developer in the correct direction...

Markfw
I've asked the seller(AliEx), I've ordered a M98 from about M99 and they say, they don't have a stock of it yet.
So, I guess it will be available for purchase within 1-2 months...

I will never do business with Aliex. They screwed be out of $100 on my first cooler. I had to go to VISA and spend 3 months to get my money back. They said that even though it was in the US when it stopped be moved, because I did not include the 9 digit zip, it was undeliverable. Thats BS, Fedex, USPS UPS and more can deliver with a 5 digit zip.

 

[vityan666]

Markfw

I will never do business with Aliex. They screwed be out of $100 on my first cooler.

I know. I remember the story quite well, with the Alibaba seller at the time with the first M97.
Thats why I didn't provide any links - the point of the message was about future availability of M99 in general and that some sellers, at least informed about the new model, and stocks are planned,
and if sellers from Ali will get it, then sellers from other platforms, such as eBay will probably too, as well.

 

[vityan666]

So, I've had another crash(Just a reboot without any SEL/IPMI entries).
This time I replaced my NF-A14 with it's Industrial counterpart which has higher min RPM, to fix the issue of IPMI going mad and constantly warning about low RPM of FAN3(Even if its above the set minimum of 140) and now they don't go below 260RPM.
I disassembled the M97 heatsink(Didn't see problem with original thermal compound spread - It was quite even across the chip cover) and, after cleaning both surfaces, visually observing the socket and the CPU, reseated the CPU and reinstalled everything.
Temps didn't change much, even the Die(Average), which I preferred to go down 10-20 degree . At least they didn't increase
Used the situation to run Memetest86 test build and it didn't fix the bug as it still shows high temp value with BIOS 1.6, however:
Memtest86 runs faster, like it was before I extra-screwed the screws back then(2-3 weeks ago), and the system no longer stutters. Also, While following the screw order from SMicro manual I saw that it's orientation is like you mount looking from the top side of the motherboard(While I install from the bottom side), so my previous install was with numbers reversed (Such as 1<->2). While it looks symmetric and should not pose a problem, this time I installed from the top of the mobo ))
I would prefer this happening after receiving the AiO but that a life...

Extra strange thing is IPMI sensors page now look fine... May-be it just needed a Power reset...
UPDATE: Or not, Now IMPI Web Sensors page is bugged again... )))
UPDATE 2:
I think I got it:
Dashboard -> Sensor Readings Is bugged
System -> Component Info -> Sensor is fine

 

[vityan666]

So, after almost month of uptime and nearly 2 hours before planned MTC(Installed Windows updates on VM's and started host updates preparations) I've got another crash(LOL) - So close to the normal reboot...
The crash caused restart again(Not BSOD) and the system was in strange state - Drives would not all get loaded, Explorer not even opening, incl User profile, and backup files on external WD drive listed as corrupted.
Ok, I through to myself - Or I have here total disaster or I'm up to something - I've loaded the RAID drivers in the recovery boot session and checked backups located in there - They was fine, so I started to suspect the external WD drive, disconnected it's USB cable,
rebooted and everything suddenly became normal. It was the stress relaxing part

Going forward I finished the MTC software updates, and came to finally testing the updated MemTest86 build image from the devs - They indeed was right about some CPU flag meaning in the new BIOS and temps displayed are now normal, during second run I also compared their temp against IPMI - full match, so it's all software related and caused by changed Tctl offset. @Iketh I asked Passmark in the mail if they could collaborate on a flag affected, will write back then I have any answer.

Then I moved to the main and largest plan of the Mtc window - AiO installation which had also some smaller minitasks I've planned to do with it.
So, I've added C14 retention sleeves(In place of my ziptie-parts hack) to the loose C14 plugs of Amazon Cables, which work quite well.
Replaced the NF-A4x20(Blowing from top) with newly arrived NF-A4x10(Side blowing, fits due to smaller height) to cool the Broadcom heatsink on a mobo.
Connected C14-C13 right angle adapter to my Cisco switch, so I can now push it further in to the wall and updated it's firmware.
Installed the freshly arrived Micron 7450 U.3 7.68TB NVMe drive with my cartoon cable support to replace my aging Samsung 860 Evo 2TB SATA SSD(Which does a perfect job) as fast drive for games and performed init and cloning using the old good Clonezilla.

This time I've used the Wera torque screwdriver adjusting from ~1.22nm to 1.35nm and I definitely feel the springs doing its work properly, providing counter-force to your torque and stopping you at the needed point(On both M97 and SE360-SP5) and regular manual screwdriver is more than enough. This time I didn't just clean it with a paper towel bu used Noctua's safe NA-SCW1 wipes + fast drying with paper towels.

System is working well at this point and during my tests in Linux I've connected the WD drive back and it worked fine in Linux. Restarted with it and everything is good now. Probably it's USB cable became loose because it sits on top of the UPS and I always rotate it to open the main door of the PC tower(Also it did not restarted for months - so maybe it's controller needed a fresh start as well - I did a poff/pon cycle on it just for the sake of it).

What can I say... ~Month of uptime is much better than 1w but this system drives me a little crazy
Dunno if it will become finally stable like my prior systems which could easily run 2.5m btw updates(As I skipped the problematic MS updates during some months). Holding the fingers.

Attached the picture with AiO and NF-Ax10 in the final build.

 

[KanoPal]

Markfw​

I've got the Smicro compatible Micron 32GB 2R RDIMM modules. The first 6 are retail by Micron, the new 3 from Newegg are same model rebranded by SuperMicro(Still waiting for them).
Micron part#: MTC20F2085S1RC48BA1R
I've selected it out of 3 compatible options(By Micron, Samsung and Hynix). Hynix was 1R and uses older manufacturing process, Micron had much more availability via eBay back then, so it was an easy choice
It wasn't available via Newegg then I've ordered but its there now.

In my case it's was just really a bad module, not a compat issue. Happens.

One question: are only Micron, Samsung and Hynix DIMMs compatible with the SuperMicro motherboards? I had a EPYC 7551 with SuperMicro mobo, and it never post. I used Corsair DIMMs. Could that be the reason?