Intel processors crashing Unreal engine games (and others)

[DrMrLordX]

Interesting! Wendel's video at least confirms the thought that Alder Lake didn't have this problem plaguing Raptor Lake.

 

[Jan Olšan]

I recall that initial leaks seemed to point to Alder Lake / Core i9-12900K having thermal velocity boost for 5.3 GHz single-core boosting, but then that got dropped and it ended up at 5.2 GHz. I wonder if the boosting behaviour is significantly changed in 13th/14th gen, given how big difference in clocks there was. I thought it is largely the same. (i9-12900KS does have TVB iirc?)

It is possible the trouble is merely less likely to happen on 12th gen though, because it has lower clocks. Also there is a process node difference, IIRC. Raptor Lake uses what is unofficially called "Intel 7 Ultra". In theory, some knob turned to allow binning up to 5.8/6.2 GHz may turn out to be unstable.

Hopefully the new clock control used in Lion Cove will caught up with AMD SMU's boosting control abilities and the FIT protections and will be advanced enough to prevent chip damage/this instability snafu.

 

[coercitiv]

It is possible the trouble is merely less likely to happen on 12th gen, because it has lower clocks.

I don't think so, Wendel says the 12900K is not anomalous at all in the telemetry data, has error rates similar to AMD products. (while 13900K/14900K are clear outliers) So on one hand data seems to say 12900K is healthy, on the other hand it is the SKU which had the most time to degrade out of all... and it seems to be doing fine.

At least one Datacenter Service Provider agrees with the info above:


Also, to complete the answer, the video contains a segment about configuring servers, and even lowering max multiplier for 13900K/14900K to 53X and lowering DDR5 speeds would not completely eliminate issue, although it is considered the most stable config. This would be very close to 12900K, and yet the new chips still fail more apparently.

 

[igor_kavinski]

Hopefully the new clock control used in Lion Cove will caught up with AMD SMU's boosting control abilities and the FIT protections and will be advanced enough to prevent chip damage/this instability snafu.

If ARL is lagging behind in Blender vs. Zen 5 in their internal benchmarks, Intel might do something desperate again.

 

[DAPUNISHER]

I retract my prediction about what this will cost Intel. Clearly I underestimated the extent of the problem. This is going to get expensive, what with needing to now factor in DCS providers steering customers to other vendors. Lost sales and RMAs in retail = meh, but it happening in DC? That hurts bad.

EDIT: I hope Wendell is right in his speculation about frustrated clients leaking data. Would love to know how many prebuilts are failing too.

 

[zir_blazer]

This is beginning to give me Pentium 3 1.13 GHz Coppermine vibes. Or SNDS (Sudden Northwood Death Syndrome) vibes. Or perhaps Intel i820 vibes. Here is another one.
Or perhaps the Intel 6 series Chipset SATA bug. Or the Atom C2000 Apollo Lake AVR54 bug.

It is technically possible that Intel hitted both things at once: Motherboard vendors aggressive settings making factory overclocked Processors to degrade faster than what they would have done, but it was above 0 so it was going to happen in a year or two anyways.

 

[Jan Olšan]

I retract my prediction about what this will cost Intel. Clearly I underestimated the extent of the problem. This is going to get expensive, what with needing to now factor in DCS providers steering customers to other vendors. Lost sales and RMAs in retail = meh, but it happening in DC? That hurts bad.

EDIT: I hope Wendell is right in his speculation about frustrated clients leaking data. Would love to know how many prebuilts are failing too.

Idunno, it seems AMD is cursed and the OEMs won't move to their stuff significantly no matter what. They are more likely to jump to whoever gives them ARM chips even with dubious track record or other disadvantages instead, even.

 

[DAPUNISHER]

I dunno, it seems AMD is cursed and the OEMs won't move to their stuff significantly no matter what. They are more likely to jump to whoever gives them ARM chips even with dubious track record or other disadvantages instead, even.

You may be right, ARM could end up being the "combo breaker". AMD is fortunate to be getting some business at Intel's expense due to the CPU failures. It's anyone's guess how much it has, or will, amount to.

It warrants consideration that where Intel is currently, is unprecedented. AMD is valued higher by investors. @NTMBK just hit me with AMD passing Intel for brand recognition. I was a little shocked by that. https://www.tomshardware.com/tech-i...15.1512832452.1717590128-304161622.1707544553

This CPU degradation problem looks to be widespread and concerning, at a time when they can ill afford the expense, loss of sales, and bad press. Board makers are unhappy, some retail customers are unhappy, and now we are hearing OEM and DC customers may be unhappy. Forget AMD's position, Intel's is looking a bit grim historically speaking.

 

[igor_kavinski]

There could be an upside. Intel goes super lean, massive job cuts and all those employees band together and get an ARM license and we have one new player in the game!

 

[Joe NYC]

You may be right, ARM could end up being the "combo breaker". AMD is fortunate to be getting some business at Intel's expense due to the CPU failures. It's anyone's guess how much it has, or will, amount to.

It warrants consideration that where Intel is currently, is unprecedented. AMD is valued higher by investors. @NTMBK just hit me with AMD passing Intel for brand recognition. I was a little shocked by that. https://www.tomshardware.com/tech-i...15.1512832452.1717590128-304161622.1707544553

This CPU degradation problem looks to be widespread and concerning, at a time when they can ill afford the expense, loss of sales, and bad press. Board makers are unhappy, some retail customers are unhappy, and now we are hearing OEM and DC customers may be unhappy. Forget AMD's position, Intel's is looking a bit grim historically speaking.

I think Intel has 2 options:
- keep making, selling and RMAing Raptor Lake and sending out more defective Raptor Lakes
- tell customers to get AMD CPU instead

The capacity Intel has at its disposal to make MTL and ARL is extremely limited. Something in range of 10-25% of client chips Intel can make are MTL, ARL or LNL, the rest of the capacity is RPL.

 

[igor_kavinski]

the rest of the capacity is RPL.

I'm sure they are working on a better stepping as we speak. That or they better backport ARL compute die to Intel 7 Ultra to avoid their fabs going idle coz if this continues, word will spread on the street to avoid anything that stinks of the codename Raptor.

 

[Joe NYC]

I'm sure they are working on a better stepping as we speak. That or they better backport ARL compute die to Intel 7 Ultra to avoid their fabs going idle coz if this continues, word will spread on the street to avoid anything that stinks of the codename Raptor.

Or go back to ADL. That one is not exhibiting the same issues as the 2 versions of RPL.

 

[poke01]

Intel is selling defective CPUs - Alderon Games

We have been encountering significant problems with Intel CPU stability. Despite all released microcode, BIOS, and firmware updates, the problem remains unresolved.

alderongames.com

this is big.

 

[Markfw]

From that article. VERY BIG

• Server Migration: We are swapping all our servers to AMD, which experience 100 times fewer crashes compared to Intel CPUs that were found to be defective.
• Hosting Recommendations: We advise anyone hosting Path of Titans servers or selling game servers to avoid purchasing or using 13th and 14th gen Intel CPUs.
• In-Game Notifications: We are adding a popup message in-game to inform users with these processors about the issue. Many users are currently unaware of why their game is crashing and what they can do about it.

The Intel server CPUs is a new thing. And this is not a biased site.

 

[Hitman928]

From that article. VERY BIG

• Server Migration: We are swapping all our servers to AMD, which experience 100 times fewer crashes compared to Intel CPUs that were found to be defective.
• Hosting Recommendations: We advise anyone hosting Path of Titans servers or selling game servers to avoid purchasing or using 13th and 14th gen Intel CPUs.
• In-Game Notifications: We are adding a popup message in-game to inform users with these processors about the issue. Many users are currently unaware of why their game is crashing and what they can do about it.

The Intel server CPUs is a new thing. And this is not a biased site.

Intel server CPUs aren't effected as far as we know. The servers they are talking about are actually desktop chips in workstation boards, so still talking about consumer 13th/14th gen chips.

 

[Hitman928]

Intel is selling defective CPUs - Alderon Games

We have been encountering significant problems with Intel CPU stability. Despite all released microcode, BIOS, and firmware updates, the problem remains unresolved.

alderongames.com

this is big.

This is one of the contacts Wendel was talking about in his video. This is by far the most concerning part:

Over the last 3–4 months, we have observed that CPUs initially working well deteriorate over time, eventually failing. The failure rate we have observed from our own testing is nearly 100%, indicating it's only a matter of time before affected CPUs fail.

Again, this isn't just their studio as development systems, but systems they use, or have contracted, as host servers. Still, probably not that significant a number, and I expect that not all 13900k/14900k CPUs will fail, but to have a near 100% failure rate across their multiple different deployments is obviously horrendous and points to something seriously wrong that maybe can't be fixed at scale (without lowering boost specs) with these CPUs.

Edit: When Intel first announced RPL, I was pretty shocked at how much they were able to increase the frequency on a tweaked node. I honestly didn't think they'd be able to get that much frequency boost without a significant process improvement. It may be that they pushed things further than they should have and the fruit is rotten. Could explain why Intel has yet to really give any insights into what is causing it or potential fixes if the real answer is that they just pushed the frequencies higher than they should have. It's possible this isn't true and they are narrowing down on a fix, but the longer they remain silent, the less hopeful the situation.

 

[Markfw]

From that article. VERY BIG

• Server Migration: We are swapping all our servers to AMD, which experience 100 times fewer crashes compared to Intel CPUs that were found to be defective.
• Hosting Recommendations: We advise anyone hosting Path of Titans servers or selling game servers to avoid purchasing or using 13th and 14th gen Intel CPUs.
• In-Game Notifications: We are adding a popup message in-game to inform users with these processors about the issue. Many users are currently unaware of why their game is crashing and what they can do about it.

The Intel server CPUs is a new thing. And this is not a biased site.

Seems odd to me that they would not use workstation CPUs in workstation boards. And they did not say workstation, they said server, which is sure confusing to me. Desktop, workstation and server are all different CPUs in Intel as well as AMD. Even AMD has ECC rated server chips in some lower end workstation server boards. Never seen ANY desktop chip used or talked about as a server chip, but I guess there is always exceptions, and that would make sense in that respect, as even at pure stock that chip appears to degrade. That would also negate what some users say that at stock they are fine.

 

[Hail The Brain Slug]

Seems odd to me that they would not use workstation CPUs in workstation boards. And they did not say workstation, they said server, which is sure confusing to me. Desktop, workstation and server are all different CPUs in Intel as well as AMD. Even AMD has ECC rated server chips in some lower end workstation server boards. Never seen ANY desktop chip used or talked about as a server chip, but I guess there is always exceptions, and that would make sense in that respect, as even at pure stock that chip appears to degrade. That would also negate what some users say that at stock they are fine.

Consumer Alder/Raptor Lake supports ECC in W680 boards. I don't think there's actually any feature gated stuff between the consumer chips and the LGA1700 Xeons - It's gated behind the platform instead.

 

[Markfw]

Consumer Alder/Raptor Lake supports ECC in W680 boards. I don't think there's actually any feature gated stuff between the consumer chips and the LGA1700 Xeons - It's gated behind the platform instead.

OK< well, so that says, chips at stock, with ECC ram, and not at full load are 100 times more unstable than AMD desktop chips. Thats really cuts out all arguments that they are not all defective eventually.

NOT GOOD For Intel.

 

[Futuremotion]

I just purchased a new build with a Core™ i9-14900KF and the processor arrived today. Someone from another thread pointed me to this thread, and now I'm really concerned I'm going to have instability problems. Has there been a consensus in this thread about what's wrong? Are there any mitigations? And ultimately, should I return my CPU?

I don't use my PC for gaming, just content creation:

Some apps I use:
Visual: Photoshop/Illustrator/Adobe
Motion: After Effects / Premier / Davinci
3D: 3DS Max / Maya / Blender
Audio: FL Studio / Ableton / Cubase / Lots of VST/VSTi plugins.

Am I going to experience a bunch of problems? I'm willing to return the 14900KF if it's really bad.

 

[Ranulf]

GN interviews Wendell over his video:

 

[Hitman928]

I just purchased a new build with a Core™ i9-14900KF and the processor arrived today. Someone from another thread pointed me to this thread, and now I'm really concerned I'm going to have instability problems. Has there been a consensus in this thread about what's wrong? Are there any mitigations? And ultimately, should I return my CPU?

I don't use my PC for gaming, just content creation:

Some apps I use:
Visual: Photoshop/Illustrator/Adobe
Motion: After Effects / Premier / Davinci
3D: 3DS Max / Maya / Blender
Audio: FL Studio / Ableton / Cubase / Lots of VST/VSTi plugins.

Am I going to experience a bunch of problems? I'm willing to return the 14900KF if it's really bad.

The latest Bios with default settings has helped reduce the issue from what I’ve seen, but has not fixed it. There’s no known cause and the only thing I’ve seen that has fixed the problem is people lowering the max frequency of the CPU to 5.3 GHz or lower.

You may not have issues with your particular workload, but there is a very real concern for the long term reliability of these CPUs at the moment and Intel hasn’t really done or said anything yet to ease concerns. You’ll have to decide yourself if you want to keep it, but personally I wouldn’t purchase one at this point in time.

 

[poke01]

I just purchased a new build with a Core™ i9-14900KF and the processor arrived today. Someone from another thread pointed me to this thread, and now I'm really concerned I'm going to have instability problems. Has there been a consensus in this thread about what's wrong? Are there any mitigations? And ultimately, should I return my CPU?

I don't use my PC for gaming, just content creation:

Some apps I use:
Visual: Photoshop/Illustrator/Adobe
Motion: After Effects / Premier / Davinci
3D: 3DS Max / Maya / Blender
Audio: FL Studio / Ableton / Cubase / Lots of VST/VSTi plugins.

Am I going to experience a bunch of problems? I'm willing to return the 14900KF if it's really bad.

Since its for pro use, I would return it now and just move to an AMD 7950X system. Better to have a working work PC.

 

[Ranulf]

Seems odd to me that they would not use workstation CPUs in workstation boards. And they did not say workstation, they said server, which is sure confusing to me.

I think it was Wendell in his video that said they wanted those chips for max single core performance. It can depend on the software you're running. I know that in a game like Space Engineers, in its dedicated server program vastly prefers high speeds on lower core counts than more cores. A streamer I used to watch would only rent servers running Intel for that game with high ghz over a 12-16 core Ryzen cpu.

 

[H433x0n]

I just purchased a new build with a Core™ i9-14900KF and the processor arrived today. Someone from another thread pointed me to this thread, and now I'm really concerned I'm going to have instability problems. Has there been a consensus in this thread about what's wrong? Are there any mitigations? And ultimately, should I return my CPU?

I don't use my PC for gaming, just content creation:

Some apps I use:
Visual: Photoshop/Illustrator/Adobe
Motion: After Effects / Premier / Davinci
3D: 3DS Max / Maya / Blender
Audio: FL Studio / Ableton / Cubase / Lots of VST/VSTi plugins.

Am I going to experience a bunch of problems? I'm willing to return the 14900KF if it's really bad.

What’s your motherboard? If it’s an Asus motherboard make sure to run BIOS 2301 or newer and select the “performance” profile, not the “extreme” profile.