Memory Scaling Reviews: Vast Conspiracy?

[UsandThem]

So, just to get a general feel on what most people here feel about memory scaling (all memory, but specifically DDR4 since it's the newest mainstream type).

Like other users I still see who are active here, I have been a member at this site for 16 years. What prompted me to join this site over all the various discussion boards at the time, was the extremely knowledgeable members and at the time, a POS Abit BP6 motherboard I couldn't get stable to save my life.

I came here and learned a lot by being open-minded, and by mostly staying out of the sub-forums like video cards and off-topic where arguing never ended, and where bans were given out like candy.

Now, to the question that I have been thinking about since I had a member 'inform' me that almost all the memory scaling reviews over the last 20 years are not only inaccurate, but seemed to think they were misleading their readers.

The standard memory review takes one motherboard, let's say one based on the z170 chipset. They then run synthetic, gaming, and real-life benchmarks to show what actual performance you would actually see going from say DDR4 2133 to DDR4 3200. And honestly, outside of the synthetic benchmarks that showed the much bigger bandwith, the actual performance increase was always something like MAYBE 3%.

During this 'informing' talk I had, I am now told that it isn't the actual real-world performance you would see, sort off. I was told that it "increases the efficiency of the CPU and is comparable to overclocking a CPU" and some other stuff. Another member suggested the only accurate method to testing RAM was using 'frame time variance'.

My argument is, if I overclock a i7-6700k and then run benchmarks, it usually shows a clear improvement in speed or performance, outside of a benchmark being GPU or disk limited. I also agree if DDR4 3200 is $10 more than DDR4 2133, then it makes sense to get it. I'm talking about paying 2x more for 'gaming' RAM or DDR4000 when a person isn't chasing benchmark scores or bragging rights.

So old timers, have we been misled by all these sites over the years, who have done review after review after review showing there is very little performance increase for most people outside of enthusiasts?

What say you?

http://www.anandtech.com/show/8959/...3200-with-gskill-corsair-adata-and-crucial/10

For professional users, we saw a number of benefits moving to the higher memory ranges, although for only very minor performance gains. Cinebench R15 gave 2%, 7-zip gave 2% and our fluid dynamics Linux benchmark was up +4.3%. The only true benchmark where 2800+ memory made a significant difference was in Redis, which is a scalable database memory-key store benchmark. Only users with specific needs would need to consider this.

http://www.silentpcreview.com/article1478-page6.html

Though the testing for this investigation was brutally tedious, the results were satisfyingly conclusive. On Skylake systems, the type of memory used is only important if you lean heavily on the integrated graphics chip. If this is the case, opting for a single stick rather than a pair of DIMMs in dual channel is a critical mistake that can cost more than 30% performance. Frequency is relevant as well with 3000 MHz offering around 10% improvement over 2133 MHz. The difference between C12 and C15 is negligible and should be ignored unless the price differential is minimal.

On the otherhand, if you only use the integrated graphics for pedestrian uses like video playback or if a discrete video card is to be utilized, the speed and latency of the system memory is inconsequential. Our general performance tests results depict the highest performing settings as having a one third of one percent advantage overall.

http://www.legitreviews.com/ddr4-me...nding-the-best-ddr4-memory-kit-speed_170340/6

Our benchmarks show that the memory bandwidth increased, but there wasn’t a tangible improvement in system performance with real applications. We ran other applications and game titles when we tested this memory kit and you mostly ended up with flat performance charts like you saw in Handbrake or any of the game titles that we tested today
Read more at http://www.legitreviews.com/ddr4-me...memory-kit-speed_170340/6#rdJCdcMW725LDhYK.99

 

[TheELF]

Just very badly thought out proceedings,take the last as an example,legitreviews sets out to test ram speed in gaming so what do they do? run in-build game benchmarks that are pre-loaded,they load into system ram (and GPU ram) before they start running,how the heck do you expect ram speed to make a difference in this scenario?


I believe that ram speed can make a difference in gaming,especially huge open world games where there is tons of things that have to be read off of the disk and into memory constantly,same reason why 16Gb has started to make sense for some games.
But I do not believe that it makes such a big difference that someone should care about it.
(unless he builds a super high end rig)

 

[dave_the_nerd]

During this 'informing' talk I had, I am now told that it isn't the actual real-world performance you would see, sort off. I was told that it "increases the efficiency of the CPU and is comparable to overclocking a CPU" and some other stuff.

I'd question that assertion.

It's true that CPUs generally are a lot faster than system ram, and the memory controller in the CPU has to kind of negotiate traffic between them. So faster RAM will generally mean the CPU waits less and can do more in the same time.

But because of that, most software devs would rather cut off their right arms than hit main memory more than they need to. Add in the predictive prefetch algorithms and the huge L3 caches in moderns CPUs, and in most applications, the actual impact of faster memory is pretty well hidden.

It's also worth pointing out that besides absolute clock speed, clock ratios (as well as CAS latency) can have an impact on the number of cycles that a CPU has to sit and wait in between memory requests. A decade or more ago, maximizing the performance of a memory-bottlenecked application sometimes even meant lowering memory speed so that it had an even multiplier with the system bus speed. (I experienced that with my Athlon XP rig.)

Another member suggested the only accurate method to testing RAM was using 'frame time variance'.

Frame Time Variance - how consistent is the time taken to draw each frame - is something that most reviews don't check. FTV combined with average FPS is probably the best way to test game benchmarks. TheELF is right that most benchmarks are canned and don't really reflect "real world" performance of, say, an open-world MMO.

It's probably effected somewhat by system RAM speeds, but it's also pretty heavily influenced by things like GPU drivers, GPU memory architecture, and was a big problem for certain cards in Crossfire mode once upon a time. (It was actually bad enough for the Casuals to notice.)

It's easy enough to test for, but RAM speed is usually pretty consistent, so if it's not bottlenecking FPS, I wouldn't expect it to have a big impact on FTV either. But I could be wrong, and I'd prefer that FTV be part of any gaming benchmark suite anyway, whether they were testing RAM, GPUs, or keyboards.

 

[nerp]

Yes. High speed memory is pretty much a waste unless you're overclocking and need the memory to run at higher frequencies.

 

[Dasa2]

I don't know whether to laugh or cry at that silentpc review claiming faster ram is only of benefit to the IGP when there GPU isnt much faster than the IGP itself
6700k, Zotac GeForce GT 640 ZONE Edition graphics card - 900 MHz core, 2GB 1782 MHz memory, fanless

As a old gamer that use to enjoy overclocking BH-5 to 265MHz 2-2-2 with Vmoded MB\PSU I agree the value of 4000mhz ram kits is questionable and spending extra on GPU is the first priority
But there is a lot of info out there that suggests CPU limited sections of gameplay can see 10-20% increase in fps just by using faster ram not just 0-3% which is little more than the margin of error in many reviews

I have linked one of the least GPU bottlnecked games in this review there is also GPU limited tests that see the typical 0-3% difference in the link

http://translate.googleusercontent.com/translate_c?depth=7&hl=en&rurl=translate.googleusercontent.com&sl=pl&tl=en&u=http://pclab.pl/art65002-4.html&usg=ALkJrhitpE4CwC0uG8dYOMAkcv8GkAafqw

There isn’t many tests that reach 4000MHz like this one although I question its accuracy lacking clear detail on timings used
It at least suggests that ram performance continues to scale fairly well over 3000MHz
They also run a test with and without SLI which shows the % gains from faster ram drop significantly with a single 980ti as the limit on the FPS shifts from CPU to GPU​

http://www.techspot.com/article/1171-ddr4-4000-mhz-performance/page3.html

The techspot review also overlooks latency which as this one shows can be of high importance to at least some games
If you click the link below for the bandwidth images you will see the lower latency ddr3 is infront of the higher bandwidth ddr4 its only once the latency catches up that ddr4 takes the lead
You will also see that the faster ram makes no difference at all to there x265 encode times​

http://www.hardware.fr/articles/940-5/cpu-ddr4-vs-ddr3-pratique.html

As the link above shows faster ram doesn’t help all programs unfortunately like higher CPU clock speed does it's kinda like extra cores in that way
Faster ram only helps software that needs random data that doesn’t fit in the CPU cache and games (especially those not on rails) seem to fit this category

Even these reviews I use as examples above are flawed in that they don’t compare the gains from different CPU models\clocks to the gains from faster ram
While there may not be many games that drop below 60fps with a shiny new 6600k\6700k as the years go by with more demanding games and a GPU upgrade the faster ram and overclocked cpu become a noticeable improvement in more & more games​

http://www.eurogamer.net/articles/d...it-finally-time-to-upgrade-your-core-i5-2500k

Another interesting test although some will dismiss it due to the low res​

http://www.hardocp.com/article/2015...76700k_ipc_overclocking_review/6#.V0ZGM2wRUuW

Here is some excellent tests done by a forum member at oc.net with great attention to detail​

An Independent Study: Does The Speed Of RAM Directly Affect FPS During High CPU Overhead Scenarios?
http://www.overclock.net/t/1487162/...affect-fps-during-high-cpu-overhead-scenarios


Not long enough yet? i can throw in some of my own testing on ARMA 3\Thief 3\Dragon Age 3 on a 2600k\3770k in another post

 

[Dasa2]

Ok here are my own ram speed tests from over the years

Dragon Age 3
% increase
ht on vs off ~5%
3.9 vs 4.2ghz ~6%
1333 vs 2400 ~17%
1600c11 vs 2400 ~14%
1600c9 vs 2400 ~11%
To benchmark it i loaded a save and ran without moving view from one side of the valley to the other which takes about 14sec as you can see once you get to the end of the valley fps are much higher than at the start where it has more to render

http://forums.anandtech.com/showthread.php?t=2455994

Thief 3
This built in benchmark didn't prefetch the files so I ran it twice in a row at each setting for comparison
After the second run all the files would be preloaded and the fps from all following runs would closely match the second run

http://forums.atomicmpc.com.au/index.php/topic/55771-cpuram-performance-in-thief/

ARMA 3
These tests were run with some fly by benchmarks that you can find find in steam workshop Stratis\Altis

A few months later i revisited these ARMA tests in more detail with different test settings to those above
There was ~1fps variance between runs though which is why not all the ram speeds scale perfectly as each step was to small to compare accurately to the test above\below it with a single run especially with the minimum fps which were very random

https://forums.bistudio.com/topic/1...mance-comparison-1600-2133-up-to-15-fps-gain/

Nearing the end here is the first tests i ran on the ARMA 3 Alpha

https://forums.bistudio.com/topic/144563-cpu-vs-ram-performance-cpu-threading-benchmarked/

 

[UsandThem]

Not long enough yet? i can throw in some of my own testing on ARMA 3\Thief 3\Dragon Age 3 on a 2600k\3770k in another post

Once you run your own hardware review site, I'd be glad to look at it.

I do have to ask you one question though. Since you do not believe all the various RAM scaling articles out there and what they show, do you contact the authors to challenge the results they publish?

 

[Dasa2]

I do have to ask you one question though. Since you do not believe all the various RAM scaling articles out there and what they show, do you contact the authors to challenge the results they publish?

Well I questioned this review in its comments thinking some of there tests seemed to good to be true
http://www.techspot.com/news/64927-ddr4-4000-mhz-does-make-difference.html#commentsOffset

But beyond that no i don't think I have as I do believe there results and I can see how they came by them just as the silent pc review did with the gt640 completely bottlnecking there results to the point where cpu speed could be dropped significantly seeing no change to the fps in there benchmarks

I dont suppose you have a copy of ARMA 3? being a old engine with bad threading and huge landscapes with lots of ai making it very cpu heavy its a easy one to test cpu bottlnecked performance to see for yourself if ram speed makes any difference on your system

 

[UsandThem]

Well I questioned this review in its comments thinking some of there tests seemed to good to be true
http://www.techspot.com/news/64927-ddr4-4000-mhz-does-make-difference.html#commentsOffset

But beyond that no i don't think I have

Based on our limited back-and-forth on RAM, you strike me as being in the enthusiast crowd. That isn't some type of a back-handed comment, nor is there anything wrong with that.

But like we discussed in the other thread, a normal everyday user:

One discrete GPU (any mid or high-end card), a non-enthusiast level CPU (not a 5930k, 5960k etc), a SSD using it to play games, doing normal tasks (work, photo, video) will see such a small actual performance increase by buying the the fastest RAM.

This is what we disagree on. You have shown there are differences when running games at low resolutions, making the test not GPU limited. This is great for enthusiasts, but Joe Smith is going to install Doom, set it to the highest settings their card can handle, and play the game. Now if they went from DDR4 2400 to DDR 3200, their FPS would increase. Maybe from 90 FPS to 93 FPS (just an example, I haven't actually looked into that game). The only large FPS increases I have seen with faster RAM are when a person is using integrated graphics.

Gaming RAM is like Asus ROG motherboards. They look pretty, they are built very well, and they offer a million different configuration options in the BIOS. However, that $150 Asus board will offer 99.8% of the performance of the board 2x its price.

It's all about when you reach the point of diminished returns for most people outside of enthusiasts.

 

[Dasa2]

Only some of the tests i linked are low res hardocp\my dragonage 3 test and arma3 alpha the other arma3 tests and games were 1080p high-max detail i think
At 4k cpu performance becomes almost irrelevant in all but a few old games

It really depends on the game the person is playing some will see no change as most games these days still run 60fps with a i5 2500 and a 980ti
A lot of people seem to be getting 144hz monitors though which does require a bit more cpu power if they want to get the fps up there

There is a few games where the difference is very noticeable to the average user at least in some parts
arma3\total war are all but totally cpu limited and as such will see big gains from ram for anyone with a decent gpu

coh2\gta v\fallout 4\witcher 3 are very cpu heavy in some places and very gpu heavy in others so only some parts of the game will see any benefit from faster cpu\ram
coh2\fallout 4 will frequently dip well under 60fps even with a overclocked 6700k sometimes its due to the gpu but in the parts of the game where the cpu is the reason this is where faster ram can increase performance around 10-20%
witcher 3\gta v run fairly well with a stock 6700k rarely going under 60fps if the gpu is powerful enough but a slower cpu is more likely to stay over 60fps with faster ram

So for most people that want to buy a overclockable cpu\mb 6600k\z170 with a 390\gtx970 level gpu i would recommend ~3000c15 ram currently ~$60-70
If they cant afford a 390\gtx970 i suggest they drop to a cpu\mb that cant be overclocked
For those that dont want to oc obviously 2133mhz ram ~$50 is the go
For those that want to run 980ti and a expensive cpu\mb overclocked the most i have ever suggested is 3200c14 ~$100 but if they want to try higher i don't discourage it other than to sometimes say that like other aspects of there build its not the best value choice as the timings loosen significantly as you go higher much of the speed gain from higher frequency is lost

I think the only part we really disagreed on was I felt that many reviews only show half the picture with a few gpu bottlnecked tests and a few synthetic cpu tests that showed minimal to no gain from faster ram and so there conclusion was that ram never makes a noticeable difference and it was this conclusion that i felt was wrong
While the few reviews that went looking for cpu limited gameplay found that there is some situations where faster ram can make a noticeable difference
Especially those that used older cpu with newer gpu vs reviews that use newer cpu with older gpu

 

[dave_the_nerd]

They also run a test with and without SLI which shows the % gains from faster ram drop significantly with a single 980ti as the limit on the FPS shifts from CPU to GPU

So... the two second version is: if you're GPU limited, it doesn't matter, unless you have an IGP?

 

[Dasa2]

So... the two second version is: if you're GPU limited, it doesn't matter, unless you have an IGP?

Yep ram only improves cpu efficiency and even then only in tasks like games that go outside the cpu cache frequently for there data
A faster cpu does nothing if the game your playing doesn't need it just as a faster gpu does nothing if the game your playing is cpu limited
But some games are a mix of both

Your 3570k at 4ghz with fast ram could easily get higher fps than a 4690k at 4ghz with slower ram in a cpu limited game
In a gpu limited game they would perform exactly the same regardless of the oc
With the same ram speed in a cpu limited game the 3570k would be ~10%? slower

 

[zlejedi]

Anandtech memory scaling tests has been useless at best and misleading at worst for years since they are clearly done in test locations where performance is depending on GPU.