GPU reviews are all flawed

[DustinBrowder]

Sorry to break it to you, but most, if not all GPU reviews are completely and utterly useless and flawed.

Do you know how these reviewers measure fps? They start the game, load to a saved game of a certain point and then run in a straight line for 30 seconds or 1 minute. In fact many times even the line is not straight enough across play throughs, and the "better" reviewers repeat the test 2-4 times to make sure its accurate.

Or they just use a built-in test if there is one, in fact most actually just start the game from the beginning and record the first 30 second or 1 minute from the game, walking a straight line!

This is completely and utterly flawed and wrong. It can't, won't and is impossible to show the REAL world performance.

A REAL test would have to involve modding the game, loading to a certain point and having it automatically play out for at least 5 minutes in order to measure the REAL minimum frames, REAL average frames, REAL maximum frames and the 99th percentile and other stuff.

Optimally you'd want the game to run for 20 minutes in order to gauge how it works when the card is hotter and the memory is filled and the system is stressed out.

So next time you look at a review, remember that it is completely useless for REAL WORLD performance, its just like an advertisement showing basically the worthless case scenario.

 

[RobertR1]

Thanks Dustin.

 

[RampantAndroid]

I think you're assuming that games are actually consistent. They're not...GTA V is one of the worst ones for consistency (or the best, for playing) - and a lot of FPSes will have enemies in the same areas, but they won't behave the same way every time. There's a lot of randomness added to make sure that the game isn't just a snoozefest the second time through or something. Open world games try to add random events more.

The benchmarks that are built in remove this randomness, so that you can actually compare any one run to another other run, assuming settings are the same (or use them to compare performance levels at different settings and help guide you to the correct settings for your system.) There's something to be said for benchmark vs in game performance, but the latter isn't really viable.

 

[cmdrdredd]

Uh huh

 

[Silverforce11]

Sample size will minimize variation. So if sites do 3 runs and get the avg, that's already telling enough.

 

[Red Hawk]

Pretty sure each review site uses different methodologies.

 

[KingFatty]

Yeah like car reviews - they only drive the cars on racetracks or test tracks. How dare they! They should drive the car in my neighborhood and go grocery shopping!

Next time you want to buy a ferrari, ignore the tests! They don't spill ice cream in the ferraris, so it makes no difference to us!

 

[tviceman]

The only GPU reviews that aren't flawed are the ones that you want to believe.

 

[Azix]

This is why its good to have a built in benchmark or what richard Huddy said they hope for in GE titles where there is a tool to create a sequence that the user finds representative.

Ideally I think games should come with a benchmark built in that represents the worst case scenario in performance.

 

[beginner99]

OP is right but he is misinterpreting the results. He takes the results as bare numeric truth. That's not meant that way. You can still compare different GPUs to each other and determine which one is better for which game. Even this flawed benchmarks tell you an intel iGPU sucks vs anything NV or AMD has to offer. It also tells you a 980 Ti is faster than fury x especially at 1440p. And so forth.

Now if you complain the benchmark showed a min FPS of 60 and you get 40 with the same setup but different place in the game, well then you are just retarded.

 

[Eymar]

Good idea, but I'd rather have much varied benchmarks then time wasted on warming up the gpu. This is for release reviews where reviewers have limited time to review. For specific vendor card reviews then sure why not. I really like the youtube benchmark reviews where you can see the actual benchmark in motion like this one: https://www.youtube.com/watch?v=gfK0Yr1xTFo

 

[Techhog]

Great. Then you should make a review site for REAL graphics card reviews.

 

[tweakboy]

I doubt it,

 

[ShintaiDK]

CPU wise, the prescripted benchmarks are horrible. Tomb Raider is a classic example with 400% CPU load difference between benchmark and in game.

However for GPU loads they tend to be fairly accurate. The issue is the CPU part.

 

[cen1]

No.

 

[Gikaseixas]

That's why you have to read several reviews to come to a conclusion and if notice most of them are aligned, showing the fastest card consistently at the top.

 

[96Firebird]

I look forward to your perfectly executed reviews, Dustin Browder.

 

[DustinBrowder]

Sample size will minimize variation. So if sites do 3 runs and get the avg, that's already telling enough.

Yeah, but its still a 30 second test on a straight line at a very specific moment.

A real test would need to be at least 5 minutes, going through all sort of gameplay and then averaging that.

Averaging on a straight line for 30 seconds is inherently flawed and doesn't accurately display performance!

 

[Phynaz]

Thanks Bro

 

[Headfoot]

cool story bro

 

[railven]

So far, from visiting the GPU and CPU sides of this forum I've noticed the following:

1) Intel is paying EVERYONE to better review/promote their products.
2) Nvidia is paying EVERYONE OUTSIDE OF THE EU to better review/promote their products.

Conclusion: If AMD actually sold more products they could perhaps pay reviewers too. AMD needs to step up their game.

 

[positivedoppler]

If only all hardware sites come together and formulate a mechanism of standardize testing.

 

[DigDog]

best thread 2015

 

[Despoiler]

I agree. Some other things I think the reviews do incorrectly.

1) They follow the reviewers guides that are designed to put the gear in the best light possible. All companies do this if they give you free gear to test. This puts to question the integrity of the review. There is a reason why Consumer Reports is well regarded. They remove bias from the equation by purchasing all of the equipment themselves.

2) The use of open test benches by review sites is common practice. They are not commonly used by most users though.

3) Certain metrics used as absolutes that everyone must consider, when they aren't. ie perf/watt. These metrics tend pop up when we have progress stalled. We've got to fight about something right?

4) The biggest number automatically wins. When see reviews gushing over the fact that something can hit 1300/1400/1500mhz I can do nothing, but roll my eyes. I don't think there is a single review that looks at FPS/mhz to see how many FPS 1 mhz gives you. If you want to see how "strong" an architecture is, then start normalizing metrics to a common metric. Then frame that by realistic expectations for clock rate. Ongoing user surveys to get an idea of how likely something is. Not golden sample performance.

 

[railven]

If only all hardware sites come together and formulate a mechanism of standardize testing.

When you factor in that reviews are primarily subjective (ie what a reviewer wants to focus on) it would be rather hard to regulate it.

Now, if they turned it into a real scientific study (ie not bloggers/guy_A "lets start a website") perhaps you can get some regulations.

As of now, just buy what you want. Use the reviews to give you an idea of what to expect.