Futuremark confirms NVIDIA "cheating" with latest driver...releases audit report

[NFS4]

http://neowin.net/comments.php?id=11230&category=main


Interesting. In a new patch released by Futuremark and after extensive testing, it was found that the Radeon 9800 Pro's score dropped by 1.9%. The GeForce FX's score dropped by 24.1%.

Futuremark page:

http://www.futuremark.com/news/?newsarticle=200305/2003052308#200305/2003052308

Futuremark audit report:

http://www.futuremark.com/companyinfo/3dmark03_audit_report.pdf

 

[nortexoid]

that's the problem w/ benchmarks...especially industry standard ones...

clearly, performance should be measured by the size of the card, not by how it performs in any one particular synthetic test.

 

[SgtZulu]

because everybody loves to play that game 3D Mark

 

[GTaudiophile]

pwned

 

[DaveSimmons]

Originally posted by: SgtZulu
because everybody loves to play that game 3D Mark

...on their MSI Neo 865 mobo

 

[GTaudiophile]

Originally posted by: SgtZulu
because everybody loves to play that game 3D Mark

No, this just poses the question: what else did they cheat on?

 

[DAPUNISHER]

Originally posted by: SgtZulu
because everybody loves to play that game 3D Mark

Did you read the audit report? It provides a good explanation of why a synthetic bench can be more useful than any particular game and why it's harder to get away with cheating. If you can provide an compelling counter argument I'm sure we'd all love to hear it, otherwise STFU

 

[sharkeeper]

nVidia didn't cheat, they found an exploit (weakness) in the skin of the Mad Onion and ran with it. So what if the numbers aren't represenative of real world performance. Are they ever with this software?

-DAK-

 

[T9D]

..

 

[DaveSimmons]

Originally posted by: xerosleep
So futuremark made a patch that basicly changes the program and now companies drivers pretty much don't support the program because it's like a "new" program now.........then they release all there findings to the public.....yeah that's ethical, psssst. One one more reason why I hate futuremark and 3dmarks.

By your "logic" a nvidia card only needs to support existing games, not new ones as they are released.

As long as futuremark's code is making standard 3D API calls any graphics driver that is working properly (and not cheating) will continue to run at roughly the sam speed. The speed of an "honest" driver would be determined by the difficulty of rendering the scene not by whether or not the results were pre-rendered and stored in the driver.

 

[NFS4]

Originally posted by: shuttleteam
nVidia didn't cheat, they found an exploit (weakness) in the skin of the Mad Onion and ran with it. So what if the numbers aren't represenative of real world performance. Are they ever with this software?

-DAK-

Did you READ the article?? I swear, people go off half cocked before even reading things.

Originally posted by: xerosleep
So futuremark made a patch that basicly changes the program and now companies drivers pretty much don't support the program because it's like a "new" program now.........then they release all there findings to the public.....yeah that's ethical, psssst. One one more reason why I hate futuremark and 3dmarks.

The program works just fine with the patch. If NE1 would bother to actually READ the findings, you'll see that all results gathered from the 330 build are directly comparable to the previous builds of 3DMark. ATI showed a less than 2% difference between builds. That's a normal difference that can be expected. NVIDIA's drivers took a 24% hit due to the hacks they were using to try to boost their score.

The only ones affected by this patch are those using the NVIDIA drivers that were exploiting the benchmark. It'd be all the same if NVIDIA made up some hacked drivers that rendered portions of Doom III incorrectly so that they could get 24% higher frame rates. Now if Carmack layed the smackdown on them with a patch, would you bitch him out too?

 

[OneOfTheseDays]

I highly doubt Nvidia would "cheat" in anything other than 3dmark. They're smarter than that I hope. They have openly criticized 3dMark2003, and its been a back and forth battle ever since.

 

[paperfist]

Nice find

It's pretty interesting how nVidia deploys 3DMark03 benchmark detection in their drivers in order to activate work a rounds for 8 parts of the benchmark and to achieve the speed boost.

It does also say that ATI might be fudging the scores a little, but not at nearly the same clip that nVidia did.

To you guys who don't think that is really cheating or not a big deal, I for one use the 3DMark03 scores that are in almost every review done on the web to determine if the card in question is worth buying. So if they get artificial speed boosts it's a big deal to me and my $400.

 

[andreasl]

In page 4 they also mention an 8% drop for the ATI Radeon 9800Pro card in one of the subtests. They are not outright accusing ATI but why would they add that if they didn't at least suspect that ATI is cheating as well? Will be interesting to hear what their conclusion about that is as well.

 

[WarCon]

Originally posted by: Sudheer Anne
I highly doubt Nvidia would "cheat" in anything other than 3dmark. They're smarter than that I hope. They have openly criticized 3dMark2003, and its been a back and forth battle ever since.

It would be almost understandable if when asked if they had optimized the drivers for this industry benchmark, that many people base their purchasing on, they had admitted to doing it. Instead they were investigating the drivers for a "bug". So they "cheated" on the benchmark and then "lied" about it and the consumer should feel comforted thats its only a struggle between Futuremark and NVidia? And to not worry that they won't comprise any other test or even game performance (like castrating ansitropic or antialiasing) just for a "boost".

It sucked when ATI cheated on the whole Quake/Quack thing and when Trident just getting caught and it sucks that NVidia is using this benchmark to cheat/fool the consumer into believing their new cards are all that when they aren't. (Freaking $500 dollar cards anyway) It don't matter if it is because they and Futuremark aren't playing nice.

 

[galperi1]

Originally posted by: andreasl
In page 4 they also mention an 8% drop for the ATI Radeon 9800Pro card in one of the subtests. They are not outright accusing ATI but why would they add that if they didn't at least suspect that ATI is cheating as well? Will be interesting to hear what their conclusion about that is as well.

So basically what you are saying is that Ati should be guilty before proven innocent while in the same exact scenariou Nvidia should get the opposite treatment. Now that is some BS

Did FM outright accuse nvidia of cheating before this report came out??? NO

Did FM investigate and reproduce the problems and come to the conclusion that Nvidia was cheating?? YES

Should they outright accuse Ati and not follow the same path they did for Nvidia?? NO

Are they currently investigating Ati's increase in GT4??? YES (if you read the audit it explicitly says FM is checking into it)

These double standards that people hold for different corps sicken me. Everyone should get the SAME exact standards to prove their innocence.

 

[The Sauce]

Originally posted by: GTaudiophile

Originally posted by: SgtZulu
because everybody loves to play that game 3D Mark

No, this just poses the question: what else did they cheat on?

It couldn't have been more than 1-2 years ago when there was the big scandal about ATI cheating with their drivers on the Quake 3 bench. I think we will just have to accept the fact that they will both cheat as much as possible to sell their cards. Part of the American business ethic nowadays.

 

[splice]

Originally posted by: nortexoid
that's the problem w/ benchmarks...especially industry standard ones...

clearly, performance should be measured by the size of the card, not by how it performs in any one particular synthetic test.

My MSI TI4400 is pretty big, does that mean mine is better than a smaller TI4000, even though benchmark performance is the same?

About the topic, is it any surprise that companies tailor to benchmark performance? All are guilty of this, because benchmark performance sells.

 

[DaveSimmons]

Originally posted by: Snatchface

Originally posted by: GTaudiophile

Originally posted by: SgtZulu
because everybody loves to play that game 3D Mark

No, this just poses the question: what else did they cheat on?

It couldn't have been more than 1-2 years ago when there was the big scandal about ATI cheating with their drivers on the Quake 3 bench. I think we will just have to accept the fact that they will both cheat as much as possible to sell their cards. Part of the American business ethic nowadays.

Blame Canada!
(nvidia are ethics-challenged Americans but ATI are quacky moose-lovers.)

 

[Harabecw]

Everyone knows the size of the card is what matters most!

 

[sharkeeper]

Did you READ the article?? I swear, people go off half cocked before even reading things.

Yes I did read the article. Perhaps there should be an icon for sarcasm!

Anyway, why even make such a stink about this since the owner of this site doesn't use this useless benchmark?

It's flamebait for the "nvidia caught cheating" crowd plain and simple.

-DAK-

 

[pm]

Originally posted by: NFS4


The only ones affected by this patch are those using the NVIDIA drivers that were exploiting the benchmark. It'd be all the same if NVIDIA made up some hacked drivers that rendered portions of Doom III incorrectly so that they could get 24% higher frame rates. Now if Carmack layed the smackdown on them with a patch, would you bitch him out too?

I'm with you on everything else but this. If somehow a driver could increase my framerates by 24% in a game, and I couldn't tell the difference in quality (or it was very minor), then all power to them. In this case, this "hack" nVidia would have been just fine to me if it had been in a game and the game had continued to work correctly. I would have taken the framerate increase and not worried about how they got it. And if a later patch came along and broke this theoretical "optimization" in the game, then I'd be annoyed, I think.

I remember that ATI had the Quake "optimization" - the so-called "Quackifier" - and this case I didn't think it was as blatantly unethical as what nVidia was described as doing recently. ATI increased framerates by taking a hit in quality that wasn't readily apparent to the user. I wasn't impressed with their achievement when I found out how they did it, but I wasn't as righteously indignant as I am in this case.

 

[NFS4]

Originally posted by: pm

Originally posted by: NFS4


The only ones affected by this patch are those using the NVIDIA drivers that were exploiting the benchmark. It'd be all the same if NVIDIA made up some hacked drivers that rendered portions of Doom III incorrectly so that they could get 24% higher frame rates. Now if Carmack layed the smackdown on them with a patch, would you bitch him out too?

I'm with you on everything else but this. If somehow a driver could increase my framerates by 24% in a game, and I couldn't tell the difference in quality (or it was very minor), then all power to them. In this case, this "hack" nVidia would have been just fine to me if it had been in a game and the game had continued to work correctly. I would have taken the framerate increase and not worried about how they got it. And if a later patch came along and broke this theoretical "optimization" in the game, then I'd be annoyed, I think.

I remember that ATI had the Quake "optimization" - the so-called "Quackifier" - and this case I didn't think it was as blatantly unethical as what nVidia was described as doing recently. ATI increased framerates by taking a hit in quality that wasn't readily apparent to the user. I wasn't impressed with their achievement when I found out how they did it, but I wasn't as righteously indignant as I am in this case.

Well, here are the "glitches" that were showing up in the 3DMark03 test b/c of NVIDIA's "optimizations"

http://common.ziffdavisinternet.com/util_get_image/2/0,3363,sz=1&i=24047,00.jpg
http://common.ziffdavisinternet.com/util_get_image/2/0,3363,sz=1&i=24044,00.jpg

To me at least, I wouldn't want a 25% increase in framerates in ANY app if it turned out looking like these pics. I've gotten to the point where I'm more of an image quality freak than a framerate freak.

 

[Krk3561]

Originally posted by: NFS4

The only ones affected by this patch are those using the NVIDIA drivers that were exploiting the benchmark. It'd be all the same if NVIDIA made up some hacked drivers that rendered portions of Doom III incorrectly so that they could get 24% higher frame rates. Now if Carmack layed the smackdown on them with a patch, would you bitch him out too?

Nvida doesnt have access to Doom III so it couldnt have hacked drivers for it. Remember reading something about Doom III and cheating before, I forgot where though.

 

[Pabster]

Everybody is cheating!

Seriously, this shouldn't be a big shock. ATi's done (doing?) it before, it is nVidia's turn.