Futuremark confirms NVIDIA "cheating" with latest driver...releases audit report

[pm]

To me at least, I wouldn't want a 25% increase in framerates in ANY app if it turned out looking like these pics. I've gotten to the point where I'm more of an image quality freak than a framerate freak.

I agree with you; what happened here was blatant. My only point is that if there were some way to hack performance in a way that is useful to the consumer, then to me it offsets the fact that it's specific to a certain application that is frequently benchmarked. You used Doom3 as an example, and I recalled ATI and their "quackifier". I didn't think that the "quackifier" was that bad because it boosted performance with a minor quality degradation. The only thing that ATI did that was unethical about that was that they didn't reveal what they were doing.

 

[gf4200isdabest]

DAPUNISHER, why do you even bother defending 3dmark? Their explanation is nonsensical and any game benchmark is more useful than 3dmark. I'm not going to be an idiot and tell people to "stfu" but I really think you need to think about why you are defending a piece of software that causes more problems than it fixes (ie: the people who complain about 3dmark scores when all their games run fine)...

 

[andreasl]

Originally posted by: galperi1

Originally posted by: andreasl
In page 4 they also mention an 8% drop for the ATI Radeon 9800Pro card in one of the subtests. They are not outright accusing ATI but why would they add that if they didn't at least suspect that ATI is cheating as well? Will be interesting to hear what their conclusion about that is as well.

So basically what you are saying is that Ati should be guilty before proven innocent while in the same exact scenariou Nvidia should get the opposite treatment. Now that is some BS

Did FM outright accuse nvidia of cheating before this report came out??? NO

Did FM investigate and reproduce the problems and come to the conclusion that Nvidia was cheating?? YES

Should they outright accuse Ati and not follow the same path they did for Nvidia?? NO

Are they currently investigating Ati's increase in GT4??? YES (if you read the audit it explicitly says FM is checking into it)

These double standards that people hold for different corps sicken me. Everyone should get the SAME exact standards to prove their innocence.

Nope, that is not what I said at all. You are making far too many assumptions about me. I don't give a rats ass about Nvidia vs ATI. because I'm not even a 3D gamer that crave for such hardware! I was simply mentioning it since no one before me had done that.

Your attitude sickens me..

 

[rwalterk]

Does this really surprise anyone? The bottom line is always sales, regardless of all the benchmarks and testing.

Don't forget that nVidia is a publicly traded company that has a responsiblity to its shareholders to maximize profits. They (and ATI) exist for only one reason - to make money, period. Their purpose is not to put smiles on the faces of fanboys; that's just a by-product of their business.

Our stock markets push companies like these to boost revenues in almost any way possible. Something like this might be unethical, but if it pushes sales and they get away with it (which they may not this time), Wall Street will see it as justified.

Nobody should be suprised. Pissed, sure, but not surprised.

 

[OneOfTheseDays]

somebody find proof that nvidia is cheating in real games and then i will care.

 

[DaveSimmons]

Originally posted by: pm

Originally posted by: NFS4


The only ones affected by this patch are those using the NVIDIA drivers that were exploiting the benchmark. It'd be all the same if NVIDIA made up some hacked drivers that rendered portions of Doom III incorrectly so that they could get 24% higher frame rates. Now if Carmack layed the smackdown on them with a patch, would you bitch him out too?

I'm with you on everything else but this. If somehow a driver could increase my framerates by 24% in a game, and I couldn't tell the difference in quality (or it was very minor), then all power to them. In this case, this "hack" nVidia would have been just fine to me if it had been in a game and the game had continued to work correctly. I would have taken the framerate increase and not worried about how they got it. And if a later patch came along and broke this theoretical "optimization" in the game, then I'd be annoyed, I think.

I remember that ATI had the Quake "optimization" - the so-called "Quackifier" - and this case I didn't think it was as blatantly unethical as what nVidia was described as doing recently. ATI increased framerates by taking a hit in quality that wasn't readily apparent to the user. I wasn't impressed with their achievement when I found out how they did it, but I wasn't as righteously indignant as I am in this case.

three points:
1. ATI hid their "optimization" and didn't give users control over turning it on or off -- they basically said "screw all the users that care about image quality" in order to bump up the framerate in reviews. Just like MSI has been caught doing with their 865 / 875 mobos (overclocking even if you don't want it to).
2. Same with nvidia and their FSAA / filter cheats.
3. Optimizing for the general performance of a game without loss of image quality is one thing, but what nvidia has been caught doing is optimizing for a specific playback of a game -- this kind of "optimization" would make Doom3 bench 25% faster on canned benchmark timedemos but would not give 25% speedup (possibly no speedup at all) when actually playing the game. If you bought a card based on this kind of benchmark, they lied to you on how the game will really run.

 

[DAPUNISHER]

Originally posted by: gf4200isdabest
DAPUNISHER, why do you even bother defending 3dmark? Their explanation is nonsensical and any game benchmark is more useful than 3dmark. I'm not going to be an idiot and tell people to "stfu" but I really think you need to think about why you are defending a piece of software that causes more problems than it fixes (ie: the people who complain about 3dmark scores when all their games run fine)...

It's been asserted by FM that 3Dmark stresses the entire card by throwing different tests at them that forces every part of the architecture to perform and that most games do not do this, so that while the card may perform very well in one type of game it may do poorly in another. Now, prove any one game's benchie is a more effective evaluation of a graphics cards strengths and weaknesses for consumers and companies to base purchasing decisions on (it's irrelevant wether they should or not, many do) than 3Dmark or STFU

EDIT: BTW, it's been my experience in providing support to users that 99% of the problems are between the chair and monitor Furthermore, using the fact that some people post about issues with 3Dmark when their games run fine is very poor supportive evidence of it "causing more problems than it fixes"

In fact, I wasn't aware it was designed to "fix" anything, just evaluate various graphics cards in a manner that provides an "Apples to Apples" comparison. In conclusion, when I see someone who's user name reflects a strong opinion and/or brand loyality to a particular manufacturers product the way your's does I immediately disregard anything they have to say on a topic involving the manufacturer they seem enamored with as it's consequently impossible to believe their opinion could be unbiased Now to paraphrase you: I think you really need to think about why your post contained no pertinent factual information that could potentially cause me or anyone else to believe that FM's explanation is "nonsensical" As to the "any game benchmark is more useful" Bullsh*t, You're kidding right? You didn't want to sound like an idiot? You failed miserably, now STFU fanboy

 

[galperi1]

Originally posted by: andreasl

Originally posted by: galperi1

Originally posted by: andreasl
In page 4 they also mention an 8% drop for the ATI Radeon 9800Pro card in one of the subtests. They are not outright accusing ATI but why would they add that if they didn't at least suspect that ATI is cheating as well? Will be interesting to hear what their conclusion about that is as well.

So basically what you are saying is that Ati should be guilty before proven innocent while in the same exact scenariou Nvidia should get the opposite treatment. Now that is some BS

Did FM outright accuse nvidia of cheating before this report came out??? NO

Did FM investigate and reproduce the problems and come to the conclusion that Nvidia was cheating?? YES

Should they outright accuse Ati and not follow the same path they did for Nvidia?? NO

Are they currently investigating Ati's increase in GT4??? YES (if you read the audit it explicitly says FM is checking into it)

These double standards that people hold for different corps sicken me. Everyone should get the SAME exact standards to prove their innocence.

Nope, that is not what I said at all. You are making far too many assumptions about me. I don't give a rats ass about Nvidia vs ATI. because I'm not even a 3D gamer that crave for such hardware! I was simply mentioning it since no one before me had done that.

Your attitude sickens me..

you're right man, I do owe you an apology. I missed one word on your first sentence and totally came to the wrong conclusion. My bad

 

[AkumaX]

Originally posted by: Harabecw
Everyone knows the size of the card is what matters most!

that sh!t cracks me up everytime i see it

and to go along w/ 90% of the threads in here:

 

[andreasl]

Originally posted by: galperi1

Originally posted by: andreasl

Originally posted by: galperi1

Originally posted by: andreasl
In page 4 they also mention an 8% drop for the ATI Radeon 9800Pro card in one of the subtests. They are not outright accusing ATI but why would they add that if they didn't at least suspect that ATI is cheating as well? Will be interesting to hear what their conclusion about that is as well.

So basically what you are saying is that Ati should be guilty before proven innocent while in the same exact scenariou Nvidia should get the opposite treatment. Now that is some BS

Did FM outright accuse nvidia of cheating before this report came out??? NO

Did FM investigate and reproduce the problems and come to the conclusion that Nvidia was cheating?? YES

Should they outright accuse Ati and not follow the same path they did for Nvidia?? NO

Are they currently investigating Ati's increase in GT4??? YES (if you read the audit it explicitly says FM is checking into it)

These double standards that people hold for different corps sicken me. Everyone should get the SAME exact standards to prove their innocence.

Nope, that is not what I said at all. You are making far too many assumptions about me. I don't give a rats ass about Nvidia vs ATI. because I'm not even a 3D gamer that crave for such hardware! I was simply mentioning it since no one before me had done that.

Your attitude sickens me..

you're right man, I do owe you an apology. I missed one word on your first sentence and totally came to the wrong conclusion. My bad

No problem. All is forgiven

 

[NOX]

OHH NOO! You mean a graphics manufacturer actually found a weakness and tired to used it to their own gain! This is an OUTRAGE! Whatever will we do? LETS GO AFTER THESE COMPANIES, WHICH USE SUCH UNDERHANDED TACTICS! Oh, but wait, if we do that there would be no more companies left for us to buy graphics cards from! Oh well, I guess we?ll just let things continue the way they have always been. :lips:

 

[sharkeeper]

Everyone knows the size of the card is what matters most!

Updated for 2003 and patched to prevent cheating!

-DAK-

 

[Mem]

As a gamer I don`t care about 3Dmark,I never buy a video card for 3Dmark,I buy it for games .
The simple fact is all I care about is that all my games work well and look good in gaming,3Dmark means jackxx to me.

Now when`s there`s a driver upgrade you have two sorts of members here,the first say "wow! I`ve got 100 more points in 3Dmark" the other looks at the stability and image quality in real games ,I`m one of the latter.

I don`t think anybody buys a video card for 3Dmark,if they do they`re one sad person.

Getting back to the real subject,Futuremark says Nvidia cheated,well too be honest I don`t really care one way or the other,look at the past history with other companies on cheating,it doesn`t stop people buying their products.I bet there`s a lot of things that go on with these companies that we don`t hear about.


In the end only you can judge how important 3Dmark software and benchmarks are,I for one don`t rate it at all.

somebody find proof that nvidia is cheating in real games and then i will care.

I can`t find any cheating in real games with their drivers,if it`s there I have`nt notice it,well it`s time for some online gaming so I`ll get back to that while 3Dmark fanatics run their benchmarks for the 1000x .

 

[Megatomic]

Originally posted by: Sudheer Anne
somebody find proof that nvidia is cheating in real games and then i will care.

Yup, and if the games still look good then I'll be happy.

 

[Megatomic]

Originally posted by: DAPUNISHER
As to the "any game benchmark is more useful" Bullsh*t, You're kidding right? You didn't want to sound like an idiot? You failed miserably, now STFU fanboy

Please be kidding...

 

[DAPUNISHER]

Originally posted by: Megatomic

Originally posted by: DAPUNISHER
As to the "any game benchmark is more useful" Bullsh*t, You're kidding right? You didn't want to sound like an idiot? You failed miserably, now STFU fanboy

Please be kidding...

Nope, I'm not, he's implying a benchmark done with say QII for instance would be a better gauge of performance than 3dmark, it's not period.

 

[Burmese]

I suspect that the level of cheating incorporated into the drivers is directly tied to the company's perceived position versus its' competitors. When nVidia was winning all the benchmarks and ruled the roost they had little interest in fudging benchmarks. Since ATi's introduction last year of the 9700 series nVidia has been under increasing pressure and the flop that was the NV30 just added to that. Pushed into a corner they've loosened the leashes on their programmers and let them write some dodgy drivers that help the company look good on at least one common benchmark.

...and then, to add insult to injury, there's the discovery that their 'Dawn' demo can be run faster on an ATi 9800-based system (using an OpenGL wrapper) then on their own top-of-the line card.

 

[sandorski]

Whether 3dmark *should* be used to compare vidcards or not is immaterial. The fact is that some people use 3dmark as a guide, why else would Nvidia take the time to cheat?

 

[SexyK]

From HardOCP front page:

NVIDIA Fires Back:
A follow up to today?s 5th Edition story about Futuremarks claims of benchmark fudging, News.com has a quote from NVIDIA that fires right back at Futuremark which targets their extremely expensive fees to be part of the ?beta program?. Here is a quote from the article:

"Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer," the representative said. "We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad."

Seems to me they may have a bit of a point there. Discuss.

 

[Megatomic]

Originally posted by: SexyK
...we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer...[/i]

I like that comment.

 

[sandorski]

Originally posted by: SexyK
From HardOCP front page:

NVIDIA Fires Back:
A follow up to today?s 5th Edition story about Futuremarks claims of benchmark fudging, News.com has a quote from NVIDIA that fires right back at Futuremark which targets their extremely expensive fees to be part of the ?beta program?. Here is a quote from the article:

"Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer," the representative said. "We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad."

Seems to me they may have a bit of a point there. Discuss.

So they are saying 3dmark cheated? I'm not sure what they're trying to say.

 

[Mem]

"Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer," the representative said. "We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad."

The fact is a lot of people are pointing the finger at Nvidia,hell even Futuremark are,the bottom line is only Nvidia really know if they cheated or not,also it comes down to how you define a cheat,if you have drivers that are giving faster benchmarks in 3Dmark but you don`t know how to code them 100% to the 3Dmark software(when you don`t want to pay big $$ for the developer version) is that a cheat or bug?



There`s no way Nvidia will pay big $$$ for the developer version now,I personally don`t blame them if they honestly feel they`ve not cheated.



Where`s Mulder? The truth is out there .

 

[SexyK]

Mem, you've wrapped up my feelings on this pretty well. I can appriciate nVidia getting tired of being forced to pay "hundreds of thousands of dollars" to FM just for their beta program. So they went and made the thing run as fast as they could, not knowing how the program really worked, and things got muffed up. I personally cant hold it against them that this happened. I run a 9500 Pro in my main rig right now, so its not like I'm an nVidiot, but I really think they are getting a lot of unwarrented bad press over this.

 

[SexyK]

Originally posted by: SexyK
Mem, you've wrapped up my feelings on this pretty well. I can appriciate nVidia getting tired of being forced to pay "hundreds of thousands of dollars" to FM just for their beta program. So they went and made the thing run as fast as they could, not knowing how the program really worked, and things got muffed up. I personally cant hold it against them that this happened. I run a 9500 Pro in my main rig right now, so its not like I'm an nVidiot, but I really think they are getting a lot of unwarrented bad press over this.

Edit: and I think its pretty low of FM to come out with this "report". To me, it stinks of a bit of an old "you wouldn't pay our fees, now were going to screw you over."

Edit: whoops, hit quote and not edit, i'm a big moron, trying to watch hockey and post at the same time!

 

[oldfart]

Did anyone actually read the PDF? They expalin exactly HOW nVidia cheated. There is no doubt.

What Are The Identified Cheats?

Futuremark?s audit revealed cheats in NVIDIA Detonator FX 44.03 and 43.51 WHQL drivers.
Earlier GeForceFX drivers include only some of the cheats listed below.

1. The loading screen of the 3DMark03 test is detected by the driver. This is used by the driver
to disregard the back buffer clear command that 3DMark03 gives. This incorrectly reduces the
workload. However, if the loading screen is rendered in a different manner, the driver seems
to fail to detect 3DMark03, and performs the back buffer clear command as instructed.

2. A vertex shader used in game test 2 (P_Pointsprite.vsh) is detected by the driver. In this case
the driver uses instructions contained in the driver to determine when to obey the back buffer
clear command and when not to. If the back buffer would not be cleared at all in game test 2,
the stars in the view of outer space in some cameras would appear smeared as have been
reported in the articles mentioned earlier. Back buffer clearing is turned off and on again so
that the back buffer is cleared only when the default benchmark cameras show outer space.
In free camera mode one can keep the camera outside the spaceship through the entire test,
and see how the sky smearing is turned on and off.

3. A vertex shader used in game test 4 (M_HDRsky.vsh) is detected. In this case the driver adds
two static clipping planes to reduce the workload. The clipping planes are placed so that the
sky is cut out just beyond what is visible in the default camera angles. Again, using the free
camera one can look at the sky to see it abruptly cut off. Screenshot of this view was also
reported in the ExtremeTech and Beyond3D articles. This cheat was introduced in the 43.51
drivers as far as we know.

4. In game test 4, the water pixel shader (M_Water.psh) is detected. The driver uses this
detection to artificially achieve a large performance boost - more than doubling the early
frame rate on some systems. In our inspection we noticed a difference in the rendering when
compared either to the DirectX reference rasterizer or to those of other hardware. It appears
the water shader is being totally discarded and replaced with an alternative more efficient
shader implemented in the drivers themselves. The drivers produce a similar looking
rendering, but not an identical one.

5. In game test 4 there is detection of a pixel shader (m_HDRSky.psh). Again it appears the
shader is being totally discarded and replaced with an alternative more efficient shader in a
similar fashion to the water pixel shader above. The rendering looks similar, but it is not
identical.

6. A vertex shader (G_MetalCubeLit.vsh) is detected in game test 1. Preventing this detection
proved to reduce the frame rate with these drivers, but we have not yet determined the cause.

7. A vertex shader in game test 3 (G_PaintBaked.vsh) is detected, and preventing this detection
drops the scores with these drivers. This cheat causes the back buffer clearing to be
disregarded; we are not yet aware of any other cheats.

8. The vertex and pixel shaders used in the 3DMark03 feature tests are also detected by the
driver. When we prevented this detection, the performance dropped by more than a factor of
two in the 2.0 pixel shader test.

The ones who were getting "screwed over" are us, the consumer, not nVidia.