Surreptitious Samsung Benchmark Shenanigans

ShintaiDK

Lifer
Apr 22, 2012
20,378
145
106
Nothing new, they also did it with the S4.

That stinks. Just as Intel AnTuTu cheat, this is not acceptable and I hope Samsung will get the treatment it deserves (and I am sure more people here will blame Samsung than they were people blaming Intel for cheating).

Using ICC vs generic compiler is not cheating.
 

liahos1

Senior member
Aug 28, 2013
573
45
91
For the purposes of keeping things intellectually honest i wanted to point this out. Since the Intel Antutu benchmark seems to reverberate in an echo chamber, while shenangians like this are diminished
 
Last edited:

ChronoReverse

Platinum Member
Mar 4, 2004
2,562
31
91
Doesn't affect me since I don't use Touchwiz but I'm sick of Samsung pulling this crap. It speeds up only benchmarks so it's a cheat in the "looks and sounds like a duck" sense.


Using ICC vs generic compiler is not cheating.
Repeating this doesn't make it true. Even Antutu revised their benchmark to remove this. The only question is where the source of cheating came from (it's theoretically possible that it's just an accident but doesn't seem likely).
 

ShintaiDK

Lifer
Apr 22, 2012
20,378
145
106
For the purposes of keeping things intellectually honest i wanted to point this out. Since the Intel Antutu benchmark seems to reverberate in an echo chamber, while shenangians like this are diminished

They got away with it easily and almost unnoted with the S4. Well played by Samsung PR and grunts.

A turbo that only works in benchmarks to make the product look better. Not much to say about it. And specially not since this is the second device to do so. Its simply really bad ethics.
 

SiliconWars

Platinum Member
Dec 29, 2012
2,346
0
0
For the purposes of keeping things intellectually honest i wanted to point this out. Since the Intel Antutu benchmark seems to reverberate in an echo chamber, while shenangians like this are diminished

Feel free to point out the arstechnica and Anandtech articles discussing Intel's AnTuTu shenanigans, I'd love to read them.
 

Nothingness

Platinum Member
Jul 3, 2013
2,767
1,425
136
Using ICC vs generic compiler is not cheating.
Still in denial heh? :biggrin: Using a compiler that explicitly improves code for a loop in a benchmark is cheating, it's exactly the same as detecting a benchmark and up frequency.

A turbo that only works in benchmarks to make the product look better. Not much to say about it. And specially not since this is the second device to do so. Its simply really bad ethics.
I totally agree.
 

ShintaiDK

Lifer
Apr 22, 2012
20,378
145
106
Still in denial heh? :biggrin: Using a compiler that explicitly improves code for a loop in a benchmark is cheating, it's exactly the same as detecting a benchmark and up frequency.

Using ICC benefits all x86 applications. Its also the fastest compiler for any x86 CPU.

While you can claim the usage could be misleading using optimized vs non optimized compilers. Then sure.
But its still not a cheat, since its something the enduser can get.
 

Exophase

Diamond Member
Apr 19, 2012
4,439
9
81
From the article:

However, if you load up just about any popular CPU benchmarking app, the Note 3 CPU locks into 2.3GHz mode, the fastest speed possible, and none of the cores ever shut off. Stopping the CPU from idling shouldn't in and of itself affect the benchmark scores a whole lot, so this was our first sign that something was wrong. Benchmarks exist to measure the performance of a phone during normal usage, and a device should never treat a benchmark app differently than a normal app.
Actually, if the benchmark is at all properly written and the DVFS as regulated by the operating system (and user settings) is at all reasonable then this is exactly what should happen when you launch a benchmark. If you're running something that pegs the CPU it should be given as much CPU time as possible within the operating limits of the device.

The problem here is the benchmark must not have enough of a warmup period to convince the OS under normal conditions that it needs to ramp up the clock speed fast enough. So by the time the OS realizes that it needs to give the app full frequency the benchmark is already over, or at least has ran enough for the score to be heavily impacted. This is exactly why any competent benchmark has a long warmup period, and another reason why AnTuTu is garbage. But it looks like even GB3 doesn't have enough of one.

But that doesn't mean it's okay that Samsung special cases around it just for the benchmarks. This should be absolutely shunned. But it really needs to feed back into another point, stop tolerating crappy mobile benchmarks. Samsung should have been raising awareness towards this instead - this is probably a big reason why they joined that mobile benchmark consortium, but unfortunately since they're comfortable cheating at the benchmarks in the mean time everyone is going to think they just joined to gain control over cheating at that.

There is another possible issue, which is just that under normal conditions it won't let you run all four cores at 2.3GHz under normal conditions at all (not sure what the limit is then, but the article says both phones are supposed to be able to do it). By running the benchmark I guess it figures that it won't run long enough that it's worth the risk of thermal shutdown. In this case the OS should still let anything run at it for some amount of time, ala boost on Intel CPUs. Then this would be a software problem that Samsung needs to deal with instead of just hacking it for benchmarks. If the benches can run comfortably anything should be able to.

Using ICC benefits all x86 applications. Its also the fastest compiler for any x86 CPU.

While you can claim the usage could be misleading using optimized vs non optimized compilers. Then sure.
But its still not a cheat, since its something the enduser can get.

You really must have no concept of what it means to exploit a broken benchmark at the compiler level.
 
Last edited:

Nothingness

Platinum Member
Jul 3, 2013
2,767
1,425
136
Using ICC benefits all x86 applications. Its also the fastest compiler for any x86 CPU.

While you can claim the usage could be misleading using optimized vs non optimized compilers. Then sure.
But its still not a cheat, since its something the enduser can get.
No it's not something you can get, because only the loop in that particular benchmark gets optimized so it benefits only the benchmark. Even turning 64-bit code generation made the loop dozens of times slower, how do you explain that?
 

Phynaz

Lifer
Mar 13, 2006
10,140
819
126
Looks at thread title, sees talk about Intel compilers.

Getting back on topic would be a good idea guys.
 

thilanliyan

Lifer
Jun 21, 2005
11,912
2,130
126
It must suck to have to cheat on your benchmarks.

Yet another reason I bought an iPhone 5S.

There's more than just Apple and Samsung for choice...for example HTC, LG, Nokia, etc.

Not excusing Samsung here (don't particularly care for their products) but Apple is far from top of the heap when it comes to honesty (not talking about bench marks). I'm sure there are other reasons why you bought a 5S but if you're going to apply the "honesty" criteria to Samsung, you should apply it to Apple too.

Not sure why Samsung would even bother with this...the VAST majority of customers would never read any of these benchmarks.
 
Last edited:

Skurge

Diamond Member
Aug 17, 2009
5,195
1
71
HTC One does this too, so it may be a Snapdragon thing, cause I don't see it mentioned for the Exynos versions of the S4, Note3 and Note10
 

liahos1

Senior member
Aug 28, 2013
573
45
91
haha amazing that half the posts here were on intel yet again. can we just all agree that everyone seems to dope their benchmarks
 

Skurge

Diamond Member
Aug 17, 2009
5,195
1
71
haha amazing that half the posts here were on intel yet again. can we just all agree that everyone seems to dope their benchmarks

Except AMD/Nvidia. They would crucified if they were caught doing this. Gamers take that stuff seriously. Don't know about you mobile folks.
 

liahos1

Senior member
Aug 28, 2013
573
45
91
Lol.. well I was going to say not anymore anyway.

as a gamer i just care about how gpus play my games. i dont put much stock in 3dmark or any of that other stuff. just give me a card(s) that will play
BF4 at 1440p on high settings at 60fps and im a happy guy :biggrin:
 

SlimFan

Member
Jul 5, 2013
91
11
71
Actually, if the benchmark is at all properly written and the DVFS as regulated by the operating system (and user settings) is at all reasonable then this is exactly what should happen when you launch a benchmark. If you're running something that pegs the CPU it should be given as much CPU time as possible within the operating limits of the device.

The problem here is the benchmark must not have enough of a warmup period to convince the OS under normal conditions that it needs to ramp up the clock speed fast enough. So by the time the OS realizes that it needs to give the app full frequency the benchmark is already over, or at least has ran enough for the score to be heavily impacted. This is exactly why any competent benchmark has a long warmup period, and another reason why AnTuTu is garbage. But it looks like even GB3 doesn't have enough of one.

Exophase, you always try to be intellectually honest, but in this case you've just really missed the mark.

What you are saying is true if the device have an advanced power management system, supporting software, and the proper hardware to detect the conditions to limit the operation. But that is not what Samsung is doing here.

They only things that can run at the high frequency are the white-listed benchmarks. What you're saying here would be true if Samsung would let anything on the platform run at the high frequency, and then throttle it if it hits some kind of platform limit. But that is not what the device is doing here.

If the "turbo" function of the platform is not generically applicable to the general behavior of the device, then it's a cheat, pure an simple. It appears that Samsung doesn't think that the "turbo" should be available to general applications, or they wouldn't have implemented it in white-list this fashion.

You cant blame the benchmark writers in this case since they have to appear in Samsung's white-list of applications approved for high frequency operation. How is it the benchmark's fault if the operating system + the DVFS software on the platform refuses to run at the higher operating point? Geekbench takes several minutes to complete. If the device cannot raise the frequency in that amount of time, then it's irrational to think that this is the fault of the benchmark.
 
sale-70-410-exam    | Exam-200-125-pdf    | we-sale-70-410-exam    | hot-sale-70-410-exam    | Latest-exam-700-603-Dumps    | Dumps-98-363-exams-date    | Certs-200-125-date    | Dumps-300-075-exams-date    | hot-sale-book-C8010-726-book    | Hot-Sale-200-310-Exam    | Exam-Description-200-310-dumps?    | hot-sale-book-200-125-book    | Latest-Updated-300-209-Exam    | Dumps-210-260-exams-date    | Download-200-125-Exam-PDF    | Exam-Description-300-101-dumps    | Certs-300-101-date    | Hot-Sale-300-075-Exam    | Latest-exam-200-125-Dumps    | Exam-Description-200-125-dumps    | Latest-Updated-300-075-Exam    | hot-sale-book-210-260-book    | Dumps-200-901-exams-date    | Certs-200-901-date    | Latest-exam-1Z0-062-Dumps    | Hot-Sale-1Z0-062-Exam    | Certs-CSSLP-date    | 100%-Pass-70-383-Exams    | Latest-JN0-360-real-exam-questions    | 100%-Pass-4A0-100-Real-Exam-Questions    | Dumps-300-135-exams-date    | Passed-200-105-Tech-Exams    | Latest-Updated-200-310-Exam    | Download-300-070-Exam-PDF    | Hot-Sale-JN0-360-Exam    | 100%-Pass-JN0-360-Exams    | 100%-Pass-JN0-360-Real-Exam-Questions    | Dumps-JN0-360-exams-date    | Exam-Description-1Z0-876-dumps    | Latest-exam-1Z0-876-Dumps    | Dumps-HPE0-Y53-exams-date    | 2017-Latest-HPE0-Y53-Exam    | 100%-Pass-HPE0-Y53-Real-Exam-Questions    | Pass-4A0-100-Exam    | Latest-4A0-100-Questions    | Dumps-98-365-exams-date    | 2017-Latest-98-365-Exam    | 100%-Pass-VCS-254-Exams    | 2017-Latest-VCS-273-Exam    | Dumps-200-355-exams-date    | 2017-Latest-300-320-Exam    | Pass-300-101-Exam    | 100%-Pass-300-115-Exams    |
http://www.portvapes.co.uk/    | http://www.portvapes.co.uk/    |