AnTuTu and Intel

[ChronoReverse]

Many of you have seen the article posted on EETimes, here's a link: http://www.eetimes.com/author.asp?section_id=36&itc=eetimes_sitedefault&doc_id=1318857 For those interested in the state of mobile benchmarketing I recommend a read, it's pretty enlightening.

You may have also seen me rant on this topic before. For those interested I figured I'd give some additional analysis behind some of the things I've said.

*snip*

Thanks for the analysis and post. I've been looking at benches for Clover Trail+ and Bay Trail and this throws a bit of light on shadows I didn't even know were there. Will be rereading it later (I've only quickly went through it).

Interesting how Antutu still doesn't use NEON with their ARM code. I had emailed them in the past before about that and the response was for compatibility purposes. I wonder why not have a "bonus" benchmark for NEON support but oh well.




Although it's kinda amusing how the aggregate Antutu score is bandied about as if it's definitive about CPU performance (nearly half the score is from the GPU for crying out loud).

 

[krumme]

I'm sure ARM licensees are going to bend over their asses to Intel over doctored benchmarks so they can join the line of Intel slaves like Asus to suffer stuff like the RDRAM fiasco.

Intel inside on a phone means: i can not fund the development myself and i am postponing the pain until i have changed job.

No wonder Intel and its army of unknowns is going excactly nowhere.

 

[Idontcare]

John from Primate Labs here (the company behind Geekbench).

I wanted to provide some details about what's going on with the floating point workloads the Silvermont architect referenced. Two of the Geekbench 2 floating point workloads (Sharpen Image and Blur Image) have a fencepost error. This error causes the workloads to read uninitialized memory, which can contain denorms (depending on the platform). This causes a massive drop in performance, and isn't representative of real-world performance.

We only found out about this issue a couple of months ago. Given that Geekbench 3 will be out in August, and fixing the issue in Geekbench 2 would break the ability to compare Geekbench 2 scores, we made the call not to fix the issue in Geekbench 2.

If you've got any questions about this (or about anything Geekbench) please let me know and I'd be happy to answer them. My email address is john at primatelabs dot com if you'd prefer to get in touch that way.

Welcome to the forums John (jfpoole)! :thumbsup:

 

[galego]

You are not quoting the settlement, you are quoting the complaint.

You are right. Corrected. But this part of the complaint was verified and used in the final settlement, where Intel was obligated to add a disclaimer to its compiler and

to pay $10M USD to establish a fund to help business customers retool their software if they were misled by Intel to think the poor performance of Intel-compiled code on AMD chips was normal.

 

[galego]

Intel is acting like they did in the old days fighting amd.

This is my impression as well:

PC -> Phone
AMD -> ARM
Sysmark -> AnTuTu
...

John from Primate Labs here (the company behind Geekbench).

I wanted to provide some details about what's going on with the floating point workloads the Silvermont architect referenced. Two of the Geekbench 2 floating point workloads (Sharpen Image and Blur Image) have a fencepost error. This error causes the workloads to read uninitialized memory, which can contain denorms (depending on the platform). This causes a massive drop in performance, and isn't representative of real-world performance.

We only found out about this issue a couple of months ago. Given that Geekbench 3 will be out in August, and fixing the issue in Geekbench 2 would break the ability to compare Geekbench 2 scores, we made the call not to fix the issue in Geekbench 2.

If you've got any questions about this (or about anything Geekbench) please let me know and I'd be happy to answer them. My email address is john at primatelabs dot com if you'd prefer to get in touch that way.

Thanks by confirming my point on this was a mere tech. issue which is going to be fixed and that the claims made by some people in this forum were only a product of their imagination.

Besides that, congrats by a product as Geekbench. It has been useful for me to obtain performance of kabini and I am using it now to estimate performance of future kaveri.

 

[PPB]

It is unlikely an old dog will learn new tricks. I think that for a company with such a big track record of billons dumped into R&D, year after year, it is really low to keep doing this kind of thing

 

[mrmt]

You are right. Corrected. But this part of the complaint was verified and used in the final settlement

Verified by who? Because you didn't quote the settlement, you quoted your own manipulative interpretation of the settlement. I'll quote the two texts and let people judge themselves.

Here's what you brought us:

to pay $10M USD to establish a fund to help business customers retool their software if they were misled by Intel to think the poor performance of Intel-compiled code on AMD chips was normal.

And here's the FTC text:

http://ftc.gov/os/adjpro/d9341/101102inteldo.pdf

Within ninety (90) days of the date on which this Order becomes final, Respondent shall implement, and shall notify its Compiler Customers that it has implemented, a program to reimburse Compiler Customers who (i) have detrimentally relied on Intel representations as to Compiler availability, functionality or effectiveness when using an Intel Compiler to compile code to be executed on a Compatible x86 Microprocessor and (ii) decide to recompile using a Compiler not developed or sold by Respondent (the “Intel Compiler Reimbursement Program&#8221. Such a notification must include a link to or a copy of this Order and specifically reference this Section VII of the Order in the notification. The features of the Intel Compiler Reimbursement Program shall include the following:

(...)

What the FTC is trying to say here is that the ICC as a product didn't function in a way customers could reasonably expect, and because of that they should be entitled to a refund if they wanted. No mention of benchmarks, no mention to AMD, no mentions to change the behavior, nothing of the things you deemed verified in your diatribe. Just disclosure and refund.

 

[Schmide]

What about? (right below what you quoted)

B. Respondent shall not misrepresent, or assist others in misrepresenting, expressly or by implication, the level of optimizations available in its Compilers for Compatible x86 Microprocessor.

 

[galego]

Verified by who? Because you didn't quote the settlement, you quoted your own manipulative interpretation of the settlement. I'll quote the two texts and let people judge themselves.

In your first post you detected a mistake, which I corrected after acknowledging you. Now you are going too far.

I didn't quote a "manipulative interpretation of the settlement". This is pretty ridiculous. I quoted one of the points in the FTC complaint.

Here's what you brought us:

to pay $10M USD to establish a fund to help business customers retool their software if they were misled by Intel to think the poor performance of Intel-compiled code on AMD chips was normal.

This is from the news reporting the FTC settlement.

 

[mrmt]

This is from the news reporting the FTC settlement.

Ok, then you quoted someone's else manipulative interpretation of the settlement, didn't mention it wasn't a quote from the settlement, didn't provide the source for the quote and implied that because of that quote the items you cherry picked from FTC complaint were verified.

They weren't, and the interpretation you brought us is as worthless as your opinion, because there is nothing on the settlement to back up none of your claims neither the quote you brought here.

 

[mrmt]

What about? (right below what you quoted)

Same thing. Disclosure. FTC is saying that when I buy a compiler it should as standard practice not to discriminate against any other vendor's processor based on the vendor ID alone. If Intel is going to discriminate, it must disclose it somewhere. And this is exactly what happens today, ICC still runs the optimal path only in Intel processors, but Intel must mention this fact in the ICC marketing material.

 

[Blandge]

Same thing. Disclosure. FTC is saying that when I buy a compiler it should as standard practice not to discriminate against any other vendor's processor based on the vendor ID alone. If Intel is going to discriminate, it must disclose it somewhere. And this is exactly what happens today, ICC still runs the optimal path only in Intel processors, but Intel must mention this fact in the ICC marketing material.

What you are saying is that Intel had to settle because they didn't include a disclaimer. Ouch.

 

[mrmt]

What you are saying is that Intel had to settle because they didn't include a disclaimer. Ouch.

The settlement is far bigger than compiler and benchmarks issues.

Here's how the FTC summarizes the settlement:

http://ftc.gov/opa/2010/08/intel.shtm

(...)

The FTC settlement applies to Central Processing Units, Graphics Processing Units and chipsets and prohibits Intel from using threats, bundled prices, or other offers to exclude or hamper competition or otherwise unreasonably inhibit the sale of competitive CPUs or GPUs. The settlement also prohibits Intel from deceiving computer manufacturers about the performance of non-Intel CPUs or GPUs.

The FTC settlement goes beyond those reached in previous antitrust cases against Intel in a number of ways. For example, the FTC settlement order protects competition and not any single competitor in the CPU, graphics, and chipset markets. It also addresses Intel’s disclosures related to its compiler – a product that plays an important role in CPU performance. The settlement order also ensures that manufacturers of complementary products such as discrete GPUs will be assured access to Intel’s CPU for the next six years.

(...)


Under the settlement, Intel will be prohibited from:

conditioning benefits to computer makers in exchange for their promise to buy chips from Intel exclusively or to refuse to buy chips from others; and
retaliating against computer makers if they do business with non-Intel suppliers by withholding benefits from them.

In addition, the FTC settlement order will require Intel to:

modify its intellectual property agreements with AMD, Nvidia, and Via so that those companies have more freedom to consider mergers or joint ventures with other companies, without the threat of being sued by Intel for patent infringement;

offer to extend Via’s x86 licensing agreement for five years beyond the current agreement, which expires in 2013;

maintain a key interface, known as the PCI Express Bus, for at least six years in a way that will not limit the performance of graphics processing chips.

These assurances will provide incentives to manufacturers of complementary, and potentially competitive, products to Intel’s CPUs to continue to innovate; and disclose to software developers that Intel computer compilers discriminate between Intel chips and non-Intel chips, and that they may not register all the features of non-Intel chips. Intel also will have to reimburse all software vendors who want to recompile their software using a non-Intel compiler.

(...)

So, yes, regarding the ICC, it's only a matter of disclosing to everybody that the product behaves differently from what you could reasonably expect.

What didn't catch people's attention at the time is that Intel didn't have to pay a single penny to the FTC, only to Nvidia. This says a lot about the strength of the claims in the complaint.

 

[SiliconWars]

What didn't catch people's attention at the time is that Intel didn't have to pay a single penny to the FTC, only to Nvidia. This says a lot about the strength of the claims in the complaint.

No it was because Intel settled quickly instead of having another drawn-out legal affair that they knew would cost them even more in the long run.

 

[Vesku]

Let's get back to the benchmarketing (I like that word for describing this kind of thing), nice to see a Geekbench dev showing they read criticism and promising some fixes with version 3.

 

[galego]

Ok, then you quoted someone's else manipulative interpretation of the settlement, didn't mention it wasn't a quote from the settlement, didn't provide the source for the quote and implied that because of that quote the items you cherry picked from FTC complaint were verified.

They weren't, and the interpretation you brought us is as worthless as your opinion, because there is nothing on the settlement to back up none of your claims neither the quote you brought here.

I love as you continue making unfounded personal accusations, whereas pretending that the mentioned point of the FTC complaint (the point relevant to what Intel is doing now with Antutu) was not verified.

 

[mrmt]

No it was because Intel settled quickly instead of having another drawn-out legal affair that they knew would cost them even more in the long run.

So the FTC had a strong case against Intel, one that would cost them billions, but as Intel settled quickly the FTC gave a discount and settled for 0 and gave Nvidia a boost? Wow, what a bunch of incompetents.

 

[sontin]

The Intel and nVidia case had nothing to do with the FTC. Intel and nVidia had a license agreement which nVidia thought Intel broke. That's the reason why Intel agreed to pay $1,5 billions to nVidia.

 

[Vesku]

Keep in mind this was after the credit crunch, going after Intel with a large fine after the EU already hit them for over $1 billion was a pretty unlikely event. Perversely companies are getting even more of a pass in these slow economic times, the US had evidence of banks aiding embargoed nations and crime cartels and they only fined them. Still, this is drifting away from the validity of benchmarks like AnTuTu.

I have yet to see any defense of the lack of a NEON code path for ARM, it's not difficult to check for what a processor supports.

 

[SiliconWars]

So the FTC had a strong case against Intel, one that would cost them billions, but as Intel settled quickly the FTC gave a discount and settled for 0 and gave Nvidia a boost? Wow, what a bunch of incompetents.

It isn't always about cash mrmt. Intel gave up a lot of their stranglehold on x86.

- modify its intellectual property agreements with AMD, Nvidia, and Via so that those companies have more freedom to consider mergers or joint ventures with other companies, without the threat of being sued by Intel for patent infringement;

It's difficult to know the exact benefits but since then AMD has been able to completely spin-off GF, fab x86 at TSMC and now collude with ARM. All of these were probably going to get the ire of Intel's lawyers before this agreement.

Intel had already recently been hit by a large cash fine by the EU and that was probably taken into consideration.

 

[krumme]

By far the most important part was the ability for amd to produce x86 outside own fabs.

 

[mrmt]

It isn't always about cash mrmt. Intel gave up a lot of their stranglehold on x86.

You should read through the entire complaint if you want to have a clearer picture of the situation, which you can find here:

http://www.ftc.gov/os/adjpro/d9341/091216intelcmpt.pdf

In the complaint the FTC listed quite a lot of things that could be liable for damages and unfair trade practices that would also yield fines. Basically, the complaint deals with:

- Nvidia and Intel practices against Nvidia (GPU restrictions and chipset restrictions)

- x86 license, which the FTC demanded be free and to Intel provide help to others companies.

- Compiler and benchmarks issues. Intel had to drop the biased dispatcher.

- Intel deals with OEMs/ODMs (very long list of complaints)

The FTC build a long list of to-do things in the complaint, located in the last three pages.

In the end, what the FTC got some nice goodies but far from everything it asked. Intel still holds all the strings in the x86 license and ICC is still the same, most of the practices argued by the FTC were long halted and forbidden by the settlement with AMD, and last, but not least, 0 payments from Intel.

That's not indicative of a very strong case, it's indicative of a significant case but that would have to be argued in a court for a looong time with unpredictable results for both parts. So in order to avoid that, both sides compromised. Intel gave away something, the FTC watered down their demands. Had the FTC a stronger case, I doubt that they would have settled for less than full compliance with their demands *and* a nice fine on Intel.

 

[mrmt]

By far the most important part was the ability for amd to produce x86 outside own fabs.

That was included in the settlement with AMD in 2009. This was Intel's bait to bring AMD to the table.

 

[Exophase]

I updated the OP to reflect on some additional findings jhu made.. thanks jhu. Also gave a link to BDTI's story on AnTuTu.

- Compiler and benchmarks issues. Intel had to drop the biased dispatcher.

Have they though? You see they now put a disclaimer everywhere that says that their compilers are optimized for their processors (http://software.intel.com/sites/default/files/m/0/1/3/opt-notice-en_080411.gif)

As of September 2010 at least they didn't drop the biased dispatcher, see the post on 2010-09-22 here:

http://www.agner.org/optimize/blog/read.php?i=49#209

 

[Exophase]

John from Primate Labs here (the company behind Geekbench).

I wanted to provide some details about what's going on with the floating point workloads the Silvermont architect referenced. Two of the Geekbench 2 floating point workloads (Sharpen Image and Blur Image) have a fencepost error. This error causes the workloads to read uninitialized memory, which can contain denorms (depending on the platform). This causes a massive drop in performance, and isn't representative of real-world performance.

We only found out about this issue a couple of months ago. Given that Geekbench 3 will be out in August, and fixing the issue in Geekbench 2 would break the ability to compare Geekbench 2 scores, we made the call not to fix the issue in Geekbench 2.

If you've got any questions about this (or about anything Geekbench) please let me know and I'd be happy to answer them. My email address is john at primatelabs dot com if you'd prefer to get in touch that way.

Thank you for the additional information on this.

Using uninitialized data would certainly explain some presence of denorms. But Jon Tyler's claim of 100% of the data set consisting of denorms seems very bizarre. If we're talking about single-precision floats the exponent field has to be 0 for a denorm and this doesn't include zero and negative zero. So given a uniform distribution there'd be a less than 1/256 (0.4%) chance of an uninitialized 32-bit floating being a denorm. Uninitialized data doesn't tend to really follow uniform distributions, it may well have a lot of stuff that looks like denorms due to containing 32-bit integers that aren't large enough (> 0x7FFFFF) to trip out of denorms. But they could also have a lot of zeroes, negative numbers, < 32-bit int datatypes, strings, pointers, code, etc etc that would tend to sway away from this. If you just so happen to have really bad luck in what sort of stuff you were doing beforehand it could have ended up with a dataset that's fully denormal, but I still find this pretty bizarre. It'd be good to get some measurements on this.

I do have to ask, is Geekbench doing anything to verify that the results of the benches are correct, like a checksum? IMO this is an important step in benchmarks, you want to at least make sure that the compiled code is doing what it's supposed to and not cutting corners that result in the wrong answers.