Like all binary benchmarks. The Geekbench author already has disclosed compilers and flags used.
What open source benchmark exists? SPEC isn't open source. Have you ever seen its source code? I have.
Also note Geekbench source code is available if you pay a license.
http://www.primatelabs.com/geekbench/versions/
In the same way you can get access to SPEC source code (though the price is not the same ).
He discussed it but proved nothing. To prove fairness of benchmark he just need to open the code... 90% of his "benchmark" are available in open source variants and have much better performance. Comparing geekbench benchmarks to SPEC benchmark is even not funny. SPEC benchmarks are testing very complicated algorithms while geekbench performs basic task or copy available open source benchmarks.
Lets see what he has in geekbench3:
Crypto(SHA, Twofish, AES) - open source code available pretty much everywhere.
BZIP2 - source code and benchmarks:
http://www.bzip.org/
PNG/JPG compressor/decompressor - available everywhere. For example:
https://code.google.com/p/jpeg-compressor/
Lua - scripting language. Don't know what script it runs.
Dijkstra - Basic tree search algorithm. Students learn to code it on the first university semester.
Black-Scholes - many open source implementations are available.
Mandelbrot - Simple code.
http://en.wikipedia.org/wiki/Mandelbrot_set
Blur/Sharpen Image - see above.
SGEMM/DGEMM/SFFT/DFFT - open source ATLAS/FFTW including benchmarks.
N-Body/Ray trace - many open source implementations are available. I'm sure the author using one of these implementations.
Pretty much every benchmark form geekbench and much, much, much more are available under linux with open source code. There is even application (
http://www.phoronix.com/) which combines many open source benchmark in one test suite.
These are very valid points. But can you prove the same doesn't apply to Geekbench for ARM?
It would not apply for ARM, because currently ARM does not support double precision vector instructions.
Can you also be sure the number of GFLOPS Geekbench reports is comparable to the one other libs report?
Yes, I can be sure, because DGEMM is just a matrix multiplication and all efficient algorithms are known for a long time.
Also what's the point of comparing a hand tuned library, while at the same time you complain this benchmark result depends on compiler and flags?
ATLAS/BLAS/LAPACK is standard open source library available for many architectures (including ARM).
Note I certainly agree with you that using this DGEMM result as representative of Intel GEMM performance would be very misleading. But using it as a way to compare what various CPU/OS/compilers do on a similar piece of code, it still has some interest.
In 5 minutes I can write DGEMM implementation (even without ATLAS) which will be at least twice faster on Intel/AMD hardware than his implementation.
Here I found something interesting:
http://support.primatelabs.com/disc...performance-of-4770k-across-linux-and-windows
Author either lying, or have no clue about things he are talking about.
For example:
We certainly agree that the same hardware should perform similarly and we strive for this. We want the scores to represent the hardware performing well, but we also intend the scores to reflect execution performance of real-world application code.
What real world application he is talking about? MAPLE, MATLAB, Mathematica are not real world?
http://math-atlas.sourceforge.net/faq.html#who
Furthermore, if the libraries are proprietary we don't know exactly what optimizations the library performs.
What the f**k he is talking about?
ATLAS uses a BSD-style license, without the advertising clause. ATLAS's license is taken almost verbatim from the example given at opensource.org
http://math-atlas.sourceforge.net/faq.html#license