Thanks for asking. I am having a PhD in math and a Master in CS.
Do not waste my time with linking stupid papers written by some students.
Read and try to understand my example from above why arithmetic mean is nonsense. Claiming that from 2 equal graphics cards, one is 25% faster is ridiculous.
Both harmonic mean and arithmetic mean are inconsistent with respect to samples of relative quantities and their reciprocal.
For consistency you would like to have the following property 1/(mean(q1,q2)) = mean(1/q1,1/q2). This translates to, that you want to have consistent results if you answer the question of how much A is faster than B or how much B ist slower than A.
But i have the feeling i am talking to a wall in your case.
.
Because there is really no straightforward way of determining what is or isn't appropriate, so by using a mixture of settings you should hopefully even out any outliers.
As I just said above, this is actually a strength of meta analysis not a weakness.
And how do you determine what is bad data?
Take TPU (seeing as this seems to be the one people are hating on), they currently have 2 games in their benchmark suite with DX12, but the DX12 implementation for both those games is more or less broken.
Do you still think they should use DX12 in their testing then?
And as I said above TPU already tests in the API that best represents the cards, so what's the problem?
A professor with a 187 publications is "some student", seriously mate just stop digging.
but what you don't seem to understand is that the same thing is just as possible with geometric means.
Since where are not dealing with an such extreme scenario it doesn't matter which mean we use.
Take TPU (seeing as this seems to be the one people are hating on), they currently have 2 games in their benchmark suite with DX12, but the DX12 implementation for both those games is more or less broken.
Do you still think they should use DX12 in their testing then?
And as I said above TPU already tests in the API that best represents the cards, so what's the problem?
Oh please, don't try to argue this, your are going to loose this argument since apparently you have no math background. It could not be more wrong to use arithmetic mean.
Want to see what's happening with arithmetic mean?
Assume we have benchmarks of 2 games:
1) RX480: 50 fps GTX1060: 100fps
2) RX480: 100fps GTX1060: 50 fps
Intuitively we would rate both cards equal, right?
Now lets look how TPU calculates:
RX480 GTX1060 100*GTX/RX
50 100 200
100 50 50
arith mean: 125
geom mean: 100
According to arithmetic mean (and also TPU!!) GTX1060 would be 25% faster in average over both (hypothetical) games....cool hu?
Sorry hitman DX12 is stable and AMD cards benefit significantly from the DX12 version. RoTR DX12 on the other hand regresses performance for Nvidia cards from DX11 version while for AMD cards the perf is almost same after the latest patch. RoTR is the worst DX12 game out there. All of the DX12 games out there work very well for AMD cards and show good gains vs DX11 except RoTR which is a Gameworks title and has a broken DX12 implementation.
Anyway wizzard has agreed that he will moving Hitman to DX12 and adding Doom Vulkan. So that should address a few of the concerns with their game suite.
https://www.reddit.com/r/Amd/comments/4tz24g/rx_480_strix_oc_8_gb_review_techpowerup/
"I rebench from time to time, last I rebenched in May. A full rebench takes me around two weeks plus time to figure out benchmark scenes etc.
After that rebench there was 1080, then Computex, then 1070, 480, 1060. Impossible to stop all graphics card review work for two weeks. It's something I'll do asap, and yes, will include DOOM Vulkan, Hitman DX12. Unsure about Warhammer DX12 which is Beta."
Good news.. DX12 and Vulkan generally improve performance on Pascal cards as well so it should be a welcome change for everyone. Including those looking to see if an upgrade from Maxwell to Pascal is worth it.Sorry hitman DX12 is stable and AMD cards benefit significantly from the DX12 version. RoTR DX12 on the other hand regresses performance for Nvidia cards from DX11 version while for AMD cards the perf is almost same after the latest patch. RoTR is the worst DX12 game out there. All of the DX12 games out there work very well for AMD cards and show good gains vs DX11 except RoTR which is a Gameworks title and has a broken DX12 implementation.
Anyway wizzard has agreed that he will moving Hitman to DX12 and adding Doom Vulkan. So that should address a few of the concerns with their game suite.
https://www.reddit.com/r/Amd/comments/4tz24g/rx_480_strix_oc_8_gb_review_techpowerup/
"I rebench from time to time, last I rebenched in May. A full rebench takes me around two weeks plus time to figure out benchmark scenes etc.
After that rebench there was 1080, then Computex, then 1070, 480, 1060. Impossible to stop all graphics card review work for two weeks. It's something I'll do asap, and yes, will include DOOM Vulkan, Hitman DX12. Unsure about Warhammer DX12 which is Beta."
How do you enable it? My option is grayed out and I am both on Win10 and have an rx480, I hope it's not because my Windows is Enterprise edition.Interesting he doesn't like Warhammer DX12 because it's a beta... i've been putting in many hours into it in DX12 mode, no issues. Extra performance is nice, especially MIN FPS in big battles.
For a beta, it's better than some other DX12 implementation for sure.
How do you enable it? My option is grayed out and I am both on Win10 and have an rx480, I hope it's not because my Windows is Enterprise edition.
DX12 works fine in AOTS and Doom screams under Vulkan.
And you can flip this around and get the same for the 480.
I don't think this is how TPU calculates their performance summary.
Just the latest patch on Steam:
I'm on regular Win 10 Pro.
Precisely. Therefore using arithmetic mean is inconsistent. With arithmetic mean you can make an argument of both cards being 25% faster than the other
Yay. Someone gets it finally
However with geometric mean, you will get the consistent answer that both cards are equally fast.
Unfortunately TPU does the calculation using arithmetic mean (precisely as i did in my example).
See here.
because garbage data is garbage data. the only way to account for outliers is to eliminate them from your data set. That is what real statisticians do with outliers.
no it isn't see above. Any metanalysis that contains widely variable numbers reflecting identical values is a metanalysis that chooses to include bad or misleading data which no statistician considers a strength. garbage in, garbage out. Further, tossing in outliers within that pool is amateur hour. There is no analysis that "compensates for outliers." This is why you simple toss them out on the high end and the low end. The N of your data set doesn't really matter, because outliers will always skew it.
Well, that's the real problem, isn't it? It always depends on your subject and the question you are trying to answer. I am a biologist and a lot of our work is bioinformatics, though I'm more a molecular biologist...data kinda puts me to sleep. :\
snip
So they chose 2 games with DX12 (while there are currently ~7?), and yet those 2 games have broken DX12 implementation compared to others? Yet, when you see something like this, you suggest that maybe DX12 as a whole is an invalid variable, and not TPU's specific selection of the 2 broken DX12 games...and for what reason would TPU do this?
I am puzzled by the way you interpret this situation. I doubt you will be able to sufficiently explain your reasoning behind that, but I will try.
As for the final, bolded claim: Either you really aren't paying attention or you are outright lying.
The 2 most obvious examples:
--They chose the known broken implementation of TroTTR in DX12--that is terrible for both cards compared to DX11 in both cards, yet they do it anyway? How does this support your claim?
--They do not test Vulkan in Doom which not only favors both cards above DX11, but highly favors AMD. How does this support your claim?
Further, they still do not use AMD's updated and current drivers which are already known to boost 480 ~3-4% in most cases above the numbers that they still report. Why do you think they do this?
I challenge you to write a letter/email to above professor, which you seem to trust so much, and ask him what mean he is suggesting for our case! He will give you precisely the answer i gave you.
Yes, for other unrelated metrics geometric means might not be the correct choice. For our particular case it is the only choice as others are at least inconsistent.
I give up arguing with you. You seem to be learn resistent.
Sorry hitman DX12 is stable
snip...
And I can easily give you an example of the exact same kind of metrics (relative fps, 2 GPUs, 2 games), where geomean gives clearly incorrect answers compared to arithmetic mean.
Making an exception to my last statement because i cannot resist this offer. Please present your easy example Always interested in extending my collection of common fallacies.
GPU A GPU B
Game 1 100 FPS 5 FPS
Game 2 0.1 FPS 5 FPSNo it isn't, "real" statisticians will first try and determine the nature of an outlier, and only if it can be determined to be due to experimental error or similar issues will they eliminate them.
A data point being an outlier doesn't make said data point faulty or garbage.
then perhaps you can clarify for me what you could possibly be inferring from this part of your post:I never said that DX12 as a whole is an invalid variable, no idea where you got that idea from.
Take TPU (seeing as this seems to be the one people are hating on), they currently have 2 games in their benchmark suite with DX12, but the DX12 implementation for both those games is more or less broken.
Do you still think they should use DX12 in their testing then?
Nor did I ever say that TPUs selection of DX12 games was optimal, I simply said, that given they games they chose, using DX11 is the correct thing to do.
Mea Culpa, I'm afraid this is a case of me not paying attention. I genuinely thought that TPU tested Tomb Raider in DX11 and not DX12. As for them not using Vulkan in DOOM, there is a very good reason for this, namely the fact that DOOM isn't part of their benchmark suite, so that has nothing to do with my "claim".
agreed. To TPU's credit, someone here posted the update that scheduling with releases and traveling was bad for him and he is currently working on updating those benches. I'd be curious to see if some members here would then stop using TPU results exclusively if they stop showing what they hope they would show.As far as drivers goes, sure it would be nice if they used the new drivers, but then again they are still quite new and not even Anandtech are using them yet. No reason to crucify TPU over that.
Right, it doesn't. But if you are trying to determine any sort of mean in a large data set, then outliers skew your data.
This is why real statisticians toss them out when performing such analysis.
...this makes me wonder what type of data you are normally working with.
then perhaps you can clarify for me what you could possibly be inferring from this part of your post:
"Take TPU (seeing as this seems to be the one people are hating on), they currently have 2 games in their benchmark suite with DX12, but the DX12 implementation for both those games is more or less broken.
Do you still think they should use DX12 in their testing then?"
"Nor did I ever say that TPUs selection of DX12 games was optimal, I simply said, that given they games they chose, using DX11 is the correct thing to do."
again, not true, because:
"Mea Culpa, I'm afraid this is a case of me not paying attention. I genuinely thought that TPU tested Tomb Raider in DX11 and not DX12. As for them not using Vulkan in DOOM, there is a very good reason for this, namely the fact that DOOM isn't part of their benchmark suite, so that has nothing to do with my "claim"."
Using the relative values and geomean would put GPU B as 58% faster than GPU A, even though GPU B is completely unplayable in both games, whereas GPU A is perfectly playable in Game 1. Arithmetic mean correctly pegs GPU A as the better GPU (although at 901% faster, the exact value is quite arbitrary).
Sorry, this is ritch. Now because you cannot find an example where geomean does not give the correct and consistent answer to the question of how much A is faster compared to B in average you have introduce 2 unrelated metrics: "playable" and "better". This was not the question at all.
The correct answer is GPU B as 58% faster than GPU A (assuming you took the mean correctly, didn't bother to check this).
I close my case here.
Just wanted to thank you for spotting this issue and arguing against using arithmetic mean. Geometric mean is clearly the only method that makes sense here. I would be very surprised if W1zzard didn't implement this method instead.Sorry, this is ritch. Now because you cannot find an example where geomean does not give the correct and consistent answer to the question of how much A is faster compared to B in average you have introduce 2 unrelated metrics: "playable" and "better". This was not the question at all.
The correct answer is GPU B as 58% faster than GPU A (assuming you took the mean correctly, didn't bother to check this). And arithmetic mean again gives wrong and inconsistent answer. Irony is, that due to inconsistency of arithmetic mean, you again come to both conclusions using arithmetic mean that A is faster than B while B is faster than A at the same time (as Enigmoid correctly concluded further up in this very thread) Thanks for proving my point.
I close my case here.
I'm surprised to see the asus card compared to those leaked 480 nitro graphs. Although I'm assuming those aren't fake, it showed the 480 performing a bit better than the 1060.
In that case if it's true it doesn't seem like the asus 480 would be worth the price increase.
Because there is really no straightforward way of determining what is or isn't appropriate, so by using a mixture of settings you should hopefully even out any outliers.
As I just said above, this is actually a strength of meta analysis not a weakness.
And how do you determine what is bad data?
Take TPU (seeing as this seems to be the one people are hating on), they currently have 2 games in their benchmark suite with DX12, but the DX12 implementation for both those games is more or less broken.
Do you still think they should use DX12 in their testing then?
And as I said above TPU already tests in the API that best represents the cards, so what's the problem?
Why would mixing bad data with good data give you a better result than simply trusting the good data?