How do you know it's still wrong after they've updated it?
And which games exactly are outdated (other than BF3, bit weird to still include that when you also have BF4?)
Also, TPU didn't use Doom Vulkan. They used Hitman DX11 and Rise of the Tomb Raider DX12. It's backwards actually when RotTR DX12 even in their previous testing, runs faster in DX11 for everybody. While Hitman DX12, AMD's lead grows.
One game can swing the results when the rest are pretty close. Examples: Anno swings wildy to NV, where a 970 is faster than Fury X. Likewise Project Cars. On AMD swing games, it's definitely Hitman DX12 and Doom w/ Vulkan.
So looking at it objectively, TPU's current testing, favors NV strongly while Computerbase's testing is neutral, because they include games that swing to NV and others that swing to AMD along with neutral titles.
That reminds me, does anyone know if any of the sites out there has gotten around to doing a proper rebenching of Tomb Raider in DX12 with the new patch?
Because Wizzard explained how they doing this here
https://www.techpowerup.com/forums/threads/tpu-math-check.224326/
This is mathematical nonsense.
Yes BF3 is a good example. In general i would rather remove games if they not sufficiently represent state-of-the-art.
Do they favor AMD or do they just not favor Nvidia? Like I see a lot of Gameworks titles on there.. and I don't think the ones you listed have any AMD involvement other than Hitman, and even then they are using DX11 instead of DX12. Even Arkham Knight I think is a "the way it's meant to be played game".That reminds me, does anyone know if any of the sites out there has gotten around to doing a proper rebenching of Tomb Raider in DX12 with the new patch?
It's not like TPU is completely devoid of games that favour AMD, i.e. Arkham Knight, COD: Black OPS 3, Hitman and Just Cause 3 (the reference 480 is faster than the reference 1060 in all 4 of those).
antihelten said:No it isn't, that's a completely normal arithmetic mean.
Do they favor AMD or do they just not favor Nvidia? Like I see a lot of Gameworks titles on there.. and I don't think the ones you listed have any AMD involvement other than Hitman, and even then they are using DX11 instead of DX12. Even Arkham Knight I think is a "the way it's meant to be played game".
I would rather not rely on metascores because as you widen the criteria there is more chance for outliers to spoil the results. Instead I would like to understand the methodology used and come to my conclusion when presented with data.3Dcenter collected and averaged the results from 12 different sites that together ran a total of 148 benchmarks.
They get the 1060 as 8.2% faster on average across all these sites, so any game that is significantly below this could be said to favour AMD and any game that is significantly above this could be said to favour Nvidia.
TPU has 5 games that are more than 5% above this number and can thus be said to favour Nvidia (Anno 2205, AC: syndicate, BF3, BF4, Crysis 3), and 4 games that are more than 5% below this number and can thus be said to favour AMD (Batman AK, COD: Black Ops 3, Hitman, Just Cause 3). The remaining 6 games fall within 5% of the above number and can thus be said to be neutral (Fallout 4, Far Cry Primal, GTA V, Rainbow Six: Siege, RotTR, Witcher 3).
See my last post. You need to use geometric mean if you want to average relative values. Using arithmetic mean is nonsense.
I would rather not rely on metascores because as you widen the criteria there is more chance for outliers to spoil the results. Instead I would like to understand the methodology used and come to my conclusion when presented with data.
But say I don't care about OpenGL performance in Doom. Because I know if I buy an AMD GPU I will play it on Vulkan with a 30%+ better performance. How many reviewers used OpenGL Doom scores? More than Vulkan based on what I've seen so far. Similar goes for Hitman.You most certainly do not need to use geometric means for relative values.
The only case in which a geomean can be considered mandatory is when you are dealing with cumulative values, such as compound interest.
One could argue that a geomean would be more apt than an arithmetic mean in this case, but that would to some degree be a subjective choice and besides the difference is almost non-existent.
Eerm no, using metascores reduces the chance of an outlier spoiling the results, that's kinda the whole reason behind metascores.
But say I don't care about OpenGL performance in Doom. Because I know if I buy an AMD GPU I will play it on Vulkan with a 30%+ better performance. How many reviewers used OpenGL Doom scores? More than Vulkan based on what I've seen so far. Similar goes for Hitman.
I would rather rely on one site that does it right than a majority who do it wrong.
antihelten said:You most certainly do not need to use geometric means for relative values.
And how would you know if any one site does it right?
I mean it's great if you could find a site that specifically tests only the games you're interested in, but for the vast majority of people that just isn't possible.
They all publish their methodologies computerbase seems to do it right.. they use the API which runs best on each card. It gives a true representation of the cards performance without gimping one in favor of the other.
And the sad fact is they are about the only one of the handful of sites who do it that way. At least for this launch.You make a good point, I forgot to raised it.
Computerbase will run DX11 in games where NV loses performance in DX12. This is a true reflection of how gamers would do it, they play with their GPU in a gaming case, not open bench, they don't play for 3 minutes at a time, but longer stretches (hence, the warm up) and they will select the best performing API for themselves.
3Dcenter collected and averaged the results from 12 different sites that together ran a total of 148 benchmarks.
They get the 1060 as 8.2% faster on average across all these sites, so any game that is significantly below this could be said to favour AMD and any game that is significantly above this could be said to favour Nvidia.
TPU has 5 games that are more than 5% above this number and can thus be said to favour Nvidia (Anno 2205, AC: syndicate, BF3, BF4, Crysis 3), and 4 games that are more than 5% below this number and can thus be said to favour AMD (Batman AK, COD: Black Ops 3, Hitman, Just Cause 3). The remaining 6 games fall within 5% of the above number and can thus be said to be neutral (Fallout 4, Far Cry Primal, GTA V, Rainbow Six: Siege, RotTR, Witcher 3).
You make a good point, I forgot to raised it.
Computerbase will run DX11 in games where NV loses performance in DX12. This is a true reflection of how gamers would do it, they play with their GPU in a gaming case, not open bench, they don't play for 3 minutes at a time, but longer stretches (hence, the warm up) and they will select the best performing API for themselves.
antihelten said:And how would you know if any one site does it right?
Because you read it? Computerbase.de typically makes an in-depth analysis of power and temperature targets in conjunction with 20 min playtime. They figured, that they would have to increase power target for RX480 STRIX in order to avoid throttling...Something what TPU did not do. Then they (TPU) increase frequency, but effective frequency was only going partially up, because power target was still in place...and then we end up getting a post from Sweepr claiming only 3% OC potential...based on the TPU review of course.
What we learn here? Avoid the incompetent review sites.
Oh please, don't try to argue this, your are going to loose this argument since apparently you have no math background. It could not be more wrong to use arithmetic mean.
They all publish their methodologies computerbase seems to do it right.. they use the API which runs best on each card. It gives a true representation of the cards performance without gimping one in favor of the other.
Those aren't 148 unique games, so saying the average is 8.2% per game is wrong.
If I have two games, 10% faster, 10% slower. If the first game is tested in 10 reviews, and the second game tested in only 5 reviews, that doesn't mean that it is faster overall just because it was faster in more reviews.
Because you read it? Computerbase.de typically makes an in-depth analysis of power and temperature targets in conjunction with 20 min playtime. They figured, that they would have to increase power target for RX480 STRIX in order to avoid throttling...Something what TPU did not do. Then they (TPU) increase frequency, but effective frequency was only going partially up, because power target was still in place...and then we end up getting a post from Sweepr claiming only 3% OC potential...based on the TPU review of course.
What we learn here? Avoid the incompetent review sites.
Sure it may very well be better than the TPU reviews but that still doesn't make it perfect and as such the meta reviews done by 3Dcenter are still the most reliable thing we have.
I never said that it was 148 unique games nor did I say that the average is 8.2% per game (which doesn't even make any sense, an average value obviously isn't per game, it's calculated across all the games used)
antihelten said:And you clearly don't have a background in computer science.
why would you include multiple tests of the same game when each test uses variable settings that in many cases are completely inappropriate?
There are many reasons that meta analysis are worthless and this is a big one
--there is no reason to include bad data that does not reflect actual user use--Wouldn't you choose to use the API that best get the performance out of your product?
Of course you would, right?
A proper benchmark model across various cards and various games should have each card performing in each game using API that best represent that card's performance, then you compare actual performance in real user events.
garbage in, garbage out.