Potential New Anandtech Video Card Benchmarking Suite?

[DominionSeraph]

How do other hardware websites do it then?

They don't.
Name one site that benchmarks every card with every driver with every patch of every game.

If you fix the test bench to 2012 games, the review becomes outdated as more demanding games come out and people start looking to other websites for their upgrade questions. Also, consider the consequences of this since we are in a period of console port stagnation. If we finalize the test bench now, AT could end up missing 3-5 GPU demanding games in 2013. You cannot be a world class GPU hardware website and not update the game selection at least once in 12 months. I think they should especially make an exception this time since Metro LL and Crysis 3 will launch next year and these 2 games are way more important for GPU upgrading than games like Dirt 3, Batman AC, Civilization V, Skyrim, Diablo 3, and Portal 2. Why buy a $500 GTX680 to play Diablo 3 or Dirt 3 when modern GPUs get > 100 fps? Do you really need to see how GTX780 will play in those old games?

Why are you anchoring on the particular FPS? As long as the test is representative it doesn't matter if the GPU gets 10 or 10000FPS in it.

Anandtech does not bench games, they bench cards.

 

[Silverforce11]

@above, AT benches cards WITH games. Nobody cares that their GPU runs an old game at 200 fps, it doesnt inform them on their future purchases. Thats what this review site is about, informing potential buyers on their best choices. It should therefore reflect games that are a) very demanding (thus the need to upgrade) and b) games that are popular but still moderately demanding. Games that are popular but not demanding doesn't inform the buyer, they know ANY cheap gpu would crush it.

Arma series need to be in there, its actually very popular online with mods and custom servers etc. Its also an extremely demanding FPS.

Dirt 3 is old and not demanding even with 8x AA, time for it to go.

Skyrim has to stay, its one of the most popular RPG and with mods it can crush GPUs. Mods are a major factor for these games to keep them new and interesting.

Witcher 2 is a beautiful game and crushes GPU, period.

Guild Wars 2 should be here because you have no MMO, and its one of the more demanding MMO currently, it also looks to be going to stay around for a long time.

Shogun 2 on Ultra destroys GPUs and its still a popular RTS (its DLCs was top of steam sales), one of the few that pushes GPUs.

BF3 Armored Kill, for the obvious reason its one of the big titles as the representative of online FPS.

Other titles should be more recent games that stresses GPU, and also save a few slots for future upcoming AAA titles: FC3, Crysis 3, Metro etc.

 

[Grooveriding]

Borderlands 2 and Diablo 3 are worthless, those games are light weights. Crysis Warhead should stay in as it is still as demanding as any other game out there, it also helps to highlight strengths/weaknesses of nvidia/amd, kepler is weak in that game compared to Tahiti; see mem bandwidth limitations. Portal 2 is a joke as well, but makes sense because source games are so popular so people care. Really anything from the past 3 years can run source games maxed though.

Now that GPUs have gone beyond what most games really need, I would like to see tests with 8xAA and SSAA being done. At this point unless you are on triple monitors or a 30" panel, a 7970GE or 680 is all you need, so let's see some tests using high level AA modes on cards like the 7970GE and multi-gpu configurations.

 

[raghu78]

They don't. Name one site that benchmarks every card with every driver with every patch of every game.

You don't need to benchmark every card with every driver with every patch. The testing should have a relevant game suite updated on a 3 month or 6 month basis. Newer titles should replace older titles. Titles which are demanding should be given the highest priority Eg: BF3, Sleeping Dogs.

Why are you anchoring on the particular FPS? As long as the test is representative it doesn't matter if the GPU gets 10 or 10000FPS in it.

representative of what ? fps does not mean anything if the game is outdated. Performance in a DX9 game is not representative of performance in DX11 games. With the latest GPUs the gamer needs to know how the card performs in the latest titles. Tesselation performance, compute shader performance are important and not only shader performance or ROP performance. Performance in the latest engines like FrostBite 2, Cryengine 3 is important.

Anandtech does not bench games, they bench cards.

The primary purpose of a graphics card is to play games. Game performance is what matters most. The games need to relevant and be what people are playing. The games need to be a reliable indicator of the hardware's performance capabilities. The selected games should push the hardware to its limits and also give a clear assessment of the graphics card's strengths and weaknesses.

 

[JarredWalton]

Since someone was kind enough to start this thread, let me share some other items with you that you may not have considered.

1) I test laptops/notebooks, not systems. While a GTX 660 might scream through the vast majority of our gaming tests at 1080p, a GTX 660 is also about as fast as the top mobile GPUs.

2) Updating the gaming suite more than once a year makes comparisons with older hardware virtually impossible, particularly on the mobile side of things. Ryan can go and retest all the old GPUs in his benchmarking testbed (and it will take days of work, if not weeks, to go through each card at the various settings); for notebooks, most systems are back with their manufacturer within a month of the review posting -- and often even before then.

3) We have two primary reviewers for notebooks (Dustin and me), which means we now have to coordinate all of the testing parameters between two people. Vivek and Anand occasionally use the mobile test suite as well. Guess what that does if you try to add and change games regularly.

4) For those that say, "An old game getting 150 FPS says nothing about new titles," I respectfully disagree with the general sentiment. If we're totally CPU limited to 150 FPS, then it's pretty useless, but if the game is simply not as demanding so that it hits higher frame rates, it will match up with general performance expectations of other titles. However, what you also need to consider is that every game tested only provides a piece of the picture, and there is no "best" title or set of titles to test with. Dropping one game to replace it with a newer game just means we lose information about, e.g. Batman and gain information about Crysis 2 or whatever.

5) Modded games are out. Sorry. Someone else already said some of this, but basically you're opening a can of worms and when you use a mod you run a major risk of unfairly penalizing companies. NVIDIA or AMD might work hard to get Skyrim to run acceptably at max settings, and then you add a mod that pushes things so far that it might overflow the RAM on a 2GB card but not on a 3GB or 4GB card. Now it looks like the 2GB card sucks at Skyrim, but the 4GB card still runs well. The reason the game didn't ship in a state that it would use more than 2GB is because the developers already looked at that and said, "That doesn't make sense."

6) Right now, testing all of the games and other benchmarks on a notebook requires a pretty solid 20-25 hours of testing and writing, just for the basic suite. If you then have to retest a bunch of titles, you add a minimum of six hours -- I've done this multiple times with the P170EM reviews. Now, if you get paid a flat rate per article, what happens when you suddenly take twice as long? Let's just use a rate of $250 for a review as a baseline. At 25 hours you would make $10 per hour. If you then take an extra six hours, you're down to $8.06 per hour, and if you add games and end up requiring 40 hours of testing you're at a rate of $6.25. If any of you are willing to benchmark and then write a quality article for $6.25 per hour, please send me an email and I'm sure we can find a use for you! But seriously, this is a full-time job for most of us and we only have so many hours in a day.

Just to conclude, we will be looking at revamping our benchmark suite in the coming months. Windows 8 will already throw out many of our previous results, so very likely I'll change up the battery life and general applications for Win8, with gaming suites up for change by the end of the holiday shopping season. Ideally, I'd love to be able to drop every single game in our current list of seven titles (on notebooks) and replace them with seven popular and yet GPU intensive titles for 2013. So when looking at the games we should add, please consider the stuff coming out in the next two months as a higher priority than stuff that has been released this summer or earlier.

If someone wants to send me an email once you reach a consensus, please do so; otherwise, I'll try to keep an eye on this thread for when it's time to really mix up the games. Very likely, we'll start in 2013 and use about seven or eight games for the whole year (and occasionally look at other games as warranted). We like to have a wide selection of titles as well, so FPS/Action, RPG, Strategy, Driving/Simulation, and MMO should all be present if possible, with multiple titles for the most popular categories (usually FPS).

Until the list is finalized, thanks for the input! I'll point Ryan at this as well, since most of you seem to be talking more towards his realm of testing (desktop GPUs).

 

[Silverforce11]

5) Modded games are out. Sorry. Someone else already said some of this, but basically you're opening a can of worms and when you use a mod you run a major risk of unfairly penalizing companies. NVIDIA or AMD might work hard to get Skyrim to run acceptably at max settings, and then you add a mod that pushes things so far that it might overflow the RAM on a 2GB card but not on a 3GB or 4GB card. Now it looks like the 2GB card sucks at Skyrim, but the 4GB card still runs well. The reason the game didn't ship in a state that it would use more than 2GB is because the developers already looked at that and said, "That doesn't make sense."

They release these games in their state with low res textures due to consolitis, you should be well aware of this. Many people consider mods as essential for single player RPGs, ie. Oblivion, Fallout, Dragon Age and now Skyrim. The entire community of these games still exists for this sole reason.

Yes, it would punish cards with low vram. That's THE POINT of going for cards with extra vram, as there's no other advantages, on the contrary, it uses extra power for that extra gddr5. If you arent going to test it for these reasons, you ARE NOT informing buyers of this critical point. The best approach would be to do both, vanilla and modded. But obviously, manpower and time aren't unlimited..

ps. VRAM is a critical factor on gpus, as much as ROPs, shaders, etc. 1gb vs 2gb variants, pay a bit extra and go for 2gb. Or 1.5gb vs 3gb etc etc. Soon it will be 2gb vs 4gb. You are deliberately not testing this factor.. why?

 

[Termie]

Thank you for your input, Jarred. If it appears there's some consensus on 7 or 8 games, I'll make sure to pass that along.

 

[f1sherman]

They release these games in their state with low res textures due to consolitis, you should be well aware of this. Many people consider mods as essential for single player RPGs, ie. Oblivion, Fallout, Dragon Age and now Skyrim. The entire community of these games still exists for this sole reason.

Yes, it would punish cards with low vram. That's THE POINT of going for cards with extra vram, as there's no other advantages, on the contrary, it uses extra power for that extra gddr5. If you arent going to test it for these reasons, you ARE NOT informing buyers of this critical point. The best approach would be to do both, vanilla and modded. But obviously, manpower and time aren't unlimited..

CAN OF WORMS

VRAM being only a single issue.

Community efforts are nice, but optimizing for every piece of hardware is not exactly their forte. Because they are often hardware/resources/knowledge limited.

Try Doom3 Sikkmod with some Soft-Shadowing/HDR/PoM/AO on your 7950.
You'll be lucky to brake 30fps .

 

[blastingcap]

Is it a system-killer that at highest settings will drive all single-GPU cards down to 60 fps or less, at 1080p?

Is it reasonably popular or at least well-reviewed (80+ at Metacritic)?

Then it's a potential candidate.

While you're at it, Anandtech, you may want to include some smoothness tests, imperfect as they may be, a la the "frame times" metric TR uses.

 

[notty22]

They release these games in their state with low res textures due to consolitis, you should be well aware of this. Many people consider mods as essential for single player RPGs, ie. Oblivion, Fallout, Dragon Age and now Skyrim. The entire community of these games still exists for this sole reason.

Yes, it would punish cards with low vram. That's THE POINT of going for cards with extra vram, as there's no other advantages, on the contrary, it uses extra power for that extra gddr5. If you arent going to test it for these reasons, you ARE NOT informing buyers of this critical point. The best approach would be to do both, vanilla and modded. But obviously, manpower and time aren't unlimited..

ps. VRAM is a critical factor on gpus, as much as ROPs, shaders, etc. 1gb vs 2gb variants, pay a bit extra and go for 2gb. Or 1.5gb vs 3gb etc etc. Soon it will be 2gb vs 4gb. You are deliberately not testing this factor.. why?

For the sake of debate. Consider this. If you owned a 5970 or were a fan of it, when it was held up as the fastest video card against the gtx 480 and even the gtx 580 when it launched. It was. Different reasons were used why card wars were debated this way. The 5970 price was not much more than the top Fermi card at various time points and it used around the same amount of power.
But every benchmark could have been 'staged' to bring the 1gb 5970 to it's knees when the gtx 480/580 had 50% more ram. Either by AA or mods. The idea of bringing community mods makes no sense to me at all.
A supported company mod, that say Steam even installs for people is more reasonable, to consider.

 

[Majcric]

They don't.
Name one site that benchmarks every card with every driver with every patch of every game.




Why are you anchoring on the particular FPS? As long as the test is representative it doesn't matter if the GPU gets 10 or 10000FPS in it.

Anandtech does not bench games, they bench cards.

This is only partially true. New cards need to be tested on the Newer games or in general games that can stress the hardware to its limit. I don't think anyone wants to see a benchmark of a GTX 780 or HD 8970 running half life 1, Starcraft, Planescape Torment, Baldurs Gate, etc. While these games are much older than Amandtech's current lineup, the point is all games eventually become old and need to be tossed in the "I know my computer can run this extremely well category" with the newer or graphical demanding games we/most want to see how that hardware pars with its newer counterparts and so forth.

 

[f1sherman]

But every benchmark could have been 'staged' to bring the 1gb 5970 to it's knees when the gtx 480/580 had 50% more ram. Either by AA or mods. The idea of bringing community mods makes no sense to me at all.
A supported company mod, that say Steam even installs for people is more reasonable, to consider.

I love modding, but AT benchmarking mods does not even sound like a serious idea.

It's an unnecessary complication, there are compatibility issues, support issues, and it's watering down of what could have been meaningful and comparable result.

With newest GTX 780 we'are getting 67.2fps in Doom 4 with BestModHD 3.1 and Monoxead Textures v2.31. using this autoexec.cfg.
If you have AMD HD8000 GPU you'll want to D/L openGLIB43.dll v5.1, else you won't be able to run the game.

Out comes BestModHD 4.0 - Nvidia gains 20% due to mod author owning NV hardware, and doing some shader optimizations, AMD stops working all together.

What now?

80% of us don't even use this BestModHD. The hell do we care about benchmarking it at all?

 

[ExcaliburMM]

As an SC2 player its rather important to me that they keep it in the benchmarks. Its one of the few I look at when comparing similarly performing cards.

 

[Dankk]

I'm gonna go ahead and agree: Mods shouldn't be included. Makes things more complicated than they already are.

 

[Magic Carpet]

The only benchmark I pay attention to, is Crysis. I think, Crysis 3 will take its place early next year, though.

 

[Majcric]

As an SC2 player its rather important to me that they keep it in the benchmarks. Its one of the few I look at when comparing similarly performing cards.

Are you not already ranging in the 60 fps? my old 9800 gtx+ would get close to this and your card is faster. But more importantly, Starcraft 2 loves cpu and it has better belonging in a cpu benchmark. It is a limiting piece of software that at this point is uterrly useless for benchmarking high-end or even mid-range GPU's of today.

 

[Silverforce11]

There's mods, then there's mods.

If you even play skyrim on a regular basis, the HD texture mod is a no brainer.

There are no can of worms when some mods are downloaded millions of times: http://skyrim.nexusmods.com/mods/607 etc etc.

Obviously the reviewer would need to actually do a few seconds of homework to pick the popular mods that improve the games visuals. But if AT does not, other review sites do, so...

 

[Majcric]

The only benchmark I pay attention to, is Crysis. I think, Crysis 3 will take its place early next year, though.

When a new high-end GPU releases, I often find myself flipping through to the Crysis and Metro 2033 benchmarks right out the gate and then later returning to reread the article.

 

[f1sherman]

There's mods, then there's mods.

If you even play skyrim on a regular basis, the HD texture mod is a no brainer.

There are no can of worms when some mods are downloaded millions of times: http://skyrim.nexusmods.com/mods/607 etc etc.

Obviously the reviewer would need to actually do a few seconds of homework to pick the popular mods that improve the games visuals. But if AT does not, other review sites do, so...

Skyrim HD - 2K Textures?
I remember this mod back when it got out 1st. It burned my eyes with just awful overdone sharpness, destroyed some pretty well done default textures, like rocks.

See that wooden fence on the right.
It will flicker from a mile

 

[MrK6]

Skyrim HD - 2K Textures?
I remember this mod back when it got out 1st. It burned my eyes with just awful overdone sharpness, destroyed some pretty well done default textures, like rocks.

See that wooden fence on the right.
It will flicker from a mile

Yep. There's more to texture mods then just dropping a high resolution texture in and calling it a day. Texturing is very much an art as well as a science, and it takes a lot of skill to get a "natural" look.

All that said, no mods unless they're official. If everyone wants to see Skyrim tests, official HQ pack only.

 

[Termie]

Ok, after reading everyone's comments, along with Anandtech Notebook editor Jarred Walton's insights, I've come up with a list of games that arguably represents a consensus:

(1) Battlefield 3 (but not Armored Kill due to repeatability problem) - to be replaced with Medal of Honor WarFighter 3 (same engine)
(2) Guild Wars 2
(3) Elder Scrolls: Skyrim SSAA or The Witcher 2 Enhanced Edition - these are both DX9 RPGs, so it's hard to see how they both belong in a 2013 test suite.
(4) Total War: Shogun 2 Ultra
(5) Crysis: Warhead - to be replaced with Crysis 3
(6) Metro: 2033 - to be replaced with Metro Last Light
(7) Dirt Showdown or F1 2012
(8) Sleeping Dogs or Max Payne 3

Assuming Anandtech will only have an 8-game core benchmarking suite, this provides a great diversity of games:
- one driving game
- one RTS
- one MMO
- one third-person shooter/adventure
- one RPG
- three FPS on different engines (FB2, CE3, Metro)

I'm happy to start a new poll to take votes on these games. I don't believe I can change the existing poll in this thread.

 

[JarredWalton]

Ok, after reading everyone's comments, along with Anandtech Notebook editor Jarred Walton's insights, I've come up with a list of games that arguably represents a consensus:

(1) Battlefield 3 (but not Armored Kill due to repeatability problem) - to be replaced with Medal of Honor WarFighter 3 (same engine)
(2) Guild Wars 2
(3) Elder Scrolls: Skyrim SSAA or The Witcher 2 Enhanced Edition - these are both DX9 RPGs, so it's hard to see how they both belong in a 2013 test suite.
(4) Total War: Shogun 2 Ultra
(5) Crysis: Warhead - to be replaced with Crysis 3
(6) Metro: 2033 - to be replaced with Metro Last Light
(7) Dirt Showdown or F1 2012
(8) Sleeping Dogs or Max Payne 3

Assuming Anandtech will only have an 8-game core benchmarking suite, this provides a great diversity of games:
- one driving game
- one RTS
- one MMO
- one third-person shooter/adventure
- one RPG
- three FPS on different engines (FB2, CE3, Metro)

I'm happy to start a new poll to take votes on these games. I don't believe I can change the existing poll in this thread.

I'll tell you right now that we're not bringing back Metro 2033 or Crysis Warhead for notebooks. Sorry. Dustin and I were both very much in agreement that while we liked both games, they're not worth continuing to test. Metro: First Light and Crysis 3, sure, but I'm not moving backwards before moving forward. Also keep in mind that we're not planning to do a piecemeal update of the gaming benchmarks, so we're using the same core seven titles until we update, most likely in late Dec/early Jan.

Something else I didn't mention earlier is that I also like to see a diversity of gaming engines. I don't know offhand all the engines in use for the various titles, but just as an example we wouldn't be interested in testing four different titles that all use Unreal Engine 3. Anyway, we'll be looking to come to some sort of consensus by January, and while I can't guarantee we'll use the titles you want we can try.

As for desktop GPU testing, that's basically up to Ryan, so his list of games is actually a lot closer to yours. Ryan also tends to test a few more games, largely because his reviews are purely GPU focused (e.g. gaming), so there's no need to discuss general application performance, build quality, LCD quality, speakers, keyboards, trackpads, etc. So keep that in mind: if you come up with 15 games, he might take eight of them and add two of his own choice, and I might use six of them with a couple others for notebooks. Our intention is generally to have the notebook gaming suite be a subset of the desktop suite.

 

[soccerballtux]

Crysis should stay on since that's the first game where people still go "wow, I still can't play this on high and maintain 60fps). It's nice for when new mid-range cards are released, so we can see "I can buy this affordable card, and I'll finally be able to play Crysis"

 

[cmdrdredd]

However, most users WILL mod their Skyrim with high res stuff. Knowing this they would probably want to know what card won't limit them as much. If a 2GB card struggles so what? Note it and say "this card cannot run the mods like this". That's Nvidia's fault for only using 2GB as standard and they SHOULD be ridiculed for it if indeed some title really has issues. I know for a fact that ENB Skyrim with 4k textures does NOT use a full 2GB. Close, but not fully.

Saying you won't use mods because it will expose the vram limitations (you mentioned 2GB vs 3GB which translates to Nvidia vs AMD) sounds like you are afraid to paint Nvidia in a poor light. Playing favorites are we? If performance tanks we deserve to know as readers. Journalists are supposed to be neutral and simply show us the facts as they find them. Let us draw the conclusions.

What you seem to be saying by removing key titles like Metro and not wanting to expose potential VRAM limitations by using simple mods for Skyrim is that I should find another site for my GPU reviews. I know you do Notebook and mobile GPUs Jarred but if I were buying a gaming laptop or desktop replacement I want to know how it handles all this stuff that my desktop could.

This is all my opinion. Not meant to be directed at anyone at all.

 

[Ryan Smith]

Hi guys.

Picking up from where Jarred left off, I'll try to answer questions about our benchmarking suite from the desktop GPU side of things. I'm not going to have the time to respond to every individual post, but I did want to respond to a few questions/suggestions.

Crysis: Warhead: It won't be going away from the GPU suite, at least not for the next iteration of it. My plan has been to keep it until we have single-GPU cards that can do 60fps @ 1920 with everything cranked up. We're not there with the current generation but we will be in the next. It's the quintessential "modern" game and the game that broke the 8800GTX. I do want to have one classic title to track for historical purposes, and I know nothing better than Crysis.

Crysis 2: No. Specifically, I'm only interested in its DX11 mode, and that mode does too much weird stuff. It goes without saying that I'm hopeful Crysis 3 will be better behaved.

MMOs: MMOs are generally a no-go on the desktop GPU benchmark. This is for a few reasons. MMOs patch early and often, which can break our ability to compare results. MMOs sometimes have significant downtime, which would impact our ability to test. And MMOs generally lack repeatability, which is a cornerstone of our testing methods. This doesn't completely exclude MMOs, but we have to be able to solve those issues if we're to use one. I'd rather have an MMO in given how popular they are, but I cannot compromise testing standards for it.

BF3/MoH/Frostbite 2: We'll see what Medal of Honor is like, but I doubt we'll use it. MoH is DICE's B-team game, and if it's anything like the previous iteration then Battlefield will continue to be the more actively played game. On that note Armored Kill is sadly out due to the lack of repeatability of MP.

Shogun 2: Our settings will be going up in the next iteration. I can safely ignore 1GB cards like the GTX 560 Ti at 1080p now that we're a generation past them.

Frame times: I've always said I like the concept, but I want to do better than FRAPS. I have a plan for this, but we'll see if it pans out.

Starcraft 2: SC2 is probably out of the next suite. 1.5 made it more difficult to test, and the load just isn't there. Perhaps Heart of the Swarm will change this but we'll see.

8x MSAA: Don't expect us to use 8x MSAA in any benchmarks. The quality improvement from 8x is extremely minor; I'd rather turn on TrAA/AAA or SSAA first than to blow performance like that, which is one of the reasons why we have Portal 2 with SSAA.

What AT Reviews: To whoever pointed out that AT reviews a lot of mobile devices, this isn't an either/or thing. My job is solely to handle GPUs; mobile devices are handled by other staff. So AT reviewing mobile devices does not preclude GPU reviews, and. What does preclude doing more GPU articles is the sheer amount of work it takes and all the other GPU projects also going on; HPC, workstation cards, GPGPU, etc. Until cloning is made available I'm only one person.

Mods: User-made mods are out of the question due to the issues Jarred mentioned. We do run Skyrim with the official HD texture pack though. But that's basically the exception to the rule since very few games have official mods of any magnitude.

Adding games mid-cycle: As Jarred already noted, the sheer amount of work it takes keeps us from frequently changing the benchmark suite (just go look at how many GPUs I have in bench, and I haven't done nearly as much SLI/CF testing this year). This doesn't preclude adding a game mid-cycle, but it means there needs to be a very good reason due to all of the prep-work required. Another factor is where we are in the GPU family launch cycle; it's far more difficult to add a game if we're in the middle of a launch blitz. E.G. if the rumors about the next generation of GPUs not arriving until Q2 are true, then it will be easier to add Q1 games ahead of that.

Finally, since we're on the subject of new games, let me throw something back at you. Coming up with first and third person games is fairly easy. My problem is that I really could use new strategy and sim games so that we aren't so FPS/TPS heavy. Is there anything you guys would like to see, keeping in mind our repeatability requirements?