[TECH Report] As the second turns: the web digests our game testing methods

Page 11 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

Homeles

Platinum Member
Dec 9, 2011
2,580
0
0
I don't think this has been posted here yet.
It has:
PCPER ditching standard FPS measuring all together and instead capturing video output.

This is so much win. Possibly the only thing missing is recording input(mouse) lag.

http://www.pcper.com/reviews/Graphics-Cards/Frame-Rating-New-Graphics-Performance-Metric
But either way, it's welcome news. I liked how Ryan Shrout was reaching out to Reddit community. Reddit doesn't like him though, because he doesn't give AMD a reacharound.
 

krumme

Diamond Member
Oct 9, 2009
5,956
1,595
136
We all know just how huge the uproar for the 5xx series was.

There is a reason why NV pays for technical marketing, user groups and invest hugely in their brand. It pays off, but you need to have the cash for it, and it takes years and talent to build.

The value of a brand is so strong the personal perception can change, contrary to the like "I've been noticing the stuttering also in games since going from a GTX 580 to the 7970 Ghz"

On top level you simply decide how to change the perception of value: du you fx. prioritize the driver team or the marketing.

I wellcome new methods of assessment. I hope we get something that can be validated in blindtests, and not another set of methods that draws unessesary attention from real progress.
 

3DVagabond

Lifer
Aug 10, 2009
11,951
204
106
There is a reason why NV pays for technical marketing, user groups and invest hugely in their brand. It pays off, but you need to have the cash for it, and it takes years and talent to build.

The value of a brand is so strong the personal perception can change, contrary to the like "I've been noticing the stuttering also in games since going from a GTX 580 to the 7970 Ghz"

On top level you simply decide how to change the perception of value: du you fx. prioritize the driver team or the marketing.

I wellcome new methods of assessment. I hope we get something that can be validated in blindtests, and not another set of methods that draws unessesary attention from real progress.

+1 Humans are very susceptible to the power of suggestion. I've personally been involved with testing where the subjects were aware of what they were comparing and then compared the same things blindly. Not one person could back their open observations when the tests were blind to any kind of statistical significance.
 

f1sherman

Platinum Member
Apr 5, 2011
2,243
1
0
+1 Humans are very susceptible to the power of suggestion. I've personally been involved with testing where the subjects were aware of what they were comparing and then compared the same things blindly. Not one person could back their open observations when the tests were blind to any kind of statistical significance.



+1

To hell with high speed cameras, benchmarking and... numbers

We need more blind tests by Joe The Trucker.

Mobile Trinity Blind Test - AMD the Clear Winner!



********
AMD Reality Check

  • System C (Intel Core i7-2700K): 40 Votes
  • System D (AMD FX-8150): 73 Votes
  • No Difference: 28 Votes
 
Last edited:

BrightCandle

Diamond Member
Mar 15, 2007
4,762
0
76
More in the second podcasts from Techreport.

http://techreport.com/review/24153/the-tr-podcast-126-as-the-second-turns-an-unexpected-journey

The parts about frame timing etc starts around the 40 minute mark.

One thing I especially like is the test done with games that have frame times built in for the purpose of testing the response and the conclusion of that test is that fraps is measuring the same thing. Which brings yet more evidence that the fraps frame times are indeed what people are seeing in regards to the game world update smoothness (52 minute mark). Mark Poppin (alienbabeltech.com) also notes that the frame time graphs correlate with the experience at that point.

Mark Poppins analysis can be found here where he investigates if fraps is a good tool for this (tldr - its pretty good): http://alienbabeltech.com/main/?p=33060
 
Last edited:

Rikard

Senior member
Apr 25, 2012
428
0
0
Which brings yet more evidence that the fraps frame times are indeed what people are seeing in regards to the game world update smoothness
Well, I think that the fluctuations FRAPS picks up are displayed to the user who will see them if they are of sufficient magnitude. However, I have noticed that when FRAPS has perfectly smooth frametimes you can still have stuttering performance, so it does not tell the full story. (I wonder if I can capture it on video, but I think Hitman here already did when doing a frame cap that gave beautiful constant frames in FRAPS but horrible stuttering video.)

FRAPS is a blunt instrument. I think using capture cards like pcper are doing is the way forward.
 

KCfromNC

Senior member
Mar 17, 2007
208
0
76
+1 Humans are very susceptible to the power of suggestion. I've personally been involved with testing where the subjects were aware of what they were comparing and then compared the same things blindly. Not one person could back their open observations when the tests were blind to any kind of statistical significance.

Yep. This happens in all fields. That's why the negative results from blind testing are interesting to me. Sure, something's going on with the measurement but as far as anyone can show, what's being measured is tough to perceive without someone telling you beforehand what you're supposed to see. This is perfectly consistent with all of the benchmarks out there, too - as others have pointed out, the measured problems with nV 5xx cards were never seen before in actual game play testing.

And yeah, I know we'll be accused of being blind. It's the same approach audio marketing takes when they want to sell $15,000 power cables (http://bmicable.com/oceanic_statement) and $200 magic stones (special sale! - http://www.musicdirect.com/p-7506-shakti-electromagnetic-stabilizer-stone.aspx) which make music sound better. Sure, you'll hear a difference with these just so long as you believe the hype. When you don't know if they're installed or not, suddenly the difference goes away. There's a reason people spend so much money (and give away so much "free" hardware) for marketing.
 

Ferzerp

Diamond Member
Oct 12, 1999
6,438
107
106
It's amazing to me that after AMD has stated that there is an issue, there is post after post claiming that there isn't (and that's putting what many of you have posted in the most generous of lights).
 

Keysplayr

Elite Member
Jan 16, 2003
21,209
50
91
And yeah, I know we'll be accused of being blind.

You do? Sounds like you've predetermined your answer before taking the blind test yourself. It's like you already know that you wont be able to tell the difference.
 

Rikard

Senior member
Apr 25, 2012
428
0
0
It's amazing to me that after AMD has stated that there is an issue, there is post after post claiming that there isn't (and that's putting what many of you have posted in the most generous of lights).
Who is? Please use the quote function so we can read what posts you are referring to.
 

railven

Diamond Member
Mar 25, 2010
6,604
561
126
It's amazing to me that after AMD has stated that there is an issue, there is post after post claiming that there isn't (and that's putting what many of you have posted in the most generous of lights).

I for one hope AMD rolls out a fix quickly. While it isn't killing me when I notice the stuttering, it be nice for it not to be there.

And I'm glad in the games I've played I only notice it in MoP.
 

KentState

Diamond Member
Oct 19, 2001
8,397
393
126
There is a reason why NV pays for technical marketing, user groups and invest hugely in their brand. It pays off, but you need to have the cash for it, and it takes years and talent to build.

The value of a brand is so strong the personal perception can change, contrary to the like "I've been noticing the stuttering also in games since going from a GTX 580 to the 7970 Ghz"

On top level you simply decide how to change the perception of value: du you fx. prioritize the driver team or the marketing.

I wellcome new methods of assessment. I hope we get something that can be validated in blindtests, and not another set of methods that draws unessesary attention from real progress.

I'm not sure if you are accusing me of brand loyalty, but the stuttering was visible to me before becoming aware that it was a reported issue. I was disappointed that my new $400 card seemed to perform poorly at times with the bundled games. For example, I loaded up Sleeping Dogs and ran the benchmark mode under video settings. The stutter was noticeable and then verifiable once I downloaded MSi Afterburner and had the monitoring on.
 

railven

Diamond Member
Mar 25, 2010
6,604
561
126
Nvidia and Keysplayr's plan seems to be moving along nicely.

Haha, someone's going to get a promotion! Maybe his own office and an assistant so he doesn't have to post anymore!

I'm just messing. Haha.
 

VulgarDisplay

Diamond Member
Apr 3, 2009
6,193
2
76
It's amazing to me that after AMD has stated that there is an issue, there is post after post claiming that there isn't (and that's putting what many of you have posted in the most generous of lights).

There is an issue, but it's being blown way out of proportion.

Nvidia just had a massive security issue with there drivers and we saw maybe 10 posts about it. AMD has an issue that probably 90% of viewers could not notice an we are on the third 15 page thread full of nvidia trolls blowing things out of proportion. That to me is amazing.
 

Hitman928

Diamond Member
Apr 15, 2012
5,622
8,857
136
It's amazing to me that after AMD has stated that there is an issue, there is post after post claiming that there isn't (and that's putting what many of you have posted in the most generous of lights).

I think it's important to reiterate what Dave Baumann (AMD) actually said about this:

DaveBaumann said:
Curious, where did you read any of that?

First off, I mentioned that we have a memory management rewrite underway for orthogonal reasons and it was at a point where we could just try it and see if it makes any improvements; it transpires that in some cases it has helped improve things. This has raised the priority to trying to get this through earlier; that doesn't mean that we "weren't going to bother", otherwise what would have been the point of investing the development time in the first place? Again, the memory manager rewrite was being done for other reasons, and in fact there was no indications from anywhere (including TR's prior reviews) was this an area that needed to be addressed.

Secondly, the memory manager change is not "magic bullet". The quote Scott originally highlighted from me was that these types of things can occur from multiple different areas of the driver - for instance the BL2 improvement primarily comes from resizing a buffer, not related to generic the memory management at all.

This is not due to solely chasing average FPS. One of the software engineers looking into this contacted me to point out that for at least two of the titles where we've made improvements the average FPS should be improved, maybe by a reasonably large degree in some cases.

I can't say that in the games I've played they have been noticeable, likewise I do not see end user feedback for this type of issue (except, maybe outside of a few known problem titles).
As far as I had seen, TR's prior reviews using this method didn't highlight anything particularly untoward.

Likewise, we all use our GPU's and, for instance, I've put may many hours in to Borderlands 2 (207 in fact :gasp:!) on other an HD 7970 Ghz Ed or a HD 7870 and I genuinely can't notice any affects on the gameplay related to this to have even flagged it up as something I can perceive. However, we take it seriously and this has occupied quite a lot of architectures and software engineers time over the past few weeks to see where this is coming from and whether it can be improved.

Yes, I think this is the area that needs further investigation. It is interesting that TR has does this type of analysis, and flagged an "issue" but to all intents and purposes I have no frame of reference that indicates that this is an "issue" that affected people in most of the cases; as mentioned nobody has really reported this an issue that affects their gameplay.
 

railven

Diamond Member
Mar 25, 2010
6,604
561
126
There is an issue, but it's being blown way out of proportion.

Nvidia just had a massive security issue with there drivers and we saw maybe 10 posts about it. AMD has an issue that probably 90% of viewers could not notice an we are on the third 15 page thread full of nvidia trolls blowing things out of proportion. That to me is amazing.

Isn't that a little hypocritical? One is "massive" yet the other is "blown way out of proportion."

If it's so "massive" NV fixed it in like a week of it being reported. AMD...according to some here has known for a year or so. Yeah you aren't helping team red much haha.
 

VulgarDisplay

Diamond Member
Apr 3, 2009
6,193
2
76
A problem where a malicious attack can destroy your system, or a problem that you most likely can't even detect. You need to realize I'm not attacking nvidia or defending AMD. I'm just drawing attention to the absurd amounts of effort certain members of this forum devote to downplaying nvidia problems and blowing AMD issues out of proportion.

Now they will reply by saying that they can detect the problem and that's why they only buy nvidia cards. They will even say that this had been a problem for years. All while people who actually use AMD GPUs have been happily playing games with no problems.
 

f1sherman

Platinum Member
Apr 5, 2011
2,243
1
0
We can fix this... series of drivers that will address the issue... many things need to be rewritten...not happy with memory manager and scheduling


listen to the podcast

no, I'm not going to flip through my earlier posts where I predicted that AMD driver team will catch little sleep this year.
 

railven

Diamond Member
Mar 25, 2010
6,604
561
126
A problem where a malicious attack can destroy your system, or a problem that you most likely can't even detect. You need to realize I'm not attacking nvidia or defending AMD. I'm just drawing attention to the absurd amounts of effort certain members of this forum devote to downplaying nvidia problems and blowing AMD issues out of proportion.

Now they will reply by saying that they can detect the problem and that's why they only buy nvidia cards. They will even say that this had been a problem for years. All while people who actually use AMD GPUs have been happily playing games with no problems.

By pointing out the faults of one to make the faults of another appear lesser...yeah that is definitely the bold.

Hey, I'm an AMD user, and I notice the stutter. It is noticeable. Why should I settle for less? Why should I defend a company that is clearly not putting me, their customer, first? Sure, I guess I can understand they are hemorrhaging money, but in the end it's all about ME, not you, not AMD, and not anyone here (sounds callous, I know), and AMD is stuttering in the number 1 game I play.

Yet my nVidia rig hasn't been hacked. I know which issue affects me most right now.
 

VulgarDisplay

Diamond Member
Apr 3, 2009
6,193
2
76
By pointing out the faults of one to make the faults of another appear lesser...yeah that is definitely the bold.

Are you saying that's what I'm doing or what the posters I was calling out were doing? I'm confused...

Long story short, both sides have had drivers issues, and will continue to have driver issues. That's just how software/driver development works. I think the people with the obvious bias towards either brand choose to ignore this in order to advance their agenda.
 

railven

Diamond Member
Mar 25, 2010
6,604
561
126
Are you saying that's what I'm doing or what the posters I was calling out were doing? I'm confused...

What both sides are doing, and it's always hilarious watching both sides doing it. Considering this is an AMD thread...why is nVidia even a subject? Oh yeah!

Long story short, both sides have had drivers issues, and will continue to have driver issues. That's just how software/driver development works. I think the people with the obvious bias towards either brand choose to ignore this in order to advance their agenda.

And water is wet, but trying to say nVidia's water is always as wet as AMD's water, and vice versa, is stupid and just shows bias.
 

Keysplayr

Elite Member
Jan 16, 2003
21,209
50
91
Are you saying that's what I'm doing or what the posters I was calling out were doing? I'm confused...

Long story short, both sides have had drivers issues, and will continue to have driver issues. That's just how software/driver development works. I think the people with the obvious bias towards either brand choose to ignore this in order to advance their agenda.

How could you not know? He was surely implying that your posts, parts of which he had bolded for you, rang hypocritical.
There is no short story here. There is a long drawn out one though. And not nearly as cut and dry as you make it out to be. While Railven and I do not often see eye to eye, he is spot on about you.

EDIT: FIXED. Sorry, my smartphone went stupid on me. Autocorrect.
 
Last edited:

AdamK47

Lifer
Oct 9, 1999
15,324
2,930
126
How could you not know? He was surely implying that your posts, parts of which he had bolded for you, rang hypocritical.
There is no short story here. There is a long drawn out one though. Adams not a thing about it is as car savings day as you make it out to be. While Railven and I do not often see eye to eye, he is spot on about you.

I don't know what you just said.
 
sale-70-410-exam    | Exam-200-125-pdf    | we-sale-70-410-exam    | hot-sale-70-410-exam    | Latest-exam-700-603-Dumps    | Dumps-98-363-exams-date    | Certs-200-125-date    | Dumps-300-075-exams-date    | hot-sale-book-C8010-726-book    | Hot-Sale-200-310-Exam    | Exam-Description-200-310-dumps?    | hot-sale-book-200-125-book    | Latest-Updated-300-209-Exam    | Dumps-210-260-exams-date    | Download-200-125-Exam-PDF    | Exam-Description-300-101-dumps    | Certs-300-101-date    | Hot-Sale-300-075-Exam    | Latest-exam-200-125-Dumps    | Exam-Description-200-125-dumps    | Latest-Updated-300-075-Exam    | hot-sale-book-210-260-book    | Dumps-200-901-exams-date    | Certs-200-901-date    | Latest-exam-1Z0-062-Dumps    | Hot-Sale-1Z0-062-Exam    | Certs-CSSLP-date    | 100%-Pass-70-383-Exams    | Latest-JN0-360-real-exam-questions    | 100%-Pass-4A0-100-Real-Exam-Questions    | Dumps-300-135-exams-date    | Passed-200-105-Tech-Exams    | Latest-Updated-200-310-Exam    | Download-300-070-Exam-PDF    | Hot-Sale-JN0-360-Exam    | 100%-Pass-JN0-360-Exams    | 100%-Pass-JN0-360-Real-Exam-Questions    | Dumps-JN0-360-exams-date    | Exam-Description-1Z0-876-dumps    | Latest-exam-1Z0-876-Dumps    | Dumps-HPE0-Y53-exams-date    | 2017-Latest-HPE0-Y53-Exam    | 100%-Pass-HPE0-Y53-Real-Exam-Questions    | Pass-4A0-100-Exam    | Latest-4A0-100-Questions    | Dumps-98-365-exams-date    | 2017-Latest-98-365-Exam    | 100%-Pass-VCS-254-Exams    | 2017-Latest-VCS-273-Exam    | Dumps-200-355-exams-date    | 2017-Latest-300-320-Exam    | Pass-300-101-Exam    | 100%-Pass-300-115-Exams    |
http://www.portvapes.co.uk/    | http://www.portvapes.co.uk/    |