info on timedemo style of benchmark

[wolf2009]

I am looking for some info on time demo style of benchmark.
My question is : While replaying a timedemo, the game engine tries to play it as fast as possible. So no 2 graphic cards can complete the timedemo in the same time.
While benchmarking that timedemo with FRAPS, how correct is FRAPS with the min-max-avg scores ?

Also if you try to make a graph out of the FRAPS outputted excel spreadsheet with all the FPS at particular Frames ( since no two graphic cards will have same number of frames in the file since they did not complete the timedemo in the sametime), how do i compare two graphs ?

I'm talking about games such as COD4, Splinter Cell Chaos Theory and Farcry with HOC benchmark.

This will give you a better idea .

http://img.techpowerup.org/080807/Capture027.jpg

Here is what I'm talking about .

How do i stretch the 7600GT FPS . I cant put more FPS in there manually .

Note 7800GTX graph ends at frame 87 , while 7600GT continues . How do get them to match up .



They match up in timedemos like the in built ones in HL2 Lost coast, COH, WIC .

This is from HL2 Lost Coast , both end at frame 145 .
http://img.techpowerup.org/080807/Capture028.jpg

 

[wolf2009]

bump

 

[wolf2009]

bump

 

[wolf2009]

bump

 

[bob4432]

well if you use the same machine and just swap out the gpus, then compare them, you would use the fastest one. or if you are comparing to online sources, take into consideration the benchmark rig and then make a certain amount of adjustment for that.

and i have compared many benchmarks w/ timedemos w/ systems similiar to mine and although not identical in performance, i have seen some that may just be a frame or 2 a second faster, usually the rig.

is this what you are asking?

 

[wolf2009]

Thank you for replying but thats not what i was asking . This will give you a better idea .

http://img.techpowerup.org/080807/Capture027.jpg

Here is what I'm talking about .

How do i stretch the 7600GT FPS . I cant put more FPS in there manually .

Note 7800GTX graph ends at frame 87 , while 7600GT continues . How do get them to match up .



They match up in timedemos like the in built ones in HL2 Lost coast, COH, WIC .

This is from HL2 Lost Coast , both end at frame 145 .
http://img.techpowerup.org/080807/Capture028.jpg

 

[bob4432]

sorry, not really sure. a timedemo, like you said is x amount of frames, so it should be the same for both cards. what game was the one that was cut off from?

 

[wolf2009]

Originally posted by: bob4432
sorry, not really sure. a timedemo, like you said is x amount of frames, so it should be the same for both cards. what game was the one that was cut off from?

Farcry , COD4 , COD 2 this happens in .

In games like HL2 Lost Coast , its the same for all cards .

 

[bob4432]

i haven't used fraps in a while, but is it mearsuring for x amount of time or x amount of frames? is there an option so it will measure to say 2000 frames?

 

[wolf2009]

Originally posted by: bob4432
i haven't used fraps in a while, but is it mearsuring for x amount of time or x amount of frames? is there an option so it will measure to say 2000 frames?

That isn't the problem .

The problem is , timedemo on 7600 GT finishes in lets say X seconds , on a faster card like 7800GTX it finishes in X-15 seconds .

 

[apoppin]

So what do you want to do ... slow the faster card down or make the slower card faster?


The faster card *will* will run 400 frames [for example] faster than the slower card will run the same 400 frames; that is the purpose of comparing them in the first place.

Which laws of physics are you trying to change?

=D

 

[bob4432]

cant you graph based off frame vs time? sorry if i am not making much sense today i have had a tension and migraine all day....so i am a bit cloudy

 

[wolf2009]

Originally posted by: bob4432
cant you graph based off frame vs time? sorry if i am not making much sense today i have had a tension and migraine all day....so i am a bit cloudy

I am doing exactly that but the problem is , the timedemo ends faster on a faster card .

 

[bob4432]

but if you graphed based of of fps @ each frame, then it wouldn't matter because there would be no time in the graph. you would still go through 2000 frames regardless of how fast - the faster card would show say frame 236 @ 60fps where the slower card would show the same frame @ 25fps (just an example), so the graph should be the same length. or am i missing someting?

 

[DerekWilson]

anything that is actually a timedemo will always run exactly the same frames on every test setup. farcry, crysis, et:Qw, UT3, hl2 ... all these games will always run the same set of the same number of frames.

therefore, if your x axis is frame number both lines will always end in the same spot.

if your x axis is time, one line could end earlier if it finished faster -- which is, i assume, not what you want.

if you put frame number on x you'll be fine.

then you can either put frame time on y or instantaneous frame rate ... either way you'll be fine.

if you are NOT using a timedemo, then you CAN make the lines match up with time on the x axis if you run the test for the same amount of time (this is how most FRAPS tests work). in this case, however, you will be running more frames on the faster card -- obviously these frames will also not match up directly like they do in a timedemo... what you will see is how much more work the faster card can do in the same amount of time as opposed to how much more efficient a card is at rendering the same set of data ... making and understanding the distinction is important if you want to dive really deep into analysis.

hope that helps.

 

[Keysplayr]

dont use the timedemo command. just use "demo". And use Fraps to record the framerates.

 

[bob4432]

Originally posted by: keysplayr2003
dont use the timedemo command. just use "demo". And use Fraps to record the framerates.

in which games?

 

[Keysplayr]

Originally posted by: bob4432

Originally posted by: keysplayr2003
dont use the timedemo command. just use "demo". And use Fraps to record the framerates.

in which games?

The games usually have just a "demo" command as well as a timedemo command. Like CoD4.
If you use the timedemo command, it tends to run the demo as fast as possible. In other words, how fast can the GPU render a set amount of frames.

Some can do it in 58 seconds, others in 1:24 seconds.

But if you use the "demo" command instead, the game will run at normal speed. That is what I had to do when I benched CoD4.

You remember benching DoomIII at all? Remember how fast the timedemo ran according to how powerful the GPU's were? Same principle.

 

[wolf2009]

The games usually have just a "demo" command as well as a timedemo command. Like CoD4.
If you use the timedemo command, it tends to run the demo as fast as possible. In other words, how fast can the GPU render a set amount of frames.

Some can do it in 58 seconds, others in 1:24 seconds.

But if you use the "demo" command instead, the game will run at normal speed. That is what I had to do when I benched CoD4.

You remember benching DoomIII at all? Remember how fast the timedemo ran according to how powerful the GPU's were? Same principle.

Thank You for your comments Derek and Keys .

The problem is what Keys stated . On the slower card the demo finishes in say 1:30 min , on faster card it finishes in 1:00 min . So the graphs plotted do not end in the same place in excel .

Now as Keys said to use demo command instead of demo, it will run at normal speed . I'll try that . I usually use HOC benchmarks, anyway to force demo command in there ?

Also if its not possible to make the game run at normal speed, is there any way to stretch the shortened graph in excel ?

 

[bob4432]

wolf,

i am not trying to be a dick, but i just can't wrap my head around the fact that if you graph off the frame instead of the time, the graph will come out the same length, with the length being dictated by x # of frames. am i missing something?

 

[wolf2009]

No need to apologize man, comments are always appreciated , both positive and negative .

Alrite the thing which maybe you are not getting is this :

Lets say Farcry timedemo contains 4000 frames .

I run the timedemo, and press the benchmarking hotkey in FRAPS .

Now FRAPS will keep on recording the FPS for every second(after every 1 second) the timedemo runs and will output that number to Excel file.

Now on 7800GTX ( Faster Card ) the timedemo runs for a shorter time which is 89 seconds . So there is 89 entries or points in excel file where FPS are taken .

On 7600GT ( Slower Card ) the timedemo runs for a longer time which is 140 seconds . So there are 140 entries or points in excel file where FPS are taken .

When you plot the graph of FPS vs time, there a 140 points for the longer graph while there are 89 points on the time axis for the shorter graph .

If you do the same thing with the inbuilt timedemo of HL2 Lost Coast , both card have equal number of points .

Now my dilemma is , either to make both the cards finish in the same time in Farcry, or if i cant do that , stretch the shorter graph somehow to match the longer graph .( if that gives accurate representation of FPS )

 

[BFG10K]

The only way to make both cards finish at the same time is to run benchmarks that maintain the engine's tick.

The benchmark playback of some titles (e.g. Crysis) removes the engine tick and plays back as fast as possible, hence the differing durations.

The only solution for such titles is to use fraps to benchmark manual game runs or to benchmark game cinematics that are rendered in real-time by the engine like the rest of the game.

 

[wolf2009]

Originally posted by: BFG10K
The only way to make both cards finish at the same time is to run benchmarks that maintain the engine's tick.

The benchmark playback of some titles (e.g. Crysis) removes the engine tick and plays back as fast as possible, hence the differing durations.

The only solution for such titles is to use fraps to benchmark manual game runs or to benchmark game cinematics that are rendered in real-time by the engine like the rest of the game.

Good advice sir, hats off to you .

I think I'll resort to benching cinematics for such games. Or not make any graph comparisons, and only show the min max avg fps .

Is there anyway to force the game engine to kepp the engine tick intact ?

 

[BFG10K]

Originally posted by: wolf2009

Is there anyway to force the game engine to kepp the engine tick intact ?

Usually not, but depends entirely on the game and usually requires research and experimentation to find such methods.

Some games for example allow parameters to be passed to benchmarking commands which can change their behaviour.