Civilization VI: Performance Analysis (Techpowerup)

[antihelten]

The point is, a single frame in a second worth of frames is pretty insignificant and why we look at the minimums over a second.

You cannot say before hand whether or not a single frame within a second is significant or not. That depends entirely upon how bad that frame is. For instance if you have one frame that takes 500 ms to render and 50 frames that takes 10 ms to render, then they would all fit within 1 second and the average FPS for that second would be 51 FPS, but you can bet your ass that the 500ms frame would result in it not feeling like 51 FPS.

I might also add, you were harping on me saying 40 FPS minimums was wrong, which is only true if you use your own personal having to use the single slowest frame. I was getting my data from the min/ave charts and the way most of us view it.

This is not my definition of minimum, it is quite literally the dictionary definition. A minimum value is the lowest value in a data set, not the lowest value after you've averaged the data points over some predefined time period.

And what min/ave charts are you looking at? The Civ 6 benchmark doesn't report min values, it only reports the average frame time value and the 99th percentile value (and a frame time distribution graph).

 

[bystander36]

Min. FPS, is the minimum per second. That's a definition of 1 second, but yes, you could choose your own time frame and it would still apply. I'm just saying most charts choose a time from from half to 1 second. Not a single frame.

 

[antihelten]

Min. FPS, is the minimum per second. That's a definition of 1 second, but yes, you could choose your own time frame and it would still apply. I'm just saying most charts choose a time from from half to 1 second. Not a single frame.

FPS doesn't have to be reported only once per second, that's just the way FRAPS chooses to do it (and as mentioned it's kind of a crappy way). Just like you don't have to drive you car for an hour to determine its speed in mph.

 

[bystander36]

I just think 1 frame is also a crappy way, and many people would agree, otherwise they wouldn't choose to report it in other ways. A single frame slightly out of time isn't going to stand out as much as many. Though a single really slow frame might. That's why both metrics are used.

 

[AtenRa]

Without green glasses that you are wearing, it looks like it runs good on everything. Also even on Nvidia we see reduced CPU overhead by going from DX11 to DX12. i5-2500k goes from 37/44 to 44/59 fps. Looks like nobody got gimped as usually with Gaming Evolved titles. Only Intel and AMD hardware consistently gain in new APIs. <wink face>

Highlight of the day:
470 is 55% faster than 1050Ti.

But but but 1050Ti doesnt need 6-pin .................................

Prices as of today from newegg,

Cheapest GTX 1050Ti 4GB = $140
Cheapest RX 470 4GB = $170 or $165 AR

You get 55% higher performance for only 21% higher price. Simple the RX 470 4GB is the best card in the $140-180 price market

 

[USER8000]

WTF?? My GTX960 4GB is beating a GTX1050TI 4GB in DX12 even though it is Maxwell. Finally!! The RX470 is getting close to 60FPS in the game. The RX480 and GTX1060 are only slightly faster too.

 

[linkgoron]

I'm sorry, the highlight is, by far, GTX 1060 beating RX 480 in a Gaming Evolved title at DX12 mode. Around 10% faster at 1440p/4K.
The results are not my fault.

What a highlight! 1 FPS difference between averages @ 1080p and 3 @ 1440p.

The real highlight is how horrible and disappointing the RX 460 card it is. Everything else between Nvidia and AMD is pretty competitive. (290x/290/470 edging barely the 970, Fury X and 980ti trading blows). Only the 980 isn't performing that well vs its competition (the Nano) @ 4k.

 

[Bacon1]

AMD seems to be still CPU bound @ ~84 max as Fury X -> Fury X CFX has no change in results and Fury X doesn't lose any FPS @ 1440 over 1080.

DX12 helps the 1080 break out of the CPU bound 1080p a lot though. Really shows that it can help a ton, even if it isn't perfectly implemented here (hence why most cards are slightly slower overall).

 

[bystander36]

Yeah, seems performance has a slight loss, unless CPU bound, which on that test, was only with the gtx 1080, but something happened with the GTX 780. It lost a significant amount of FPS.

 

[Dave2150]

DX12 should have been delayed a year or two to iron out these bugs. Can't expect widespread adoption when the API is worse than DX11 in a few games.

 

[bystander36]

DX12 should have been delayed a year or two to iron out these bugs. Can't expect widespread adoption when the API is worse than DX11 in a few games.

It's hard to say if it is worse from those benchmarks. The difference is 1-3% less, but if that comes with more consistent frame times, and lifts CPU bottlenecks on weaker systems, that would still be a benefit.

 

[AtenRa]

The most important part of DX-12 in this game is the CPU performance, Im sure if you play with a slower CPU and GPU the performance increase over DX-11 will be significant.

 

[Bacon1]

The most important part of DX-12 in this game is the CPU performance, Im sure if you play with a slower CPU and GPU the performance increase over DX-11 will be significant.

Yep the game is heavily CPU bound and DX12 will help with that, but it seems like the game is still heavily unoptimized for how low the perf is for the visuals and considering its turn based there shouldn't be tons of CPU usage all the time. I don't think they are culling objects properly.

But yes the DX12 version brings a 6300 upto i5 4670k. and that i5 4670k upto i7 6700. Thats a free CPU upgrade or two.

 

[FredWillson]

is dx 11 the only treat added to civ 6? Or we should wait for DLC`s, as usual?

 

[EightySix Four]

is dx 11 the only treat added to civ 6? Or we should wait for DLC`s, as usual?

I've been enjoying it but it definitely needs some patches/DLC to workout some of the new gameplay mechanics... the usual for a Civ game.

 

[antihelten]

I just think 1 frame is also a crappy way, and many people would agree, otherwise they wouldn't choose to report it in other ways. A single frame slightly out of time isn't going to stand out as much as many. Though a single really slow frame might. That's why both metrics are used.

I very much doubt many people if any at all would agree with you. The ones who report averaged values don't do so because they think it's superior, they do so either because it's easier (less time consuming), or because they don't know any better.

It's not about a single frame standing out (that would be what minimum FPS looks at), it's about a whole bunch of frames standing out (for instance if you're playing Civ 6 at an average FPS of 80, for 2 hours, then the 99th percentile would cover a total of 5760 frames and the 95th percentile would cover 28,800 frames). Obviously many frames being bad (i.e. having high frame times), is worse than a few frames being bad, but a few frames (say 1% or 5%) being bad can absolutely have a noticeably negative effect.

The simple fact is that averaging FPS over a second, like FRAPS does for it's FPS log, offers absolutely nothing of value compared to logging the FPS on a frame by frame basis (which is what FRAPS does for its frametime log). Obviously if your frametimes are already smooth and stutter free, then doing it on a frame by frame basis won't show anything you different from the FPS log, but the entire reason why you do it on a frame by frame basis is that games are not always smooth and stutter free, and measuring on a frame by frame basis is the only way to catch this.

Techreport did a nice article on why using smoothed FPS graphs are flawed, and why measuring on a frame by frame basis is strictly superior.

From the above Techreport article:

Averaging results over a single second can obscure some big and important performance differences between systems.

 

[bystander36]

Considering how many people use a FPS cap 1 below their refresh rate and use V-sync, goes to show a single frame in 60 doesn't disrupt most people, as long as that long frame isn't taking too much of that second (a single frame that is 16.7ms longer). A single frame that is 7ms longer than the rest of your FPS doesn't even show up on most peoples radar (your example going from 28 FPS from 40 FPS based on a single frame).

EDIT:
But a single frame that lasts 100ms probably would disrupt a user's experience. And having an average of 28 FPS over the course of a second probably would too. That's why both things are tested. Even in many FCAT reviews, they explain that small variances are showing good results, and large ones are bad. I think Tom's Hardware tried to place a value on when it was bad and not so bad.

 

[bystander36]

Ok, so you can shut up already. From a technical standpoint, the minimum is 28 FPS. From a practical stand point, the minimum was 40 FPS.

And god help us if reviewers ever started showing a single frame measurement as their minimum. So many people get so hung up on numbers, and prevents them from playing the game if something appears it could be a problem, even when there isn't an issue.

 

[antihelten]

Considering how many people use a FPS cap 1 below their refresh rate and use V-sync, goes to show a single frame in 60 doesn't disrupt most people, as long as that long frame isn't taking too much of that second (a single frame that is 16.7ms longer).

FPS caps generally work on a frame by frame basis (at least I don't know of any software that uses the smoothed FPS value to calculate the delay to meet the cap value) so not sure why you would think that supports your argument in anyway.

VSync is actually one of the most important reasons for why you want to measure FPS on a frame by frame basis. Say for instance that your game is constantly oscillating between 59 FPS and 61 FPS. Using a smoothed FPS value you would see your game running at a constant 60 FPS and would thus think that VSync would also run at 60 FPS, but in reality VSync will constantly switch between 30 and 60 FPS, thus producing horrible stuttering. This can only be caught if you measure FPS on a frame by frame basis.

A single frame that is 7ms longer than the rest of your FPS doesn't even show up on most peoples radar (your example going from 28 FPS from 40 FPS based on a single frame).

First of all the difference between 28 FPS and 40 FPS is 11 ms not 7 ms (35.7 ms vs. 25 ms, although it's worth noting that the final 99th percentile score was 34.349 ms corresponding to 29 FPS). And are you seriously claiming that people wouldn't notice going from 40 FPS down to 28 FPS, because I very much doubt anyone would agree with that, we're talking about a 30% dip in performance after all.

Obviously if it's only a single frame then it probably wont be too noticeable, but no one's talking about a single frame, we're talking about multiple frames dipping down to 28 FPS over the entire benchmark/game session (in the benchmark in question roughly 100 frames would be at 28 FPS or worse).

EDIT:
But a single frame that lasts 100ms probably would disrupt a user's experience. And having an average of 28 FPS over the course of a second probably would too. That's why both things are tested.

Testing on a frame by frame basis tests both of these things, so there is no need for calculating a smoothed FPS value.

As I said before testing on a frame by frame basis is strictly superior to smoothed FPS values, since it achieves everything the smoothed value does and more. It will show long lasting dips down to 28 FPS, and also short dips down to 10 FPS (100 ms).

Even in many FCAT reviews, they explain that small variances are showing good results, and large ones are bad. I think Tom's Hardware tried to place a value on when it was bad and not so bad.

Obviously some dips will be worse than other I have never said otherwise. But just because some dips wont noticeably affect smoothness doesn't mean that we shouldn't test for dips altogether, which is what you do when you look at smoothed FPS values.

And it's not like you somehow can't tell a small insignificant dip from a large significant dip, which is of course also why sites like THG can even categorize them in the first place. So there is never any harm in testing on a frame by frame basis.

Ok, so you can shut up already.

No ones forcing you to have a discussion on this discussion forum. If it's too much for you, then just walk away from the keyboard.

From a technical standpoint, the minimum is 28 FPS. From a practical stand point, the minimum was 40 FPS.

How in the world are you determining that the minimum was 40 FPS from "a practical stand point"?

The Civ benchmark reports the average frame rate, the 99th percentile frame rate and a frame rate graph, so where in the world are you pulling this 40 FPS number from, and why is that number the "practical minimum"?

And god help us if reviewers ever started showing a single frame measurement as their minimum.

Reviewers are already reporting a single frame measurement as their minimum, since that how many (if not most) in game benchmarks measure it.

But again I don't see how that's relevant here, I never said that reviewers should report minimums (it's a largely useless metric), but should instead use 99th or 95th percentile scores (which is exactly what the Civ 6 benchmark does).

So many people get so hung up on numbers, and prevents them from playing the game if something appears it could be a problem, even when there isn't an issue.

Seriously just read the Techreport article I linked. It seems obvious that you don't understand the problems with using smoothed FPS values instead of the raw frame rate times.

And this isn't just about Civ 6 (obviously an isometric turn-based game isn't as sensitive to bad frame rates as say an FPS), but is a more general issue. There really isn't any scenario where using smoothed FPS will give you a more accurate picture of performance than what using non smoothed values (i.e. individual frame times) will, at best it won't be any worse (when your frame rate is very stable and smooth).

 

[bystander36]

What ever you say man.

I will only mention one thing, as I know you aren't so hard headed (please don't prove me wrong) as to not understand.

Using a 59 FPS cap with V-sync on, assuming triple buffering is used, means you will get 59 FPS, one frame of which will last 33ms, instead of 17ms. It works this way, as many, many people use it, and that is what they get.

They don't noticed that one long frame. 1 frame is not nearly same as 60 frames in a row.

 

[antihelten]

I know you aren't so hard headed (please don't prove me wrong) as to not understand.

Oh the irony.

Using a 59 FPS cap with V-sync on, assuming triple buffering is used, means you will get 59 FPS, one frame of which will last 33ms, instead of 17ms. It works this way, as many, many people use it, and that is what they get.

They don't noticed that one long frame. 1 frame is not nearly same as 60 frames in a row.

Using a 59 FPS cap with VSync on and triple buffering will result in 59FPS with one frame per second at 33.3 ms and the rest at 16.7, if and only if your setup is capable of rendering all frames at 16.7 ms in the first place. The only way to know for certain whether or not your system is capable of this, is to measure the frame time on a frame by frame basis.

If you setup is not capable of rendering all frames in 16.7 ms or less (which could be the case even if you're smoothed FPS value is 60 or more), then VSync with FPS cap plus triple buffering can quite easily have more than just 1 frame at 33.3 ms per second (how many more obviously depends upon how slow your setup is).

 

[bystander36]

Oh the irony.



Using a 59 FPS cap with VSync on and triple buffering will result in 59FPS with one frame per second at 33.3 ms and the rest at 16.7, if and only if your setup is capable of rendering all frames at 16.7 ms in the first place. The only way to know for certain whether or not your system is capable of this, is to measure the frame time on a frame by frame basis.

If you setup is not capable of rendering all frames in 16.7 ms or less (which could be the case even if you're smoothed FPS value is 60 or more), then VSync with FPS cap plus triple buffering can quite easily have more than just 1 frame at 33.3 ms per second (how many more obviously depends upon how slow your setup is).

Sorry, but that is not true. They just need to average at about 17ms (what ever 59 FPS requires). Triple buffering will make the first frame double up, just barely missing its window and allowing it to finish that frame for the next vblank window almost immediately, allowing time to finish most of another frame by then. By the time the 59th frame comes along, it'll be just barely able to squeeze it in for vblanking mode. The key here is triple buffering allows frames to be rendered ahead of time. And if fluctuations happen enough, you may end up losing another FPS, but they'd have to be pretty large fluctuations.

Not only does the math allow it, but the proof is in the fact people do this all the time successfully. I have tested this too, but with a 120hz monitor, I don't feel the need.

EDIT: And you are missing the point here. 1 frame in 60 being even double, does gives a far different experience than 60 frames that are long. This is why a minimum FPS over a period of time is used, as a small blip is near meaningless.

 

[antihelten]

Sorry, but that is not true. They just need to average at about 17ms (what ever 59 FPS requires). Triple buffering will make the first frame double up and get a big head start for the next frame. By the time the 59th frame comes along, it'll be just barely able to squeeze it in for vblanking mode. The key here is triple buffering allows frames to be rendered ahead of time. And if fluctuations happen enough, you may end up losing another FPS, but they'd have to be pretty large fluctuations.

Now only does the math allow it, but the proof is in the fact people do this all the time successfully. I have tested this too, but with a 120hz monitor, I don't feel the need.

Triple buffering may very well allow you to render ahead, but that doesn't help you when the GPU can barely keep up with the targeted frame rate as is. And no the fluctuations do not have to be large to lose FPS when using VSync, since you only barely have to miss the refresh window for it to dip down to 30 FPS.

Obviously if you have a GPU that is capable of running at above 60 FPS for all frames, then you will get exactly what you described, which is 1 frame out of 60 being displayed after 33.3 ms, and the rest after 16.7 ms.

This is due to the fact that the first frame misses the initial monitor refresh at 16.7 ms (since it takes 16.95 ms to render with a 59 FPS cap), and thus gets rendered at the second monitor refresh at 33.3 ms. However thanks to triple buffering this delay can effectively be used as a time buffer (or head start as you put it) to render the next frame, thus ensuring that it shows up on time at the third refresh interval (at 50 ms). But since each frame takes about 0.3 ms longer than the refresh rate to render, you will slowly but surely eat away at this time buffer until you have a dip down to 30 FPS after 60 frames and thereby reestablish the time buffer.

The problem however is if not all of your frames finish in 16.95 ms, because then your buffer time will disappear faster and you will thus have more dips down to 30 FPS. The only way to determine if some of your frames take longer than 16.95 ms to render is to measure the frame times on a frame by frame basis.

And of course this is all before we even get into the latency issues triple buffering and render ahead has, which is why most people prefer to not use it (Nvidia's new Fast Sync is of course an exception to this).

EDIT: And you are missing the point here. 1 frame in 60 being even double, does gives a far different experience than 60 frames that are long. This is why a minimum FPS over a period of time is used, as a small blip is near meaningless.

I think you're the one missing the point here, seeing as I never said that 1 frame in 60 taking twice the time is the same as 60 frames taking longer than average. And again it's not about a single blip, it's about multiple blips.

 

[bystander36]

Triple buffering, which lets the system get up to 16.6ms ahead, will smooth out most variances, with a rare added lost frame.

The point being, an occasional long frame isn't a big deal, and why review sites, and benchmark programs don't use the single frame to determine the minimum FPS of a game.

That is my last response which I'm sure you are set in your idea anyway. We can agree to disagree. Which does leave me a little puzzled, as if I recall, you have used a lot of SLI or CF in your systems, which causes some problems with that occasional long frame.

I guess this would be important to add:

I think you're the one missing the point here, seeing as I never said that 1 frame in 60 taking twice the time is the same as 60 frames taking longer than average. And again it's not about a single blip, it's about multiple blips.

If it's multiple blips in a second, that would drive the min FPS down with the typical min FPS system, so it still shows up with the "smoothing" out method. If it's spaced out a few seconds a part, or even a second a part, it's just not going to matter much unless it was a really long frame, which again will drive the smoothed out minimum FPS.

 

[antihelten]

Triple buffering, which lets the system get up to 16.6ms ahead, will smooth out most variances, with a rare added lost frame.

Triple buffering (again assuming a FIFO render ahead queue here) will only smooth out most variances if you don't combine it with an FPS cap, but that is exactly what we're doing here.

With an 59 FPS cap, all of your frames will be slower than the refresh rate. If your GPU is fast enough to render all frames at the 59 FPS cap, then it is true that dips down to 30 FPS will be rare (once every 60 frames), but if your GPU is not fast enough to render all frames at the 59 FPS cap, and have multiple frames being rendered slower than that then you will experience more dips down to 30 FPS when using VSync. How many dips you experience will of course depend upon how many frames are rendered slower than 59 FPS, and how much slower they are rendered, both of which can only be determined by measuring you frame rate on a frame by frame basis.

The point being, an occasional long frame isn't a big deal, and why review sites, and benchmark programs don't use the single frame to determine the minimum FPS of a game.

First of all I don't know why you're so hung up on minimum FPS. Minimum FPS is a crappy measurement regardless of whether it is determined from a single frame or from 1 seconds worth of averaged values.

What were talking about here is 99th percentile scores (which is what the Civ 6 benchmark uses).

With that being said though tons of sites and benchmark programs use single frames to determine the minimum FPS of games. Just because FRAPS smoothes its FPS log over 1 second, doesn't automatically mean that this is the same for all other games/benchmarks.

That is my last response which I'm sure you are set in your idea anyway. We can agree to disagree. Which does leave me a little puzzled, as if I recall, you have used a lot of SLI or CF in your systems, which causes some problems with that occasional long frame.

Using mGPU systems is actually one of the things that drove people to start looking at frame graphs on a frame by frame basis in the first place, since a lot of people were reporting that the gameplay didn't feel as smooth as the average FPS value would otherwise indicate.

The issue here was of course microstutter, which happens on a frame by frame basis, and is thus obviously only apparent when measuring on a frame by frame basis, instead of using smoothed values.

Going back to the Techreport article I linked you earlier, a mGPU setup when measured on a frame by frame basis looks like this (this was before AMD introduced improved framepacing):

If you had instead measured this using the smoothed FPS values it would have looked similar to this (just ignore the values on the axes here):

Which obviously means that the smoothed FPS value would have been incapable of picking up the microstutter.

If it's multiple blips in a second, that would drive the min FPS down with the typical min FPS system, so it still shows up with the "smoothing" out method. If it's spaced out a few seconds a part, or even a second a part, it's just not going to matter much unless it was a really long frame, which again will drive the smoothed out minimum FPS.

Obviously fluctuations in individual frame times affects the smoothed FPS value (and thus the minimum smoothed FPS value, although as noted above, minimum values are largely pointless), but that doesn't mean that the smoothed value will suddenly contain all of the information that the individual frames does.

Imagine having a game that constantly switches between rendering a frame 20 FPS and then at 100 FPS (obviously no games would be this extreme, but this is just a hypothetical example). Obviously such a game would be completely unplayable due to stutter, and yet looking at the smoothed FPS value from FRAPS you would see a constant 60 FPS value, which would indicate that this game should be perfectly playable. So clearly in this case the smoothed value is useless. If you used a frame by frame measurement instead then the extreme stutter would be perfectly clear.

I can understand if you don't want to carry on this discussion anymore, especially considering that every time you tried to bring up an example of where smoothed FPS values were better than raw frame by frame values, you managed to do the exact opposite and actually given examples were frame by frame measurements are superior. Frame by frame measurements are much better predictors for frame rate behavior under VSync scenarios compared to smoothed values. Likewise frame by frame measurements are also clearly superior for predicting behavior in mGPU scenarios.