Multi-GPU microstutter: Workaround solution

[boxleitnerb]

Introduction

SLI and CF are plagued by so called microstuttering. It is inherently bound to the AFR rendering technique that todays multi-GPU setups are using. Microstuttering occurs mostly and most notably when the GPU load is very high (aka deep GPU bottleneck). This usually is tied to low fps (of course, there are exceptions).

It all depends on the game engine, driver and probably many other different factors. However, it can (very roughly) be condensed in something like this:






From left to right (every box equals 10ms):

1. Perfect synchronization between CPU and GPUs. No component has to wait for the other, the frames are displayed evenly. 33fps. You could say there are both, a CPU and a GPU bottleneck - or no bottlenecks at all. I guess it's a matter of how you look at it.
2. CPU bottleneck. 1 GPU could render 1 frame in 60ms, 2 GPUs could render 2 frames in 60ms (parallel), effectively putting out 1 frame every 30ms if their work was synced perfectly. That would be again 33fps. However, the CPU is so slow that GPU 2 can only begin its work when GPU 1 is almost finished. The CPU limits the effective framerate to 20fps, 1 frame is displayed every 50ms
3. Heavy GPU bottleneck. The majority of the time it takes to prepare a frame for display, the GPUs are at work. Instead of beginning in the middle of GPU 1's cycle, GPU 2 can begin much earlier since it gets its data from the CPU so quickly. The timing of the frames displayed (in ms) is 10-50-10-50...it is uneven and perceived as microstutter.
4. Lighter GPU bottleneck. The CPU takes a bit longer (in relative terms) each frame, so the stutter is lessened
5. Even lighter GPU bottleneck. Still, the GPU is the dominating factor for performance here, but we're slowly approaching the situation described in 1. Less microstutter than in 4. or 3.

To verify this, I have benchmarked GTA IV and Mafia 2 in extremely taxing settings (ultrahigh resolution and SGSSAA respectively) with two CPU speeds. The diagrams show frametimes, not fps!

My system:
2600K@4300
GTX580 3GB SLI
16GB DDR3-1333
Win7x64



At 4300MHz we have a GPU bottleneck. At only 1600MHz, the bottleneck is slowly shifted towards the CPU, thus significantly lessening microstutter.


Workaround solution

Underclocking the CPU is obviously a stupid solution, no one would do that. There is a better way: The use of a fps limiter that caps fps to a certain fixed value. Programs that can be used to that extent are:

• Dxtory (a video capturing program, that includes a fps limiter).
  http://dxtory.com/v2-home-en.html
  

  Trial restrictions:
  The logo goes into the file generated by capture.
  The logo is displayed in the DirectShow output.
  A license purchase site is displayed, when a program is terminated.

• Bandicam (same as Dxtory, freeware)
• There is a fps limiter in the Nvidia drivers, starting with Forceware 280.26. Accessible via Nvidia Inspector. Later maybe via the control panel as well

Below I will present several examples, always with and without a fps cap. Again, this is a frametime measurement, not fps!

1. Crysis 2

There are less spikes when a limiter is used. Standard frametime deviation is reduced considerably.

WIP, more examples will be added soon.

 

[boxleitnerb]

2. Skyrim


3. Battlefield 3

I don't know what these large spikes are with the cap, but I can say it definitely feels alot smoother with the cap. Almost perfect. Smooth 360 degree turn around. Without the cap it feels quite jerky in the section displayed in the screenshot, because it is around 40fps compared to 50-60 if looking in the other directions.
Also interesting here: fps don't necessarily have to suffer. The displayed fps are roughly the same, but their temporal distribution is far better if capped. However, in areas where you could get more fps, you lose them, obviously. It's a tradeoff, but one that can be worthwhile.
For example: Always smooth 40fps vs. 50% jerky 40fps and 50% smooth 40+fps.

 

[BrightCandle]

What would you recommend as an FPS cap, just using Vsync and targetting performance for 60Hz or some piece of software you haven't mentioned?

 

[boxleitnerb]

The actual cap value depends on your fps. You should never fall below the cap. So if your lowest fps are in the low 30's range, 35 as cap would be a good start.
With VSync and 60Hz, thus 60fps, you are almost always CPU bottlenecked. Then, microstuttering are not really an issue.

 

[BrightCandle]

What software do I use to apply a cap?

 

[boxleitnerb]

Personally I use Dxtory. For starters I would recommend Bandicam because it's free. I had some problems with it, though, but I would still give it a shot.

 

[Porter_]

i tried this method a while back using bandicam and i didn't notice any difference in BF3. that's not to say it doesn't work, it just doesn't work for me in that game.

 

[JAG87]

There is a fps limiter in the Nvidia drivers, starting with Forceware 280.26. Accessible via Nvidia Inspector. Later maybe via the control panel as well

Thank you! Great info. I can get rid of Bandicam now

Anyway, the solution to microstuttering is just vsync. It's not necessary to limit FPS, but limiting FPS to 1 below the refresh rate reduces input lag, so they should be used in conjunction.

 

[imaheadcase]

I've only had sli for a few months gtx 580 and never noticed any micro studder. Is it because i'm not CPU limited at 4.8ghz for games? (bf3,batman AC, etc)

I think its even sillier to put a cap on fps when you invest so much into cards imo.

 

[boxleitnerb]

Thank you! Great info. I can get rid of Bandicam now

Anyway, the solution to microstuttering is just vsync. It's not necessary to limit FPS, but limiting FPS to 1 below the refresh rate reduces input lag, so they should be used in conjunction.

Vsync was active in all my examples. Vsync does not cure microstuttering in my experience. Also, any kind of limit does not work if you go below it. And I go below 59/60fps alot of the time. With a real limiter, I can set the value to anything I like. But I'm with you on the 1 below refresh rate. It's amazing that that works concerning the input lag.

I've only had sli for a few months gtx 580 and never noticed any micro studder. Is it because i'm not CPU limited at 4.8ghz for games? (bf3,batman AC, etc)

I think its even sillier to put a cap on fps when you invest so much into cards imo.

It's the other way around. And it is entirely possible that you're still a bit CPU limited, even with your fast processor...because you have double the graphics power now. It all depends on the settings you play at. I partly agree with you that the cap can be silly. I don't cap every game, only the ones where I fall a good deal below 60fps anyway. 60fps with vsync is perfect, but I like to use very high resolutions and SGSSAA and then even a 580 SLI shows strain.

 

[Grooveriding]

A framerate cap is an excellent way to keep a smoother experience. This is good info. I do find if you are not already maintaining a high frame rate that it is not much help.

If you want to try this without using an external program, some games allow you to cap FPS without using vsync.

In Battlfield 3 you can create a file called user.cfg in the main directory for the game and have this line in the file GameTime.MaxVariableFps 65

That is my setting, adjust the 65 accordingly to what you would use. I find setting it to 60 even does not give a smooth experience, but setting it slightly above my max refresh rate does.

 

[JAG87]

Vsync was active in all my examples. Vsync does not cure microstuttering in my experience. Also, any kind of limit does not work if you go below it. And I go below 59/60fps alot of the time. With a real limiter, I can set the value to anything I like. But I'm with you on the 1 below refresh rate. It's amazing that that works concerning the input lag.

That's not possible. What did you use to measure frame times? If vsync was on you would see a straight line (straight for uncapped assumes you render more than 60 fps). At least your 30 fps line should flat with vsync on, unless you dip below 30.

 

[boxleitnerb]

That's not possible. What did you use to measure frame times? If vsync was on you would see a straight line (straight for uncapped assumes you render more than 60 fps). At least your 30 fps line should flat with vsync on, unless you dip below 30.

Fraps. I moved the mouse around, so fps varied a bit.
Why only 30 and 60? Today's games use triple buffering or a comparable technique. Every fps value between 0 and 60 is possible with vsync on.

 

[soccerballtux]

intresting. this thread delivers.
I prefer the non-software solution, IE just massively underclocking the CPU.

 

[Jacky60]

Never had microstutter. Its a feature of low frame rates generally so if your fps is high then you don't get it. Get a fast cpu and fast gpu's x whatever number and hey presto! No microstutter.

 

[TakeNoPrisoners]

Never had microstutter. Its a feature of low frame rates generally so if your fps is high then you don't get it. Get a fast cpu and fast gpu's x whatever number and hey presto! No microstutter.

You have more then 2 GPU's which is known to reduce microstutter much more then anything else.

The power has some to do with it also.

 

[Lonbjerg]

Never had microstutter. Its a feature of low frame rates generally so if your fps is high then you don't get it. Get a fast cpu and fast gpu's x whatever number and hey presto! No microstutter.

False:
http://techreport.com/articles.x/21516
You might not percieve it, but that don't mean others won't...

 

[Tempered81]

afterburner has a built in frame rate limiter

 

[boxleitnerb]

afterburner has a built in frame rate limiter

Does that always work or only when capturing video? Fraps has one as well, with exactly this limitation.

 

[BFG10K]

An interesting thread, but you’re basically capping the framerate to your minimum value. That would be something I’d never do, especially not on an expensive SLI/CF setup. If I ever cap games, it’s at 120 FPS.

I’d rather do SFR or multi-GPU AA to fix micro-stutter.

 

[boxleitnerb]

An interesting thread, but you’re basically capping the framerate to your minimum value. That would be something I’d never do, especially not on an expensive SLI/CF setup. If I ever cap games, it’s at 120 FPS.

I’d rather do SFR or multi-GPU AA to fix micro-stutter.

Does SFR even work with modern engines? Nvidias multi-GPU AA has a bad image quality and is only based on MSAA/CSAA. AMD has a 16xSGSSAA CF-AA mode that is horribly slow. At least for me this is not a practical solution.

I agree that you have to find a scenario where capping makes sense. If I want to invest heavily in image quality and my fps are low anyway, I only lose a little performance but get rid of microstutters completely.

 

[Lonbjerg]

Does SFR even work with modern engines? Nvidias multi-GPU AA has a bad image quality and is only based on MSAA/CSAA. AMD has a 16xSGSSAA CF-AA mode that is horribly slow. At least for me this is not a practical solution.

I agree that you have to find a scenario where capping makes sense. If I want to invest heavily in image quality and my fps are low anyway, I only lose a little performance but get rid of microstutters completely.

AFAIK SFR died and AFR won...with microstutter as a fallout.

SFR didn't scale geometry, so it had it's own drawbacks too.

 

[BrightCandle]

Microstutter is a big big problem. If a game is pushing your GPU down to 30 fps and it stutters very badly its equivalent to having 15fps. But even when you are at 60 fps its like 30 fps. I think this is the reason I have never been happy with the frame rate my 5970 provides in some games, because even at 60 it isn't fluid enough. Yet I know in the past I've been happy with 60 fps, back before I started using xfire.

Personally I find that vsync often makes the problem worse. The increased lag and reduced frame rate is likely playing its part to make the problem that much more prevalent than it was when I was getting some tearing.

I was thinking maybe this is something AMD/NVidia could build into the drivers, but actually the problem is the game loop itself. So really this needs fixing in DX and the game so that its aware that the game is stuttering and delays the next frame a bit. Its simple enough logic to detect and fix it but its not something that is easy to do outside of the games programming.

 

[boxleitnerb]

All true!
But for the fixing part...how would the game know how much to delay the next frame(s)? It would have to look into the future, know where the player is looking at and what fps will ensue from this.

 

[BrightCandle]

All true!
But for the fixing part...how would the game know how much to delay the next frame(s)? It would have to look into the future, know where the player is looking at and what fps will ensue from this.

It can only use the historical measure. No game can predict how long the GPU will really take on the next frame, but at the same time the delay of 1 or 2 frames when actually all of a sudden the load has increased dramatically isn't going to make much difference and if you are only looking back for less than 100ms then it shouldn't really ever register to the human something bad just happened, we are talking about 8ms at 60 fps and 16ms at 30.

It is a trade off because occasionally you'll get a frame later than otherwise because the game/DX predicted better performance than it got, reducing slightly the average. But it still started before the other card had finished, at the predicted 50% point, so it will still achieve better than single card performance while eliminating stutter in the process. It does however require that the use of multiple frames in parallel rendering be exposed to DX and the game and hence it leaks the abstraction. But its a necessary tweak that could make the frame rate "quality" on multiple cards considerably better.