[Idea] think about: Frame Interpolation

[Borealis7]

Hi,
I'm in the market for a big flat screen TV and I saw all those 200/400/600Hz TVs from Samsung, LG, Sony etc'. I stood in the store with all the TVs mounted on a wall and stared at them to grasp the effect of this massive frame overdrive and I clearly noticed that while the input (blu ray movies) was still at 24 or 30 FPS the picture was moving a lot smoother on the monitors compared to non-overdriven ones.

due to budget constraints I am going to go with a slightly older Samsung F6100/6400 TV, if anyone has any advice against it, please let me know.

All this research into TV technology got me thinking, could the same frame interpolation be implemented in PC monitors for high FPS gaming?
I know the technology isn't perfect and that there are some artifacts and stuttering still, but the same principal should work with PCs.

what do you think? it could act as a post-processing effect like MPAA/FSAA except it would run on the monitor and not on the GPU.

<conspiracy>
BTW...how can we know that the GPU companies don't already do this on the GPU level?
</conspiracy>

 

[Flapdrol1337]

For interpolation you need the next frame, this means you have to build in a delay.

You'll have a smooth picture, but massive input lag, the whole reason people want high fps in the first place is to have lower input lag.

 

[Pottuvoi]

For games monitor is the wrong place to do this.
Right way is to implement proper system within the actual game.
http://advances.realtimerendering.com/s2012/bosch/Bidirectional%20Iterative%20Reprojection(Siggraph2012).pptx

http://research.microsoft.com/en-us/um/people/hoppe/proj/bireproj/

It's actually possible to reduce input lag.

 

[Borealis7]

For interpolation you need the next frame, this means you have to build in a delay.

You'll have a smooth picture, but massive input lag, the whole reason people want high fps in the first place is to have lower input lag.

the monitor already displays frames in delay, as it has a buffer and a clock which determines when to sample the buffer. that is why you get tearing without VSynch for example.

obviously it wouldn't do much if you get 120FPS in a game, but it could help 4K gaming with weaker hardware. perhaps make 20FPS feel like 30-35.

 

[CakeMonster]

I really like Frame Interpolation and I think its excellent for video, as opposed to many videophiles and purists who desperately want everything to stay at 24fps like in the 1920s. But for gaming, the delay is a huge issue. You can test it yourself by sending an input signal of 24 or 30 via HDMI from your PC to a TV that supports FI and turn the feature on and then off.

 

[BrightCandle]

Any extra calculation you put to an image is going to add latency. If you interpolate between two models or two images both have to be created to form the intermediate frame. Rather than showing the most up to date frame instead we display an intermediate one, yikes no thanks.

 

[Borealis7]

you would still display the most updated frame, as the interpolated frame will only be used to "smooth out" the motion. i believe the processors on these monitors are capable of calculating the interpolated frame in a reasonable time.
think of a pixel moving a distance of 10 pixels at 6FPS. you would see the pixel at positions 0,2,4,6,8,10 for the length of 166ms each frame (10 refresh cycles because monitor is 60hz)

with frame interpolation, you would see the pixel at 0,1,2,3,4,5,6,7,8,9,10 for an approximate length of 83ms (5 refresh cycles) - that is almost like having 12FPS.

i may be off with the numbers but you still get positions 0,2,4,6,8,10 at the same "frame start time" you would get them without interpolation.

without interpoation:
pos 0 @ 0 , pos 2 @ 166 , pos 4 @ 333 , pos 6 @ 500 , pos 8 @ 667 , pos 10 @ 833

with:
pos 0 @ 0, pos 1 @ 83, pos 2 @ 166, pos 3 @ 250, pos 4 @ 333, pos 5 @ 416, pos 6 @ 500, pos 7 @ 583, pos 8 @ 667, pos 9 @ 750, pos 10 @ 833.

the screen polls the buffer every 16ms or so, and the ideal time for calculating the interpolated frame would have to be less than that.

 

[Wall Street]

i may be off with the numbers but you still get positions 0,2,4,6,8,10 at the same "frame start time" you would get them without interpolation.

without interpoation:
pos 0 @ 0 , pos 2 @ 166 , pos 4 @ 333 , pos 6 @ 500 , pos 8 @ 667 , pos 10 @ 833

with:
pos 0 @ 0, pos 1 @ 83, pos 2 @ 166, pos 3 @ 250, pos 4 @ 333, pos 5 @ 416, pos 6 @ 500, pos 7 @ 583, pos 8 @ 667, pos 9 @ 750, pos 10 @ 833.

I don't think you get it. Here is the idea of why it adds lag:

How can you know that position 1 is 83 (average of 0 and 166) until after you know that position 1 is 166? How can you know that position 3 is 250 (average of 166 and 333) without knowing that position 4 is 333? There is lag because the interpolated frame is calculated from the data of the frame before it and the frame AFTER it. Meaning the frame after it has already been calculated.

TVs currently do this by showing position 1 after receiving the data for position 2 and showing position 3 after receiving the data for position 4, which is input lag. It is possible to make an algorithm that predicts future positions, but if you were to do so, you would have a "snap back" effect if the prediction doesn't match the actual data of what moved where when the next frame of data arrives.

 

[BrightCandle]

In order to interpolate two things you need them both. So that means the earliest moment you can put out the interpolated frame is when the second frame is already completed. At that point you can then start to process the frames and calculate the intermediate one. That is going to take a while, lets say 8-16ms which is around about what most scalars take as a minimum, but interpolation takes more calculation than just screen based anti aliasing but its a good starting point.

So everything is delayed by half the frame rate + the interpolation time. So at 120hz (8ms frames) you would be looking at 16ms of extra time in order to smooth it out. In a game like CS:Go which achieves about 40ms that is nearly 50% more latency, just for the purpose of producing some blurry intermediate frames. That is not worth it.

Eizo does have a 240hz monitor, but it doesn't work like this. What it does is use the flashed backlight technique to flash the same frame effectively twice, initially longer and then a shorter one after it, the end result is a clearer motion overall. They call it 240hz, its really 120hz with a special low persistence mode. But their technique critically doesn't introduce and extra latency. Monitors don't buffer whole frames, they did at one point but they really shouldn't anymore. They take the pixels as they come and display with minimal latency. If we were to look at practically the extra latency here its 2 frames + interpolation time compared to a current monitor, ie at 60hz its more like 40ms, which is more than a lot of games take in total from moue input to the screen.

Its a bad idea, a terrible one for gaming, it would be horrible to play games with that amount of additional latency. The extra 16ms or so of IPS is bad enough especially with the blur, the last thing we need is a doubling of that and even more blur in between. I like my images on screen to be readable, blur is a nasty horrible unrealistic effect and it needs to go away as soon as possible. This technique just adds more latency, more blur (no interpolation is perfect, I know I have written quite a few for real time ray tracers) and most importantly the extra frames it produces don't matter much. Most of us now agree its the latency reduction and the blur reduction not the additional frames that make the 144hz monitors awesome for gaming.

 

[zebrax2]

You can't display the latest frame then use the interpolated frame to smooth things out since the motion in interpolated frame happens before the latest frame.

 

[Borealis7]

but you do have some frames in advance in the monitor's buffer. unless the GPU is pegged at 100%, it is able to render some frames ahead of time.

 

[BrightCandle]

but you do have some frames in advance in the monitor's buffer. unless the GPU is pegged at 100%, it is able to render some frames ahead of time.

Ummm no, because that "time" hasn't happened yet. Unless the CPU has prepared the updated frame for the world updates, which depends on the users input and other player movements etc then the GPU can't render anything. If the GPU is rendering it then the game tick has happened, the GPU is doing its thing and the moment its ready it needs to output it.

Basic premise here is you can't predict magically ahead of time, you need a fully made thing to be working on. That adds latency, lots and lots of it.

 

[Rakehellion]

but you do have some frames in advance in the monitor's buffer. unless the GPU is pegged at 100%, it is able to render some frames ahead of time.

As everyone else has already said and you ignored, it's possible but slow.

Input lag for gaming is bad. Input lag for movies is fine.

 

[BrightCandle]

As everyone else has already said and you ignored, it's possible but slow.

Input lag for gaming is bad. Input lag for movies is fine.

With a movie you just the delay the audio 40ms and no one notices, infact the movie appears to play smoother so long as the cadence is all OK. But that extra latency is death for an interactive experience it just doesn't work when you control the image on screen you loose everything that matters and gain nothing of importance.