Discussion AMD Gaming Super Resolution GSR

[DisEnchantment]

New Patent came up today for AMD's FSR

Patent Public Search | USPTO

appft.uspto.gov

https://www.freepatentsonline.com/y2021/0150669.html

20210150669 GAMING SUPER RESOLUTION

Abstract
A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display

[0008] Conventional super-resolution techniques include a variety of conventional neural network architectures which perform super-resolution by upscaling images using linear functions. These linear functions do not, however, utilize the advantages of other types of information (e.g., non-linear information), which typically results in blurry and/or corrupted images. In addition, conventional neural network architectures are generalizable and trained to operate without significant knowledge of an immediate problem. Other conventional super-resolution techniques use deep learning approaches. The deep learning techniques do not, however, incorporate important aspects of the original image, resulting in lost color and lost detail information.

[0009] The present application provides devices and methods for efficiently super-resolving an image, which preserves the original information of the image while upscaling the image and improving fidelity. The devices and methods utilize linear and non-linear up-sampling in a wholly learned environment.

[0010] The devices and methods include a gaming super resolution (GSR) network architecture which efficiently super resolves images in a convolutional and generalizable manner. The GSR architecture employs image condensation and a combination of linear and nonlinear operations to accelerate the process to gaming viable levels. GSR renders images at a low quality scale to create high quality image approximations and achieve high framerates. High quality reference images are approximated by applying a specific configuration of convolutional layers and activation functions to a low quality reference image. The GSR network approximates more generalized problems more accurately and efficiently than conventional super resolution techniques by training the weights of the convolutional layers with a corpus of images.

[0011] A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display.

[0012] A processing device is provided which includes memory and a processor configured to receive an input image having a first resolution. The processor is also configured to generate a plurality of non-linear down-sampled versions of the input image via a non-linear upscaling network and generate one or more linear down-sampled versions of the input image via a linear upscaling network. The processor is also configured to combine the non-linear down-sampled versions and the one or more linear down-sampled versions to provide a plurality of combined down-sampled versions. The processor is also configured to convert the combined down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution by assigning, to each of a plurality of pixel blocks of the output image, a co-located pixel in each of the combined down-sampled versions and provide the output image for display.

[0013] A super resolution processing method is provided which improves processing performance. The method includes receiving an input image having a first resolution, generating linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generating non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The method also includes converting the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and providing the output image for display.

It uses Inferencing for upscaling. As will all ML models, how you assemble the layers, what kind of parameters you choose, which activation functions you choose etc, matters a lot, and the difference could be night and day in accuracy, performance and memory

 

[GodisanAtheist]

Interesting!

So If I am understanding this correctly, AMD's implementation is still centered around a trained neural network, it will simply use additional "non-linear" data in its upscaling techniques and the algo will be run entirely on shaders since AMD's cards don't have tensor cores sitting around.

I wonder if this technique will actually end up benefitting NV even more, thanks to the vast but inefficiently used shader array on their cards.

Hell, now I'm starting to wonder if the shader doubling was so they could ditch tensor cores on their next arch design, bring DLSS 3.0 over to shader processing, and gain that space back...

 

[coercitiv]

Hell, now I'm starting to wonder if the shader doubling was so they could ditch tensor cores on their next arch design, bring DLSS 3.0 over to shader processing, and gain that space back...

How are you able to asses the compute requirements of DLSS based on the compute requirements of another image reconstruction technique?

 

[GodisanAtheist]

How are you able to asses the compute requirements of DLSS based on the compute requirements of another image reconstruction technique?

- Magic

 

[Gideon]

As people have mentioned elsewhere, AMD filed for this patent in the end of 2019, roughly 2-3 months before DLSS 2.0 was even announced. I'm quite convinced the actual implementation of GSR has evolved a lot since then. Possibly to the point of having nothing anything in common with this patent.

Somebody on reddit also mentioned that this seems to require per-game training and all-in-all looks more similar to DLSS 1.

 

[Shamrock]

RedGamingTech said today, that FSR will almost certainly be able to be used at least back to Vega, and maybe even Polaris!

 

[Gideon]

RedGamingTech said today, that FSR will almost certainly be able to be used at least back to Vega, and maybe even Polaris!

[RANT warning]
I hate it when people attribute to "sources" and "leaks" stuff that AMD reps themselves has said in interviews.

There was a long interview last year (I think it was this one) where both Scott Herkelman and Frank Azor essentially said that while they initially they something that would only work on their cards, game devs really wanted something platform-agnostic (a la Freesync) they could only implement once so they decided to go that way. (and if I remember correctly Frank Azor even confirmed it in his interview with the very same RedGamingTech not 2 weeks ago)

The gamedevs can only implement it "once" if it works on Nvidia. AMD also wants it to work on consoles as well, so it must be API agnostic as well. All of this means that it 99% works on Vega which has rapid-packed math. The only reason it might not work on polaris is if the performance hit (running some 4bit calculations) is too much.
[/RANT]

So essentially getting info from "top secret sources" that the top brass of the company confirmed months ago.

At least RedGamingTech actually seems to be quite decent compared to the disasters of MLID or Adored.

 

[DisEnchantment]

Somebody on reddit also mentioned that this seems to require per-game training and all-in-all looks more similar to DLSS 1.

It may be.
But, from a pure ML modelling perspective, it is hard to tell how the training is done.
The description in the patent is so generic that you can just say we assembled the network in some special way and select some specific activation function and achieve better results. Thats what the patent is about which basically means anything from ML perspective.
When doing ML modelling for a specific problem, it is not uncommon that you eventually figure out a more deterministic way to represent the problem as an actual mathematical model and this happens regularly in highly automated driving.

So essentially getting info from "top secret sources" that the top brass of the company confirmed months ago.

Video Thumb is actually Exclusive info from USPTO

 

[Dribble]

The patent looks very like DLSS, I suspect because AMD want some protection if they make a DLSS variant in the future (i.e. something that needs the AMD equivalent of tensor cores). Doesn't mean that's what FSR on the current cards will be.

 

[VirtualLarry]

So If I am understanding this correctly, AMD's implementation is still centered around a trained neural network, it will simply use additional "non-linear" data in its upscaling techniques and the algo will be run entirely on shaders since AMD's cards don't have tensor cores sitting around.

This... talk of "non-linear" scaling... intrigues me.

Remember back when, CD-ROM was a new thing? Well, there was an early title, called "Compton's Multi-Media Encyclopedia" on CD-ROM. In order to fit the "Multi-Media" onto the disc, they used compression. A fairly unique type of it, it's called "Fractal Compression" (look it up, it's patented, or was). The really unique thing about "Fractal Compression", was that, once the original image was compressed that way, the resultant output when de-compressed, could easily and probably nearly infinitely scaled-up in resolution. It was really revolutionary technology, if a bit obscure.

I really wonder if AMD is now essentially doing those steps IN REAL-TIME on the shader processors. That would be, to me, quite frankly rather amazing, if true. It was a fairly compute-intensive process, as I understand it, to fractally-compress an image file.

 

[Stuka87]

This... talk of "non-linear" scaling... intrigues me.

Remember back when, CD-ROM was a new thing? Well, there was an early title, called "Compton's Multi-Media Encyclopedia" on CD-ROM. In order to fit the "Multi-Media" onto the disc, they used compression. A fairly unique type of it, it's called "Fractal Compression" (look it up, it's patented, or was). The really unique thing about "Fractal Compression", was that, once the original image was compressed that way, the resultant output when de-compressed, could easily and probably nearly infinitely scaled-up in resolution. It was really revolutionary technology, if a bit obscure.

I really wonder if AMD is now essentially doing those steps IN REAL-TIME on the shader processors. That would be, to me, quite frankly rather amazing, if true. It was a fairly compute-intensive process, as I understand it, to fractally-compress an image file.

I might still have that sitting around here someplace. It helped in many high school reports!

The patent looks very like DLSS, I suspect because AMD want some protection if they make a DLSS variant in the future (i.e. something that needs the AMD equivalent of tensor cores). Doesn't mean that's what FSR on the current cards will be.

The chances of AMD ever going with proprietary hardware are less than one of us being struck by lightning. There is no need for custom hardware for upscaling (we have been through this already), and there is zero reason they would come up with something that would not work on the consoles.

 

[Panino Manino]

We won't need to wait much longer, Farcry 6 will finally be properly revealed soon and I bet that this will be the first game to ship with AMD's feature.

 

[Dribble]

The chances of AMD ever going with proprietary hardware are less than one of us being struck by lightning. There is no need for custom hardware for upscaling (we have been through this already), and there is zero reason they would come up with something that would not work on the consoles.

definition of proprietary:
relating to an owner or ownership.
"the company has a proprietary right to the property"

That would be everything in an AMD gpu being as it's all patented.

The whole gpu is custom (which for some reason you called proprietary) hardware - it's all specialised to do certain tasks (draw, texture, etc). This is no different - having hardware specialised for machine learning tasks such as image manipulation is to be expected because it's the most efficient - even phones have it.

 

[Gideon]

While not AMD's solution Unreal Engine's built-in Temporal Super Resolution looks quite cool:

Unreal Engine 5.0 Release Notes | Unreal Engine 5.0 Documentation

And it looks to be optimized for RDNA2 as well

before. To meet these demands, we've written a Temporal Super Resolution algorithm from scratch that replaces UE4's TemporalAA for higher-end platforms.

Temporal Super Resolution has the following properties:

• Output approaching the quality of native 4k renders at input resolutions as low as 1080p, allowing for both higher framerates and better rendering fidelity.
• Less ghosting against high-frequency backgrounds.
• Reduced flickering on geometry with high complexity.
• Runs on any Shader Model 5 capable hardware: D3D11, D3D12, Vulkan, PS5, XSX. Metal coming soon.
• Shaders specifically optimized for PS5's and XSX's GPU architecture.

While It's obviously a WIP It still looks significantly better than UE's out of the box TAA.

Unfortunately the comparison images are rather useless as Nanite is resolving geometry to fit a 1080p image. Still I'll post them here just for reference:

Spoiler: 4K Native

Spoiler: 1080p TSR (with nanite vastly reducing geometry detail

All in all it still looks WAY better than UE4's TAA IMO. Hopefully I get the time to make some more detailed images in the engine at some point ...

 

[Kepler_L2]

TAA and DLSS are good for static images but have issues with motion. We won't know how TSR compares to either until games built with UE5 come out.

Also FSR is likely to be better than all 3 solutions at motion clarity but worse at static images. Will be interesting to see what people prefer.

 

[Gideon]

TAA and DLSS are good for static images but have issues with motion. We won't know how TSR compares to either until games built with UE5 come out.

Also FSR is likely to be better than all 3 solutions at motion clarity but worse at static images. Will be interesting to see what people prefer.

That is true for FSR. UE5 TSR can be checked out right now (and it's quite different from UE4 TAA), though it's WIP. Just download the Early Access Unreal engine and load the Valley of the Ancient demo. TSR is enabled by default, but you can easily disable it. From there on out just capture your screen and do comparisons.


I'll try to get to it myself some day when I'm not stomped by work.

EDIT:

And while the terrible JPG resolution and Nanite trying to optimze a polygon per pixel (meaning the upscaled image has near 4x less geometry detail) skews things, It's still pretty evident the new algorithm (TSR) is way better than the older one (TAA).

Here is a zoomed in crop of the screenshots shared above. I tried to focus on an area that should use similar mesh-quality (stuff further away). Unfortunately JPG artifacts totally murder the quality, but it isn't completely useless

TSR (1080p -> 4K)

Native 4K (20 FPS):

TSR upscaled from 1080p (44 FPS):

This looks pretty decent for a 2x upscale.

TAA (70% -> 100% equivalent to 1512p -> 4K upscaling)

70%

100%

Look at the poor detail resolving of the fence. Actually the entire image looks blurred in the fullscreen comparison despite the source resolution being WAY higher.

TSR should also be better in motion as they promise:

• Less ghosting against high-frequency backgrounds.
• Reduced flickering on geometry with high complexity.

I really hope to see a good Native vs TAA, TSR and DLSS (as Nvidia's plugin seems to be updated to work with UE5) )comparison in motion, preferrably side-by-side.
Nanite is configurable to produce the same level of detail and maps can be used where it's off.

I wish I had the time to do it, maybe someday

 

[Gideon]

So I got around to play a little with the Unreal Engine 5 Early Access and the Ancient Valley demo

I tired out the new Unreal Engine Temporal Super Resolution (TSR) upsampling method (not to be confused with GSR) and compared it to it's original TAA upsampling and the Bilinear (naïve) method.

If It's not FidelityFX GSR why bother?

• Because Unreal Engine is very widely used and it's TAA is the defacto standard for last-gen UE4 games using upsampling. The same will probably be true for TSR for next-gen UE5 games.
• TSR is a lightweight algorithm (no Tensor units required) with shaders specifically optimized for PS5’s and XSX’s GPU architecture(source). It's a very good baseline for what GSR can and needs to accomplish.
• It has some properties required for good upscaling that TAA doesn't have and GSR needs to have (temporal stability, less ghosting via motion-vector use) Here's what Epic has to say about it:

• Output approaching the quality of native 4k renders at input resolutions as low as 1080p, allowing for both higher framerates and better rendering fidelity.
• Less ghosting against high-frequency backgrounds.
• Reduced flickering on geometry with high complexity.
• Runs on any Shader Model 5 capable hardware: D3D11, D3D12, Vulkan, PS5, XSX. Metal coming soon.
• Shaders specifically optimized for PS5's and XSX's GPU architecture.

Test Setup
All of the tests were run at 1440p on my home rig: Radeon 6800XT, Ryzen 3700X, 32GB of DRR4 @ 3600CL14

Caveats:
UE5 is still in early beta, and so are most of the algorithms used. Expect Nanite, Lumen and TSR to improve. There are still bugs particularily around character (just look this native 4K video around the character's hair). That's Lumen only. Once you add TSR at times you get additional ghosting on the character(well visible on my video with 480p -> 1440p).

Still Image Comparison:

Spoiler: 1440p native

Spoiler: 720p TAA

Spoiler: 720p TSR

Spoiler: 1080p TAA

Spoiler: 1080p TSR

Here is a collage (I added a downsampled 2880p version for good measure, to see if it makes any major difference to geometry due to how Nanite operates)

Here is the entire image gallery in PNG format including some easter eggs (e.g. 288p retro campfire). The ones in the collage above are behind the "load more" button:


Video Comparison:

Youtube (a bit blurrier but with chapters):

Vimeo:

Here is a list of resolutions and methods I ran through* the console commands and approximate performance I got:

Resolution / Upscaling method	Console command	Average FPS (approx near the starting rock)
1440p Native	r.TemporalAA.Upsampling 0 r.ScreenPercentage 100	44 FPS
720p Bilinear (naïve method)	r.TemporalAA.Upsampling 0 r.ScreenPercentage 50	85 FPS
720p TAA	r.TemporalAA.Upsampling 1 r.TemporalAA.Algorithm 0 r.ScreenPercentage 50	81 FPS
720p TSR	r.TemporalAA.Upsampling 1 r.TemporalAA.Algorithm 1 r.ScreenPercentage 50	79 FPS
1080p Bilinear	r.TemporalAA.Upsampling 0 r.ScreenPercentage 75	61 FPS
1080p TAA	r.TemporalAA.Upsampling 1 r.TemporalAA.Algorithm 0 r.ScreenPercentage 75	61 FPS
1080p TSR	r.TemporalAA.Upsampling 1 r.TemporalAA.Algorithm 1 r.ScreenPercentage 75	58 FPS

* In the video I accidentally skipped vanilla 1080p (Bilinear) and played around with 480p + TSR in the end (r.ScreenPecentage 30)


Conclusion:
Overall TSR looks really really good IMO considering the circumstances.

In actual gameplay in motion it fixes most of the problems I have with TAA upsampling (e.g. why I can't stand it in Cyberpunk below 90%)

Positives:
+ Very small performance hit
+ No exotic hardware requirements (works even with Vega)
+ Excellent temporal stability and no flickering on faraway objects with complex geometry
+ Looks considerably better than TAA, particularily on the edges of faraway objects. 720p TSR sometimes even beats 1080p TAA

Negatives:
+ Still bugs and artifacts on moving objects/charactes
+ Nanite optimizes geometry to show 1 polygon per pixel. This means with upsampling scene geometry is considerably simplified (essentially 4x for 50% upscale)

Unfortunately I don't have a 4K screen, but from 1080p up the upscaling seems excellent (so 50% Screen Percentage for 4K should be very decent).

Even if AMD can't beat TSR with it's GSR it would still be a very good upscaling method (well worth it for consoles at least). If they manage to do even slightly better it's a true and honest DLSS competitor.

Overall, I'm quite impressed

 

[Gideon]

Well, it's coming already this month!

AMD Formally Unveils FidelityFX Super Resolution: Open Source Game Upscaling

www.anandtech.com

The not-so-good news is that it's indeed not using motion vectors at all (according to Ian Cutress it's per-frame only). The quality of the GTX 1060 screenshot also leaves a lot to be desired.

I still don't understand why don't they use motion vectors (they hopefully at least use Z-buffer data). Information is better than no information. You can't upscale from "thin air".

I wonder how well it does with moving objects like thin wires. I'm afraid not so well. Imo they should just copy what UE5 did. It looks awesome in motion and has next to no performance hit.

 

[exquisitechar]

The not-so-good news is that it's indeed not using motion vectors at all (according to Ian Cutress it's per-frame only). The quality of the GTX 1060 screenshot also leaves a lot to be desired.

I still don't understand why don't they use motion vectors (they hopefully at least use Z-buffer data). Information is better than no information. You can't upscale from "thin air".

That's really a letdown after all the waiting.

 

[Tup3x]

Based on the screenshots the lower resolution is very obvious. DLSS1 level at best. It should be very straight forward to implement but it does not have any further in UE5 for example.

 

[CakeMonster]

That could be disappointing, but I mean why complain when it will almost certainly be better than the old resolution scaling in-game anyway. I'm sure they'll work on it for some time, and probably improve quality, and if not, at least performance.

 

[GodisanAtheist]

Imo, so long as these IQ reducing features don't get baked in and hidden from the end user or in some way cripple or prevent PC gamers from native res gaming, bring them all on.

My line with DLSS has always been if I need it I bought too little GPU, and when I should want it is to squeeze a bit more out of an aging card to get it up to a FPS number that's functional for me. I feel the same way about FSR or whatever other tech.

If this keeps AMD cards lean and mean in the future (free of hardware bloat) and fighting on the native res but can be touted as feature parity and makes them more competitive in all the benchmarks then I'm all for it.

 

[ThatBuzzkiller]

There's still potential to improve image quality for FSR if we use higher internal resolutions like 75% of native with a minimum output resolution of 1440p. FSR isn't going to work very well with low internal resolutions but the same applies to DLSS as well ...

 

[Dribble]

That could be disappointing, but I mean why complain when it will almost certainly be better than the old resolution scaling in-game anyway. I'm sure they'll work on it for some time, and probably improve quality, and if not, at least performance.

I don't know - I would have thought the game engine teams will have spent more time working on good scaling then AMD - particularly for any engine that works on a console as they scale all the time. Would be interesting to compare UE5 built in scaling to this for example.

 

[Heartbreaker]

Well, it's coming already this month!

AMD Formally Unveils FidelityFX Super Resolution: Open Source Game Upscaling

www.anandtech.com

From the Link:
" FSR is going to be a purely spatial upscaling technology; it will operate on a frame-by-frame basis, without taking into account motion data (motion vectors) from the game itself."

I still don't understand why don't they use motion vectors (they hopefully at least use Z-buffer data). Information is better than no information. You can't upscale from "thin air".

Once you start requiring motion vectors, it then requires more developer work and more tie ins to the game engine, so less support. That and you actually need to do the work of developing a good algorithm, which can be a lot of work.

This is just a dumb simple spatial resize algorithm (AKA what we already had), likely with a bit extra work applied to the sharpening. Overall it's not much more than formalizing what AMD already does as DLSS competitor.

They are just hoping people will use it in reviews for comparing to DLSS for frame rates.