Discussion AMD Gaming Super Resolution GSR

[DisEnchantment]

New Patent came up today for AMD's FSR

Patent Public Search | USPTO

appft.uspto.gov

https://www.freepatentsonline.com/y2021/0150669.html

20210150669 GAMING SUPER RESOLUTION

Abstract
A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display

[0008] Conventional super-resolution techniques include a variety of conventional neural network architectures which perform super-resolution by upscaling images using linear functions. These linear functions do not, however, utilize the advantages of other types of information (e.g., non-linear information), which typically results in blurry and/or corrupted images. In addition, conventional neural network architectures are generalizable and trained to operate without significant knowledge of an immediate problem. Other conventional super-resolution techniques use deep learning approaches. The deep learning techniques do not, however, incorporate important aspects of the original image, resulting in lost color and lost detail information.

[0009] The present application provides devices and methods for efficiently super-resolving an image, which preserves the original information of the image while upscaling the image and improving fidelity. The devices and methods utilize linear and non-linear up-sampling in a wholly learned environment.

[0010] The devices and methods include a gaming super resolution (GSR) network architecture which efficiently super resolves images in a convolutional and generalizable manner. The GSR architecture employs image condensation and a combination of linear and nonlinear operations to accelerate the process to gaming viable levels. GSR renders images at a low quality scale to create high quality image approximations and achieve high framerates. High quality reference images are approximated by applying a specific configuration of convolutional layers and activation functions to a low quality reference image. The GSR network approximates more generalized problems more accurately and efficiently than conventional super resolution techniques by training the weights of the convolutional layers with a corpus of images.

[0011] A processing device is provided which includes memory and a processor. The processor is configured to receive an input image having a first resolution, generate linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generate non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The processor is also configured to convert the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and provide the output image for display.

[0012] A processing device is provided which includes memory and a processor configured to receive an input image having a first resolution. The processor is also configured to generate a plurality of non-linear down-sampled versions of the input image via a non-linear upscaling network and generate one or more linear down-sampled versions of the input image via a linear upscaling network. The processor is also configured to combine the non-linear down-sampled versions and the one or more linear down-sampled versions to provide a plurality of combined down-sampled versions. The processor is also configured to convert the combined down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution by assigning, to each of a plurality of pixel blocks of the output image, a co-located pixel in each of the combined down-sampled versions and provide the output image for display.

[0013] A super resolution processing method is provided which improves processing performance. The method includes receiving an input image having a first resolution, generating linear down-sampled versions of the input image by down-sampling the input image via a linear upscaling network and generating non-linear down-sampled versions of the input image by down-sampling the input image via a non-linear upscaling network. The method also includes converting the down-sampled versions of the input image into pixels of an output image having a second resolution higher than the first resolution and providing the output image for display.

It uses Inferencing for upscaling. As will all ML models, how you assemble the layers, what kind of parameters you choose, which activation functions you choose etc, matters a lot, and the difference could be night and day in accuracy, performance and memory

 

[Hail The Brain Slug]

I think Nvidia managed to skew people's perception about upscalers since they needed DLSS to be a crutch for RTX performance.

There's nothing stopping us from using native rendering in conjunction with an upscaler such as FSR to get better IQ. The performance hit is worth it.

Too bad that mode of DLSS that was part of the initial announcement never materialized, and FSR doesn't yet offer a mode for that either.

 

[Mopetar]

It is a bit of a dissapointment than 1080p -> 4K is a blurrfest, while TSR and other temporal algorithms are still very good at it.

Still an excellent result overall for a first version. I stilldo hope we eventually also get a temporal one, but even in it's current implementation it can be very useful. It just needs wider game support

Honestly I'm rather impressed given AMD's previous track record with delivering software or driver enhancements during the Vega era.

Their first release is just barely not quite as good as DLSS and there's no doubt room for AMD to make improvements. They also created something that will probably see broader industry support. Intel will no doubt hop on this train because it helps their iGPUs a lot and could give a boost to their fledgling dGPUs. Consoles will definitely use it and all the ports get it for free.

I was really skeptical about AMD's earlier claims about delivering something effective that worked across platforms. It just seemed far too good to be true or that the cross-platform would come at the expense of performance in order to have something that could work everywhere. Never mind that I assumed the quality wouldn't really be much better than the initial DLSS release and that we'd get the usual "work in progress" schtick we've heard a hundred times before.

This still isn't something I'm personally interested in and I think that we're going to be moving to a future filled with really petty squabbles over differences in IQ and whether higher FPS matters and a lot of other crap I'm not looking forward to at all. However being able to set a higher resolution so that it uses a native render as the target for the sharpening is certainly interesting. It certainly worked in the example posted earlier in this thread and if it's not too much of a performance hit I could see myself using it that way.

 

[Leeea]

I give to you FSR "Uber" Quality mode:



1. Enable Virtual Super Resolution in the AMD driver.
2. Set in-game resolution to 2X the native resolution.
3. Enable FSR Performance mode to render the game at half the resolution (effectively going back to native resolution).
4. Observe what FSR does with native resolution content.

The crop above is from a 1440p render. Native is to the left, VSR(5120x2880)+FSR Performance to the right.

Someone mentioned this can be even implemented through some injectors like Reshade, hmm

The potential of the above two things combined is the holy grail of retro gaming.

 

[moinmoin]

I do hope they can continue to improve it to make it even more valuable

Since it's open source (AMD writes code will be released on Github under permissible MIT license in mid July) I fully expect people and companies to develop it further even in case AMD itself doesn't. With FSR likely being widely supported on consoles and PC the potential of improvements being a drop in replacement for existing FSR implementations should boost the development further, especially considering it's not reliant on specific hardware (vendors).

 

[DAPUNISHER]

Seems like there are some big gains to be had with 1080p after all. At least with APUs. Here is a 4700U getting huge gains in rift breaker. Yeah, it is going to look extra bad with anything below quality, but on an APU, decent fps is a nice trade off.

https://twitter.com/x/status/1407596151428161541

 

[Fanatical Meat]

I give to you FSR "Uber" Quality mode:

View attachment 46126

1. Enable Virtual Super Resolution in the AMD driver.
2. Set in-game resolution to 2X the native resolution.
3. Enable FSR Performance mode to render the game at half the resolution (effectively going back to native resolution).
4. Observe what FSR does with native resolution content.

The crop above is from a 1440p render. Native is to the left, VSR(5120x2880)+FSR Performance to the right.

gotta be honest. Those two screens look virtually identical to me

 

[Leeea]

gotta be honest. Those two screens look virtually identical to me

look at the rocks
also look at the plants in the upper part of the screen

considerably more visible definition

 

[uzzi38]

https://www.reddit.com/r/hardware/comments/o6sx19

TL;DR - TAAU disables depth of field which makes 1080p TAAU on the character DF showed actually look better than 4K native.

Making it a really bad comparison point vs 1080p internal resolution FSR with DoF enabled.

 

[Timorous]

https://www.reddit.com/r/hardware/comments/o6sx19

TL;DR - TAAU disables depth of field which makes 1080p TAAU on the character DF showed actually look better than 4K native.

Making it a really bad comparison point vs 1080p internal resolution FSR with DoF enabled.

Alex going to Alex I am afraid. Makes Durante's comments come off like they are a DF fanboy.

 

[uzzi38]

Alex going to Alex I am afraid. Makes Durante's comments come off like they are a DF fanboy.

Durante always has been. I saw the discussions on Reddit - you take him too seriously.

 

[coercitiv]

gotta be honest. Those two screens look virtually identical to me

Here's a 3X zoomed crop, native to the left, native+FSR to the right.


I used the two colored squares to highlight important areas with what I consider the best and worst of FSR in this crop.

1. Red square - FSR sharpening applied on textures with 1 pixel highlights will result in even higher contrast 1 pixel artifacts. (will discuss more on this bellow)
2. Greeen Square - edges of the rocks are not only sharper (as expected) but also better anti-aliased. If this holds true for more game content the this alone would warrant it's usage, as it has the potential to smooth out many troublesome areas.
3. Not seen in this crop is how much better the grass textures look with FSR. There's no contest here.

Overall FSR appears to be over-sharpening this scene, although I believe the low quality textures aren't helping either. We had people warn us that AAA games with high quality assets won't see the same perceived benefits as the modest ones, but the irony is they won't see this many artifacts either.

One last note: this game has a lot of blur while moving the camera (presumably from TXAA and maybe other post-processing), and the camera moves even when pointing your character in a different direction or when opening and closing the menu to change settings. The screenshots above were taken after waiting for the camera movement to stop, so they lack one other layer of blur that is observed constantly during gameplay. Ironically this means that some of the artifacts that I'm pointing out with FSR and also my subjective take on over-sharpening are well dampened during gameplay. This may also be the reason why some people will claim FSR can offer both better performance and better IQ, this game has a poor native presentation which really favors FSR even when upsampling from lower than native resolution.

 

[coercitiv]

Things are evolving fast I see: people are testing Dota 2 with FSR, I don't have time to check results right now but it appears FSR does a lot better in terms of IQ with better quality assets (at least better than it does with Riftbreaker).

Las but not least, Digital Foundry's assessment of FSR is put under the microscope. Not necessarily their fault, but still good to know what happened.

 

[Gideon]

Alex also added Godfall shots with TAA vs FSR vs Bilinear. FSR does way better than in Kingshunt IMO (considering it's performance mode)

A temporal solution would still be better though , as clearly seen when compared to TAA. Against something like TSR it would be a no-competition (as it can recreate an almost indistinugishable from native 4K picture from 1080p)

 

[Timorous]

Alex also added Godfall shots with TAA vs FSR vs Bilinear. FSR does way better than in Kingshunt IMO (considering it's performance mode)

A temporal solution would still be better though , as clearly seen when compared to TAA. Against something like TSR it would be a no-competition (as it can recreate an almost indistinugishable from native 4K picture from 1080p)

KitGuru showed FSR UQ vs TAAU 77% render scale and preferred FSR because of the shimmering TAAU added. Performance for both is pretty similar so you can't do FSR Ultra Quality and get similar performance to TAAU 67% render scale with better IQ.

It really looks like Alex went hunting for comparisons to prove his pre testing conclusion rather than doing the testing and coming to a conclusion based on the findings.

DOTA2 is showing good results with 90% FSR looking better than native. DOTA 2 Comparisons

 

[Timorous]

Durante always has been. I saw the discussions on Reddit - you take him too seriously.

He has a certain amount of clout though so I find it irritating that DF produce a video with a flawed conclusion that TAAU > FSR despite only testing at one resolution scale setting and not proving it for the others. Durate then backs up that video by saying 'I do game porting for a living and this is THE BEST analysis in the history of analysis' and then all of a sudden TAAU > FSR becomes the 'truth' when it was a faulty conclusion from a limited data set.

EDIT. That is not to say that FSR is flawless and these kinds of scenarios should be hidden but that only showing these scenarios presents a misleading picture of what FSR can and cannot do.

 

[leoneazzurro]

Alex also added Godfall shots with TAA vs FSR vs Bilinear. FSR does way better than in Kingshunt IMO (considering it's performance mode)

A temporal solution would still be better though , as clearly seen when compared to TAA. Against something like TSR it would be a no-competition (as it can recreate an almost indistinugishable from native 4K picture from 1080p)

Point is, why comparing only with the FSR mode that AMD itself says says it is the worst for IQ? Why not doing the comparison for all FSR modes and comparable TAAU modes, looking also at the performance hit? Why not doing the same comparison with DLSS in tha games supporting it?

TSR IMHO is the future, hardware agnostic AND built-in the graphic engine. FSR is a stopgap, but it must be judged fairly.

 

[Timorous]

Point is, why comparing only with the FSR mode that AMD itself says says it is the worst for IQ? Why not doing the comparison for all FSR modes and comparable TAAU modes, looking also at the performance hit? Why not doing the same comparison with DLSS in tha games supporting it?

TSR IMHO is the future, hardware agnostic AND built-in the graphic engine. FSR is a stopgap, but it must be judged fairly.

The future is a CRT resurgence so we don't even need upscaling anymore.

 

[uzzi38]

KitGuru showed FSR UQ vs TAAU 77% render scale and preferred FSR because of the shimmering TAAU added. Performance for both is pretty similar so you can't do FSR Ultra Quality and get similar performance to TAAU 67% render scale with better IQ.

It really looks like Alex went hunting for comparisons to prove his pre testing conclusion rather than doing the testing and coming to a conclusion based on the findings.

I mean, it says enough that he decided to switch game rather than showing updated comparisons using Kingshunt.

 

[Timorous]

I mean, it says enough that he decided to switch game rather than showing updated comparisons using Kingshunt.

It does but if his conclusion was that at 50% render scales TAAU > FSR in the games he has tested I would have little issue with it. The issue is taking that information and generalising to TAAU > FSR in all cases which does not seem true based on other testing that has been done by outlets that did not blunder their testing scenarios.

 

[coercitiv]

DOTA2 is showing good results with 90% FSR looking better than native. DOTA 2 Comparisons

That "better than native" is still subjective, to me it looks overshapened, but I guess this is understandable given that the processing was tuned for 75% and bellow. It's also something that can be tuned further, maybe even on a game-by-game basis.

 

[uzzi38]

That "better than native" is still subjective, to me it looks overshapened, but I guess this is understandable given that the processing was tuned for 75% and bellow. It's also something that can be tuned further, maybe even on a game-by-game basis.

I think it looks better without the 60% RIS that was added in those screenshots as well.

Cheshire Merc Cat on Twitter: "Which one is FSR? https://t.co/ooJk7Bewxv" / Twitter

This serves as an excellent example. If possible it would be best if nobody here votes on the poll now that I'm saying which is which, but the right hand side is with FSR applied on 99.99% internal resolution, and left is native

 

[moinmoin]

Every mention of "better than native" only shows one's preference for post-processing. Sure, it may (subjectively) look better to many people, but at that point one could as well start a competition of e.g. ReShade artists who manages to tweak a game's look to the most favored way (that may end up totally different from what native looks like, it's subjective after all).

 

[Krteq]

Alex also added Godfall shots with TAA vs FSR vs Bilinear. FSR does way better than in Kingshunt IMO (considering it's performance mode)

A temporal solution would still be better though , as clearly seen when compared to TAA. Against something like TSR it would be a no-competition (as it can recreate an almost indistinugishable from native 4K picture from 1080p)

Lol, on those screenshots you can clearly see that DoF is disabled (or bugged) once TAAU is active

 

[Timorous]

So it seems as though the same DoF issue may be happening on Alex's Godfall images as well. Such expert, wow, top tier.

 

[Shivansps]

Seems like there are some big gains to be had with 1080p after all. At least with APUs. Here is a 4700U getting huge gains in rift breaker. Yeah, it is going to look extra bad with anything below quality, but on an APU, decent fps is a nice trade off.

https://twitter.com/x/status/1407596151428161541

720 or 1080p low? yet another one that is talking about APUs whiout ever using one. But yeah, "Quality FSR" would be the setting to use here, that should be the equivalent to 900P native and you can see why you should not use 1080p on a APU.