AMD's Super Resolution patent details a potential DLSS-beater

AMD's Gaming Super Resolution patent answers a lot of questions we had about the red team's upcoming FidelityFX Super Resolution (FSR)… and actually gives us a whole load more to ask. Whenever AMD is ready to talk about its potential answer to DLSS, that is.

We've been pouring over the patent filing published today (via Videocardz), though it was actually originally filed back in November 2019. From what we can discern from the patentese speak, and from an often confused back and forth between Jacob and I, this implementation of AMD's Super Resolution feature should be platform agnostic, could work on any game, and doesn't necessarily need to run on a specific GPU.

If that is indeed how AMD's DLSS-a-like feature works, then it will have a far more wide-ranging remit than Nvidia's more restrictive technology. Though by being trained on a per-game basis, rather than a "wholly learned environment", DLSS could end up being more accurate in games it's specifically coded for. Yet this patent suggests its approach might lead to AMD's version being rather impressive in its own right.

The more we hear about AMD's FSR, the more it sounds like the super-sampling equivalent of the FreeSync vs. G-Sync battle. AMD's tech is more open, and works pretty darned well, while Nvidia's is a closed design, but potentially that little bit more effective. 

In the end, if AMD FSR is good enough, then it will be a far more useful, and more widely used technology. And that's pretty exciting.

Anyways, back to the specifics of this patent filing. What seemingly differentiates FSR from DLSS is in the way it uses multiple up-scaling networks, both linear and non-linear, to attempt to preserve important visual data from the original image when it blows it up. But also in that it aims to "efficiently super resolve images in a convolutional and generalizable manner." To us that seems to mean not having to train a neural network on a specific game before it can be used.

Image 1 of 2

The patent details the basic flow of this Gaming Super Resolution as taking a low resolution image (which can be rendered quickly by the hardware in question) and then uses two deep-learning networks, one linear and one non-linear, to down-sample it into multiple different images. These differently down-sampled images allow FSR to pull out a large number of different features—for example colours, objects, and curves—from a low-res starting point and then use that to create a more detailed, accurate high-resolution picture at the end.

"The combination of the linear and non-linear upscaling," says the patent, "facilitates both the preservation of color and larger scale features (large objects and shapes that are more easily perceived by the human eye) of the image from linear upscaling as well as the preservation of finer features (e.g., curved features and features that are not easily perceived in low resolution) of the image from non-linear upscaling. 

"Linear operations use only input data, while non-linear operations use both input data and other data (i.e., non-input data) to augment the input data. Non-linear functions facilitate accurately determining complex features (e.g., curves) of an image more efficiently than non-linear functions (e.g., convolution operations)."

The multiple down-sampled images are then pulled together to create the pixels of the high-res image using a further 'pixel shuffle' stage. This involves using multiple transposed images to create nine-by-nine blocks of pixels, per low-res pixel, using eight shifted images, and one non-shifted image of the down-sampled pictures. 

After this pixel shuffle process some more clean-up operations are performed before a final high-res image is created.

Seems like a lot to do for each frame, but then it always seems like that when you look at what we're expecting a graphics card to do when we throw games at them.

That's just the simplified process, but the patent details the fact the super resolution feature can use any or all of these elements, in any combination. That might suggest that when used on different types of devices it is able to adapt to the hardware on offer. It is claimed that it can be used on "a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer."

And that it also doesn't necessarily need to be run on a GPU, although that does afford it the parallelism needed to do the linear and non-linear portions at the same time. "When hardware does not support the processing in parallel," reads the patent, "the linear upscaling processing and the non-linear upscaling processing are not performed in parallel."

It even states that super resolution could be run in software, though inevitably that would be far, far slower, and would likely negate the benefits.

Still, it sounds rather exciting, especially if it can really run on, well, anything. If it can run effectively on old graphics cards, or even, potentially, an Nvidia GPU it could be an incredibly powerful tool. That will, however, depend on how exactly FSR is enabled. 

If it's a switch in the Adrenalin driver on PC, that makes it tough to see how it would ever be enabled by Nvidia, though there's a chance it could be baked into games, or game engines, like some of AMD's current FidelityFX features.

Perhaps it could be both. It's possible there could be a game-agnostic version that lives in the AMD graphics driver, and another that is enabled on a game-by-game basis.

As I said at the start of this patent screed, this publication leaves a whole load more questions to find answers to. Fingers crossed we'll find out more soon—Computex is coming at the start of next month, with a Dr. Su keynote, and virtual E3 is following close behind. If the June rumours for a FidelityFX Super Resolution release come to pass, those are the two best bets for an unveiling.