AMD FSR and Unity

Yesterday AMD dropped more details on FidelityFX Super Resolution (FSR), which is AMD's response to Nvidia DLSS. It is due for release on the 22nd. While it is unknown yet how good the quality is, the big surprise in their announcement was that it will support both AMD and Nvidia GPU's. FSR will support GTX cards going back to the 10 series.

With good cross hardware support, it would make sense for a game engine like Unity to prioritize support for it. Unity is supposed to have HDRP support for DLSS in 2021.2, but I'm wondering if Unity has any plans yet for FSR.

 

I seriously doubt it's as good as DLSS since the latter has custom hardware it can utilize made for it.

Nonetheless, it's really cool that we now have an alternative that works on 'almost' all GPUs used nowadays.

I for one can't use DLSS :/
Unity took it's time implementing DLSS, but hopefully FSR gets support as well.
Maybe if the other engine did it first again, they'll cave in eventually.
To be fair though, I feel like it'll be a pain for them having to juggle these two.

 

Technically there is another but it's restricted to Unreal Engine 5.

https://www.pcgamer.com/unreal-engi...romises-near-4k-quality-at-the-cost-of-1080p/

 

It's due out June 22 so when can we add it to our Unity games?

2 x performance boost on lowest of four quality settings.

Side question could this be added to browsers to enable it in WebGL games or videos?

 

I am very disappointed in AMD. I would have thought they would learn from the mistake of NVIDIA when they launched DLSS 1.0 and it was a blurry smeary mess. Yet here we are with a solution from AMD that is a blurry smeary mess. I would not want to turn this on for my games. Unity's dynamic resolution looks better than this.

Spoiler

 

hmm FSR have 4 quality mode isn't? seeing from the quality comparison the ultra settings seems giving good result

 

Well that's... hideous. That's the kinda results I'd expect from just using bilinear filtering and an unsharp mask

 

I'll reserve my opinion when I can see it on my own screen or see better screenshots. We shouldn't judge videos. Screenshots I've found have motion blur in it, so it's really hard to say how good it will be.

 

You can do that if you wish but just be aware that the technique they're using is identical to the technique used by NVIDIA for DLSS 1.0. NVIDIA switched away from that technique because it couldn't produce acceptable results and more often than not introduced artifacts into the final result. Thus they deemed it a failure.

Spatial's main advantage is that it can be applied after everything has been rendered whereas a temporal solution needs to be built into the game by the developer. We'll have to wait and see if AMD can produce better results but seeing how their drivers have been for years I'm not holding my breath.

https://www.anandtech.com/show/1672...x-super-resolution-open-source-game-upscaling

 

Oh dear, okay. DLSS 1.0 wasn't any good. Have you found more technical details? I mean, there has to be some difference. Using an outdated technique would be pretty weird.

 

What I've mentioned above is all we currently know. Once the source is revealed we'll know more. I'm just hoping that it being open source means it will have better quality but again I'm not holding my breath here. That image I presented in an earlier post? That's the quality setting.

Looking at it again there is a bit of motion blur on the sides but I can't see any in the middle. Check the pillar. It's texture is very blurred compared to the non-upscaled pillar.

 

AMD's technique has to be different than DLSS 1.0. With DLSS 1.0 they would pre-train a neural network against super sampled images on a per game basis. At runtime the neural network would run on the tensor cores to do the upscaling based on what it was "taught" a good upscaled image should look like from its training on that game. This is why DLSS has been restricted to only Nvidia cards with tensor cores. (and also why they named it Deep Learning Super Sampling)

AMD FSR though doesn't require tensor cores, and says it doesn't require per game training.

 

No. DLSS is not restricted to tensor cores in the sense that it cannot run without them. DLSS is restricted due to a combination of performance considerations and to sell cards. Tensor cores are not special in the sense you're making them out to be. All they do is multiple two 4x4 FP16 matrices then add a third matrix to the result.

https://en.wikipedia.org/wiki/Volta_(microarchitecture)

Tensor uses a fused multiply-add which allows this to be a single operation. CUDA would take several steps to do the same task which is why the dedicated hardware exists.

https://en.wikipedia.org/wiki/Multiply–accumulate_operation
https://www.quantstart.com/articles/Matrix-Matrix-Multiplication-on-the-GPU-with-Nvidia-CUDA/

That said you're most likely accurate with this statement since it is a neural network.

 

Looks like FSR is actually pretty good. Much better than DLSS 1.0. FSR is pretty competitive at high resolutions, but DLSS 2.0 is noticeably superior at low resolutions. Seems to be a good showing for an FSR 1.0 initial release, and I notice Unity is on the list of studios working on integrating it.

 

I tried it in a demo game and it's a nice option to have for 2k/4k monitors getting more framerate. Good to know it will be coming to unity.

 

Eh, the performance and balanced profiles look like ass (performance especially literally looks like bilinear upsampling plus unsharp mask). The other two look a bit better. I would say it's a bit better than DLSS 1.0, because it doesn't super smear, but I don't think final image is that impressive.

 

Maybe with adaptive resolutions rendering areas of more complexity and proximity could be rendered at a higher resolution e.g. centre of screen in FPS?

 

HDRP's upscale is quite basic but effective.

FSR is similar but with sharpening and factoring motion vectors, but honestly without the zoom 3x for comparison, you probably can't tell for most games if it is on or not. If you didn't have a comparison shot you would never actually know.

Devs are the worst possible audience.

 

looks like an improvement to me.

I was playing a shooter game from around 2015 last night. I was fiddling with the graphics settings a bit just cause, and I realized that the game felt and looked better if i turned the resolution down to like 1280x720, rather than 1920x1080.

What I realized is that the eye hardly notices the resolution decrease, and in fact the slight bluriness actually helps unify the graphics, hiding a lot of those hard CG lines and bad transitions. Materials blend better together, and the whole thing gets a sort of charm to it, whereas with too much fidelity it just felt like slightly dated realistic graphics.

Anyway, my point is, i think a little bit lack of crispness can be a good thing.

 

Can UI still be at native resolution?

 

Most people don't even notice their TVs have motion smoothing on, but that doesn't mean it's good. Most games are run without the default settings being changed, even the resolution, but that doesn't mean they look good. The end user won't notice all sorts of things without having them compared right in front of them just like this.

The thing is? This doesn't matter. It does not matter if they'll notice.

As developers, we're evaluating technology based on how it will affect the games we create. If a game is going to take a visual hit because of this technology, we have to consider that! Our entire goal is to deliver a game to somebody, and often we probably want to deliver the best possible version of that game. If we just go "oh, the user won't notice" then we ignore that whole part.

 

I think that if user is not affected on some level - conscious or not - you literally wasted time.

Without the user, there is no game.

 

There's literally a whole-rest-of-the-post that covers this.

 

Well I think you have a "everything must grow infinitely" mindset. I am suggesting a "does it make a real difference?" mindset.

We know one thing, newer prettier graphics do get the LCD excited to spend money. So I think there is always a reason to improve tech - it's an arms race essentially. If somebody elses game is prettier than yours, it will catch more eyes.

But I think when a developer develops a sense of "what is the best possible version of a thing" separate from the consumer - that's counter-productive. Assuming you are making games for commercial release. The consumer is the target - there is no other metric.

Like many here I've been playing games almost since the beginning of games. They haven't gotten better. They are pretty much the same - sometimes worse. But how much more money and man-hours do they take to create these days?

 

That's a wild leap to make. The things you think do not make a difference do not make a difference if you don't let them. Again, there's a whole rest of the post.

They never see those graphics themselves unless they're in the higher-end of gaming spaces. The places they do see those graphics are from streamers and promotional materials. Their games don't look like that, but they look like the idea of the game does to them.

But they would have a marked better experience if the default is better.

Again, you are divorcing all meaning from the post you're replying to itself and completely decontextualizing things. I never said it was separate from the consumer, you said I did.

This has nothing to do with anything I've been talking about.

 

I guess I don't follow you then.

I thought your post could be paraphrased like this:

Who cares what consumers think, they are vegetables.

Developers owe it to themselves to use the best possible tech to make the best possible game because...reasons. Whether or not consumer knows the difference your hard work made doesn't matter.

 

Again, you're saying these things. None of these things are in the post.

 

haven't read much here so excuse me but, If i'm struggling with performance and can get better performance without the user noticing a visual difference, trust me, I'll enable that so fast the feature will feel abused

the marketing content doesn't need to have it on

 

No actually I think the "the user won't notice X and Y" is attitude that is treating consumers like vegetables. "No one will notice" is already simplifying and looking down on your target audience.

You can't really know what people will notice or not. And you can't really quantify it (will some people notice? What is that worth? Will some notice, but not consciously? What is that worth?). Unless you have concrete answers for these things (which I don't know how you could), the only thing you can do is do what you think is best.

 

Well if you have the means this is what testing is all about.

For tiny indies we can only do our best to know the target audience. Humans aren't a total enigma, there is pretty observable patterns. It's not hard to catch a person in a trap if you want to.

But my main point is to push back against this notion that there is some higher truth than consumer perception. It can be a real time/money sink if developer gets focused on things that don't register with consumers because they get an idea about the right or wrong ways to do things. I think for indies especially we have to think outside of the tech arms race, find ways to get the job done more efficiently by being more focused.

 

My main point is that we can't really "know" and quantify consumer perception (or at least, it's a task even bigger than making a game), in which case there is no point of trying to cater to a hypothetical.

 

My understanding is FSR is intended to run before the UI is applied, so the UI should be native if done as AMD recommends.

 

Well, I think we can, and it is perhaps the most important part of making a game. I think so anyway.

Without big data like AAA has, we do have to rely more on assumptions, and if its the best you can do, its the best you can do. But I think we have to be really cognizant of knowing what we know, what we are assuming, and not conflating personal desires with professional needs.

 

"Most people won't notice X" is useless for indies. Maybe 5% will notice. Maybe 1% will notice. Maybe 0.1% will notice and will really appreciate it and then make a youtube video raving about your game which will result in more eyes on your game.

AAA need to cater to lowest common denominator since they need to sell a gazillion copies. Indies need to sell way less than that, and few people really appreciating things in your game can have a big impact.

If you accept that, creating a hypothetical "average consumer" in your mind and catering to that is of little use, especially if you're looking down and being dismissive "most people won't notice that, or that", because with a few steps forward, the logical conclusion is that most people won't notice most things, so why bother doing anything.

Most people won't use most of Unity's features, so Unity should just have like 2% of the feature set it currently has... right?

(with that said, I'm not against *reality checks* in terms of "wtf am I doing, why am I implementing this, maybe it doesn't matter", those are useful, also keeping in mind that your game is supposed to have an audience is important, just be careful of the hubris of thinking you absolutely know what your audience likes and what they will notice or not)

 

Yes I agree. Aren't we saying the same thing here?

 

I am not sure I did have an inkling we don't really disagree all that much, just approaching from a different side.

 

Loads of games already use upscaling and play at non-native resolutions. If this does a better job than the algos people were using before, then neat.

I honestly think that 4K for games is a bit gratuitous in most cases, considering our current, brute force rendering techniques. It's quadruple the work for our already hot and hungry GPU's for a visual impact that our eyes are often going to have a hard time perceiving.

Plus, consider how many pixels are rendered and then thrown away. Things like blur effects which take our crisp, native resolution pixels, then smush them together before sending them off to be displayed.


Personally, I'm looking forward to seeing experimentation with non-uniform-resolution rendering. Increased pixel density only makes a difference where there is high enough frequency information in the image, or where there are edges. In other words, where nearby pixels will be different from one another. For low-frequency interior areas of shapes you can render far fewer pixels and with simple bilinear filtering our eyes won't notice the difference. Even with a magnifying glass. It's just not the kind of thing our eyes / brains are tuned to pick up on.

This principle is one part of how image and video compression works. Identify information which can be lost without being noticed. One major catch to this is that it relies on knowing where that information will be in advance.

Eye tracking could boost this further. We only need full resolution where we're focusing, so if the renderer knows that it can basically render at lower res everywhere else.

The idea is that we could potentially fill our screens at native percieved resolution while only rendering a fraction of the pixels,without any post procesding guesswork magic required.

 

All three options (DLSS, TSR, and FSR) are superior to traditional dynamic resolution upscaling methods and I would be surprised if they weren't superior to the upscalers embedded in most televisions.

I disagree but only because these techniques work best when the output resolution is high. DLSS 8K for example is actually 1440p internally which is much easier to render than 4K (only a 75% increase in pixel density) and reviews mentioned the results were impressive even though it uses the "Ultra Performance" setting.

Meanwhile trying that same setting with an output resolution of 1440p was just awful with many artifacts thanks to the internal resolution being sub-480p.

 

Doubt any of this will come to URP anyway. URP is the biggest money spinner for small indie teams but HDRP is what Unity pumps money into because HDRP is considered the future of Unity.

Switch and Quest will probably not benefit from FSR anyway, the returns would be far smaller given the weaker hardware, so I don't think it's much of a loss.

 

URP recently gained support for motion vectors which is one of the requirements to use DLSS.

https://portal.productboard.com/unity/1-unity-platform-rendering-visual-effects/c/218-motion-vectors

 

Yeah I don't think it's coming though. Love to be super proved wrong. Lets see.

 

If you haven't already go submit a request for it. I just finished submitting it on my end.

https://portal.productboard.com/uni...s/c/452-didn-t-find-what-you-were-looking-for

It's my understanding too that it's trivial to implement it in existing games.

https://wccftech.com/amd-fidelityfx-super-resolution-can-be-implemented-in-a-day-or-two-devs-say/

 

So, at least 6 months for Unity to implement.

 

This is what I mean when I think the blurrier one actually looks better.

It's like when you got one of those magnifying mirrors, then you can see all your disgusting pimples and blemishes in full glory. Even really good CG is still far cry from photorealistic, so if we use resolution that can show a dirt spec on a booger dangling from a nose hair I don''t think it's actually making a positive effect.

 

I wonder if it'll be possible to implement ourselves or if engine source code is absolutely required. I guess we'll know in Mid-July when the source code is open at GPUOpen as MIT.

According to GPUOpen (they will have the sourcecode mid july some time), you can make it implemented after Anti-Aliasing and Tonemapping and that's where it should be at. So I wonder if we can implement it as a post process effect in HDRP for example. https://gpuopen.com/fidelityfx-superresolution/

I really hope Unity doesn't slack in implementing this if we can't do it ourselves. A team of thousands should be able to do it just as fast as a single team of normal game-devs. Granted yeah Unreal is open-source so they had extensive knowledge of the engines source code to implement it. But just like them, Unity of course has extensive knowledge of their own engine, so it should be rather simple for them to implement it. (Yes I know not all thousands of Unity employees work on the same thing), was just trying to emphasize the numbers that a small team can do it quickly, so could Unity.

And the truth is, it would greatly benefit them if they did it, heck even as a package manager asset idc. But give us the choice, even if it doesn't live up to expectations, at least we still have a choice for our customers for better performance.

 

Yeah, I was thinking they could probably quickly implement this as an alternate modified HDRP package tagged as experimental. That would satisfy a lot of game devs here in the short term. Then work on integrating it into the official HDRP and maybe URP packages for future fully supported releases. I expect Unity to not add it for the built in pipeline.

 

Dunno if it works well with desktop and PS4/5 VR but here's hoping.

 

Convince Unity team to make FSR as a package manager asset asap and find out for us once we get it

 

The more the merrier. +1-ed it.

 

I did it a few weeks ago myself