DLSS in Built-In

To preface, I know Unity is done with the built-in pipeline.

I'm curious if anyone knows if DLSS could be implemented in built-in on DX11, OR what the limitations are and why it can't be done.

I found community AMD FSR and there's dynamic resolution, but both of those solutions require DX12, which is still a slow mess. The editor won't even launch for me in DX12 mode.

Thank you!

 

It should be possible, we’ve been working on implementing FSR 2 for all renderpipelines, also the built-in. If there is enough animo, we might also give DLSS a go.

https://forum.unity.com/threads/amd-fsr-2-0-support.1357190/

 

Yes, we've implemented DLSS 2.3 and FSR 1.0 for the built-in render pipeline in Crowfall on DX11.

 

Awesome! Are you thinking of creating an asset for the DLSS implementation?

 

please! DLSS for build-in!

 

By the way I just bought your FSR 2 asset and I would by a DLSS build-in asset too.

 

Same here @Alterego-Games .
The developer 'Bolt' (Bolt Scripts) of the other DLSS plugin has backed off offering the plugin on the asset store due to push-back and all the edge cases introduced by post processing.

There's also an open source effort to support FSR2 in BIRP.

Personally I would prefer to pay for both and have proper integration with the Post Processing Stack, as well as the support that is due for a paid product.

So far we have found FSR2 (from your plugin, not the open source one) to produce many artifacts and ultimately look worse than CTAA in motion.
(To anyone reading this: NO I do not recommend CTAA if you can avoid it, it looks really good but the developer obfuscates his code and has what I would call a hostile business model.)
We can't guarantee perfect motion vectors and FSR2 seems to be crippled by that.

DLSS handles incorrect motion vectors wildly better than FSR2 and with post processing disabled completely we can run Bolt's DLSS plugin (not on the store anymore), which produces stellar results basically all the time.
When PostFX stack is enabled though, the input textures seem to get upscaled horribly before DLSS gets them and the result is a blurry mess.

I would suggest reaching out to Bolt, he seems like a nice guy and he may be willing to share his work on DLSS support. He indicated that he simply didn't have time to figure out all of the edge cases, which he didn't expect when he started out.
Here is a way to reach him: https://discord.gg/NYhucQJkAP

 

If anyone wants to have a go at it, here is what I remember (I no longer work for the company):

- You need to write a native Unity plugin and integrate the DLSS SDK
- Motion vectors must be enabled on the camera
- DLSS needs the color buffer (before UI and post-processing is applied), depth buffer, and motion vectors as input. I think I got them by calling Shader.GetGlobalTexture in the Camera.onPreRender on onPostRender callback (don't remember if you have to pass them to Init() or the Upscale() method).
- You need to apply jitter to the camera. You can copy the function which generates a Halton sequence from the post-processing stack. I don't remember how I applied it exactly, but most likely exactly how the post-processing stack does it for TAA. There is Camera.projectionMatrix and Camera.nonJitteredProjectionMatrix, so I'm guessing I just assigned them in a pre-render callback or so. But again, you can look it up in the TAA implementation.
- A mip map bias must be applied. You can do this per texture but that's not how I did it. Don't remember exactly, but I think I changed the macro that is used to sample textures. You can copy the standard shaders and includes into your asset directory to modify them - just keep in mind that you need to copy the entire include chain, otherwise Unity will fall back to the includes in the install directory. For example, if you modify C.cginc which is included in B.cginc which is in turn included in A.shader, you also need to copy B.cginc because otherwise you'll end up with A.shader (in asset directory) -> B.cginc (in install directory) -> C.cginc (in install directory, not the one in assets).
- Upscaling must happen before post-processing and UI. Therefore we used two cameras, one for the world and one for the UI. Don't remember exactly how I copied the upsampled image over (canvas background maybe) but I do remember that it was tricky to find the right place to do it. I think the problem was that even the AfterEverything camera event runs before UI, but don't remember why that was an issue.
- Some post-processing effects need a depth buffer. Either run them on the world camera before upsampling, which is not correct but works fine most of the time or upsample the depth texture as well. We did end up upsampling the depth texture because we needed it to depth test nameplates in world-space. There is a gotcha, though: You will have some jitter in the upsampled depth which can look bad depending on what you use it for. I've tried to average it out but didn't work.
- You should implement a feature to notify DLSS about camera cuts.
- Whenever the window is resized or DLSS quality is changed, you have to re-initialize DLSS
- There are some requirements about how you present DLSS options in the UI for the player

 

@Alterego-Games Hey I figure you'd want to see this. (And my post above)

 

Thanks, this is generally the same ballpark as FSR. Only I haven’t checked out the DLSS source. If the only way to get it is via a DLL, it might not be worth the hassle, as it’ll be windows only. Unlike FSR 2 which can run on almost any platform…

 

Yes, DLSS is closed source and Windows/NVidia only. However, it is generally considered slightly better quality than FSR2. Also, at the moment only DLSS3 supports frame generation until FSR3 comes out.

 

That's understandable, but I think for many people, that's exactly what is desired. FSR2 and DLSS offered where applicable.
FSR2 is a decent multi-platform option, but certainly DLSS is on an entirely different level when it comes to quality, and we would pay to have DLSS in our game in addition to FSR2.

For our specific case FSR2 is actually kind of unusable... It cannot handle incorrect motion vectors well and I've had test cases where it generates pretty bad artifacts.
DLSS on the other hand generates simply superior sub-pixel detail, less noise, and minimal artifacts on areas with incorrect motion vectors. It's also sublime when set to work as DLAA, which is something FSR2 can't even begin to do.

Currently we're using CTAA3 because it's more stable than FSR2 while being faster, even if its ability to resolve sub-pixel detail is the worst of the three options.

There is yet to be a DLSS implementation that works with BIRP and post processing.

 

I'll take a look at it soon and see if I can get some decend results

 

It's a long shot but I just discovered that Unity exposes the DLSS API that is used in SRP, I presume. So if you can use this, it would be much easier:
https://docs.unity3d.com/ScriptReference/NVIDIA.GraphicsDevice.html

 

I may, or may not, have something working already (URP and BIRP)

That said, it will take a while before I have anything ready to release and we'll have to get in the Unity asset store queue again...

Guess the DLSS quality level:

 

OOoooooo nice!
Uh.. Performance?

 

that would be nice to have for sure, i dont plan on using URP for any project yet

 

Very like FSR 2, remember, its rendering at a lower resolution that gives the performance boost. FSR and DLSS, “only” make it look good.

And this screenshot is taken on dx11

 

Oh oh, I meant is it the "performance" setting for DLSS?

Is this with the post processing stack?

 

Ah, woops my bad, forgot my own question! Wrong it was ultra performance!

Current version is without PP, but I will built in into the BiRP PP stack like I did for version 1.3 of the FSR 2 asset

Here is the forum link to the DLSS asset!
https://forum.unity.com/threads/in-review-nvidia-dlss-2-upscaling-for-unity.1470576/#post-9201642

 

I thought to implement DLSS in Unity (built-in) from scratch:

https://forum.unity.com/threads/how-to-use-cuda-in-unity.1396699/#post-9016981

but I see that API is exposed via UnityEngine.NVIDIA:

https://docs.unity3d.com/ScriptReference/UnityEngine.NVIDIAModule.html

Tested with Unity 2021.3.25, built-in, empty project then activate package
Window > Package Manager > Packages > Built-in > NVIDIA



test:

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.NVIDIA;
3.  
4. public class DLSS : MonoBehaviour
5. {
6.     void Start()
7.     {
8.         GraphicsDevice device = GraphicsDevice.CreateGraphicsDevice();
9.         bool result = device.IsFeatureAvailable(UnityEngine.NVIDIA.GraphicsDeviceFeature.DLSS);
10.         Debug.Log(result);
11.     }
12. }

Example how to use API:

https://github.com/Unity-Technologi...Runtime/RenderPipeline/RenderPass/DLSSPass.cs

 

That's what I said above but thanks for elaborating on it! You still have to write code for jittering, mip bias and separate UI from world rendering but it saves you from having to write a native plugin. Downside might be that this probably doesn't support DLSS 3, yet.

 

For anyone that didn’t see, I released the DLSS asset for Built-in and URP!

https://assetstore.unity.com/packages/tools/utilities/dlss-upscaling-for-unity-262761