Question DLSS for upscaling a texture

I have a texture that is rendered using path tracing, and want to upscale it 2x.

Is this possible using DLSS?

The details around DLSS are quite flaky. From what I understand it might take metadata like motion vectors, the previous frame as well. So, how to upscale an already rendered texture without any other buffers?

 

As you already guessed, NVidia DLSS and AMD FSR 2.0 do need the previous frame, motion vectors and depth. However, you could use AMD FSR 1.0 or the NVidia Scaling SDK instead if you don't have that data. Just don't expect miracles. The reason why DLSS and FSR 2.0 work that well is because they use jitter and history, so after 4 frames they have all the information to reconstruct the high-res image perfectly when there is no movement.

It is also worth mentioning that those upscaling techniques are fairly expensive - so usually you only want to use them once for the final frame before rendering the UI.

The easiest and fastest alternative would be to use a simple bilinear blit to a larger render target.

PS: What do you mean with the requirements are "flaky"? They are stated pretty clearly in the SDK docs at page 2:
https://github.com/NVIDIA/DLSS/blob/main/doc/DLSS_Programming_Guide_Release.pdf

 

Thank you for your insight, this is very helpful. Flaky is probably a poor choice of words, but I was unable to find any detailed information in any of the Unity docs, or in any forums. The actual Unity implementation is what matters to us.

I could not find any info about DLSS upscaling for pathttracing for example, which is a typical example of where it would be super useful since this is a very expensive rendering technique. From my tests the Unity implementation of DLSS simply does not work with PT at all. Only the simple upscaling algorithms, like catmullrom, seems to be enabled in PT mode. And the upscaling in PT mode is also not very efficient, as they happen for every pass, and if you do say 500 SPP to render the final image, that is 500 useless upscalings before the final denoise.

Maybe DLSS is a poor choice anyway as it is optimized for real time use, while we need max quality.

This is what I want to achieve: We are making pre-rendered graphics with Unity, and want to cut down the render time by rendering lower res images then upscale 2x. It is not realtime, so a time expensive but good scaling algorithm is fine. We can offload the scaling to a separate thread anyway.

I looked at Topaz Gigapixels, and it delivers great results, but sadly it cannot be used programmatically, which is a deal-breaker for our use case.

Do you happen to know any good AI-based upscaling algorithms (quality is the most important factor) that could be called command-line?

 

I'm not aware of an AI-based upscaling SDK for offline use, unfortunately. Even if there is such a SDK, there is a good chance that its results are not temporarily coherent.

If you don't have realtime requirements, why not just render at the target resolution? That will give you the best quality.

Another idea would be to render the scene with rasterization first to get the meta data, overwrite the result with your path traced image and then use DLSS.

 

Because it is very time consuming to render at the target res. It would take weeks.

 

It seems that most path tracing renderers boost speed by lowering down the SPP.

Maybe you can try this idea with a good denoiser.

 

Yes, of course we use a good denoiser, but that will never compensate quality wise for a low SPP. And besides, even with as low as 10SPP denoised we are taking many days of rendering. We have played with these settings to death already.

We have been doing this a long time, and optimized it like crazy, both the scene and everything else. AI-upscaling is by far the best option and gives us good results, however, we need to do be able to do it programmatically. But now I do not think DLSS is the best option anyway, we will most likely go for another AI-upscaler since we do not need real time upscaling, but high quality. Just need to find a good one, like Topaz Gigapixel, but that can be hooked into our automatic workflow.

 

I think that they forget the use case of combining recorder's accumulation with upscaling.

I suggest submitting this idea on HDRP's roadmap because Unity prefers real-time movies and may not pay enough attention to boosting the HDRP path tracing rendering speed.

If no one mentions this, I think they will never know. (better way to use accumulation & upscaling)

 

Thanks, I have been trying to suggest PT specific features to Unity multiple times, but have long since giving up this approach, could just as well give it to my cat. Same result.

From what I see, Unreal has everything we need. They got a really good system for pre-rendering high quality images, for use in video or otherwise. Unfortunately, we are too heavily invested already in Unity to make the switch atm, so desperately and continuously trying to find workarounds for so many PT related issues, upscaling is only the tip of the iceberg.