Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Denoiser for HDRP pathtracing

Discussion in 'High Definition Render Pipeline' started by molestium, Jan 20, 2021.

  1. molestium

    molestium

    Joined:
    Dec 16, 2018
    Posts:
    22
    I searched whole internet, and could not find a clear answer. Is there any way denoise image in real-time using patch tracing? I know that Unreal engine already has it (DLSS), but for the unity, I did not find anything. Please help, maybe there are some other projects which add denoiser in Unity.
     
  2. Remy_Unity

    Remy_Unity

    Unity Technologies

    Joined:
    Oct 3, 2017
    Posts:
    690
    There is no denoiser for path tracing currently, but it may come in the future (no ETA yet).
    But anyway, pathtracing is not meant for real time rendering.

    There is denoising in raytracing (for GI, shadows, reflections ...)

    Also, DLSS is not denoising, but more an upscaling method.
     
    molestium likes this.
  3. molestium

    molestium

    Joined:
    Dec 16, 2018
    Posts:
    22
    Thanks! Strange, on my GeForce 3080 Unity patchtracing works is very fast. And for me it was a big disappointment that I cannot turn on denoiser :( . I would like to get something like Quake 2 RTX, but it seems not destiny. Primarily for realistic realtime lighting and shadows.

    Hopefully in the future developers will add this function, because people with RTX cards getting more.

    Yeah, DXR rendering is very cool, but it does not provide realistic lighting as with patch tracing.
     
    Last edited: Jan 22, 2021
  4. molestium

    molestium

    Joined:
    Dec 16, 2018
    Posts:
    22
    I'm found this:
    Source: https://blogs.unity3d.com/2019/04/16/introducing-unity-2019-1/#optix

    This for only lightmaps?
     
  5. soleron

    soleron

    Joined:
    Apr 21, 2013
    Posts:
    546
    [QUOTE="molestium, post: 6753148, member: 2441404"This for only lightmaps?[/QUOTE]

    ye
     
  6. soleron

    soleron

    Joined:
    Apr 21, 2013
    Posts:
    546
    Correct but since you added it, do the right thing and let us get some fast and nice renders for marketing purposes. ;) It should not be that difficult to implement.
     
    valentinwinkelmann and molestium like this.
  7. MassiveTchnologies

    MassiveTchnologies

    Joined:
    Jul 5, 2016
    Posts:
    87
    Why is this taking so long? Been on the roadmap for months.
     
  8. The-Real-Mag

    The-Real-Mag

    Joined:
    Nov 7, 2012
    Posts:
    5
    Now with dlss 2.0, pathtracing works realy great. But if a realtime denoiser could be connected to this, it's realy going to be game changing. We work a lot with Chaos Vantage and they have a similar approach and this dramastically changed our workflow. So if a realtime game engine like Unity would have this, it would be amazing.
     
    Last edited: Jul 5, 2021
  9. MCYCShadow

    MCYCShadow

    Joined:
    Jan 1, 2018
    Posts:
    14
    Im using unity by now as my main renderer for cinematics, the usage of having all in place beeing able to easily integrate effects, simulations and particles is just very comfortable.

    But yes I would also love to have an update on the pathtracing which includes a denoiser to reduce render times.
     
    wm-VR likes this.
  10. wm-VR

    wm-VR

    Joined:
    Aug 17, 2014
    Posts:
    123
     
    Last edited: Nov 15, 2021
  11. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,133
    Been using the path tracer lately, a denoiser would really be helpful. :D
     
    newguy123 likes this.
  12. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Posts:
    1,347
    Yes, denoiser - please!
     
    newguy123 and DEEnvironment like this.
  13. LumaPxxx

    LumaPxxx

    Joined:
    Oct 3, 2010
    Posts:
    340
    Denoiser! please!
     
  14. MCYCShadow

    MCYCShadow

    Joined:
    Jan 1, 2018
    Posts:
    14
    Well, of course its not but hey it is a nice thing for rendering.
    And it can open the door for more people using it to make offline media instead.

    Voila, you can see how noisy it is still. But overall its pretty nice.
     
  15. newguy123

    newguy123

    Joined:
    Aug 22, 2018
    Posts:
    1,248
    +1 on denoiser for pathtracing please!
     
    fuzzy3d likes this.
  16. MassiveTchnologies

    MassiveTchnologies

    Joined:
    Jul 5, 2016
    Posts:
    87
    There has been a card for it in the roadmap for quite a while now. Any ETA on it please?
     
  17. chap-unity

    chap-unity

    Unity Technologies

    Joined:
    Nov 4, 2019
    Posts:
    751
    Sadly no ETA, but the plan is still standing !
     
    olavrv and chadfranklin47 like this.
  18. MassiveTchnologies

    MassiveTchnologies

    Joined:
    Jul 5, 2016
    Posts:
    87
    We are at point where we are almost going to develop our own denoiser from scratch
    Ah thanks for the update. We will implement own temporal denoiser in that case.
     
  19. MCYCShadow

    MCYCShadow

    Joined:
    Jan 1, 2018
    Posts:
    14

    Will you make this aviable on the asset store by any chance?
     
  20. Matjio

    Matjio

    Unity Technologies

    Joined:
    Dec 1, 2014
    Posts:
    108
    Hi, good news, work on the pathtracing denoiser has started! (card also moved to in progress on the roadmap)
     
  21. newguy123

    newguy123

    Joined:
    Aug 22, 2018
    Posts:
    1,248
    Awesome news, thanks all!
     
  22. MassiveTchnologies

    MassiveTchnologies

    Joined:
    Jul 5, 2016
    Posts:
    87
    Amazing! Any plans for it to support animation (temporal denoising)?
     
    Last edited: Oct 20, 2021
    chadfranklin47 likes this.
  23. The-Real-Mag

    The-Real-Mag

    Joined:
    Nov 7, 2012
    Posts:
    5
    Finally! Can't wait to see it in action!
     
    chadfranklin47 likes this.
  24. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    We ended un using Intel Open Image Denoiser which we invoke using a command line process when the image is converged. Results are pretty great and integrated seamlessly using a noise image as input. We added albedo and normals AOVs for better result.
     
    The-Real-Mag and chap-unity like this.
  25. The-Real-Mag

    The-Real-Mag

    Joined:
    Nov 7, 2012
    Posts:
    5
    Is this a solution you would like to share?
     
  26. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    It's pretty simple, really. You can download a command-line version of the denoiser at:

    https://github.com/DeclanRussell/IntelOIDenoiser

    You can simply pass in a beauty pass (the noisy image) and it will be denoised. If in addition, you are rendering out AOVs, you can add an Albedo and a Normal pass for better results. Something like this should work:

    Code (CSharp):
    1.  
    2.     public static void SaveFrame(string outputDenoisedImagePath, string noisyImagePath, string albedoImagePath = null, string normalImagePath = null)
    3.     {
    4.         string denoiserPath = Path.Combine(Application.streamingAssetsPath, "oidn\\Denoiser.exe");
    5.         string args = $"\"& \'{denoiserPath}\'\" -i \'{noisyImagePath}\' -o \'{outputDenoisedImagePath}\' -hdr 0 -affinity 1 -clean_aux 1";
    6.         if (albedoImagePath != "")
    7.         {
    8.             args += $" -a \'{albedoImagePath}'";
    9.         }
    10.         if (normalImagePath != "")
    11.         {
    12.             args += $" -n \'{normalImagePath}'";
    13.         }
    14.         System.Diagnostics.ProcessStartInfo startInfo = new System.Diagnostics.ProcessStartInfo();
    15.         startInfo.FileName = "powershell.exe";
    16.         startInfo.Arguments = args;
    17.         startInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
    18.         startInfo.CreateNoWindow = true;
    19.         System.Diagnostics.Process processTemp = new System.Diagnostics.Process();
    20.         processTemp.StartInfo = startInfo;
    21.         processTemp.EnableRaisingEvents = true;
    22.         try
    23.         {
    24.             processTemp.Start();
    25.         }
    26.         catch (Exception e)
    27.         {
    28.             Debug.Log(e);
    29.             throw;
    30.         }
    31.     }
    32.  
     
  27. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    Be sure to look at the command line denoiser parms to make sure you set things the way you need. In our case, we were using sRGB input, so HDR was off, we use affinity to maximize core count and clean aux since the AOVs seem to come out clean (although not antialiased).

    Also note in our case we placed the denoiser in the streaming assets folder so it was available on standalone builds.
     
    Last edited: Nov 8, 2021
    chadfranklin47 likes this.
  28. eturquin

    eturquin

    Unity Technologies

    Joined:
    May 17, 2019
    Posts:
    3
    Hi,
    The solution provided will be along the same lines (we are evaluating Open Image Denoise, along with the NVidia and AMD denoisers), without having to go through cmd line of course. ;) As denoising can take a bit of time (few tenths of a second), it will only be applied after your image has reached max samples. AOVs will also be computed from the path tracer: they should be pretty close to their G-buffer counterparts however, so using those in the meantime (as I assume you may have done) will also give good results.
     
    Matjio, PutridEx and newguy123 like this.
  29. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    We are very happy with the results of this approach. Easier than waiting for the next HDRP version, and will literally produce the same results if given the same inputs and the same denoiser. That same author has a command-line flavor for the Optix Denoiser, but as far as I've tested on Blender, Open Image Denoiser tends to preserve details a bit better IMO, as I don't mind about the denoising speed too much, as renders in our Unity-based path tracing use case are being done offline :D
     
  30. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    Hopefully, you make the denoising call public so that if devs are not using the multi-frame accumulation APIS, like in our case, we can still enjoy denoising features, just providing three Texture2D objects, or even better it's applied to the Camera prior to firing the IsComplete flag, so that it's done automatically.
     
    Last edited: Nov 11, 2021
    chadfranklin47 likes this.
  31. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    @etur I see there is a pt_denoiser branch in the github repo. I tried to use it but it seems to require a package called com.unity.rendering.denoising, but no idea how to install it. Hope to get a glimpse of how it will work and provide feedback as we are actively working on offline rendering based on path tracing on an ongoing project.
     
    newguy123 and PutridEx like this.
  32. The-Real-Mag

    The-Real-Mag

    Joined:
    Nov 7, 2012
    Posts:
    5
    Many thanks Alejandro!
     
  33. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    Incidentally, you can also make a wrapper for the Open Image Denoise DLL and invoke it directly. I saw a couple of implementations of this out there on a google search. It might be the better option to avoid invoking a command line or powershell app.
     
    Last edited: Nov 16, 2021
  34. eturquin

    eturquin

    Unity Technologies

    Joined:
    May 17, 2019
    Posts:
    3
    This package is not available yet. ;)
     
  35. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Posts:
    1,347
    Just being curious, will the denoiser be applied during the rendering-process or when it´s finished?
     
  36. MassiveTchnologies

    MassiveTchnologies

    Joined:
    Jul 5, 2016
    Posts:
    87
    Could you guys give us updated timeline for the denoiser? Also if it will support temporal denoising (for consistent results across frames in an animation?)
     
    wm-VR likes this.
  37. MassiveTchnologies

    MassiveTchnologies

    Joined:
    Jul 5, 2016
    Posts:
    87
    Also, it would be nice if it works with the frame accumelation API (for offline rendering and cinematic motion blur)
     
  38. Pranav_Redstoneinvente

    Pranav_Redstoneinvente

    Joined:
    Apr 13, 2018
    Posts:
    121
    Is there any third part denoiser, maybe from intel or amd? Or how could we make one?
     
  39. Pranav_Redstoneinvente

    Pranav_Redstoneinvente

    Joined:
    Apr 13, 2018
    Posts:
    121
  40. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    You can integrate either Intel Open Image Denoise or Nvidia AI denoiser. I found Intel easier as there is a DLL you can make a wrapper around. For NVIDIA, you would need to first create a DLL and then a wrapper. The Intel DLL is actually already in the unity editor install path, so you only need to create a wrapper. I tried this one and it works great in Unity:

    https://github.com/teamclouday/PathTracer/blob/main/Assets/Scripts/Denoise.cs

    You can use the accumulation APIs or you can create your own simply updating to collect all samples and using the new API renderPipeline.IsFrameCompleted().
     
    wm-VR, Cascho01 and chadfranklin47 like this.
  41. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Posts:
    1,347
    If it´s that easy it should be definitely in one of the next Unity-Versions. o_O
     
  42. Pranav_Redstoneinvente

    Pranav_Redstoneinvente

    Joined:
    Apr 13, 2018
    Posts:
    121
  43. chadfranklin47

    chadfranklin47

    Joined:
    Aug 11, 2015
    Posts:
    220
    Do you have a way to perform AA on the AOVs?
     
  44. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    not really, I'm using the AOVs as they are outputted by unity, but images look pretty good IMO using Open Image denoise.
     
  45. chadfranklin47

    chadfranklin47

    Joined:
    Aug 11, 2015
    Posts:
    220
    I'm seeing that the un-antialiased AOVs are introducing the jagged edges back into the final denoised output, so I'm just leaving off the AOVs for now.
     
  46. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,133
    Anyone knows why some meshes don't show up when using the pathtracer in 2021.2? Even the HDRP template has this issue, most meshes don't show up at all.

    I did a test with 7 meshes, all the same material, and are cubes extended to create a room but only 2 show up. Any ideas?
     
  47. Pranav_Redstoneinvente

    Pranav_Redstoneinvente

    Joined:
    Apr 13, 2018
    Posts:
    121
    Are they LOD?
    If yes then this is the problem
    Else, try restarting unity or try reactivating cameras
     
  48. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,133
    Nope, they're just gameobject cubes.
    The problem happens when I press play and go in-game. In the scene it renders correctly.
     
  49. Carl201010

    Carl201010

    Joined:
    Jan 21, 2016
    Posts:
    16
    Just going to make the case for a denoiser being given a higher priority if it hasn't been already. I'm using Unity / HDRP as a preferred method for producing fairly large amounts of video content for commercial clients. I use Unity for a whole bunch of different reasons, not the least of which is I just plain like it. It also fits well with my workflows and the renders you get from HDRP can be outstanding. I try to avoid tools such as TwinMotion because they have almost zero animation capabilities. Yeah, there's Unreal but project sizes end up being very large and Unity just feels a bit more flexible to me somehow.
    Path tracing is becoming an increasingly big deal for photorealistic renders though. Some recent tutorial videos of TM's path-tracer also happened, almost incidentally, to show just how effective denoising is in improving image quality and reducing production time, which is at the end of the day the primary reason for using a game engine as opposed to offline renderers. I guess basically I want it all - quality, speed, ease of use, flexibility, marmalade even. Denoising seems like an important component in making this happen.
    Just saying folks, but either way - keep up the good work, it's all appreciated.
     
    PutridEx and chadfranklin47 like this.
  50. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,133
    Denoiser is actually in progress: [HDRP] Path tracer denoiser by pmavridis · Pull Request #6652 · Unity-Technologies/Graphics (github.com)