Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Official Question from the team: Accumulation Motion Blur

Discussion in 'Virtual Production' started by marc_tanenbaum, Nov 30, 2021.

  1. marc_tanenbaum

    marc_tanenbaum

    Unity Technologies

    Joined:
    Oct 22, 2014
    Posts:
    637
    Hi folks!

    The Recorder team would like to get your take on how/if you use Accumulation with HDRP and the Recorder. Accumulation uses multiple sub-frames to converge over time on a higher quality final image.
    1. Do you use Accumulation today?
    2. If so, how well does it meet your needs?
    3. If not, is it because:
      1. It lacks features you need?
      2. It’s not performant enough?
      3. It’s too hard to use?
      4. You didn’t know about it?
    4. What features does Accumulation lack or need to do better?
    Your feedback will help drive our roadmap…so please be vocal and as blunt as necessary to let us know how we can improve.

    P.S. If we get high quality feedback using this approach, we'll likely ask on this forum more regularly.

    Thanks!
     
    PutridEx, vladala and antoinecharton like this.
  2. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    588
    Love that you are asking the question so I will reply (even though low quality reply).

    1. I don't use Accumulation today - I don't use HDRP at this stage because I wanted to master the basics first and I had problems with some tools and Unity asset store purchases trying to use HDRP. So I have stuck to the default render pipeline for now.
    3. I did not know about Accumulation.


    For me (off topic) it is show stopper bugs I would address before adding new features. Example (based on the world revolving around ME! ME! ME!), I would love to use Animation Rigging and Sequences together. E.g. do a live recording, but then override it so hand reaches out to grip door handle (hard to line up in a live recording). But they throw exceptions. (Bug has been registered.) So the feature is completely unusable.

    Or another one is Recorder with MP4 generates very low quality video files if the scene gets too complicated. You cannot tell Unity to "create good quality video no matter what". Fancy tools to make the creation better are not useful if I cannot generate a good quality video file when done (for Film making).

    Once the basics work reliably, then I am interested in fancy features (e.g. richer depth of field blur etc). (But I appreciate you may have different internal teams working in parallel, so Accumulation might be the roadmap for a different team. I just have no experience with that. Reading docs it looks fairly advanced, so no interest to me until I am convinced the basics are rock solid and my skills improve.)
     
  3. yokenstein

    yokenstein

    Joined:
    Sep 19, 2017
    Posts:
    84
    I don't use it because I don't have an NVidia graphics card to support path tracing within Unity
     
  4. vladala

    vladala

    Unity Technologies

    Joined:
    Mar 3, 2017
    Posts:
    189
    Thanks for the reply.
    There are 2 features that are very close: dxr that needs directX and nvidia, and accumulation without any special hardware requirements, that allows for very slow but pretty rendering. Both are supported for HDRP.
     
  5. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    1. I did not, although i knew about for a while and i only use HDRP for the most part,
    2. -
    3. I'd say (3), not exactly though, looking at the doc its pretty easy to use, you did all the work already.
    But it doesn't come with HDRP, as in, I don't just click a few buttons in editor and its on, or add a component that comes with HDRP. Although if/when I need it bad enough, I'll use it, it's not a deal breaker, but I probably would've already used it if it came with HDRP. I realize my laziness here but this is my honest guess as to why I haven't used it so far :D

    --
    Can't give feedback on it feature/quality wise as I haven't used it so far.
     
    Last edited: Dec 4, 2021
  6. yokenstein

    yokenstein

    Joined:
    Sep 19, 2017
    Posts:
    84
    I see. From what I understand from the article, it combines physically based motion blur + path tracing and other post processing effects to create high quality renders. Last I checked, I can't access path tracing (from the volume settings) because I don't have an NVidia card. For physically based motion blur, i'm already using Deckard Render from the asset store. It also cleans up other post processing effects like SSR and is fast, but doesn't have path tracing . What I want from Unity is a path tracing solution that supports AMD graphics cards. Considering making the switch to Blender because they have solutions for AMD.
     
    Last edited: Dec 5, 2021
  7. yokenstein

    yokenstein

    Joined:
    Sep 19, 2017
    Posts:
    84
    Btw, out of curiosity, I tried out Accumulation with HDRP in the Recorder, set it at 128 samples, and it was very very slow, and the results weren't that good. Could still see significant aliasing and motion blur was just okay. Deckard Render from the asset store is blazing fast and the results are very high quality. So, maybe you guys could work on the speed of the Accumulation feature
     
    olix4242 likes this.
  8. antoinecharton

    antoinecharton

    Unity Technologies

    Joined:
    Jul 22, 2020
    Posts:
    197
    Still interesting feedback to know if people used it or not. :)

    You don't need an NVIDIA card to use accumulation. It combines path tracing when it's enabled but it's not a requirement.

    If you have some result you could share feel free to do so. It shouldn't be this bad with 128 samples. There might be some issue here (Maybe open an other thread).

    It's a feature that's intended for quality more than real time. Motion blur might be more adapted in that case.
     
  9. hugokostic

    hugokostic

    Joined:
    Sep 23, 2017
    Posts:
    84
    I am using recorder actually for an HDRP project that mean to be a video, but I didn't try the accumulation method because i didn't know it exist (until now), but I will give a try asap for sure!
    It could have a gorgeous effect on very organic scenes.
     
  10. Enigma229

    Enigma229

    Joined:
    Aug 6, 2019
    Posts:
    136
    I use it in HDRP for cinematics that I render out to an image sequence and then take into After Effects.

    I am not happy with the motion blur in HDRP. I find it hard to master and always get ghosting and other artifacts.

    I seem to remember it was a lot easier to use and had good results in the SRP.
     
  11. yokenstein

    yokenstein

    Joined:
    Sep 19, 2017
    Posts:
    84
    Use Deckard Render (see asset store) for motion blur and DOF that is physically modeled
     
  12. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    We tried accumulation/recorder to render timeline sequences with Path Tracing but found it not too performant compared to manually accumulating samples. We had to tweak a bit of accumulation source to enable manual accumulation while managing the timeline manually but found our solution to be ~4x faster than using Recorder/Accumulation. On our tests, a frame that took 41 sec using the method above took 8.5s manually managing everything. We had to implement the motion blur based on the accumulation technique by manually advancing the timeline at sub -rame speed, but were able to set custom shutter speeds, and on top, we integrated Open Image Denosie to get clean frames, which is also not possible with recorder ATM.,

    So to the question:
    1. Lacks Denoising
    2. It's not performant, at least in our use case.

    We still don't understand where the raw speed of the HDRP is being throttled, but it would be awesome to allow users to manually accumulate samples or get to the root cause. Our project is still unreleased but if UT is interested, I can share some videos and source files.
     
    PutridEx likes this.
  13. yokenstein

    yokenstein

    Joined:
    Sep 19, 2017
    Posts:
    84
    If you release that as an asset, I would be happy to test it out, whether paid or free
     
  14. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    Unfortunately, it's the property of the gaming company we work for, plus it was very custom-made for our use case, but I'm super puzzled as to why the Recorder/Accumulation path is much slower than it should (at least for us).
     
    PutridEx likes this.
  15. yokenstein

    yokenstein

    Joined:
    Sep 19, 2017
    Posts:
    84
    It is the same case with me, it's very slow with the recorder. + I noticed that the samples don't accumulate "intelligently" in darkly lit areas. Unity could use some kinda AI powered sampler or something to make it production worthy
     
  16. vladala

    vladala

    Unity Technologies

    Joined:
    Mar 3, 2017
    Posts:
    189
    Thanks so much for the answer.
    Are you using the public HDRP accumulation API or are ticking a timeline manually at a crazy small Dt.
    I presume you get the higher performance since you need less samples because you de-noise the resulting images ?
    If so, what's the typical number of samples you need ?


     
  17. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    I tried the first approach but it was somehow slow, so I added a flag to avoid the auto clean calls and are moving the timeline ourselves. Our tests are based on the same number of samples (360).
     
    antoinecharton and vladala like this.
  18. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    One thing we found, which might be related (we have not tested recently, was that path tracing was actually capped when vSync was active to 60 samples per second. I understand that vsync should cap FPS, but seems to be capping Path tracing sampling since a frame is in effect a sample. Not sure if this was s=affecting our tests with Recorder, back then, but the main reason we walked away from recorder/accumulation was the fact we needed to create clients that could actually process the frames for us in runtime, and Recorder is Editor bound.
     
    antoinecharton likes this.
  19. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    Hi. Wanted to share that after querying the premium support team, we found the culprit why accumulation was slower than the manual override we added to the HDRPsource ourselves. Apparently, there is a property called
    Time.captureDeltaTime
    that is set to 0 by default which makes the physics system go nuts generating significant load on the CPU.

    I set the variable using the reciprocal property
    Time.captureFramerate
    and now the accumulation API runs as fast as my manual method, although it limits a bit on what we do, as it can only move forward by calling the
    renderPipeline.PrepareNewSubFrame();
    method. Having a manual accumulation option allows devs to decide what to draw on every sample, for example in our use case, move manually across a timeline. This freedom allows us, for example, to calculate the motion blur as nth of a second and even use blur lengths over that of a frame if we wanted by setting for example 1/12s on a 24fps clip.

    For anyone wanting to give this route a go, to achieve manual accumulation we simply added a flag in the PathTracing.cs file as follows:

    Code (CSharp):
    1.         public bool ManualAccumulation = false;
    2.  
    3.         TextureHandle RenderPathTracing(RenderGraph renderGraph, HDCamera hdCamera, TextureHandle colorBuffer)
    4.         {
    5.  
    6. ...
    7.  
    8.             if (!m_SubFrameManager.isRecording && !ManualAccumulation)
    9.             {
    10.                 // Check if things have changed and if we need to restart the accumulation
    11.                 camData = CheckDirtiness(hdCamera, camID, camData);
    12.  
    13.                 // If we are recording, the max iteration is set/overridden by the subframe manager, otherwise we read it from the path tracing volume
    14.                 m_SubFrameManager.subFrameCount = (uint)m_PathTracingSettings.maximumSamples.value;
    15.             }
    16.  
    17. ...
    18.  
    19.         }
    To use it, we simply set
    renderPipeline.ManualAccumulation = true;
    to start accumulation and
    renderPipeline.ManualAccumulation = false;
    when we are done with the frame, which essentially is triggered by
    renderPipeline.IsFrameCompleted
    .

    It would be really wonderful to have such flag available within the official version versions.
     
    Last edited: Feb 3, 2022
    vladala, PutridEx and marc_tanenbaum like this.
  20. vladala

    vladala

    Unity Technologies

    Joined:
    Mar 3, 2017
    Posts:
    189
    We use captureDeltaTime extensively in the recorder. This ensures that the engine is ticked in "frames" rather than forcing it to run at unpredictable framerates. The fact that setting it to 0 makes physics go haywire is odd, especially since that is the default unity way of running.

    Good to know more details about your use case.
     
    alejobrainz likes this.
  21. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    Sure. We basically record mp4 sequences off timelines, but we do it in standalone clients. Few reasons for that, but essentially since we are doing thousands of permutable characters, doing it on the editor was not viable. Also, we needed unity to deliver finished assets, and the recorder's mp4 quality, even at high preset was really poor. On top of that, we needed denoised images, and Unity's denoiser for Path Tracing was still under development. So we ended up writing our own code to render off the timeline to mp4, denoising frames in the process.

    In that process we found that the accumulation API was running slower, and ended up adding the manual flag on the HDRP source, which we just now found was due to the captureDeltaTime flag. The reasoning on CPU load and physics was actually told to us by a member of the raytracing team via premium support. I can point you to the premium support thread if you PM me.

    Anyway, since we were doing our own accumulation, we needed to supplement the motion blur bit, so we made our own implementation that can take samples, even out of the bounds of the current frame. This allowed us to get extra blur if we wanted to simulate ridiculously slow shutter speeds. Here are a few examples (all rendered at 256 samples and denoised using OIDN with a custom implementation of the OIDN DLL, that is actually shipped with the editor):

    No Blur:
    no-mb.jpg

    1/24th of a second (kind of similar to using (0,1) in the shutter begin end params in the accumulation API:
    24th.jpg

    1/8th s, not easily viable using the accumulation API, unless you tamper with the animation speed or playback time:
    8nth.jpg

    1s shutter, again not viable with the accumulation API by itself:
    1nth.jpg

    Again, this was a byproduct of hitting a wall with the rendering speeds, due to the lack of documentation in the accumulation API examples, but it does provide enormous power having the ability to manually override the accumulation/check dirtiness process.

    Also, we found that vsync will clamp the rendering speeds to whatever Hz your client is running on. In our case it clamped to 60 samples per second, so you should also document that accumulation speed will be affected by that.

    Cheers,

    Alejandro.
     
    Last edited: Feb 3, 2022
    antoinecharton likes this.
  22. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    Also to note, all examples above took the same time to render, just changed the time within the timeline from which samples are taken.
     
  23. yokenstein

    yokenstein

    Joined:
    Sep 19, 2017
    Posts:
    84
    @alejobrainz Why don't you try out Deckard Render from the asset store?
    Pros:
    -Fast. Faster than any path tracing engine in existence
    -Noiseless
    -Physically accurate motion blur and DOF
    -Capability to simulate soft shadows
    -Amazing anti aliasing options, without temporal and spatial aliasing issues
    Cons:
    -Doesn't have any in built physically accurate lighting options like path tracers. So combine with real time Raytraced AO + GI + Reflections + Shadows and you're good to go.
     
  24. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    Thanks for the heads up. TBH, The Unity Path tracer exceeded our expectations. It's still rough around the edges but allowed us to create CGI quality images at blazing speed, orders of magnitude faster than path traced counterparts in Max, Maya, or Blender. It's far from perfect, but It's got enormous potential. We have reported to the unity team several crash bugs, and visual issues but see it coming together quite nicely. In our exploration phase, we toyed around with otoy octane but it was editor bound and not flexible enough for our use case. didn't get around to try Deckard, but HDRP Path tracing + Denoising allowed us to generate thousands of animations in record time.
     
    PutridEx likes this.
  25. yokenstein

    yokenstein

    Joined:
    Sep 19, 2017
    Posts:
    84
    What's the render time looking like, per frame? And on what hardware?
     
  26. PavlosM

    PavlosM

    Unity Technologies

    Joined:
    Oct 8, 2019
    Posts:
    31
    Thanks for the feedback. I just want to clarify that this is something that we could fully implement with the existing API, without a "manual override". In your example, you could get a 1/12s "shutter speed" [*] on a 24fps clip if you set a shutter interval of 2 (in the API, the shutter interval is relative to the frame duration, as mentioned in the doc). However, to keep things as efficient as possible, the existing implementation allows values only up to 1 (no frame overlap), but it's something that we could add in the future.

    The idea about a manual override is something orthogonal to this, and I can see that the extra flexibility can be valuable in some cases.

    [*] I use the term "shutter speed" here because this is how it's used in photography, but from a physics perspective this is a time interval (since it's measured in seconds), and that's why the API calls it an "interval", and not "speed" as a photographer might expect.
     
    Last edited: Feb 4, 2022
    alejobrainz likes this.
  27. Enigma229

    Enigma229

    Joined:
    Aug 6, 2019
    Posts:
    136
    I have a i9 and 3070 GPU and it takes around 10 minutes to render out a 5-10 second image sequence.
     
    yokenstein and alejobrainz like this.
  28. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    288
    In our case, we benchmarked a sequence on Vray that took around 60 minutes to render, and we matched the lighting, which consisted of around 5 Area Lights, including indirect light with around 6 bounces, raytraced DOF (available when using the physical camera), plenty of Subsurface scattering, reflections, iridescent translucency effects, no screen space effects but path traced, plus motion blur to 6 minutes on an AMD 3900 and an RTX 2070 Super. We rendered at 360 samples and used the OIDN denoiser, and TBH both were pretty comparable results at 10x speeds. Maybe on heavily indirect lit scenes results may differ, but for our use case is pretty fine.

    Also, since we needed to render thousands of sequences, we were able to deploy a client to several NVIDIA T4 GPU-powered servers on AWS, which was pretty sweet too.

    So I do think that Unity Path Tracing has tons of potential as an offline renderer.
     
    yokenstein and PavlosM like this.
  29. chmodseven

    chmodseven

    Joined:
    Jul 20, 2012
    Posts:
    120
    I've tried using Accumulation today and for some reason I can't figure out it is bailing out early and not completing the number of interval seconds for the clip. No error messages or anything, it just quits after a small number of frames, even at really low samples. These are the settings I'm using. Any ideas?

    upload_2022-2-7_8-51-14.png
     
    alejobrainz likes this.
  30. Enigma229

    Enigma229

    Joined:
    Aug 6, 2019
    Posts:
    136
    You might want to try rendering out and image sequence and then converting to an mp4.

    Rendering out as image sequences will make Unity render per frame so there are no performance issues.
     
  31. cguertin

    cguertin

    Unity Technologies

    Joined:
    Aug 20, 2019
    Posts:
    83
    Hey @chmodseven, thanks for reporting this! It looks like you stumbled on bug. I have logged it into our system for fixing.

    This is definitely not the expected behavior but the issue here happens specifically because:
    • Accumulation is enabled
    • The "Recording Mode" is set to "Time Interval (Sec)"
    Until this is fixed, I would suggest changing the "Recording Mode" to "Frame Interval" instead. It's definitely not ideal for your use case but multiplying your range in seconds by the frame rate should give you the same range (1 to 2 seconds @24fps == 24 to 48 frames)

    Let us know if you are still having issues with these changes. Cheers!
     
    alejobrainz likes this.
  32. chmodseven

    chmodseven

    Joined:
    Jul 20, 2012
    Posts:
    120
    Cheers, thanks for the update and workaround.
     
  33. avnphong3bao8

    avnphong3bao8

    Joined:
    Feb 5, 2019
    Posts:
    4
    when using recorder with 1 sample, final image result is kinda fine. But with samples > 2, my ABC and FBX files in the timeline got desynced, even with v-sync on. That means my FBX file sometime faster or slower 1 frame (exactly 1 frame) than ABC file
     
  34. vladala

    vladala

    Unity Technologies

    Joined:
    Mar 3, 2017
    Posts:
    189
    Hey,

    I'd need more info to diagnose the problem. Please start a new thread.
    Mind telling me the recorder and alembic versions ?
    Is the FBX animation funky in any way? Constant keys, etc ? I presume you drive your FBX models with animation clips.
     
  35. avnphong3bao8

    avnphong3bao8

    Joined:
    Feb 5, 2019
    Posts:
    4
  36. eliphat

    eliphat

    Joined:
    Jul 4, 2017
    Posts:
    48
    Sorry to bump the thread.
    A critical feature preventing me to use the feature i think is to use accumulation within timeline.
    Using recorder leads to some 50ms-100ms out of sync between the recording and the timeline, and it is just not that convenient within a timeline-centric production workflow.
    I actually had a post processing effect for accumulation written myself, and hacked the recorder package to add an option inside the timeline to write only 1 frame every k frames :)
     
  37. cguertin

    cguertin

    Unity Technologies

    Joined:
    Aug 20, 2019
    Posts:
    83
    Hey @eliphat, thank you for reporting this! It does sound like a potential bug, could you please start a new thread and include the necessary information to reproduce the problem you are describing? Just add the tag
    recorder
    to the thread and our team will see it.

    Alternatively, you can submit a bug directly inside Unity using these instructions.
     
  38. eliphat

    eliphat

    Joined:
    Jul 4, 2017
    Posts:
    48
    cguertin likes this.
  39. jbergs

    jbergs

    Joined:
    Feb 13, 2014
    Posts:
    26
    Does Accumulation in Recorder not work with the Physical Camera? When recording It reverts back to the normal camera. When I turn Accumulation off the Physical Camera renders fine. Is this a bug?
     
  40. marc_tanenbaum

    marc_tanenbaum

    Unity Technologies

    Joined:
    Oct 22, 2014
    Posts:
    637
    Physical Camera should work fine. I've definitely used it myself. Let me check with the team to see if anyone can offer some advice. In the meantime, can you give us details (platform, Unity version, Recorder version, render pipeline, and anything else you think might be informative).
     
  41. jbergs

    jbergs

    Joined:
    Feb 13, 2014
    Posts:
    26
    Sure..
    - Unity 2022.2.12f1
    - Mac Ventura 13.2.1 on M1 Max
    - HDRP
    - Recorder 4.0.0

    I think I might have narrowed it down to something. It seems to be when I animate a property of the camera in the timeline, in my case Rotation, it no longer records the camera as a Physical Camera. But if the camera has no animation on it, or it's Animation timeline track is muted, in records fine.
     
  42. marc_tanenbaum

    marc_tanenbaum

    Unity Technologies

    Joined:
    Oct 22, 2014
    Posts:
    637
    Update: our engineers are looking at this and may have found a repro case. One of them should be getting back to you soon.
     
  43. jbergs

    jbergs

    Joined:
    Feb 13, 2014
    Posts:
    26
    Thank you! This is definitely important for us and I'm not quite sure of a work around. I know as far as rotation I could add a parent game object to the camera and animate that, but we're also animating Field of View and Focal Length of the camera. I'm not sure of a work around for that.
     
  44. rooose

    rooose

    Unity Technologies

    Joined:
    Jun 8, 2022
    Posts:
    30
    Hi! We found the root of the problem. There is no workaround, but I created a public issue link if you want to stay updated on the progress!
     
    jbergs likes this.
  45. jbergs

    jbergs

    Joined:
    Feb 13, 2014
    Posts:
    26
    I also wrote this in the issue tracker, but it doesn't seem to be just the physical camera. If you use Accumulation on the Non-Physical camera and you have a property of that camera animated, for example field-of-view. The recorder still ignores the animation.
     
    rooose likes this.
  46. AquaGhost

    AquaGhost

    Joined:
    Jan 27, 2013
    Posts:
    94
    @rooose we got a script we have been using when we brought up this bug a while ago, given to me by the unity folks.
    we use cinemachine, so we make sure we have the MainCamera tagged on the actual non CM cam.
    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4.  
    5. public class ForcePhysicalCam : MonoBehaviour
    6. {
    7.  
    8.     void LateUpdate()
    9.     {
    10.         Camera.main.usePhysicalProperties = true;
    11.     }
    12. }
     
    rooose and jbergs like this.