Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Is this amount of TAA ghosting on a skinned renderer expected?

Discussion in 'High Definition Render Pipeline' started by PutridEx, May 11, 2022.

  1. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    Case 1426984
    Default blending TAA value:



    I'm almost positive now there's a motion vectors issue. There's no way HDRP's TAA causes this much ghosting.
    Not sure where the issue is though, skinned motion vectors is ticked. Maybe it's related to the camera controller somehow, or HDRP's motion vectors have issues with skinned MV.
    Or it's related to the terrain? not sure..
     
    Last edited: May 11, 2022
  2. EricFFG

    EricFFG

    Joined:
    May 10, 2021
    Posts:
    183
    Did you maybe turn off motion vectors somewhere?
     
  3. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    I don't think, didn't touch any HDRP settings, and motion vectors show up when using the debugger and moving the camera
     
  4. Win3xploder

    Win3xploder

    Joined:
    Dec 5, 2014
    Posts:
    159
    no it's just that bad
     
  5. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    851
    I guess you also switched "skinned motion vectors" on already on SMR? Some kind of ghosting will always be there, especially on high contrasts. I try to avoid this by using darker colors for scene objects which get often overlayed by moving objects.
     
  6. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    Yeah, skinned motion vectors is ticked
    I mean I understand some ghosting, but this is pretty severe. People hate TAA when it has that much ghosting, and honestly I can't blame them. And with HDRP you kind of need to use TAA, unlike URP/built-in.

    If this is considered expected behavior, I'll be hoping for further improvements to TAA because this will be immediately noticed by players. It's not a tiny amount of ghosting that you need to zoom in or slow things down to notice. It's very noticeable at normal speed of turning the character or moving the mouse. I believe jumping as well.

    And this is with the minimum amount of TAA blending, default values for blending would make it more noticeable.
     
  7. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    default blending value:


    I'm almost positive now there's a motion vectors issue. There's no way HDRP's TAA causes this much ghosting.
    Not sure where the issue is though, skinned motion vectors is ticked and . Maybe it's related to the camera controller somehow, or HDRP's motion vectors have issues with skinned MV.
     
  8. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Try without the terrain. Does it still ghost? Use debug motion vector view so you can see the motion vectors.
     
  9. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    Yeah considered that so I tried it with a cube, using the same soil texture, ghosting was still there with the cube in the background
    I did use the motion vector debugger and motion vectors seem to be written, not sure if they're correct though
     
  10. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    You sure? Because I don't see bad motion vectors over the sky or orange cube.
     
  11. francescoc_unity

    francescoc_unity

    Unity Technologies

    Joined:
    Sep 19, 2018
    Posts:
    193
    What are your settings on the camera? Also it's hard to tell what could be wrong in this situation unfortunately.

    In theory skinned meshes should write proper motion vectors.
     
  12. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136



    camera:



    Unity_ySsGVpLr3K.png

    Unity_o8J9ODKdpY.png

    You can download the project from the bug # report
     
  13. francescoc_unity

    francescoc_unity

    Unity Technologies

    Joined:
    Sep 19, 2018
    Posts:
    193
    You can try decreasing the anti flickering too and play with the blend factor - I can't look at the bug right, but it'll fall under our eyes hopefully soon :)
     
    Last edited: May 11, 2022
  14. Deleted User

    Deleted User

    Guest

    Are u using motion blur??? I think i also had this issue but after disabling motion blur it fixed it!?
     
  15. olavrv

    olavrv

    Joined:
    May 26, 2015
    Posts:
    515
    I tried to add some seagull flock animation, but got issues with skinned mesh renderers and DLSS due to MV's I think.
     
  16. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    I actually did play with blending/flickering settings, and originally I posted a gif with the minimum amount of blending possible, unfortunately the ghosting was still very apparent.
    Nope, motion blur disabled in default global settings & scene volume
     
  17. jjejj87

    jjejj87

    Joined:
    Feb 2, 2013
    Posts:
    1,115
    I've been saying this for a long time.

    Unity's motion vector will break randomly, and not work.

    This is only noticeable when TAA is on.

    I've been saying this for like 5 years now.

    It starts to paint the screen when the color contrast is high.

    If the object is near camera, it smudges.

    If the material uses triplanar, it smudges.

    Vertex animated objects with Motion Vector? -> it smudges.

    And then restart PC -> fixed.

    I've given up on TAA long time ago.

    Solution: Don't use TAA, turn on DLSS and let AI fix it.

    Funny thing is, even if you disable motion vector for vertex animated materials, it will fix it to some degree.

    Love nVidia for this.
     
  18. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    Thing is you have to use TAA, not everyone has an RTX card.
    I really don't want to go back to SMAA/FXAA, it's 2022 and we're doing PBR games. Flickering shouldn't be a thing.

    Hopefully this is sorted. Maybe TAA will receive improvements in 2022.
    And then, there's the issue of terrain grass and trees having no motion vectors.. (anything instanced loses MV)
     
  19. Onigiri

    Onigiri

    Joined:
    Aug 10, 2014
    Posts:
    469
    Is this even possible to fix? Even UE has TAA ghosting everywhere(this is why i cant watch any UE related videos on youtube, my eyes start bleeding when i see how badly everything smearing in UE editor)
     
  20. Deleted User

    Deleted User

    Guest

    But it's definitely more better than unity!!
     
  21. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    Very possible. Many games have amazing TAA implementation that even do well on small meshes (like wires/fences).
    A good TAA implementation is amazing, and should be a top-priority for all 3D PBR engines.

    It will literally be on-screen all the time, effecting everything. One of the most important things to get right.
    UE4 TAA is also not that good, but it's fairly stable and works. Skinned charcters are fine with it, almost no ghosting.
    I did many tests with it, and played many UE4 games that use TAA.
    UE4 (don't know about 5) has the same grass issue as unity though, instanced grass loses motion vectors. Which leads to ghosting/blurriness.

    In UE5 They introduced TSR, mainly for upscaling, but it also does anti-aliasing, it's basically better TAA compared to UE4 TAA, but a bit more expensive.

    They also introduced TAA gen5 in 4.26 (or near that version), I believe it's similar to old TAA but with some adjustments, it does a slightly better job when it comes to flickering/small meshes and blur for most projects.

    Unlike unity, unreal only has TAA and FXAA, so you're basically forced to use TAA.
    Hence, almost every unreal game out there uses TAA. It does have it's fair share of issues though, prior to UE5 at least.
    And, Unreal engine for whatever reason has really bad aliasing when TAA is disabled, worse than you'd normally expect, they seem to rely on TAA a lot.

    Many HDRP titles so far only use SMAA, possibly because TAA wasn't in a good situation prior to 2021. (all games I checked are on 2020 or older)

    Hopefully this bug gets looked at soon.
     
    Last edited: May 15, 2022
    Deleted User likes this.
  22. Deleted User

    Deleted User

    Guest

  23. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    I don't think this project fits his comment, it's a 3rd person and not really fast moving.
    He was more talking about first person, very fast moving cameras.

    In the gif, even a jump leads to noticeable ghosting.
    And as you can see, the camera was moving very slowly at the start, unfortunately ghosting is still there. I believe it's possibly a motion vector issue, maybe something wrong with the character.

    Also while exploring in this project, I found a (possibly, could be wrong) bug with speed rejection and sent it to unity, bug case # 426305
     
    Last edited: May 15, 2022
  24. chap-unity

    chap-unity

    Unity Technologies

    Joined:
    Nov 4, 2019
    Posts:
    760
    Thanks for reporting the issue, technically, there's multiple things here about TAA and I'll try to separate them for the sake of clarity.

    First, there's the "popping" issue when you increase speed rejection.
    This indeed shoudn't happen and as far as I'm concerned this is what your #1426305 bug is about.

    Second, keep in mind that TAA settings are hard to set and really depends on what's happening on screen.
    There will be some sort of ghosting in most cases and in your case, the algorithm is having a hard time differentiating between pixel color from the 3rd person character and the background since it can find the same colors in both.
    To be certain, I tried with regular mesh renderers on the side with your project to be sure that nothing on the skinned mesh renderer motion vectors is broken and nothing seems to be. (meaning the static mesh renderer with the same mesh ghosts as much).
    That being said, the whole algorithm can probably be still improved, we can't promise anything but It's tracked as a side task.

    Lastly, there's also probably something fishy happening with TAAU (The dynamic resolution filter) and TAA. If you use TAAU at 100% resolution, you should in theory have the same result but in reality you get a bit less ghosting.
    This is also tracked on our side and most probably something that will be fixed as well and depending on the fix, it might help your case.
     
    PutridEx likes this.
  25. jjejj87

    jjejj87

    Joined:
    Feb 2, 2013
    Posts:
    1,115
    Actually, I think the issue is not that.

    In my experience, objects near and on focus to the camera, seems to be having this issue. For example, in an FPS environment, the player hand/gun does that, but not much with close by environment.

    Also, try the following, triplanar seems to ghost like crazy, I've tried literally same texture but one with normal repeat mapping and the other triplanar and ghosting occurs on triplanar only (probably why it happens with terrain as well in the first post)

    But for some reason, as long as the object is moving in screen space, ghosting is not as severe.

    So, my guess, if the object is moving in world space but not moving on screen space, (so, character on screen, FPS hands and guns) then this ghosting starts to go crazy. The same happens with racing games near tires.

    I know that most racing games with TAA implement a mask layer to handle this, and perhaps Unity can look into a similar approach.

    If you want to confirm my theory, then put a finger on the rotating character's head, and see where ghosting occurs. Ghosting is happening where the head hasn't been. Which means the motion vectors are generated correctly, but the TAA is not taking camera rotation/movement into account. Or if it is, then the code needs revision. TAA is notorious for ghosting but not like this.

    Regardless, TAA needs work.
     
  26. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    107
    HDRP's TAA implementation increases history blending (decreases the current frame's effect on the final output) when there's high color variation in a given area and the previous frame's color at that pixel was dark. This is likely one of the bigger contributors to the smearing in the OP's video (you can see mostly dark colors being smeared there).

    Here's the problem code: https://github.com/Unity-Technologi...g/Shaders/TemporalAntialiasing.hlsl#L699-L704

    In my own project I've been experimenting with changing that function to decrease its impact when there's some motion in the pixel and to be less dependent on the history color's brightness (since that part also causes broken-looking brightness shifts when anything about the history blending % changes), with some success so far.

    With improvements to that and to history rejection and a few other aspects like the area of influence used by TAA (and maybe some sort of material ID rejection system, even if it's limited), HDRP's TAA wouldn't be too far off from other implementations in terms of artifact issues. It's not there yet though.
     
    Last edited: May 22, 2022
    PutridEx likes this.
  27. larsbertram1

    larsbertram1

    Joined:
    Oct 7, 2008
    Posts:
    6,899
    sounds pretty reasonable.
    regarding motionvectors: first video would not really need per object motion vectors as there isn't anything moving but the camera. so looking into the contrast thing absolutely makes sense here.
     
    PutridEx likes this.
  28. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    Was curious how built-in TAA would handle this, same texture:
    (ignore the low FPS, that's just the video)


    As you can see there's no ghosting, almost as if TAA isn't on.
    Built-in TAA also offers a jitter paramater which is great to finetune the amount of TAA, for the video it wasn't changed though
    The same at high speeds & camera movement
     
    Deleted User likes this.
  29. EricFFG

    EricFFG

    Joined:
    May 10, 2021
    Posts:
    183
    Since switching to 2021 the TAA is really broken
    On 2020 TAA worked perfectly fine, now after our switch to latest and 2021 LTS its glitchy
    There is heavy blurring, instability, never wanting to stabilize, outlines when particles are behind the object, and a general strange white fringe on everything if you compare with photoshop.

    I can not see the ghosting effect of OP however on 2021 and this is definitely not normal.




    Bug - 2021.3 - instability and varied bugs (TAA broken / Visual Environment breaks) - Unity Forum

    Since there has been a lot of work on DLSS, FSR and Dynamic resolution + Camera tweaks,
    some things must have broken there
     
    Last edited: May 27, 2022
    Deleted User and newguy123 like this.
  30. larsbertram1

    larsbertram1

    Joined:
    Oct 7, 2008
    Posts:
    6,899
    have you looked into the fps for both hdrp and birp? i guess birp will run at 150-200fps, while hdrp might only run at 60-90?
    assuming that taa will use the last 9 frames for reconstruction these 9 frames would mean way less time = change in birp resulting in less artifacts just do to fps.
     
  31. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    tested with 60 FPS in built-in, same results
    plus, I seriously doubt FPS would be the cause. I can see it happening if I was running at 20-40 FPS, but 60 and over there is very little difference.

    It just needs improvements as chap stated, to be honest I didn't think HDRP's TAA would have such significant issues.
    (Hopefully instanced grass gets motion vectors one day as well :D)
     
    Last edited: May 28, 2022
  32. Deleted User

    Deleted User

    Guest

    IDK but from my limited testing yesterday, i was not getting any smudging or blurriness because of TAA on grass(painted with GPU instancing).... I am using TAAU and there is an option in the last part of the materials properties of something called motion vectors for vertex animation which I have ticked... also i am using 2022.2 B13 version
     
  33. francescoc_unity

    francescoc_unity

    Unity Technologies

    Joined:
    Sep 19, 2018
    Posts:
    193
    Some changes are coming soon.

    See this pathological case (high contrast, high speed, noisy background texture) -- NOTE: All side by sides use same settings.



    And this is from the project mentioned above (note frame rate around 40-45fps in there)

    With the sent parameters

    Pushing the anti flickering to 1 (so more ghosting is expected, but can reduce base blend factor here for example)



    No speed rejection used in any of the videos. Note that speed rejection might still be needed with some content, but should be needed way less often than before.

    As an additional set of changes, a tiny bit of better central filtering for the high preset (leads to a bit less blurriness)


    (The video is with very high zooming)

    And added a jitter scale (only smaller than 1) option in case stuff gets too jittery and/or blurry and you are fine with a more aliased result. Again video taken with extreme zoom.




    I have other ideas for TAA improvements, but I don't have much time for it now so I've made a mental dump and will look into them in the future. No promises on when though.

    These upcoming changes should already give you a substantial ghosting reduction (and as a side bonus, it is a teeny tiny bit faster too)
     
    Last edited: Jun 3, 2022
    Rowlan, JoNax97, hippocoder and 12 others like this.
  34. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    107
    Awesome! Is there a publicly available branch of the Graphics repository with these changes we can try, or will we have to wait for it to be merged into the main branch?
     
  35. francescoc_unity

    francescoc_unity

    Unity Technologies

    Joined:
    Sep 19, 2018
    Posts:
    193
    Unfortunately will have to wait a bit, the PR is in the process of being created on the private repo. I can try and remember to ping this thread when a mirror with the change appears.
     
    slime73 and newguy123 like this.
  36. jjejj87

    jjejj87

    Joined:
    Feb 2, 2013
    Posts:
    1,115
    Thanks for the changes, much appreciated, but it is also alarming that the community has been saying this and it took years for someone at Unity to actually try it and see.

    Also, I am concerned that there is no dedicated team for TAA as nowadays TAA or temporal related rendering is pretty much rendering quality itself. Shouldn't there be a dedicated team for this all the time? I mean, Unity isn't a powerhouse rendering engine like UE, and can only handle small scale scenes, but at least Unity should be trying harder to make that better.

    I often get the feeling that Unity is not that much interested in rendering quality. Even if the market is moving on.

    Just my 2 cents.

    Also, I hope that the fix is actually released for 2021 and not for 2022. Please don't say that 2021 is in LTS and further features won't be added...I've had this so many times now and I am afraid...

    My current project is on 2021 and I have been battling to keep TAA in line and now I see why...
     
  37. francescoc_unity

    francescoc_unity

    Unity Technologies

    Joined:
    Sep 19, 2018
    Posts:
    193
    Once it lands with 2022 I'll try to backport it as far back as it is possible. No promises on how far back, but I will try as we understand it is an important fix.

    Regarding staffing, the HDRP team is smaller than you would think :) Resources are limited unfortunately. I of course am not the person that can comment much further on this.
     
    Last edited: Jun 10, 2022
    shikhrr, chap-unity and Qleenie like this.
  38. jjejj87

    jjejj87

    Joined:
    Feb 2, 2013
    Posts:
    1,115
    Thank you for the honest feedback. Really appreciated. Also thank you for being honest about backports. If you need anything from us, please ask. I would love to help given the HDRP team size, now that I hear it. Although I think it is a managerial misstep to not give the HDRP team the resources they deserve. After all, HDRP is spearheading Unity.
     
  39. newguy123

    newguy123

    Joined:
    Aug 22, 2018
    Posts:
    1,248
    Fully agree!
    I coudnt care less about URP or built-in
     
    PutridEx likes this.
  40. francescoc_unity

    francescoc_unity

    Unity Technologies

    Joined:
    Sep 19, 2018
    Posts:
    193
    The fixes I mentioned above have now landed in our latest version (not yet released, but will be in the next 23.1 alpha) and I did manually backport up until 21.3 (22.2, 22.1, 21.3) . Hopefully the backports will also land soon.


    Unfortunately backporting to 20.3 is fairly complex so I don't think it'll happen
     
  41. nehvaleem

    nehvaleem

    Joined:
    Dec 13, 2012
    Posts:
    429
    Awesome, both for the improvements and for the backports (especially for the 21.3!). Can't wait to test it out, maybe I'll give a TAA another shot.
     
  42. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    It's true but in the movies, a small group of heroes saves the day. I think you guys all saved my eyes at least <3
     
    Lex4art likes this.
  43. nehvaleem

    nehvaleem

    Joined:
    Dec 13, 2012
    Posts:
    429
    I don't want to be that guy that constantly asks "when", but since those are improvements that I am particularly excited about I have to ask it at least just once. Is there any eta about the backports landing (especially in 21.3)?
     
  44. francescoc_unity

    francescoc_unity

    Unity Technologies

    Joined:
    Sep 19, 2018
    Posts:
    193

    Hey, sorry but I don't know at the moment the release schedule, I will try and check tomorrow, however I know that it has landed internally and should be in 2021.3.6f1 (or bigger) whenever that releases
     
    shikhrr, jiraphatK and nehvaleem like this.
  45. newguy123

    newguy123

    Joined:
    Aug 22, 2018
    Posts:
    1,248
    @francescoc_unity are these changes already in the latest 2022.2 beta and 2023 alpha?
     
  46. francescoc_unity

    francescoc_unity

    Unity Technologies

    Joined:
    Sep 19, 2018
    Posts:
    193
    2023.1.0a2 and 2022.2.0b1 should contain it
     
    newguy123 likes this.
  47. seoyeon222222

    seoyeon222222

    Joined:
    Nov 18, 2020
    Posts:
    176

    https://unity3d.com/kr/unity/whats-new/2021.3.6

    Is it Updated?
     
  48. chap-unity

    chap-unity

    Unity Technologies

    Joined:
    Nov 4, 2019
    Posts:
    760
    The what's new won't include that information but the improvement have indeed landed in that version 2021.3.6f1
     
    jiraphatK and PutridEx like this.
  49. seoyeon222222

    seoyeon222222

    Joined:
    Nov 18, 2020
    Posts:
    176
    Thank you for your answer
    I'll have to update it and test it
     
  50. Rowlan

    Rowlan

    Joined:
    Aug 4, 2016
    Posts:
    4,200
    Awesome. Had to try it. It's been annoying me for quite some time:



    Quite visible with the right wing, ie the one that has the wall in the background.

    Great job, thank you!
     
    chap-unity and PutridEx like this.