Search Unity

  1. The 2022.1 beta is now available for testing. To find out what's new, have a look at our 2022.1 beta blog post.
    Dismiss Notice

Motion vectors are coming in beta 16

Discussion in '5.4 Beta' started by elbows, Apr 21, 2016.

  1. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,495
    I hope Tim C doesn't mind me posting about this now but since his messages on the subject (and its use to make a TAA effect) are in the Unity Cinematic Effects thread I thought some people may not have heard the news....

    Tim's post about this

    Exciting times :)
     
    Peter77 likes this.
  2. freekstorm

    freekstorm

    Joined:
    Nov 11, 2010
    Posts:
    86
    I realise that some people are going to be excited about this. But I for one would rather they fixed the existing stuff and made it useful again, before adding more stuff into it.

    Fix the current bug list, then add some more features. It a tried and proven way to build solid code.
     
  3. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,495
    I think they already did enough to respond to that by changing the release schedule, opening up the beta and concentrating on stability in 5.3.4.

    I know it won't satisfy people who are badly bitten by particular bugs right now, but they have to balance these two different things, and plenty of people would moan if they added no new stuff at all. Motion vectors enable some much-needed functionality and I applaud their inclusion at this stage.
     
  4. Erind

    Erind

    Joined:
    Jul 23, 2014
    Posts:
    56
  5. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,495
    Yay, here we see the motion vector stuff in the beta 16 release notes:

    • Graphics: Added motion vector support to Unity:
      • Requires RG16 texture support.
      • Motion vectors track the screen space position of an object from one frame to the next and can be used for post process effects.
      • See the API docs for: Renderer.motionVectors, Camera.depthTextureMode, SkinnedMeshRenderer.skinnedMotionVectors, PassType.MotionVectors, and DepthTextureMode.MotionVector.

    I do find it a bit disconcerting or odd that sometimes great new features like this end up in the fixes section of the release notes!
     
  6. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    28,363
    It's fixing unity's erm.... features!
     
  7. Alex-Lian

    Alex-Lian

    Guest

    Thanks for catching that. Will be fixed in final release notes rollup....
    Item was added to the wrong section of our tracking sheet.
     
    elbows likes this.
  8. Erind

    Erind

    Joined:
    Jul 23, 2014
    Posts:
    56
    can we have a vertex color painter integrated in the engine?
     
  9. Alex-Lian

    Alex-Lian

    Guest

    Sorry, beta isn't about requests but testing the features and changes in the offering. If you want to make a request - head towards feedback.unity3d.com
     
  10. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    Hm, I tried making a motion blur effect use the motion vectors. When I just dump the motion vector's R and G channels to the screen in the form float4( vec.r, vec.g, 0, 1 ) it's just pure black. Are the values negative I wonder?

    EDIT: Scratch that. Seems to be related to a separate issue where moving objects via the transform inspector causes hiccups/lag (reported as a bug). If I move via the transform handles with the game and scene view side by side, it seems to work.

    EDIT2: OK, so I've got a simple motion blur effect implemented. Definitely needs work, however it was really easy to set up now that we have native motion vectors XD
     
    Last edited: Apr 27, 2016
    nxrighthere and elbows like this.
  11. Tim-C

    Tim-C

    Unity Technologies

    Joined:
    Feb 6, 2010
    Posts:
    2,163
    You are going to have to post a picture :)
     
  12. K-E-I

    K-E-I

    Joined:
    Oct 6, 2012
    Posts:
    17

    Attached Files:

    Last edited: Apr 28, 2016
    nxrighthere and elbows like this.
  13. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    Well looks like someone beat me to the pictures XD so I'll post a video instead, from one of the projects I'm working on.
    If you pause the video especially at the part where I'm strafe jumping across the large empty space, you can really see the blur effect on the ground.
     
  14. Tim-C

    Tim-C

    Unity Technologies

    Joined:
    Feb 6, 2010
    Posts:
    2,163
    Oh cool :)

    I really miss quake style physics in games :(
     
    JonPQ and spikything like this.
  15. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    Heh, thanks. Thinking about making it an asset for the store. Really kind of a niche asset, but w/e maybe somebody will want it.
    Also, the motion vector buffer relies on RGHalf, right? I notice it doesn't work on WebGL. So in my image effect would this work to check for support?

    Code (csharp):
    1.  
    2. if (!SystemInfo.SupportsRenderTextureFormat(RenderTextureFormat.RGHalf))
    3. {
    4.     enabled = false;
    5.     Debug.LogWarning("Platform does not support motion vector buffers");
    6. }
    7.  
    EDIT: Nevermind. Answered it myself. The answer is yes - I tested using this code on a WebGL build and it does disable the effect as expected.
    Be nice if motion vectors did work on WebGL, but oh well. Guess we'll have to wait until WebGL 2.0 becomes standard...
     
    Last edited: Apr 28, 2016
  16. theprotonfactor

    theprotonfactor

    Joined:
    Oct 10, 2014
    Posts:
    140
    They could theoretically be stuffed into an 8 bit texture but it's not desirable because of the LDR range and lack of precision. If anything I think the resulting effect could be worse than the old motion blur. Just hypothetical thinking...I'd gladly like to be proved wrong :)
     
  17. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    Hm, makes me wonder what the actual range of values are on motion vectors. I get the feeling they used RGHalf because it's a floating point format which makes storing non-unit values a lot easier. I bet you could still pack the values into an RGBA32 with some effort, but at the cost of shader complexity (use pairs of 8-bit channels for encoding a value, so X might be encoded in RG and Y encoded in BA, giving 16 bits for each, but could prove to be more trouble than it's worth? not sure)
     
  18. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    28,363
    Packing would be slower though?
     
  19. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    Seems very likely. Just some idle thoughts I wanted to throw out there.
     
  20. theprotonfactor

    theprotonfactor

    Joined:
    Oct 10, 2014
    Posts:
    140
    Come to think of it, they won't normally go above 1. So I reckon it is about properly storing fractions and signed values.
    It would indeed, but not massively. The real issue is that packing can leave you with an approximation of the original vector. And an approximation of a fraction can be wildly inaccurate. Not really worth it for me ;)
     
  21. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    OK, here's a question. I'm experimenting with writing a custom replacement for the 'UnityObjectToClipPos' function which multiplies the clip-space Z position of a vertex by 0.1 in order to make it render on top of things. It looks like this:

    Code (csharp):
    1.  
    2. #if defined(FIRST_PERSON)
    3.     // Tranforms position from object to homogenous space
    4.     inline float4 UnityObjectToClipPos( in float3 pos )
    5.     {
    6.     #if defined(UNITY_SINGLE_PASS_STEREO) || defined(UNITY_USE_CONCATENATED_MATRICES)
    7.         // More efficient than computing M*VP matrix product
    8.         float4 ret = mul(UNITY_MATRIX_VP, mul(unity_ObjectToWorld, float4(pos, 1.0)));
    9.     #else
    10.         float4 ret = mul(UNITY_MATRIX_MVP, float4(pos, 1.0));
    11.     #endif
    12.  
    13.         ret.z *= 0.1;
    14.         return ret;
    15.     }
    16. #else
    17.     // Tranforms position from object to homogenous space
    18.     inline float4 UnityObjectToClipPos( in float3 pos )
    19.     {
    20.     #if defined(UNITY_SINGLE_PASS_STEREO) || defined(UNITY_USE_CONCATENATED_MATRICES)
    21.         // More efficient than computing M*VP matrix product
    22.         return mul(UNITY_MATRIX_VP, mul(unity_ObjectToWorld, float4(pos, 1.0)));
    23.     #else
    24.         return mul(UNITY_MATRIX_MVP, float4(pos, 1.0));
    25.     #endif
    26.     }
    27. #endif
    28.  
    So if a shader doesn't have the FIRST_PERSON keyword, it uses the original function, otherwise it uses the new version that multiplies the Z position, idea being you can just add "#pragma shader_feature FIRST_PERSON" to a surface shader or even just a plain old vertex/frag shader to add support for first person view to it.
    This works great, EXCEPT that it does not work properly with the motion buffer now. It seems to render some very strange values which result in MASSIVE blur in my motion blur shader.

    Is there something I need to add or something else I need to override in order to explicitly support motion vectors with this?
     
  22. theprotonfactor

    theprotonfactor

    Joined:
    Oct 10, 2014
    Posts:
    140
    I'm guessing it works fine with first person disabled right? If so, it might not using that function to calculate the old position. So it would constantly see it as movement in z...I'm hoping its something like that and the fix won't have to modify any matrices.
     
  23. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    I found the issue, actually. It turns out objects are rendered with a replacement shader to render the motion vectors. This shader doesn't have the FIRST_PERSON define, therefore it uses the other version of UnityObjectToClipPos.

    Unfortunately I've hit a bit of a snag here. Problem is, even if I download the source for that shader, put it in my project, and set it to be my motion vector shader, and I added the #pragma shader_feature FIRST_PERSON" to it, but apparently when motion vectors are rendered it does not respect keywords which were enabled on the object, so it still doesn't use the first person variant of the motion vector shader.

    It does work if I just use "#define FIRST_PERSON", but then it multiplies the entire scene's Z by 0.1 in that shader. Which seems to work(?) but really bugs me.

    EDIT: Turns out I might have been overthinking this entire thing. After some research, apparently in UT4 they just scale the weapon down really, really small. So I just tried it - similar effect (super small so it doesn't penetrate any scene geometry), but now it works correctly with all of the other stuff (SSRR, AO, and motion blur) sorta.

    [deleted other stuff here since it's offtopic]

    EDIT2: OK, back on topic: When I switch to Forward, sampling from the motion vector buffer appears to be vertically flipped. I've heard of using texel size to check if the UVs should be flipped, but I'm not sure how to get at this information for the motion vector buffer. Any ideas?
     
    Last edited: May 1, 2016
  24. Tim-C

    Tim-C

    Unity Technologies

    Joined:
    Feb 6, 2010
    Posts:
    2,163
    If you have things that really need custom per object motion vectors you can add a custom pass to their shader, something like this:

    Code (csharp):
    1.  
    2. Shader "Custom/StandardShaderWithCustomMovec" {
    3.     Properties {
    4.         _Color ("Color", Color) = (1,1,1,1)
    5.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
    6.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
    7.         _Metallic ("Metallic", Range(0,1)) = 0.0
    8.     }
    9.     SubShader
    10.     {
    11.         Tags { "RenderType" = "Opaque" }
    12.         LOD 200
    13.  
    14.         CGPROGRAM
    15.         // Physically based Standard lighting model, and enable shadows on all light types
    16.         #pragma surface surf Standard fullforwardshadows
    17.  
    18.         // Use shader model 3.0 target, to get nicer looking lighting
    19.         #pragma target 3.0
    20.  
    21.         sampler2D _MainTex;
    22.  
    23.         struct Input {
    24.             float2 uv_MainTex;
    25.         };
    26.  
    27.         half _Glossiness;
    28.         half _Metallic;
    29.         fixed4 _Color;
    30.  
    31.         void surf(Input IN, inout SurfaceOutputStandard o) {
    32.             // Albedo comes from a texture tinted by color
    33.             fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
    34.             o.Albedo = c.rgb;
    35.             // Metallic and smoothness come from slider variables
    36.             o.Metallic = _Metallic;
    37.             o.Smoothness = _Glossiness;
    38.             o.Alpha = c.a;
    39.         }
    40.         ENDCG
    41.  
    42.         Pass
    43.         {
    44.             CGINCLUDE
    45.             struct MotionVertexInput
    46.             {
    47.                 float4 vertex : POSITION;
    48.                 float3 oldPos : NORMAL;
    49.             };
    50.  
    51.             struct v2f_motion_vectors
    52.             {
    53.                 float4 pos : SV_POSITION;
    54.             };
    55.  
    56.             v2f_motion_vectors vert_motion_vectors(MotionVertexInput v)
    57.             {
    58.                 v2f_motion_vectors o;
    59.                 o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
    60.                 return o;
    61.             }
    62.  
    63.             float4 frag_motion_vectors(v2f_motion_vectors i) : SV_Target
    64.             {
    65.                 return float4(1.0, 0.0, 0.0, 1.0);
    66.             }
    67.             ENDCG
    68.  
    69.             Tags{ "LightMode" = "MotionVectors" }
    70.             Name "MOTIONVECTORS"
    71.  
    72.             ZTest LEqual
    73.             Cull Off
    74.             ZWrite Off
    75.  
    76.             CGPROGRAM
    77.             #pragma vertex vert_motion_vectors
    78.             #pragma fragment frag_motion_vectors
    79.             ENDCG
    80.         }
    81.     }
    82.     FallBack "Diffuse"
    83. }
    84.  
    It might not be the motion vector buffer, but the normal texture, this has more info: http://docs.unity3d.com/Manual/SL-PlatformDifferences.html[/code]
     
  25. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    Ah, thanks. I ended up using a different solution for first person weapons anyway (for compatibility with AO and SSRR) but that will be useful if I need a custom vertex shader for something (vertex animated wavy cloth perhaps?) :)

    Hm, the image effect itself isn't flipped. Additionally, checking for UNITY_UV_STARTS_AT_TOP doesn't seem to help, since it's true in both cases (running DX11).
     
  26. FuzzyQuills

    FuzzyQuills

    Joined:
    Jun 8, 2013
    Posts:
    2,875
    HELL... YES. About time a clean solution for easy implementation of vector blur's done! :)
     
    nxrighthere and Tim-C like this.
  27. XRA

    XRA

    Joined:
    Aug 26, 2010
    Posts:
    257
    does anyone know what is going on under the hood for the motion vectors?

    Is it the same as reconstructing world position from depth and comparing that with the previous frame's depth buffer to world position?
     
  28. FuzzyQuills

    FuzzyQuills

    Joined:
    Jun 8, 2013
    Posts:
    2,875
    Same here, I wish I could answer this... Unity?
     
  29. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    Taking a look at the motion vector pass, it basically seems to multiply a vertex by the current MVP, then by the previous frame's MVP, and outputs the difference in the fragment shader in red and green channels.
     
    richardkettlewell likes this.
  30. FuzzyQuills

    FuzzyQuills

    Joined:
    Jun 8, 2013
    Posts:
    2,875
    Hmmm... Would that handle skinned meshes, I wonder? Because just using that matrix wouldn't work with skinned meshes, from my understanding.
     
  31. PhobicGunner

    PhobicGunner

    Joined:
    Jun 28, 2011
    Posts:
    1,813
    It does appear to work on skinned meshes. Haven't looked too closely to figure out how or why it works in that scenario ;)
     
  32. TheSniperFan

    TheSniperFan

    Joined:
    Jul 18, 2013
    Posts:
    700
    I have a question regarding this feature.
    Since you can enable/disable motion vectors on a per-renderer basis, I wonder why you should do this? Is there a performance penalty, if they are enabled? Should you disable them for static objects?
     
  33. pvloon

    pvloon

    Joined:
    Oct 5, 2011
    Posts:
    591
    Had a similair question, the renderer interface is cluttered enough as is - it seems you'd just want a project wide toggle to set them on/off (which there already is). Having them off for one object seems like it's always wrong - except if you're going for glitch graphics I guess :) (at least, from the perspective of TXAA & motion blur. Are there other more crazy use cases?)
     
  34. theprotonfactor

    theprotonfactor

    Joined:
    Oct 10, 2014
    Posts:
    140
    They do come with a performance penalty, but if you're going to use them for image effects like TAA or motion blur then you definitely shouldn't disable them on static or stationary meshes. The motion it tracks is in screen space, so even if it can't move in the world it can still move across the screen.

    I don't know much about how untiy implements them, but I believe Tim said that they use different techniques depending on the mesh type. Skinned meshes and cloth are harder to generate motion vectors for. So disabling them might give you performance boost depending on how they're implemented.
     
  35. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,495
  36. Tim-C

    Tim-C

    Unity Technologies

    Joined:
    Feb 6, 2010
    Posts:
    2,163
    It does a few things.
    *Pass 1: render camera motion vectors (reconstructed from depth buffer and last frame vp matrix)
    *PAss 2: for each object that is moving, rerender it into the motion buffer using a custom motion shader. For non skinned meshes this just uses the last mvp matrix and the normal vertex positions. For skinned motion vectors this uses the last map matrix and the last skinned vertex position (internally we double buffer the skinned mesh result)

    This generated render texture is then passed through as the motion buffer texture.
     
  37. Tim-C

    Tim-C

    Unity Technologies

    Joined:
    Feb 6, 2010
    Posts:
    2,163
    Per renderer motion vectors have a small cost (one draw call per object), and skinned renderers have a memory cost also (Double buffered vertex buffer on the gpu). If motion vectors are enabled you will always get them for camera motion (done from reconstructed depth) but you won't get object motion or skinned motion.

    You may want to disable per objet motion vectors if you don't want motion blur for that object or you have some weirdness with the shaders you are using.
     
  38. Tim-C

    Tim-C

    Unity Technologies

    Joined:
    Feb 6, 2010
    Posts:
    2,163
    Yeah! It's so good. Will be coming to the effects package soon :)
     
    elbows, hippocoder and Detniess like this.
  39. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    28,363
    I guess the effect can work with LOD - further away simply untick velocity on mesh renderers, or if they're moving quite slow, untick again. Could work out quite performance happy with a little care? :)

    I assume anything that doesn't have that ticked will become background blur when the camera moves quickly?

    That gives me a thought though. Would this result in a distracting pop?

    Also a feature: Instead of binary on / off, perhaps float? This way we can fade them out to 0 or perhaps boost them beyond 1, so at 2 the effect is twice as strong for that object (multiplied by this value).

    In films they can boost the effect slightly to create a better effect, or more pronounced special move... or maybe it just doesn't work so hot so we want a little less motion blur on the player and so on.
     
    Last edited: May 21, 2016
  40. pvloon

    pvloon

    Joined:
    Oct 5, 2011
    Posts:
    591
    Agreed with Hippo, I still think we could do without either a float/bool (if people really want to exclude some stuff probably better off doing that from shader), but at least a float would give a pretty wide range of behaviour
     
  41. FuzzyQuills

    FuzzyQuills

    Joined:
    Jun 8, 2013
    Posts:
    2,875
    Well, that explains it. Not for mobile then... (motion vector blur on a mali-400?! Yea, not gonna happen... :p)
     
  42. Funny-Face

    Funny-Face

    Joined:
    Nov 6, 2013
    Posts:
    17
    Is there an option to disable this for entire project? For projects which targets low end devices.
     
  43. Tim-C

    Tim-C

    Unity Technologies

    Joined:
    Feb 6, 2010
    Posts:
    2,163
    It's only enabled when you set the flag on the camera. See the scripting docs for depthtexturemode.
     
  44. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,495
    Last edited: May 29, 2016
  45. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,495
  46. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    28,363
    Motion vectors viz is available afaik as a debug option on the post that uses it and Kino motion has been around a long time, just updated for vectors. I guess it will be included at some point :) I hope they'll allow us to set the intensity motion per renderer or such so some things can pretend to have more blur than others.
     
  47. FuzzyQuills

    FuzzyQuills

    Joined:
    Jun 8, 2013
    Posts:
    2,875
    I wonder how that could be used... :D
     
  48. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    28,363
    Well the key reason for wanting more blur on some objects and less on others is the same reason hollywood post production adds it in varying degrees - there are only so many frames available. If you want to give the impression of great power or movement but STILL render it, you can't really move it at a practical speed - that would allow practical speed and the feeling of energy at the same time.
     
  49. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,495
    The pre-motion vectors version of the blur had a different name, and after initially adding motion vector support to it I think he decided to start a completely new project for it, hence my post.

    I'm pretty sure the motion vectors visualizer is way better than the debug options that were included before. It shows the vectors as little arrows/lines rather than just colouring meshes.
     
  50. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    28,363
    Oh that's very cool! I had no idea.
     
unityunity