CustomPass using different camera: how to render MotionVectors, and depth with Tessellated objects?

First, I wanted to thank @antoinel_unity for the extremely helpful CustomPass examples repository. It's been very useful as I've been learning the CustomPass and HDRP APIs.

I'm currently using 2019.3.9f1 with HDRP 7.3.1.

I've been trying to work on an effect extending from the repository's depth capture example, where I'd like to capture the depth buffer and motion vectors for a separate camera that's not being rendered to screen. I'm running into the following roadblocks:

1. When a MeshRenderer is using tessellation, such as HDRP's LayeredLitTessellation Material, the Frame Debug window shows that object being is being rendered during the CustomPass, using the DepthOnly pass, but the depth texture is unchanged. The INSTANCING_ON keyword is set, and settings in the Frame Debug window seem similar to those in the standard DepthPrepassDeferredForDecals pass, which of course correctly captures the depth.
   Note that using a Lit override material when setting up the RendererListDesc will cause the object to be rendered correctly, though the depth will not match that of the true tessellated version. Additionally, I can reproduce the same behaviour in the repository's depth capture example.
2. I'm trying to render MotionVectors from the same camera, but calling HDUtils.DrawRendererList doesn't cause any objects to be rendered (even using Lit material, irrespective of tessellation).
   MeshRenderers have Motion Vectors set to Per Object Motion.

I'll try to summarize the relevant code snippets here.

For the camera setup, I ensure the depthTextureMode is set, though this doesn't seem to make a difference.

_bakingCamera.depthTextureMode |= DepthTextureMode.Depth | DepthTextureMode.MotionVectors;

For the depth pass, setup is the same as the repository's example:

Code (CSharp):

1. _bakingCamera.TryGetCullingParameters(out ScriptableCullingParameters cullingParams);
2. cullingParams.cullingOptions = CullingOptions.ShadowCasters;
3. cullingParams.cullingMask = (uint)_layerMask.value;
4. cullingResult = renderContext.Cull(ref cullingParams);

The same is true to the capture during Execute:

Code (CSharp):

1. var heightRendererListDesc = new RendererListDesc(_depthShaderTags, cullingResult, _bakingCamera)
2. {
3.     rendererConfiguration = PerObjectData.None,
4.     renderQueueRange = RenderQueueRange.all,
5.     sortingCriteria = SortingCriteria.BackToFront,
6.     excludeObjectMotionVectors = false,
7.     stateBlock = new RenderStateBlock(RenderStateMask.Raster) { rasterState = new RasterState(depthClip: false) },
8. };

Code (CSharp):

1. CoreUtils.SetRenderTarget(cmd, _depthBakeBuffer, ClearFlag.Depth);
2. HDUtils.DrawRendererList(renderContext, cmd, RendererList.Create(heightRendererListDesc));

For the motion vectors pass, I tried to use the same GraphicsFormat (RG16) that it seems Unity normally uses:

Code (CSharp):

1. motionVectorsBuffer = RTHandles.Alloc(_depthCaptureTexture.width, _depthCaptureTexture.height, colorFormat: GraphicsFormat.R16G16_SFloat, depthBufferBits: DepthBits.None, dimension: TextureDimension.Tex2D, name: "Motion Vectors");
2.  
3. _motionVectorsShaderTags = new ShaderTagId[2]
4. {
5.     new ShaderTagId("MotionVectors"),
6.     new ShaderTagId("Motion Vectors"),
7. };

Then, during Execute:

Code (CSharp):

1. var motionVectorRendererListDesc = new RendererListDesc(_motionVectorsShaderTags, cullingResult, _bakingCamera)
2. {
3.     rendererConfiguration = PerObjectData.MotionVectors,
4.     renderQueueRange = RenderQueueRange.all,
5.     sortingCriteria = SortingCriteria.BackToFront,
6.     excludeObjectMotionVectors = false,
7. };
8.  
9. CoreUtils.SetRenderTarget(cmd, _motionVectorsBuffer, ClearFlag.All);
10. RendererList motionVectorsRendererList = RendererList.Create(motionVectorRendererListDesc);
11. HDUtils.DrawRendererList(renderContext, cmd, motionVectorsRendererList);

The cullingResult is the same as for the depth pass, though no objects are rendered, even the ones that appear in the depth pass.


Any help or advice would be greatly appreciated!

 

An update on my investigations on capturing motion vectors from a different camera in a CustomPass: it seems that the MOTIONVECTORS pass is disabled for a normal MeshRenderer, even if Motion Vectors is set to Per Object Motion and the object is moving (eg. by setting transform.position each frame). Enabling 'Add Precomputed Velocity' (even though we have so such velocity to supply) in the MeshRenderer's Material seems to enable the pass again, and HDUtils.DrawRendererList does indeed perform a pass for such objects if using the MotionVectors shader tags in the RendererListDesc.

However, it seems as though the per object motion vector data is not being set in this pass. Particularly, the following shader properties show up in the pass, but the entries are all zero: unity_MotionVectorsParams, _WorldSpaceCameraPos, and unity_MatrixPreviousM. Now, _WorldSpaceCameraPos can be set at the command buffer level, which is also how we'd need to set _PrevViewProjMatrix and _NonJitteredViewProjMatrix (in my case, I know that the different camera doesn't move, so we can re-use its current view-projection matrix). As a result, no motion is rendered in this pass.

Does anyone know why unity_MotionVectorsParams and unity_MatrixPreviousM would not be set, even though rendererConfiguration = PerObjectData.MotionVectors is being set in the RendererListDesc?

Thanks for reading, and any input would be greatly appreciated!

 

Thank you for putting this together! I've installed Unity 2020.3.6f1 and HDRP 10.4 and can confirm that motion vectors are indeed captured. Downgrading the example to 2019.3.9f1 with HDRP 7.3.1 doesn't work; objects are not picked up by the DrawRendererList call. Nonetheless, I should be able to update to 2020.3.

However, I have run into issues trying to bake motion vectors for a different (non-rendered) Camera. The main problem is that CustomPassUtils.RenderFromCamera won't work for motion vectors. From trial and error, it seems that the ShaderTagId used in the RendererListDesc for CoreUtils.DrawRendererList is necessary; otherwise, the pass seems to cycle from frame to frame between rendering all moving objects, and all other objects.

Unfortunately, DrawRendererList doesn't work nicely when trying to bake from another camera. I've found that HDRenderPipeline.OverrideCameraRendering will let me set up the baking camera parameters. However, that struct is currently internal. Also, it causes the baking camera's aspect ratio to be changed to match the main camera, which is incorrect for my use case (the baking camera's position, aspect ratio, and field of view are specified ahead of time).

Making a custom version of the HDRP package to work around the current limitations of HDRenderPipeline.OverrideCameraRendering was thankfully straightforward, and allows me to successfully bake motion vectors. Perhaps some API changes could allow for a little more flexibility when using a CustomPass to bake to another camera.

Thank you again for your help. With it, I'm able to move forward with my project.

 

This is an useful info, thanks

 

Nice! What HDRP version this PR will get into?