Accessing the depth texture and motion vector texture

I have a camera that is configured to provide depth and motion vector textures. I'm trying to extend an Amplify Shader Editor template (essentially just a shader) to access these. I have the following code where the comments //MY ADDITION indicate... uh... my additions to the template shader.

Code (CSharp):

1. // Upgrade NOTE: replaced 'mul(UNITY_MATRIX_MVP,*)' with 'UnityObjectToClipPos(*)'
2.  
3. Shader /*ase_name*/ "ASETemplateShaders/PostProcess" /*end*/
4. {
5.     Properties
6.     {
7.         _MainTex ( "Screen", 2D ) = "black" {}
8.         _CameraMotionVectorsTexture("Motion Vectors", 2D) = "black" {} // MY ADDITION
9.         _CameraDepthTexture("Depth Texture", 2D) = "black" {} //MY ADDITION
10.         /*ase_props*/
11.     }
12.  
13.     SubShader
14.     {
15.         Tags{ "RenderType"="Opaque" }
16.        
17.         ZTest Always
18.         Cull Off
19.         ZWrite Off
20.         /*ase_pass*/
21.  
22.         Pass
23.         {
24.             CGPROGRAM
25.  
26.             #pragma vertex vert_img_custom
27.             #pragma fragment frag
28.             #pragma target 3.0
29.             #include "UnityCG.cginc"
30.             /*ase_pragma*/
31.  
32.             struct appdata_img_custom
33.             {
34.                 float4 vertex : POSITION;
35.                 half2 texcoord : TEXCOORD0;
36.                 /*ase_vdata:p=p;uv0=tc0*/
37.             };
38.  
39.             struct v2f_img_custom
40.             {
41.                 float4 pos : SV_POSITION;
42.                 half2 uv   : TEXCOORD0;
43.                 half2 stereoUV : TEXCOORD2;
44.         #if UNITY_UV_STARTS_AT_TOP
45.                 half4 uv2 : TEXCOORD1;
46.                 half4 stereoUV2 : TEXCOORD3;
47.         #endif
48.                 /*ase_interp(4,7):sp=sp.xyzw;uv0=tc0.xy;uv1=tc1;uv2=tc2;uv3=tc3*/
49.             };
50.  
51.             uniform sampler2D _MainTex;
52.             uniform half4 _MainTex_TexelSize;
53.             uniform half4 _MainTex_ST;
54.  
55.             uniform sampler2D _CameraMotionVectorsTexture; // MY ADDITION
56.             uniform sampler2D _CameraDepthTexture; // MY ADDITION
57.  
58.             /*ase_globals*/
59.  
60.             v2f_img_custom vert_img_custom ( appdata_img_custom v /*ase_vert_input*/ )
61.             {
62.                 v2f_img_custom o;
63.                 /*ase_vert_code:v=appdata_img_custom;o=v2f_img_custom*/
64.                 o.pos = UnityObjectToClipPos ( v.vertex );
65.                 o.uv = float4( v.texcoord.xy, 1, 1 );
66.  
67.                 #if UNITY_UV_STARTS_AT_TOP
68.                     o.uv2 = float4( v.texcoord.xy, 1, 1 );
69.                     o.stereoUV2 = UnityStereoScreenSpaceUVAdjust ( o.uv2, _MainTex_ST );
70.  
71.                     if ( _MainTex_TexelSize.y < 0.0 )
72.                         o.uv.y = 1.0 - o.uv.y;
73.                 #endif
74.                 o.stereoUV = UnityStereoScreenSpaceUVAdjust ( o.uv, _MainTex_ST );
75.                 return o;
76.             }
77.  
78.             half4 frag ( v2f_img_custom i /*ase_frag_input*/) : SV_Target
79.             {
80.                 #ifdef UNITY_UV_STARTS_AT_TOP
81.                     half2 uv = i.uv2;
82.                     half2 stereoUV = i.stereoUV2;
83.                 #else
84.                     half2 uv = i.uv;
85.                     half2 stereoUV = i.stereoUV;
86.                 #endif  
87.                
88.                 half4 finalColor;
89.  
90.                 // ase common template code
91.                 /*ase_frag_code:i=v2f_img_custom*/
92.  
93.                 finalColor = /*ase_frag_out:Frag Color;Float4*/half4( 1, 1, 1, 1 )/*end*/;
94.  
95.                 return finalColor;
96.             }
97.             ENDCG
98.         }
99.     }
100.     CustomEditor "ASEMaterialInspector"
101. }
102.  

Now, from what I've read over the past few hours, this should be assigning the depth and motion textures to the appropriately named properties so that I can access them in ASE's editor. Unfortunately, when I try and output them in any capacity, I'm left with a pure black image. This makes me think that something must be going wrong with the following lines:

Code (CSharp):

1.             uniform sampler2D _CameraMotionVectorsTexture; // MY ADDITION
2.             uniform sampler2D _CameraDepthTexture; // MY ADDITION

I don't expect clean results like a perfect greyscale depth texture, but rather the mess of colours a depth texture usually reports as if you don't convert it. Does anyone know what I'm doing wrong here? I'm sure it's something simple.

 

Okay, I found the problem, kind of. After some more searching and banging my head against this problem, I discovered this:

Graphics.Blit Does Not Copy RenderTexture Depth

It gives me a way to do this for the depth texture, but I can't for the life of me figure out how I'd do the same for the motion vectors. I feel like the fact that the motion vectors contain a range of -1 to 1 might be a problem here?

 

DepthTextureMode

There's no need to have the textures be properties of the shader. Those textures are set as global parameters accessible by all shaders as long as they exist. You just need these lines in your shader:

sampler2D_float _CameraDepthTexture;
sampler2D_half _CameraMotionVectorsTexture;

Then sample the textures like any other using the screen UVs.

The depth texture is going to be a value from 0.0 to 1.0, and the velocity texture is going to be something like -1.0 to 1.0. Note, whether a depth of 0.0 is at the near or far plane depends on the platform being rendered on, which you can check for in the shader by if UNITY_REVERSED_Z is defined. However Unity has built in functions for converting the depth texture values into world space distances or linear 0.0 to 1.0.

That link isn't useful for you here. There's no need to make a copy of the texture. If it exists it's already going to be passed to the shader.


The real question is, do they exist?

My guess is they don't because you've not enabled them on the camera. To do that you need to write a script that enables both the depth texture and velocity texture for the camera.depthTextureMode.

GetComponent<Camera>().depthTextureMode |= DepthTextureMode.Depth | DepthTextureMode.MotionVectors;

 

They are enabled on the camera, but...

Code (csharp):

1.  
2.             Graphics.Blit(cam.targetTexture, undevelopedPhotos[totalPhotos], noiseMaterial);
3.  

This is the code I'm using to pass things to the material. When you blit, as far as I understand, you lose the depth texture information because that isn't copied over.

 

When you blit you loose the depth buffer, not the depth texture. The depth texture remains as a globally bound texture for the rest of the frame regardless of what you do. Unless you're using multiple cameras and need the depth texture from a specific one (and it's not the camera you're currently rendering to) then there's no need to copy it.

 

Alright, with your help I managed to get a depth texture outputting, but it seems ASE doesn't recognise sampler2D_half, so I guess I'll have to go to them for support now. Thanks!

 

Actually, upon further testing, it seems I'm getting a depth texture, but not the correct one.

This is the rendertexture output to an image:


But this is the depth texture output to an image:


For some reason, I'm getting the main camera's depth texture when I call sampler2D_float _CameraDepthTexture; instead of the camera I want. If I disable the main camera, effectively rendering nothing to the screen, I get the correct depth texture.

 

You want _LastCameraDepthTexture then.
https://docs.unity3d.com/Manual/SL-CameraDepthTexture.html

 

Tried that, no joy. I get the exact same result.

 

Is the depth texture enabled for that camera? When are you trying to access it?

 

Yes, I can access the depth texture so long as the main camera isn't active.

Here's the code I'm using to send the RenderTexture around where it needs to be:

Code (CSharp):

1.     IEnumerator TakePhoto()
2.     {
3.         if (totalPhotos < photoRollSize)
4.         {
5.             Shutter();
6.             AdjustSettings();
7.  
8.             yield return new WaitForEndOfFrame();
9.  
10.             Graphics.Blit(cam.targetTexture, undevelopedPhotos[totalPhotos], noiseMaterial);
11.             totalPhotos++;
12.         }
13.         else
14.         {
15.             Debug.Log("Photo roll full!");
16.         }
17.     }

This is called whenever the player presses the assigned camera shot button.

 

Ah. You're waiting for the end of the frame. The depth texture for that camera has been destroyed at that point if the main camera is active, so yeah, it's not accessible.

If you want to access the depth texture for that camera you do need to make a copy while that camera is active. To do this you'll want to use a command buffer assigned to the camera instead of a coroutine.

 

Hello All,

I have a related question so I will highjack the thread slightly.
I am trying a very simple setup where, I try to read the depth texture from one camera, use it in a shader, and visualise in a UI image via a material using that shader. I have only one camera in the scene for this test.

I have the following code on the camera to make sure the depth texture mode is se to depth.

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. public class CameraDepthUtility : MonoBehaviour {
6.  
7.     Camera cam;
8.  
9.     [SerializeField] Material mat;
10.  
11.     // Use this for initialization
12.     void Start () {
13.         cam = GetComponent<Camera> ();
14.         cam.depthTextureMode = DepthTextureMode.Depth;
15.     }
16.  
17.     void OnRenderImage(RenderTexture src, RenderTexture dest)
18.     {
19.         Graphics.Blit(src, dest, mat);
20.     }
21. }

The material has the following shader, where I sample the _CameraDepthTexture:

Code (CSharp):

1. Shader "DepthTest"
2. {
3.     SubShader
4.     {
5.         Tags { "Queue"="Transparent" "RenderType"="Transparent" }
6.  
7.         ZWrite Off
8.         Blend SrcAlpha OneMinusSrcAlpha
9.  
10.         Pass
11.         {
12.             CGPROGRAM
13.             #pragma vertex vert
14.             #pragma fragment frag
15.             #pragma target 3.0
16.             #include "UnityCG.cginc"
17.  
18.             // vertex shader inputs
19.             struct appdata
20.             {
21.                 float4 vertex : POSITION; // vertex position
22.                 float2 uv : TEXCOORD0; // texture coordinate
23.             };
24.  
25.             // vertex shader outputs ("vertex to fragment")
26.             struct v2f
27.             {
28.                 float2 uv : TEXCOORD0; // texture coordinate
29.                 float4 vertex : SV_POSITION; // clip space position
30.                 float4 scrPos : TEXCOORD1;
31.             };
32.  
33.             // vertex shader
34.             v2f vert (appdata v)
35.             {
36.                 v2f o;
37.                 o.vertex = UnityObjectToClipPos(v.vertex);
38.                 o.uv = v.uv;
39.                 return o;
40.             }
41.          
42.             sampler2D _CameraDepthTexture;
43.  
44.             // pixel shader; returns low precision ("fixed4" type)
45.             // color ("SV_Target" semantic)
46.             fixed4 frag (v2f i) : SV_Target
47.             {
48.  
49.                 fixed4 colCameraDepth = tex2D(_CameraDepthTexture, float2(i.uv.x , i.uv.y));
50.                 float r = colCameraDepth.r;
51.                 r = 1 - Linear01Depth(r);
52.                 colCameraDepth = fixed4(r,0,0,1);
53.  
54.                 return colCameraDepth;
55.  
56.             }
57.             ENDCG
58.         }
59.     }
60. }

And the same material is assigned to a UI image for visualising purpose.
However I cant get it to show the depth.

I have the feeling that I am missing something very obvious about using the _CameraDepthTexture but cant figure out.

All help is very much appreciated.
Cheers,
Doruk

 

Like the situation for the previous post, if your UI is being rendered as a screen space overlay, the depth texture has already been “destroyed” by the time the UI renders. (Technically it likely still exists, it’s just not being passed to shaders anymore.)

Solutions would be to use a camera or world space UI element, or to copy the depth texture using a command buffer, or maybe even just assign it as a global texture using a different name.

 

@bgolus thank you for the help. The UI image was in the world space indeed. I could not get this work. But found a workaround that solves my situation.

I make the camera render the depth info to a render texture and then feed that one in a shader to do the maths.

Thanks again for the help!