Resolved Reconstructing world space position from depth texture

(Using Unity 2020.3.12f1)

Hi everyone,

I am trying to reconstruct the world space position from the depth normal texture for custom lights. Most of the information I've found is for an image effect shader, but I would like to achieve this in a per-object fashion. From what I understand, the process of reconstructing the world space position involves the following:

1. Retrieve depth from the depth normal texture and remap from (0, 1) to (-1, 1)
2. Create clip space position with X and Y set to the screen position (remapped to (-1, 1), Z set to the remapped depth, and W set to 1
3. Calculate Inverse View Projection matrix in separate C# script and set as global shader property for access in shader
4. Multiply Inverse VP matrix with clip space position to calculate world space position
5. Divide the world space position XYZ by its W value to compensate for perspective

This seems to be the same process used in this tutorial although in SRP rather than the built-in renderer I'm currently using.

However, my current implementation of the above doesn't seem to be working properly.

In camera script:

Code (CSharp):

1.  
2.     void Start()
3.     {
4.         cam = GetComponent<Camera>();
5.         cam.depthTextureMode = DepthTextureMode.DepthNormals;
6.     }
7.  
8.     private void Update()
9.     {
10. //Code for MVP Matrices https://answers.unity.com/questions/12713/how-do-i-reproduce-the-mvp-matrix.html[/INDENT]
11.         bool d3d = SystemInfo.graphicsDeviceVersion.IndexOf("Direct3D") > -1;
12.         Matrix4x4 V = cam.worldToCameraMatrix;
13.         Matrix4x4 P = cam.projectionMatrix;
14.         if (d3d) {
15.             // Invert Y for rendering to a render texture
16.             for (int i = 0; i < 4; i++) {
17.                 P[1,i] = -P[1,i];
18.             }
19.             // Scale and bias from OpenGL -> D3D depth range
20.             for (int i = 0; i < 4; i++) {
21.                 P[2,i] = P[2,i]*0.5f + P[3,i]*0.5f;
22.             }
23.         }
24.         Matrix4x4 VP = P*V;
25.         Matrix4x4 VP_I = VP.inverse;
26.         Shader.SetGlobalMatrix("VP_I", VP_I);
27.     }
28.  

In shader:

Code (CSharp):

1.  
2.  
3. //Pass settings
4.        Cull Front
5.        Ztest Always
6.        Zwrite off
7.  
8.  
9. fixed4 frag (v2f i) : SV_Target {
10.  
11.        //Other code here
12.  
13. // Retrieve Depth from texture
14. float2 scrUV = i.screenPos.xy / i.screenPos.w;
15. float4 depthnormal = tex2D(_CameraDepthNormalsTexture, scrUV);
16. float depth;
17. float3 normal;
18. DecodeDepthNormal(depthnormal, depth, normal);
19.  
20. // Remap depth to (-1, 1)
21. depth = depth * 2.0 - 1.0;
22.  
23. // Clip space position
24. float4 posCS = float4(scrUV * 2.0 - 1.0, depth, 1.0);
25.  
26. // Calculate world space position using inverse VP matrix
27. float4 posWS = mul(VP_I, posCS);
28.  
29. //Compensate for perspective
30. posWS.xyz /= posWS.w;[/INDENT]
31. }
32.  

When I apply this shader to a basic sphere, this is the result I get:

Before:



After:



I'm not too sure what I can do to fix this problem, as the depth, clip space position, and matrices seem to be correct.

Thank you in advance.

 

I know basically every example out there says you need to use a script to pass the inverse view projection matrix to your shader to reconstruct the world position from the depth buffer... but you don't. Not even for post processing. Unity passes that to the shader already.

You also don't even need the inverse view projection matrix at all unless it's for a post process. For an object in the world all you need is the world position of the surface.

In the below example I use the camera relative world position, which can fix some possible precision issues when you're far from the world origin, but it's fine to pass the world position and subtract the camera position in the fragment shader if you prefer.

Code (CSharp):

1. Shader "Unlit/WorldPosFromDepth"
2. {
3.     Properties
4.     {
5.     }
6.     SubShader
7.     {
8.         Tags { "Queue"="Transparent" "RenderType"="Transparent" "IgnoreProjector"="True" }
9.         LOD 100
10.  
11.         Pass
12.         {
13.             CGPROGRAM
14.             #pragma vertex vert
15.             #pragma fragment frag
16.  
17.             #include "UnityCG.cginc"
18.  
19.             struct appdata
20.             {
21.                 float4 vertex : POSITION;
22.             };
23.  
24.             struct v2f
25.             {
26.                 float4 pos : SV_POSITION;
27.                 float4 projPos : TEXCOORD0;
28.                 float3 camRelativeWorldPos : TEXCOORD1;
29.             };
30.  
31.             UNITY_DECLARE_DEPTH_TEXTURE(_CameraDepthTexture);
32.  
33.             v2f vert (appdata v)
34.             {
35.                 v2f o;
36.                 o.pos = UnityObjectToClipPos(v.vertex);
37.                 o.projPos = ComputeScreenPos(o.pos);
38.                 o.camRelativeWorldPos = mul(unity_ObjectToWorld, float4(v.vertex.xyz, 1.0)).xyz - _WorldSpaceCameraPos;
39.                 return o;
40.             }
41.  
42.             bool depthIsNotSky(float depth)
43.             {
44.                 #if defined(UNITY_REVERSED_Z)
45.                 return (depth > 0.0);
46.                 #else
47.                 return (depth < 1.0);
48.                 #endif
49.             }
50.  
51.             half4 frag (v2f i) : SV_Target
52.             {
53.                 float2 screenUV = i.projPos.xy / i.projPos.w;
54.  
55.                 // sample depth texture
56.                 float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, screenUV);
57.  
58.                 // get linear depth from the depth
59.                 float sceneZ = LinearEyeDepth(depth);
60.  
61.                 // calculate the view plane vector
62.                 // note: Something like normalize(i.camRelativeWorldPos.xyz) is what you'll see other
63.                 // examples do, but that is wrong! You need a vector that at a 1 unit view depth, not
64.                 // a1 unit magnitude.
65.                 float3 viewPlane = i.camRelativeWorldPos.xyz / dot(i.camRelativeWorldPos.xyz, unity_WorldToCamera._m20_m21_m22);
66.  
67.                 // calculate the world position
68.                 // multiply the view plane by the linear depth to get the camera relative world space position
69.                 // add the world space camera position to get the world space position from the depth texture
70.                 float3 worldPos = viewPlane * sceneZ + _WorldSpaceCameraPos;
71.                 worldPos = mul(unity_CameraToWorld, float4(worldPos, 1.0));
72.  
73.                 half4 col = 0;
74.  
75.                 // draw grid where it's not the sky
76.                 if (depthIsNotSky(depth))
77.                     col.rgb = saturate(2.0 - abs(frac(worldPos) * 2.0 - 1.0) * 100.0);
78.  
79.                 return col;
80.             }
81.             ENDCG
82.         }
83.     }
84. }

You'll also notice I'm using the

_CameraDepthTexture

here, and not the

_CameraDepthNormalTexture

. You can use the camera depth normal texture if you want, but the depth information is much, much lower precision as it's storing the depth value as a 16 bit integer in two 8 bit color channels. The

_CameraDepthTexture

is a 32 bit float. So most likely you'll have to sample both textures to do your lighting.

 

What a much simpler method! Thank you for the help, hopefully, this will work perfectly.

Just for future reference and because of curiosity, is there anyway to properly pass an inverse VP matrix to the shader? Just from my debugging, the built-in unity variable and the calculated inverse matrix seem to be different. Was it simply a mistake on my part?

 

To calculate the inverse matrix in c#, you want to use the

Gl.GetGPUProjectionMatrix()

function.
https://docs.unity3d.com/ScriptReference/GL.GetGPUProjectionMatrix.html

Code (csharp):

1. Matrix4x4 VP = GL.GetGPUProjectionMatrix(cam.projectionMatrix, false) * cam.worldToCameraMatrix;
2. Matrix4x4 VP_I = VP.inverse;

However, you kind of don't want or need to do this. Unity already passes the inverse projection matrix and inverse view matrix to the shader. This example shader is calculating the surface normal from the depth buffer, but it calculates the view position using the inverse projection matrix as part of that.
https://gist.github.com/bgolus/a07ed65602c009d5e2f753826e8078a0

That won't work for orthographic cameras, but Unity's own screen space shadow shader does handle that!
https://github.com/TwoTailsGames/Un...sExtra/Internal-ScreenSpaceShadows.shader#L63

The main key is the

unity_CameraInvProjection

both my example and Unity's code are using are passing in the

camera.projectionMatrix.inverse

as is without going through

GetGPUProjectionMatrix

. This might seem crazy, but it means it's exactly the same no matter what hardware you're running which means it's actually a bit easier to use if you're trying to reconstruct from the screen UV.

 

@bgolus, thanks for showing this method. Could you clarify what is a "1 unit view depth"? I am having trouble understanding how this is different from a 1-magnitude vector. Also, what is stored inside

unity_WorldToCamera._m20_m21_m22

?

 

As I'm assuming you understand, a normalized vector has a magnitude of 1. Lots of things in graphics programming / shaders rely on normalized vectors.

The

viewPlane

isn't a normalized vector. This is the offset from the camera to a flat plane that is 1 unit in front of the view. An easier to understand example might be something like this:

Code (csharp):

1. // transform the world space view direction (camRelativeWorldPos) from world space to view space
2. float3 viewSpaceViewDir = mul((float3x3)unity_WorldToCamera, i.camRelativeWorldPos.xyz);
3.  
4. // divide view space view dir by its z so that it represents the offset from the camera to a plane that is 1 unit
5. // in front of the camera view
6. float3 viewSpaceViewPlane = viewSpaceViewDir / abs(viewSpaceViewDir.z);
7.  
8. // transform view plane back into world space
9. float3 viewPlane = mul((float3x3)unity_CameraToWorld, viewSpaceViewPlane);

The bit of code in the example shader is an optimization version of of all that. The

unity_WorldToCamera._m20_m21_m22

is the camera's forward vector in world space, and a dot product of an arbitrary vector with a normalized vector gets you the width of the arbitrary vector align the normalized vector.