Calculating inverse view projection for a shader

Hi all,

I'm trying to apply a Z bias to an object with a surface shader.
Since surface shaders call UnityObjectToClipPos() automatically, I'm trying to convert to clip space, apply my Z bias, and then convert back to world space in the vertex shader.

I put together some code based on this example: https://forum.unity.com/threads/surface-shader-vertex-unwrap.780611/

And it almost works but not quite, the object is displaced and inside out.

Here's the C# code:

Code (CSharp):

1.     void Update()
2.     {
3.         // get GPU projection matrix
4.         Matrix4x4 projMatrix = GL.GetGPUProjectionMatrix(Camera.main.projectionMatrix, false);
5.    
6.         // get GPU view projection matrix
7.         Matrix4x4 viewProjMatrix = projMatrix * Camera.main.worldToCameraMatrix;
8.    
9.         // get inverse VP matrix
10.         Matrix4x4 inverseViewProjMatrix = viewProjMatrix.inverse;
11.  
12.         m_Material.SetMatrix("_InverseViewProjection", inverseViewProjMatrix);
13.     }

Here's the shader: (I'm not applying the Z bias yet, just trying to make sure my inverse projection matrix is correct, it's not)

Code (CSharp):

1.         void vert(inout appdata_full v)
2.         {
3.             v.vertex = UnityObjectToClipPos(v.vertex);
4.  
5.             //do z bias here
6.  
7.             v.vertex = mul(_InverseViewProjection, v.vertex);
8.             v.vertex = mul(unity_WorldToObject, v.vertex);
9.         }

Here's the result: The objects (a shield and set of swords) should be where the shadow is on the ground but instead it's floating above it and looks to be both rotated wrong and inside out. It also moves as the camera moves.


The problem is definitely in my inverse view projection because if I replace it in the shader with inverse(UNITY_MATRIX_VP) then it works fine, but adding an inverse() function to the shader is really expensive and not ideal.

Can anyone see what's wrong with my inverse view projection?

 

There's no way to 100% match z biasing if you're having to convert back to object space (or even view or world space). It only really makes sense in the 4 dimensional clip space. So I would say don't bother trying to mimic this way as you won't get good results.

Instead I'd recommend you go the easy route and push / pull the vertices by the view direction. That doesn't require any extra c# script, and will be just as effective (and possibly more easily controllable) than the z bias approach.

Code (csharp):

1. float3 worldPos = mul(unity_ObjectToWorld, float4(v.vertex.xyz, 1.0)).xyz;
2. float3 viewDir = worldPos - _WorldSpaceCameraPos.xyz;
3. viewDir /= dot(viewDir, -UNITY_MATRIX_V[2].xyz);
4. worldPos += viewDir * _Bias;
5. v.vertex.xyz = mul(unity_WorldToObject, float4(worldPos, 1.0)).xyz;

 

That kind of works in perspective, but I have an orthographic camera and the object appears to move as it's bias changes.

Even in perspective though it messes with the lighting, the shadows cast on the biased object are visibly in the wrong place. I guess that's unavoidable.
Here you can see the white rectangle is casting a long shadow on the ground but a very short shadow on the shield because it's biased. Maybe this isn't the right approach.


What I'm actually trying to do is I want the objects to fully render over each other based on a controllable sort order and not intersect even if they are in the same place. Essentially I want them to act like 2D sprites stacked on top of each other but I'm not using sprites because I want full PBR and dynamic lighting/shadows.

I tried turning off Z Writing and this mostly works but I get this artifact where the shadows clip through the objects. Here you can see the shadow on the shield in the bottom right is cast by an object that's underneath it. The objects is really clipping through it and making a shadow but rendered underneath so only the shadow is visible. I don't want that shadow visible. If I could fix that, this approach would be fine.


Do you know if there's another approach I could take that would better achieve what I'm after or is this just not really possible?

 

If you have an orthographic camera, then you can just do:

Code (csharp):

1. worldPos -= UNITY_MATRIX_V[2].xyz * _Bias;

 

Okay thanks, that fixes the orthographic part.
Still doesn't work right with shadows and sorts incorrectly with other nearby objects so this definitely isn't the right approach. The other approach of turning of Z writing works better except for the shadow artifacts.

Any idea how that can be solved?

 

The problems with this method in general is you’re moving the object, even if you’re using the original clip space Z offset. If you move the object that changes where it receives shadows. It’s also moving it during the shadow caster pass... actually it’ll be wrong for that since UNITY_MATRIX_V will be for the light’s “view” in that case.

I guess the question is, what exactly are you trying to do that needs biasing? Any time you do that with anything related to receiving or casting shadows is going to cause unexpected problems.

 

It was just a way to try and get one object to render on top of another. I now think it's completely the wrong approach.

So now I'm looking for either a solution to the shadow artifacts on the Z write approach or some other way to force an object to fully occlude another object that is in the same position.

Here's a gif to show what I'm trying to do: I want to be able to drag any object onto any other and have it render on top. You can see this working here but there are shadowing artifacts. The way this is working is that they have a shader with Z Write turned off, and I am adjusting their Render Queue order based on which object was dragged on top.



The Z bias was just a test to see if it would do this better, it doesn't so I'm dropping the Z bias idea.

 

Yeah, the shadow receiving makes this kind of impossible. They’ll always give away the “real” order because they don’t give a f what your draw order or what kind of bias towards the camera you have.

If you use a vertex offset of any kind in a Surface Shader, it’s going to actually move the object in 3D space. That means the shadow receiving for lights (apart from the main directional light) are going to be calculated as if it’s in that different position. That’s what you’re seeing now.

If you use a vertex fragment shader instead you can use the original position instead of the offset position to calculate the shadow receiving position. But now it’ll be receiving shadows as if it’s in the original position, and those other objects are presumably penetrating each other in the original position (which is where the shadows are being calculated as) which means you’ll see stuff like the shield shadowing the sword when the sword is supposed to be under it.

If you modify the shadow caster pass to do another offsets there, either the shadow positions move with the objects, meaning they cast in the wrong place (because they too have been moved in 3D space towards the camera) or they self shadow / have no shadow because they’re being pushed away / towards the light. There’s no way to do this in a way that doesn’t look like crap.

You could use a stencil pass to prevent objects from drawing, forcing a specific order to the visuals w/o needing to use a z bias. But this won’t fix the shadow problem, especially since Unity totally ignores the draw order when rendering shadow maps and only allows a single shadow caster pass per object. So you’d have to manually inject your object’s shadow pass & stencil pass into lights’ shadow map rendering to ensure their draw order. But it would fix most of the issues you’re having.

Basically you need your current Surface Shader, but with a Stencil check that skips any pixels already rendered to w/ a specific ref value. Then a second pass after the Surface Shader that’s empty except for

ColorMask 0

and writing to that stencil value. Then you render your objects top to bottom with that shader (use a single late opaque queue for all gear, and the RendererComponent’s sortingOrder value from script to control their order). But don’t give it a Fallback, which will mean it won’t have a shadow caster pass. Then you’ll need a second shader that is just a shadow caster pass w/ the stencil test, and manually render objects in the correct order into shadow casting lights using command buffers for both the shadow caster depth pass and the stencil only pass.

 

Or you could disable shadow casting on your objects and use shadow proxy meshes that are just flat 2D shapes just above the ground.

 

Wow, thanks for all that info, that should be helpful.

The shadow proxy method sounds like it would have too many problems, like if an item is next to a wall it should cast a shadow on the wall but with a shadow proxy it wont.
I'm going to attempt the more complex approach with stencil checks and command buffers.

I haven't settled on a render pipeline yet, I have branches of my project for fixed, URP, and HDRP so I can experiment in all of them, and I have my current shaders working as SurfaceShaders and as ShaderGraphs.

Would this technique be easier or harder in any specific pipeline? If one of them will make it easier I might use that one.

 

Probably easiest with the built in path, but I haven’t spent a ton of time with the URP or HDRP. Both of the SRPs I just assume would require modifying the SRP code directly, which is either easier or much harder depending on how you want to look at it.

 

Thanks again for the help with this.
I have most of it working: I have the stencil buffer working on my existing surface shader and I have a second shader that just casts a shadow.

I'm a bit stuck on how to use Command Buffers though. I've been reading up on it and followed a few tutorials on doing post effects with them but haven't quite worked out how to apply them in this case. I assume I would make a CommandBuffer, bind my shadow caster material to it and insert it into the right hook in the renderer, but not quite sure if this would be done from a light, from the camera, or from each item, or exactly which hooks to use.

Would you be able to provide a bit more info or an example?

 

For each of your objects you'll want to create a command buffer that's basically two

cmd.DrawRenderer();

commands. One to render the mesh with your shadow caster material, with the pass index of the shadow caster pass, and one that's the stencil pass from the other material. You might be able to stick both in the shadow caster pass and use a pass index of -1, but I've never tried that when rendering to depth maps.

That's the easy part.

Next you need to add the command buffer for each object to each shadow casting light it might affect, and in the order you want them to render in. This means you have to iterate over every light every frame, remove all of their command buffers, compare the light's position & radius against the position & bounds radius of every object that you want to have sorted, and add those that interact to that light's

LightEvent.AfterShadowMapPass

command buffer. There's no right way to do this... (though there are a lot of wrong ways, anything that involves calling something like FindObjectsOfType every frame is one of the very wrong ways). Put all of the lights you care about in a big list or array, and all of the objects you care about with the special command buffer in a list, and iterate through them. That might be a FindObjectsOfType once to get all of the lights, check if they're shadow casting, and add them to a list at runtime. Or adding a script file to every light you know you'll care about that OnEnable adds itself to some global list. Presumably you already have a list of all of the objects you're looking to sort. You can jobify the iteration and/or update only when things change, that depends on how you have your scene and objects setup.

 

Thanks that helps a lot.

I have this almost working now. The shadows are being drawn with the CommandBuffer but it still has the shadow artifacts so the stencil part isn't working for the shadows.



I'm not too worried about this being performant just yet, I'll get it working first then optimize, so right now I only have 2 lights and 4 items in the scene and just dragged them onto public arrays in the inspector.
I'm not sure why the stencil buffer isn't working for the shadows (works fine for the colour pass), so I guess I'll just share my scripts, can you see anything obviously wrong?

Item Shader:

Code (CSharp):

1. Shader "Custom/ItemShader"
2. {
3.     Properties
4.     {
5.         _Color ("Color", Color) = (1,1,1,1)
6.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
7.         _BumpMap("Bumpmap", 2D) = "bump" {}
8.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
9.         _Metallic ("Metallic", Range(0,1)) = 0.0
10.     }
11.     SubShader
12.     {
13.         Tags { "Queue" = "Geometry" }
14.         LOD 200
15.  
16.         Stencil
17.         {
18.             Ref 1
19.             Comp notequal
20.         }
21.  
22.         CGPROGRAM
23.         // Physically based Standard lighting model, and enable shadows on all light types
24.         #pragma surface surf Standard fullforwardshadows
25.      
26.         // Use shader model 3.0 target, to get nicer looking lighting
27.         #pragma target 3.0
28.            
29.         sampler2D _MainTex;
30.         sampler2D _BumpMap;
31.  
32.         struct Input
33.         {
34.             float2 uv_MainTex;
35.             float2 uv_BumpMap;
36.         };
37.  
38.         half _Glossiness;
39.         half _Metallic;
40.         fixed4 _Color;
41.  
42.         // Add instancing support for this shader. You need to check 'Enable Instancing' on materials that use the shader.
43.         // See https://docs.unity3d.com/Manual/GPUInstancing.html for more information about instancing.
44.         // #pragma instancing_options assumeuniformscaling
45.         UNITY_INSTANCING_BUFFER_START(Props)
46.         // put more per-instance properties here
47.         UNITY_INSTANCING_BUFFER_END(Props)
48.  
49.         void surf (Input IN, inout SurfaceOutputStandard o)
50.         {
51.             // Albedo comes from a texture tinted by color
52.             fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
53.             o.Albedo = c.rgb;
54.             o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap));
55.  
56.             // Metallic and smoothness come from slider variables
57.             o.Metallic = _Metallic;
58.             o.Smoothness = _Glossiness;
59.             o.Alpha = c.a;
60.         }
61.         ENDCG
62.  
63.         Pass
64.         {
65.             Name "ItemStencilPass"
66.             ColorMask 0
67.  
68.             Stencil
69.             {
70.                 Ref 1
71.                 Comp always
72.                 Pass replace
73.             }
74.         }
75.     }
76.     //FallBack "Diffuse"
77. }
78.  

Shadow caster shader:

Code (CSharp):

1. Shader "Custom/ItemShadowCaster"
2. {
3.     SubShader
4.     {
5.         Pass
6.         {
7.             Stencil
8.             {
9.                 Ref 1
10.                 Comp notequal
11.             }
12.  
13.             Tags { "Queue" = "Transparent" "LightMode" = "ShadowCaster" }
14.             ZWrite On ZTest Less Cull Off
15.             Offset 1, 1
16.  
17.             CGPROGRAM
18.             #pragma vertex vert
19.             #pragma fragment frag
20.             #pragma multi_compile_shadowcaster
21.             #pragma fragmentoption ARB_precision_hint_fastest
22.  
23.             #include "UnityCG.cginc"
24.  
25.             struct v2f
26.             {
27.                 V2F_SHADOW_CASTER;
28.             };
29.  
30.             v2f vert(appdata_base v)
31.             {
32.                 v2f o;
33.                 TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
34.                 return o;
35.             }
36.  
37.             float4 frag(v2f i) : COLOR
38.             {
39.                 SHADOW_CASTER_FRAGMENT(i)
40.             }
41.             ENDCG
42.         }
43.     }
44. }
45.  

Shadow renderer on the camera (does most of the work):

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.Rendering;
5.  
6. //-------------------------------------------------------------------
7. //-------------------------------------------------------------------
8. [ExecuteInEditMode]
9. public class ItemShadowRenderer : MonoBehaviour
10. {
11.     public Material m_ShadowCasterMaterial = null;
12.  
13.     public GameObject[] m_aLights = null;
14.     public GameObject[] m_aItems = null;
15.  
16.     private Light[] m_aLightScripts = null;
17.     private ItemBase[] m_aItemScripts = null;
18.  
19.     //-------------------------------------------------------------------
20.     //-------------------------------------------------------------------
21.     private void ResetRenderer()
22.     {
23.         if(m_aLights == null)
24.             return;
25.  
26.         m_aLightScripts = new Light[m_aLights.Length];
27.         for(int i = 0; i < m_aLights.Length; ++i)
28.         {
29.             if(m_aLights[i])
30.             {
31.                 Light light = m_aLights[i].GetComponent<Light>();
32.                 light.RemoveAllCommandBuffers();
33.                 m_aLightScripts[i] = light;
34.             }
35.         }
36.  
37.         m_aItemScripts = new ItemBase[m_aItems.Length];
38.         for(int i = 0; i < m_aItems.Length; ++i)
39.         {
40.             if(m_aItems[i])
41.             {
42.                 m_aItemScripts[i] = m_aItems[i].GetComponent<ItemBase>();
43.                 m_aItemScripts[i].CreateCommandBuffer(m_ShadowCasterMaterial);
44.             }
45.         }
46.     }
47.  
48.     //-------------------------------------------------------------------
49.     //-------------------------------------------------------------------
50.     private void OnDisable()
51.     {
52.         ResetRenderer();
53.     }
54.  
55.     //-------------------------------------------------------------------
56.     //-------------------------------------------------------------------
57.     private void OnEnable()
58.     {
59.         ResetRenderer();
60.     }
61.  
62.     //-------------------------------------------------------------------
63.     //-------------------------------------------------------------------
64.     private void SortItemList()
65.     {
66.         for(int i = 0; i < m_aItemScripts.Length - 1; ++i)
67.         {
68.             for(int j = 0; j < m_aItemScripts.Length - 1 - i; ++j)
69.             {
70.                 if(m_aItemScripts[j].GetRenderDepth() < m_aItemScripts[j + 1].GetRenderDepth())
71.                 {
72.                     ItemBase swap = m_aItemScripts[j];
73.                     m_aItemScripts[j] = m_aItemScripts[j + 1];
74.                     m_aItemScripts[j + 1] = swap;
75.                 }
76.             }
77.         }
78.     }
79.  
80.     //-------------------------------------------------------------------
81.     //-------------------------------------------------------------------
82.     private void Update()
83.     {
84.         ResetRenderer();
85.  
86.         if(m_aLightScripts == null)
87.             return;
88.  
89.         if(m_aItemScripts == null)
90.             return;
91.  
92.         SortItemList();
93.  
94.         foreach(Light light in m_aLightScripts)
95.         {
96.             foreach(ItemBase item in m_aItemScripts)
97.             {
98.                 light.AddCommandBuffer(LightEvent.AfterShadowMapPass, item.GetCommandBuffer());
99.             }
100.         }
101.     }
102. }
103.  

On each items script to create the command buffer

Code (CSharp):

1. //-------------------------------------------------------------------
2.     //-------------------------------------------------------------------
3.     public void CreateCommandBuffer(Material shadowCaster)
4.     {
5.         if(shadowCaster == null)
6.             return;
7.  
8.         if(m_CommandBuffer == null)
9.         {
10.             MeshRenderer renderer = GetComponent<MeshRenderer>();
11.  
12.             m_CommandBuffer = new CommandBuffer();
13.             m_CommandBuffer.name = "Item Shadow Caster";
14.             m_CommandBuffer.DrawRenderer(renderer, shadowCaster, 0, 0);
15.  
16.             int nStencilPass = renderer.sharedMaterial.FindPass("ItemStencilPass");
17.             m_CommandBuffer.DrawRenderer(renderer, renderer.sharedMaterial, 0, nStencilPass);
18.         }
19.     }
20.  
21.     //-------------------------------------------------------------------
22.     //-------------------------------------------------------------------
23.     public CommandBuffer GetCommandBuffer()
24.     {
25.         return m_CommandBuffer;
26.     }

Any idea why the stencil buffer may not be working for the shadows?

 

Hmm ... I have a feint memory of Unity totally disabling stencils during some passes, even when using command buffers. They may have done that to the shadow passes, which would be unfortunate.

I’d check to make sure both passes are actually showing up in the frame debugger. Then I’d try using a copy of the shadowcaster pass instead of an empty pass for the stencil only pass. Then I’d check with RenderDoc to make sure the stencil state is set when they’re being rendered.

 

Yep, both passes are showing up in the frame debugger.
Changing it to a copy of the shadowcaster pass didn't seem to make a difference.
I had a look with RenderDoc, I've never used it before so I wasn't too sure about how, but I think it was indicating that the CommandBuffer never wrote to the stencil buffer. When looking at the colour pass I could see when it did the stencil test and afterwards the stencil showed as 1. But during the shadow pass it was always 0, even after rendering the shadow for an item.

Also I found another weird issue: Previously the items were illuminated only by a point light, but I tried using a directional light instead and now they cast shadows on themselves (I haven't changed any code, only the lights in the scene):

Any idea why that's happening?

 

I conducted some more tests: made a fresh project with just a couple of quads in it and made them a really simple shader that just wrote to the stencil buffer. Attached it to a command buffer and injected it into the one light in the scene. Tried every different hook but RenderDoc always shows the stencil buffer as being zero during the shadow pass. It works fine during the colour pass.

So I guess Unity just doesn't allow you to change the stencil buffer during the shadow pass? Is there any other way around this?

 

Yep, I didn’t bring this up because your examples were only with point lights. For the main shadow casting directional light on desktop and consoles, they cast onto the camera depth texture, which uses the same shadow caster pass to render a depth texture from the camera’s view (hence the name). You’d have to do the same two pass command buffer injection setup there too. On the camera’s AfterDepthTexture event.

In terms of being able to use stencils in the depth rendering passes? No. It seems like they really did totally lock it down, which is really annoying.

The work around would be you’d have to generate a mask using a second render texture that you alternate rendering to, and then sample from to reject pixels.

 

Sorry to keep bugging you about this, I really appreciate the help

I'm trying to fix the directional light but it's not casting any shadow for the custom shaders (it does cast shadows for normal objects). I'm injecting the same command buffer that works for the point lights into the camera.

This script is on the camera and I checked that Camera.current is valid.

Code (CSharp):

1. private void OnPreRender()
2. {
3.     ResetRenderer();
4.  
5.     if(m_aLightScripts == null)
6.         return;
7.  
8.     if(m_aItemScripts == null)
9.         return;
10.  
11.     SortItemList();
12.  
13.     foreach(ItemBase item in m_aItemScripts)
14.     {
15.         if(!item.gameObject.activeInHierarchy)
16.             continue;
17.  
18.         if(Camera.current)
19.             Camera.current.AddCommandBuffer(CameraEvent.AfterDepthTexture, item.GetCommandBuffer());
20.  
21.         foreach(Light light in m_aLightScripts)
22.         {
23.             if(light.gameObject.activeInHierarchy && light.enabled && light.type == LightType.Point)
24.             {
25.                 light.AddCommandBuffer(LightEvent.AfterShadowMapPass, item.GetCommandBuffer());
26.             }
27.         }
28.     }
29. }

Any idea why that doesn't work?

 

Honestly, no idea why it's not working. That looks correct to me. I'd again go back to the frame debugger and make sure you see them showing up in the list of render calls, and then check to see if they're showing up properly in the depth texture (you may need to mess with the slider at the top so you can see anything). If they are in the list but not visible go back to RenderDoc to see if you can figure out why they aren't showing up. Render doc should show you where the mesh is being rendered to in the depth buffer, SceneCamera.Render > UpdateDepthTexture > DepthPass.Job. Goto the Texture Viewer tab, select DS texture in the Outputs, select Wireframe Mesh in the Overlay. Then arrow through the list until you find your object. If you can't find it, try selecting the Mesh Viewer tab until you see the mesh you expect there. Make sure it's in the spot on screen you expect it to be (taking into account Unity renders upside down). If it's where you expect, and it's just not rendering anything, check the Pipeline State's Output Merger. Depth State should have both enabled & write checked, and Func should be Greater Equal.

 

Everything looks fine, it's showing in both the frame debugger and in RenderDoc but not in the final render.

That circle is it


Pipeline state:


As I arrow through in RenderDoc, it's there for Depth-only Pass #1 which appears to be the directional light shadows and for Depth-only Pass #2 which looks like maybe ambient occlusion.
Then it does Depth-pass #3 which appears to be point lights and so it doesn't appear there as it's not near any point lights, and then there's the Color pass and no shadow appears.

Now sure how the shadow is being lost between the depth pass and the final render.

 

I spent a little time trying to look into this. As best I can tell the stencil state is set for the camera depth and shadow map rendering, and I could get it to work for the camera's depth texture. Again, I have no idea why it's not working for you as it really "just worked" for me with essentially the same code as what you posted. But I could not get it to work for the shadow maps, and this appear to be because Unity is using 16 bit depth render textures for shadow maps which don't support stencils.

As for the "render your own mask" option. It doesn't seem like this is workable either due to various reasons. But basically it seems to be impossible to render to another texture during the shadow map passes and ever render to the shadow map again.

If this is a path you truly want to go down, the answer may be you have to write your own SRP.

 

Oh I didn't try setting the stencil during the camera depth, only during the point light's shadow pass so that makes sense.
Shame this won't work.

Thanks for all your help with this anyway.