Why does addshadow cause shadows to be projected on top of the caster if the caster is stenciled?

If you have #pragma addshadow defined on a stenciled object, its shadow will be projected on top of it instead of behind it. Other depth effects are broken too. Removing addshadow and having just fullforwardshadows leads to the expected behaviour. (Unity 2018.4.11)


Sample Shader

Code (CSharp):

1. Shader "IslandTest"
2. {
3.      Properties
4.      {
5.          _Color0("Color 0", Color) = (0.2270826,0.764151,0.2394226,0)
6.          [HideInInspector] __dirty( "", Int ) = 1
7.      }
8.      SubShader
9.      {
10.          Tags{ "RenderType" = "Opaque"  "Queue" = "Geometry+0" "IgnoreProjector" = "True" }
11.          Cull Back
12.          Stencil
13.          {
14.              Ref 1
15.              Comp Equal
16.          }
17.          CGPROGRAM
18.          #pragma target 3.0
19.          #pragma surface surf Standard keepalpha addshadow fullforwardshadows exclude_path:deferred nodynlightmap noforwardadd
20.          struct Input
21.          {
22.              half filler;
23.          };
24.          uniform half4 _Color0;
25.          void surf( Input i , inout SurfaceOutputStandard o )
26.          {
27.              o.Albedo = _Color0.rgb;
28.              o.Alpha = 1;
29.          }
30.          ENDCG
31.      }
32.      Fallback "Diffuse"
33.      CustomEditor "ASEMaterialInspector"
34. }

 

The main issue here is with out

addshadow

the shader is using the shadow caster pass from the fallback shader. The fallback shader doesn't have any stencil tests, so it always renders. The way Unity does its main directional light's shadows is by casting onto the camera depth texture to generate a screen space shadow texture rather than sampling the shadow map directly in the object's shader. The camera depth texture is generated by rendering the view using each opaque objects' shadow caster pass.

When you do add

addshadow

your shadow caster passes now are doing the stencil compare, but aren't getting rendered since either the stencil object itself doesn't have a shadow caster pass of it's own, or because Unity doesn't respect the queue when rendering the depth texture so the order the mask object gets rendered in isn't easily controlled.

The one thing I don't understand is why your objects are still casting shadows. Are you using hidden shadow only objects? If your stencil mask does have a shadow caster pass, you might just be getting "lucky" and it's rendering before those objects meaning the stencil value is set before they render.

 

Thanks for the thorough explanation! This is just a simple surface shader spit out by Amplify Shader Editor. When I do the checkbox for stencil testing, this is what it gives me. I've been able to work around it being a bit strange but was curious what the cause was.