Question Jagged shadows under custom shadow

(Code at the end of the thread.)

Hello there,

I am working on a toon unlit shader that calculates its own shadow from the main light in the frag function and adds the shadows it receives from nearby objects, like so :





So far so good, but I have also added a dithering effect to clip pixels that are too close to the player's camera, allowing me to do a smooth transition to prevent the player from clipping immediately through the model. However, when I get too close to the shadows, there is a wierd shadow with the same color as the ground that is not supposed to be there :



As you can see, it's hidden beneath the yellow shadow (the one calculated in the shader), and is very jagged. It also happens to cover other objects that pass behind it, preventing me from seeing them entirely. The same thing happens with received shadows :



This is really annoying, since the only ways I've found out to make this artifact dissapear is to either :
- Change the bias on the main light (but then it applies to all other shadows, making them look very ugly)
- Change the quality settings (changing the shadows to high resolution and four cascades only mitigated the issue while not making it completely dissapear)
- Change the render mode from Alpha Test to Transparent, but then I cannot receive shadows at all anymore.

Is this a problem related to the shader itself or with how Unity calculates lighting ? In the script, I calculate the shadow attenuation and multiply it with the lightIntensity to get all the areas where the texture must be replaced with the colored shadow. Here is the code (stripped down to the essential for understanding) :

Code (CSharp):

1.  
2.  
3. Properties
4.     {
5.         [Header(Textures)]
6.         [Space(20)]
7.         _MainTex("Texture", 2D) = "white" {}
8.  
9.        
10.         [Space(20)]
11.         [Header(Global Parameters)]
12.         [Space(20)]
13.  
14.         _Alpha("Global Alpha", Range(0, 1)) = 1
15.  
16.         [HDR] _MainTexColor("Main Texture Color", Color) = (1, 1, 1, 1)
17.         [HDR] _LitSideColor("Lit Side Color", Color) = (1, 1, 1, 1)
18.        
19.         [Space(20)]
20.         [Header(Lighting Parameters)]
21.         [Space(20)]
22.  
23.         [NoScaleOffset] _ShadowTex("Shadow Texture", 2D) = "white" {}
24.         [Toggle] _UseShadow("Use Shadow", float) = 1
25.         [Toggle] _UseShadowTex("Use Shadow Texture", float) = 0
26.        
27.         [Space(20)]
28.  
29.         [Toggle] _DrawShadowAsSolidColor("Draw Shadow As Solid Color", float) = 0
30.         [Toggle] _LightingAffectsShadow("Lighting Affects Shadow", float) = 0
31.        
32.         [Space(20)]
33.  
34.         _ShadowTexSize("Shadow Texture Size", float) = 0.1
35.         _ShadowTexRotation("Shadow Texture Rotation", Range(0.0, 360)) = 0
36.         _ShadowTexTransparencyCoef("Shadow Texture Transparency Coef", Range(0.0, 1.0)) = 1.0
37.  
38.         [Space(20)]
39.  
40.         [HDR] _ShadowColor("Shadow Color", Color) = (0.5, 0.5, 0.5, 1)
41.         [IntRange]_ShadowNbSteps("Shadow Steps", Range(1, 16)) = 2
42.         _ShadowStepWidth("Shadow Step Width", Range(0, 1)) = 0.25
43.         _ShadowFalloff("Shadow Falloff", Range(-10, 10)) = 0
44.  
45.     }
46.  
47.             struct appdata
48.             {
49.                 float4 vertex : POSITION;
50.                 float4 uv : TEXCOORD0;
51.                 float3 normal : NORMAL;
52.                 float4 tangent : TANGENT;
53.             };
54.  
55.             struct v2f
56.             {
57.                 float4 pos : SV_POSITION;
58.                 float3 worldNormal : NORMAL;
59.                 float2 uv : TEXCOORD0;
60.                 float3 viewDir : TEXCOORD1;
61.                 float4 screenPos : TEXCOORD2;
62.                 // Macro found in Autolight.cginc. Declares a vector4
63.                 // into the TEXCOORD3 semantic with varying precision
64.                 // depending on platform target.
65.                 SHADOW_COORDS(3)
66.  
67.                
68.                
69.             };
70.  
71.  
72.             v2f vert(appdata v)
73.             {
74.                 v2f o;
75.  
76.                
77.  
78.                 o.pos = UnityObjectToClipPos(v.vertex);
79.                 o.worldNormal = UnityObjectToWorldNormal(v.normal);
80.                 o.uv = TRANSFORM_TEX(v.uv, _MainTex);
81.                 o.viewDir = WorldSpaceViewDir(v.vertex);
82.                 o.screenPos = ComputeScreenPos(o.pos);
83.  
84.                 // Defined in Autolight.cginc. Assigns the above shadow coordinate
85.                 // by transforming the vertex from world space to shadow-map space.
86.                 TRANSFER_SHADOW(o)
87.  
88.                
89.  
90.  
91.                     return o;
92.                 }
93.  
94. float4 frag(v2f i) : SV_Target
95.             {
96.  
97.                 float4 white = float4(1, 1, 1, 1);
98.                 float4 black = float4(0, 0, 0, 0);
99.  
100.                 float3 viewDir = normalize(i.viewDir);
101.  
102.                 // Samples the shadow map, returning a value in the 0...1 range,
103.                 // where 0 is in the shadow, and 1 is not.
104.                 float shadowAtten = SHADOW_ATTENUATION(i);
105.  
106.              
107.                 ///////////////////////////// LIGHT CALCULATION ///////////////////////////////
108.  
109.  
110.                 // Calculate illumination from directional light.
111.                 // _WorldSpaceLightPos0 is a vector pointing the OPPOSITE
112.                 // direction of the main directional light.
113.                 float NdotL = dot(_WorldSpaceLightPos0, i.worldNormal);
114.  
115.                 //stretch values so each whole value is one step
116.                 NdotL = NdotL / _ShadowStepWidth;
117.  
118.                 //Pour permettre de décaler la limite entre ombres et lumière
119.                 NdotL += _ShadowFalloff;
120.  
121.                 //make steps harder
122.                 float lightIntensity = floor(NdotL);
123.  
124.                 // calculate smoothing in first pixels of the steps and add smoothing to step, raising it by one step
125.                 // (that's fine because we used floor previously and we want everything to be the value above the floor value,
126.                 // for example 0 to 1 should be 1, 1 to 2 should be 2 etc...)
127.                 float change = fwidth(NdotL);
128.                 float smoothing = smoothstep(0, change, frac(NdotL));
129.                 lightIntensity = lightIntensity + smoothing;
130.  
131.                 // bring the light intensity back into a range where we can use it for color
132.                 // and clamp it so it doesn't do weird stuff below 0 / above one
133.                 lightIntensity = lightIntensity / _ShadowNbSteps;
134.                 lightIntensity = saturate(lightIntensity);
135.  
136.  
137.  
138.  
139.                 //Adding shadows to the lightIntensity
140.  
141.                 float attenuationChange = fwidth(shadowAtten) * 0.5;
142.                 float shadow = smoothstep(0.5 - attenuationChange, 0.5 + attenuationChange, shadowAtten);
143.                 lightIntensity = lightIntensity * shadow;
144.  
145.                
146.  
147.  
148.  
149.                 //Invert lightIntensity for shadow
150.                 float inverselightIntensity = 1 - lightIntensity;
151.  
152.  
153.  
154.  
155.                 ///////////////////////////// DITHERING ///////////////////////////////
156.  
157.  
158.  
159.                 //value from the dither pattern
160.                 float2 screenPos = i.screenPos.xy / i.screenPos.w;
161.                 float2 ditherCoordinate = screenPos * _ScreenParams.xy * _DitherPattern_TexelSize.xy;
162.                 float ditherValue = tex2D(_DitherPattern, ditherCoordinate).r;
163.  
164.                 //get relative distance from the camera
165.                 float relDistance = i.screenPos.w;
166.                 relDistance = relDistance - _MinDistanceMesh;
167.                 relDistance = relDistance / (_MaxDistanceMesh - _MinDistanceMesh);
168.                 //discard pixels accordingly
169.                 clip(relDistance - ditherValue);
170.  
171.  
172.  
173.  
174.  
175.                 ///////////////////////////// COLORS ///////////////////////////////
176.  
177.  
178.  
179.  
180.                 //calculate final color
181.                 float4 color;
182.                 float4 inverseColor;
183.                 float4 shadowTex;
184.  
185.                
186.                
187.                 float4 tex = tex2D(_MainTex, i.uv) * _MainTexColor * _TextureEmissionIntensity * occlusion;
188.  
189.  
190.  
191.  
192.                 //To get the shadow texture in screen space pos
193.                 float2 screenUV = GetScreenUV(i.screenPos, _ShadowTexSize);
194.                 screenUV = rotateUV(screenUV, _ShadowTexRotation);
195.                 shadowTex = tex2D(_ShadowTex, screenUV) * _ShadowEmissionIntensity;
196.  
197.                 //The shadow texture adapted to lighting
198.                 float4 shouldLightingAffectShadow = lerp(white, _LightColor0, _LightingAffectsShadow) * _ShadowColor;
199.                 shadowTex = shadowTex * _UseShadowTex * (inverselightIntensity) + (inverselightIntensity) * (1 - _UseShadowTex);
200.                 shadowTex = lerp((inverselightIntensity), shadowTex, _ShadowTexTransparencyCoef);
201.                 shadowTex = shadowTex * shouldLightingAffectShadow;
202.                
203.  
204.  
205.                 float4 lightenSide = _LitSideColor * tex * _LightColor0;
206.  
207.                 float4 shadowSide = shadowTex;
208.                
209.  
210. //Reverts the color for the shadow
211.                 color = lerp(shadowSide * tex, lightenSide, lightIntensity);
212.                 inverseColor = lerp(shadowSide, lightenSide, lightIntensity);
213.                 color = lerp(color, inverseColor, _DrawShadowAsSolidColor);
214.  
215.  
216.                 //Alpha
217.                 color.a = _Alpha;
218.  
219.                 return color;
220.             }
221.  
222.  
223.  

 

If you dither the main lit pass, you also need to dither the shadow caster pass, but only when being used for the camera depth texture. This is because the main directional light's shadows are rendered to a screen space texture using the depth texture as the shadow receiver. You're seeing the "background color" show through because the shadow receiver doesn't know it shouldn't be shadowing the object on those pixels, and then the background object is sampling the screen space shadow texture which still has the shadows for that object.

So that means you'll need to write a custom shadowcaster pass, and only do the dither when it's rendering to the camera depth texture. Unfortunately there's no 100% accurate way to know when you're rendering to the camera depth texture vs a shadow caster pass. This is the closest / most consistent method I know of:
https://forum.unity.com/threads/dif...d-shadow-receiver-in-5-2.362653/#post-2353713

Note, the case that's labelled "camera depth" will also happen for shadow casting spot lights if the bias is set to zero. That's pretty rare for anyone to do, but it is still an issue.

 

Thanks for your input, I guess I'll have to dig deeper into the shadowcaster pass then (i've found one in the Unity manuals but it's barebones).

So far I've used an alternate solution that gets rid of the additional shadows but brings even more problems : I've set the ZWrite to a custom boolean that allows me to get rid of the green shadows, but that prevents me from casting shadows or receiving them properly, which is even worse.