Shadow blending

Hey there!

I'm trying to have my (dynamic) objects cast real time shadow on baked lighting in my scene.

I've found out that shader Mobile/Diffuse, Directional Light with Mixed mode and lightmapper with Subtractive mode do exactly what I need it.

However... It would be great if I could achieve that in a custom vertex/fragment shader.

So far I've done this and it almost works well:

Code (CSharp):

1. Shader "Main/Enviroment/Enviroment LightMap" {
2.   Properties {
3.     _MainTex ("Tex", 2D) = "white" {}
4.   }
5.     SubShader {
6.         Pass {
7.  
8.  
9.             Tags { "LightMode" = "ForwardBase" }
10.  
11.             CGPROGRAM
12.  
13.             #pragma vertex vert
14.             #pragma fragment frag
15.             #include "UnityCG.cginc"
16.  
17.  
18.             #pragma multi_compile_fwdbase
19.             #pragma noforwardadd
20.  
21.             #include "AutoLight.cginc"
22.  
23.             uniform sampler2D _MainTex;
24.  
25.             struct vertexInput
26.             {
27.               half4 vertex : POSITION;
28.               half4 coords : TEXCOORD0;
29.               half4 lightmap : TEXCOORD1;
30.             };
31.  
32.             struct vertexOutput
33.             {
34.                 half4 pos : SV_POSITION;
35.                 half4 uv0 : TEXCOORD0;
36.                 half4 lightmap : TEXCOORD1;
37.                 LIGHTING_COORDS(2,3)
38.             };
39.  
40.  
41.             vertexOutput vert(vertexInput v) {
42.                 vertexOutput o;
43.  
44.  
45.                 o.pos = UnityObjectToClipPos (v.vertex);
46.  
47.                 o.uv0 = v.coords;
48.  
49.  
50.                 #ifdef LIGHTMAP_ON
51.                   o.lightmap.xy = v.lightmap.xy * unity_LightmapST + unity_LightmapST.zw;
52.                 #endif
53.  
54.                 TRANSFER_VERTEX_TO_FRAGMENT(o);
55.  
56.                 return o;
57.             }
58.  
59.             fixed4 frag(vertexOutput i) : COLOR {
60.  
61.                 half4 col = tex2D(_MainTex, i.uv0);
62.  
63.                 fixed atten = LIGHT_ATTENUATION(i);
64.  
65.                 #ifdef LIGHTMAP_ON
66.                         col.rgb *= DecodeLightmap(UNITY_SAMPLE_TEX2D(unity_Lightmap, i.lightmap));
67.                 #endif
68.  
69.  
70.                 return col * atten;
71.  
72.             }
73.  
74.             ENDCG
75.         }
76.         Pass
77.         {
78.           Name "ShadowCaster"
79.           Tags { "LightMode" = "ShadowCaster" }
80.  
81.           ZWrite On ZTest LEqual Cull Off
82.  
83.           CGPROGRAM
84.           #pragma vertex vert
85.           #pragma fragment frag
86.           #pragma target 2.0
87.           #pragma multi_compile_shadowcaster
88.           #include "UnityCG.cginc"
89.  
90.           struct v2f {
91.             V2F_SHADOW_CASTER;
92.           };
93.  
94.           v2f vert( appdata_base v )
95.           {
96.             v2f o;
97.             UNITY_SETUP_INSTANCE_ID(v);
98.             TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
99.             return o;
100.           }
101.  
102.           float4 frag( v2f i ) : SV_Target
103.           {
104.             SHADOW_CASTER_FRAGMENT(i)
105.           }
106.           ENDCG
107.         }
108.     }
109. }
110.  

In my shader everything seems to be working fine except for shadow blending - what am I missing? What am I doing wrong? What other magical MACROS do I need to use?

In default mobile/diffuse shader the "Realtime Shadow Color" in Lighting/Mixed Lighting controls how much dynamic shadows blends with lightmaps. How can I achieve that?

EDIT:

I've added some pictures

 

Sorry for bumping this topic but this seems achievable.. Any help would be welcome.

 

You're multiplying the shadows on top of the lightmaps so the result you're getting makes sense.

Try something like this: Change:

col.rgb *= DecodeLightmap(UNITY_SAMPLE_TEX2D(unity_Lightmap, i.lightmap));

to

col.rgb *= min(DecodeLightmap(UNITY_SAMPLE_TEX2D(unity_Lightmap, i.lightmap)),atten);

and remove the * atten at the return.

 

Thank you for your reply!

min() returns the minimum values of A and B - A is my shadow which is completly black (0) and B is the lightmap (other values in range of 0-1). That way this function always returns 0 where my shadow is - this has the exactly same effect as previously.

In Mobile/Diffuse the shadow color and transparency depends on the "Realtime Shadow Color" in Lighting-> Mixed Lighting (Subtractive). If I could somehow 'connect' this color picker value to my shader I could probably make the min() function work just by setting shadow other values than black - that would result in blending between those values.

 

You're right. The way I did it a while back was I had a script that set my own shadow color with Shader.SetGlobalColor and then I'dd add that to atten. But there probably is a way to get the Realtime Shadow Color as well. I just don't know it from the top of my head.

 

I've looked inside compiled shader variant for Mobile/Diffuse but there are so many magical Unity macros that 'do stuff' that I am pulling my hair out to figure this thing out Did you ever tried to make that work?

 

Ok, so I am a lot closer now.

I've figured out that there is a "unity_ShadowColor" variable and it is responsible for "Realtime Shadow Color" color picker in Lighting -> Mixed Lighting (Subtractive).

Code (CSharp):

1. Shader "Lit/Diffuse With Shadows"
2. {
3.     Properties
4.     {
5.         [NoScaleOffset] _MainTex ("Texture", 2D) = "white" {}
6.           _shadowColor ("shadow", color) = (1,1,1,1)
7.     }
8.     SubShader
9.     {
10.         Pass
11.         {
12.             Tags {"LightMode"="ForwardBase"}
13.             CGPROGRAM
14.             #pragma vertex vert
15.             #pragma fragment frag
16.             #include "UnityCG.cginc"
17.             #include "Lighting.cginc"
18.  
19.             // compile shader into multiple variants, with and without shadows
20.             // (we don't care about any lightmaps yet, so skip these variants)
21.             #pragma multi_compile_fwdbase
22.             // shadow helper functions and macros
23.             #include "AutoLight.cginc"
24.  
25.             struct vertexInput {
26.                 half4 vertex : POSITION;
27.                 half3 normal : NORMAL;
28.                 half4 texcoord : TEXCOORD0;
29.                 half4 lightmap : TEXCOORD2;
30.                 UNITY_VERTEX_INPUT_INSTANCE_ID
31.             };
32.  
33.             float3 _shadowColor;
34.  
35.             struct vertexOutput
36.             {
37.                 half4 pos : SV_POSITION;
38.                 half2 uv0 : TEXCOORD0;
39.                 UNITY_SHADOW_COORDS(1) // put shadows data into TEXCOORD1
40.                 half3 diff : COLOR0;
41.                 half3 ambient : COLOR1;
42.                 half4 lightmap : TEXCOORD2;
43.                 half3 worldPos : TEXCOORD3;
44.             };
45.             vertexOutput vert (vertexInput v)
46.             {
47.                 vertexOutput o;
48.  
49.  
50.  
51.                 half3 worldNormal = UnityObjectToWorldNormal(v.normal);
52.  
53.                 half3 worldPos = mul(unity_ObjectToWorld, v.vertex).rgb;
54.                 /*half nl = max(0, dot(worldNormal, _WorldSpaceLightPos0.xyz));*/
55.                 /*o.diff = nl * _LightColor0.rgb;*/
56.                 o.ambient = ShadeSH9(half4(worldNormal,1));
57.                 // compute shadows data
58.  
59.                 o.pos = UnityObjectToClipPos (v.vertex);
60.  
61.                 o.uv0 = v.texcoord;
62.  
63.                 o.worldPos = worldPos;
64.  
65.                 #ifdef LIGHTMAP_ON
66.                   o.lightmap.xy = v.lightmap.xy * unity_LightmapST + unity_LightmapST.zw;
67.                 #endif
68.  
69.                 UNITY_TRANSFER_SHADOW(o, o.lightmap.xy)
70.  
71.                 return o;
72.             }
73.  
74.             sampler2D _MainTex;
75.  
76.             half4 frag (vertexOutput i) : SV_Target
77.             {
78.                 UNITY_SETUP_INSTANCE_ID(i)
79.                 half4 col = tex2D(_MainTex, i.uv0);
80.  
81.                 half3 lightMap = DecodeLightmap(UNITY_SAMPLE_TEX2D(unity_Lightmap, i.lightmap));
82.  
83.                 half3 shadow = (1 - UNITY_SHADOW_ATTENUATION(i, i.worldPos));
84.  
85.                 half3 substractedLight = lightMap - shadow;
86.  
87.                 col.rgb *= max(substractedLight, unity_ShadowColor);
88.  
89.                 return half4(col.rgb,1);
90.             }
91.             ENDCG
92.         }
93.  
94.         Pass
95.         {
96.           Name "ShadowCaster"
97.           Tags { "LightMode" = "ShadowCaster" }
98.  
99.           ZWrite On ZTest LEqual Cull Off
100.  
101.           CGPROGRAM
102.           #pragma vertex vert
103.           #pragma fragment frag
104.           #pragma target 2.0
105.           #pragma multi_compile_shadowcaster
106.           #include "UnityCG.cginc"
107.  
108.           struct vertexInput {
109.               half4 vertex : POSITION;
110.               half3 normal : NORMAL;
111.               half4 texcoord : TEXCOORD0;
112.               UNITY_VERTEX_INPUT_INSTANCE_ID
113.           };
114.  
115.           struct vertexOutput{
116.             V2F_SHADOW_CASTER;
117.           };
118.  
119.           vertexOutput vert( vertexInput v )
120.           {
121.             vertexOutput o;
122.             UNITY_SETUP_INSTANCE_ID(v);
123.             TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
124.             return o;
125.           }
126.  
127.           half4 frag(vertexOutput i) : SV_Target
128.           {
129.             SHADOW_CASTER_FRAGMENT(i)
130.           }
131.           ENDCG
132.         }
133.       }
134.     }
135.  

This works almost well. I can use the "Realtime Shadow Color" to change the 'blackness' of the shadows in order to blend them. However using this current formula I am adjusting lightmaps too - this isn't ideal cause in order for shadows to blend I need to push this color rather far resulting in lightmap losing its darkness and changing overall look. Perfect scenario would be - "Realtime Shadow Color" adjusts only the realtime shadow colors and leave lightmaps out of it.

 

hey gaxx, thank you so much for this shader. this was exactly what i was looking for.
i had issues to get an unlit lightmp shader to support shadows correctly but thanks to you they work exactly like expected! thumbs up!

Edit:
i checked it a little more deeply and it seems to have some issues with the blending between dynamic received shadows and the lightmap baked shadows

EDIT 2:
i fixed the issues that i had with your shader with these changes:

Code (CSharp):

1.  
2. float atten = 1-LIGHT_ATTENUATION(i);
3. half3 substractedLight = lightMap - atten * i.diff; // - shadow;
4. col.rgb *= min(lightMap, max(substractedLight, unity_ShadowColor));

nonetheless, big thanks!

 

Hey!

I'll check your solution as soon as I'll dig up that thing - it's been a while since then.

Thanks for reply

EDIT:

Ok, I did check how this changes would affect and yes, this blends same as Mobile/Diffuse! Great that you solved it!

There are problems with this approach in general. In order to blend correctly between lightmap and dynamic shadows, you need to know the correct color for unity_ShadowColor. Too much black and you have overlapping shadows, too much white and your shadow is not visable. Another problem is when you have multiple levels with different lightning scenarios. This value must be controlled for proper blend. This is unpractical approach unfortunately and propably leave to much free space for errors.