[SOLVED] Using the shadows as a mask for surface shaders?

Is there a way to access the shadows of the scene inside of a surface shader?

The only way i know is to use a custom lighting model, where you have access to the attenuation value.
But i would like to stay away from custom lighting functions for this one! I want to mask my fresnel effect to only appear in the shadows of my object-

 

If by "in the shadows" you mean "in the main directional light's shadow for desktop", then yes. To get the main shadow only really requires the screen position.

Code (csharp):

1. // input struct
2. struct input
3. {
4.     // other stuff
5.     float4 screenPos;
6. }
7.  
8. // in surf function
9. float shadow = 1;
10. #if defined(SHADOWS_SCREEN) && !defined(UNITY_NO_SCREENSPACE_SHADOWS);
11. shadow = unitySampleShadow(IN.screenPos);
12. #endif

Alternatively you could use a custom lighting function that just calls the LightingStandard functions.

 

Yes, just the directional lights shadows! Nothing happend though- As far as my tests go, the if function never gets used. I also tried different setups like foward/deferred or baked/realtime shadows, but makes no difference. (In this case, the cube in the middle should start glowing only in the lit parts (just for testing purposes)

Code (CSharp):

1. Shader "Custom/testShadowMask" {
2.     Properties {
3.         _Color ("Color", Color) = (1,1,1,1)
4.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
5.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
6.         _Metallic ("Metallic", Range(0,1)) = 0.0
7.     }
8.     SubShader {
9.         Tags { "RenderType"="Opaque" }
10.         LOD 200
11.  
12.         CGPROGRAM
13.  
14.         #pragma surface surf Standard fullforwardshadows
15.         #pragma target 3.0
16.  
17.         sampler2D _MainTex;
18.  
19.         struct Input {
20.             float2 uv_MainTex;
21.             float3 viewDir;
22.             float4 screenPos;
23.         };
24.  
25.         half _Glossiness;
26.         half _Metallic;
27.         fixed4 _Color;
28.  
29.         void surf (Input IN, inout SurfaceOutputStandard o) {
30.            
31.             float shadow = 1;
32.            
33.             #if defined(SHADOWS_SCREEN) && !defined(UNITY_NO_SCREENSPACE_SHADOWS)
34.                 shadow = unitySampleShadow(IN.screenPos);
35.                 o.Emission = 1 * shadow;
36.             #endif
37.  
38.             fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color * shadow;
39.             o.Albedo = c.rgb;
40.             o.Metallic = _Metallic;
41.             o.Smoothness = _Glossiness;
42.             o.Alpha = c.a;
43.         }
44.         ENDCG
45.     }
46.     FallBack "Diffuse"
47. }
48.  

 

So, the above trick will only work with forward rendering and real time shadows. It takes a bit more work to get the lightmaps to check if you're in shadow, and it's impossible to get access to the shadows during the surface shader with deferred as those aren't rendered until long after the object.

Also there's a "fun" feature with Surface Shaders. When it does the initial code generation, it looks over the shader to figure out what Input values and surf output values are actually used. When it does that look over the shader it doesn't setup most of the defines, so because IN.screenPos and o.Emission is only used inside that #if block it doesn't think the shader uses them and chooses to skip passing & setting the screenPos, or using the o.Emission...

Like I said, "fun".

There's some secret define to detect when you're doing the shader gen pass, but it got mentioned to me once on twitter like 3 years ago and I've never been able to find the tweet again.

Anyway, the hacky work around is to use those values in a stupid way.

Code (csharp):

1. // outside a function define a dummy value, no need to put into the properties
2. // this will always be initialized as 0.0
3. half _DummyZero;
4.  
5. // in the surf function
6. half fresnel = pow(1 - saturate(dot(IN.viewDir, IN.worldNormal)), 4);
7. half3 edgeGlow = fresnel * _Emission;
8.  
9. // set the emission to the screenPos multiplied by the dummy uniform
10. // this will always be zero, because _DummyZero is always zero, but the shader gen doesn't know that
11. // the result is the shader generator will make sure both screenPos and Emission code gets added
12. o.Emission = IN.screenPos.x * _DummyZero;
13.  
14. #if defined (SHADOWS_SCREEN) && !defined(UNITY_NO_SCREENSPACE_SHADOWS)
15.     half shadow = unitySampleShadow(IN.screenPos);
16.     o.Emission = edgeGlow * (1 - shadow);
17. #endif

 

Thanks for the long answer! That's super helpful to know!xD

 

Also another approach right here with commandbuffers:

I just found the solution! So I was also searching for a way to sample the screen space shadow texture with a commandbuffer and noticed, that they use this technique in the 3D GameKit from Unity! And it just seems to work straight out of the box this way! I saw many questions about this and nobody ever posted his working code, so here ya go!

Code that should be attached to the main directional light:

Code (CSharp):

1. using UnityEditor;
2. using UnityEngine;
3. using UnityEngine.Rendering;
4.  
5. namespace Gamekit3D
6. {
7.     [ExecuteInEditMode]
8.     public class CopyShadowMap : MonoBehaviour
9.     {
10.         CommandBuffer cb = null;
11.  
12.         void OnEnable()
13.         {
14.             var light = GetComponent<Light>();
15.             if (light)
16.             {
17.                 cb = new CommandBuffer();
18.                 cb.name = "CopyShadowMap";
19.                cb.SetGlobalTexture("_DirectionalShadowMask", new RenderTargetIdentifier(BuiltinRenderTextureType.CurrentActive));
20.                 light.AddCommandBuffer(UnityEngine.Rendering.LightEvent.AfterScreenspaceMask, cb);
21.             }
22.         }
23.  
24.         void OnDisable()
25.         {
26.             var light = GetComponent<Light>();
27.             if (light)
28.             {
29.                 light.RemoveCommandBuffer(UnityEngine.Rendering.LightEvent.AfterScreenspaceMask, cb);
30.             }
31.         }
32.     }
33. }

And in the shader, access the map likes this:

Code (CSharp):

1. sampler2D _DirectionalShadowMask;

and in the surf/fragment shader:

Code (CSharp):

1. float shadowmask = tex2D(_DirectionalShadowMask,screenUV).b;

Theres also this Macro written there, but I'm not sure for what it's used:

Code (CSharp):

1. UNITY_DECLARE_SHADOWMAP(_DirectionalShadowMap);

(I just didnt used it and it still worked) Also thanks to @bgolus for clearing this up:
That's for defining a shadow map texture object ... which technically the screen space shadow texture is not so can be ignored. You've already defined it as a sampler2D, which is what it is.

Anyways, I hope this helps someone!