Adding Point Lighting and Shadows to a custom fragment shader

Hey everyone, I've been working on a billboarding shader with some special behaviour for a project of mine, and thanks to some helpful folks on these forums (namely @bgolus - thanks a million!) I've ended up with something that does more or less what I'd like.

This fragment shader supports transparent sprites that face the camera all the time, but calculate their z-depth as if they were standing completely upright, to avoid clipping into geometry behind them.

Because I'm apparently a glutton for difficult tasks, I've been trying to add support for some basic lighting and shadows, with limited success. Using a single pass and ForwardBase I've managed to make something properly responsive to directional lights, but none of the examples I've followed seem to allow this transparent shader to register point/spot lights in the environment, or receive shadows cast over it. Likewise, attempts at adding a second ForwardAdd pass haven't worked out, likely because all of the examples I've seen are meant for opaque shaders.

I've considered using a surface shader for this, but I'm reasonably certain the billboarding behaviour I want needs to make adjustments in clip space to fake that the billboard is standing upright, which surface shaders don't expose. If I'm wrong, feel free to correct me.

Ideally, what I'd like to achieve as far as lighting is:

• Receives light from nearby sources and directional lights, according to its current normals/facing.
• Receives shadows and draws darker when in areas of shadow, but pixel-perfect accuracy isn't necessary. Simply tinting the whole sprite if its origin were in darkness would be fine.
• Does not need to cast shadows. In fact, due to the nature of the billboard in this 3D world, cast stenciled shadows might look very distorted.
• Specular reflections and shininess aren't necessary - these are hand-drawn sprites with hand-drawn highlights.

Here's a shot using placeholder sprites of what I have working now:



Note how directional vertex lighting is working (all the sprites are tinted blue), but shadows and point lights are ignored, despite my attempts in the shader to implement them.

The shader code is below. Am I missing something, or overlooking some workaround for transparent sprites that receive lighting/shadows? Thanks in advance!

Code (CSharp):

1. Shader "Sprites/Billboard_VLit_ZDepth"
2. {
3.     Properties
4.     {
5.         _Color("Color", Color) = (1,1,1,1)
6.         _MainTex("Texture", 2D) = "white" {}
7.     }
8.  
9.     SubShader
10.     {
11.         Tags{ "Queue" = "Transparent" "IgnoreProjector" = "True" "RenderType" = "Transparent" "DisableBatching" = "True" } //"LightMode" = "ForwardBase" }
12.  
13.         ZWrite Off
14.         Cull Off
15.         Blend SrcAlpha OneMinusSrcAlpha
16.  
17.         Pass
18.         {
19.             Tags{ "LightMode" = "ForwardBase" }
20.  
21.             CGPROGRAM
22.             #pragma vertex vert
23.             #pragma fragment frag
24.  
25.             #pragma multi_compile_fog
26.             #pragma multi_compile_fwdbase
27.  
28.             #include "UnityCG.cginc"
29.             #include "AutoLight.cginc"
30.             #include "Lighting.cginc"
31.          
32.  
33.             struct appdata
34.             {
35.                 float4 vertex : POSITION;
36.                 float2 uv : TEXCOORD0;
37.                 float3 normal : NORMAL;
38.                 fixed4 color : COLOR;
39.             };
40.  
41.             struct v2f
42.             {
43.                 float4 pos : SV_POSITION;
44.                 float2 uv : TEXCOORD0;
45.                 UNITY_FOG_COORDS(1)
46.                 LIGHTING_COORDS(2, 3)
47.                 SHADOW_COORDS(4)
48.                 fixed4 color : COLOR0;
49.             };
50.  
51.             sampler2D _MainTex;
52.             float4 _MainTex_ST;
53.             fixed4 _Color;
54.  
55.             float rayPlaneIntersection(float3 rayDir, float3 rayPos, float3 planeNormal, float3 planePos)
56.             {
57.                 float denom = dot(planeNormal, rayDir);
58.                 denom = max(denom, 0.000001); // avoid divide by zero
59.                 float3 diff = planePos - rayPos;
60.                 return dot(diff, planeNormal) / denom;
61.             }
62.  
63.             v2f vert(appdata v)
64.             {
65.                 v2f o;
66.                 o.uv = v.uv.xy;
67.  
68.                 // billboard mesh towards camera
69.                 float3 vpos = mul((float3x3)unity_ObjectToWorld, v.vertex.xyz);
70.                 float4 worldCoord = float4(unity_ObjectToWorld._m03, unity_ObjectToWorld._m13, unity_ObjectToWorld._m23, 1);
71.                 float4 viewPos = mul(UNITY_MATRIX_V, worldCoord) + float4(vpos, 0);
72.  
73.                 o.pos = mul(UNITY_MATRIX_P, viewPos);
74.  
75.                 // calculate distance to vertical billboard plane seen at this vertex's screen position
76.                 float3 planeNormal = normalize(float3(UNITY_MATRIX_V._m20, 0.0, UNITY_MATRIX_V._m22));
77.                 float3 planePoint = unity_ObjectToWorld._m03_m13_m23;
78.                 float3 rayStart = _WorldSpaceCameraPos.xyz;
79.                 float3 rayDir = -normalize(mul(UNITY_MATRIX_I_V, float4(viewPos.xyz, 1.0)).xyz - rayStart); // convert view to world, minus camera pos
80.                 float dist = rayPlaneIntersection(rayDir, rayStart, planeNormal, planePoint);
81.  
82.                 // calculate the clip space z for vertical plane
83.                 float4 planeOutPos = mul(UNITY_MATRIX_VP, float4(rayStart + rayDir * dist, 1.0));
84.                 float newPosZ = planeOutPos.z / planeOutPos.w * o.pos.w;
85.  
86.                 // use the closest clip space z
87.                 #if defined(UNITY_REVERSED_Z)
88.                 o.pos.z = max(o.pos.z, newPosZ);
89.                 #else
90.                 o.pos.z = min(o.pos.z, newPosZ);
91.                 #endif
92.  
93.                 //Calculate Lighting
94.                 // get vertex normal in world space
95.                 half3 worldNormal = UnityObjectToWorldNormal(planeNormal); //v.normal
96.                 // dot product between normal and light direction for
97.                 // standard diffuse (Lambert) lighting
98.                 half nl = max(0, dot(worldNormal, _WorldSpaceLightPos0.xyz));
99.                 // factor in the light color
100.                 o.color.rgb = nl * _LightColor0;
101.                 o.color.rgb += ShadeSH9(half4(worldNormal, 1));
102.                 o.color.a = v.color.a;
103.  
104.                 UNITY_TRANSFER_FOG(o,o.vertex);
105.                 TRANSFER_VERTEX_TO_FRAGMENT(o);
106.                 TRANSFER_SHADOW(o);
107.                 return o;
108.             }
109.  
110.             fixed4 frag(v2f i) : SV_Target
111.             {
112.                 float attenuation = LIGHT_ATTENUATION(i);
113.                 fixed shadow = SHADOW_ATTENUATION(i);
114.                 fixed4 col = tex2D(_MainTex, i.uv) * _Color * i.color * attenuation * shadow;
115.                 //clip(col.a - 0.2); //For if using cutout shader
116.                 UNITY_APPLY_FOG(i.fogCoord, col);
117.  
118.                 return col;
119.             }
120.             ENDCG
121.         }
122.     }
123.     Fallback "VertexLit"
124. }

 

For receive shadows from directional light you need write "shadowcaster pass" with use your vertex shader, i don't know what needed for other lights types.

 

https://medium.com/@bgolus/anti-aliased-alpha-test-the-esoteric-alpha-to-coverage-8b177335ae4fThe

Fallback "VertexLit"

has a shadow caster pass already, albeit one that won't work for this shader properly.

There are a few problems here.

Vertex lit shaders by their nature don't receive shadows. You're working around this by still calculating the shadows in the fragment shader, which would work if it wasn't for the second problem.

Transparent queue materials do not receive shadows in Unity's built in rendering paths. There are now decade old forum threads on that topic. The URP and HDRP finally add support for that (and the URP only recently, it's been broken since before the LWRP was renamed to URP). If you want to receive shadows, you must use an opaque queue (like

Geometry

or

AlphaTest

). If you're using an opaque queue, you can't use alpha blending and have things render properly, so you have to use alpha testing (

clip(alpha - 0.5)

) or alpha to coverage (

AlphaToMask On

). But that alone won't solve the problem.

Like @mouurusai mentioned, you need a shadow caster pass (

"LightMode"="ShadowCaster"

). The shadow caster pass used for both shadow casting and shadow receiving, at least for the main directional light. For this shader you'd need to write a custom one rather than using anything built in. There are built in shadow caster passes that work with alpha testing (alpha to coverage doesn't work for shadows casters), but it also needs to do the same adjustments to the sprite plane so it matches the rendered mesh exactly.

After that, you'll still need to add a

"LightMode"="ForwardAdd"

pass to handle any additional lights beyond the main directional light, which is the only thing the

ForwardBase

pass handles is the single main directional light and ambient lighting.


Technically there is also a technique for getting the main directional light's shadow maps for use on transparent objects. But this only works for the main directional light, not point or spot lights. There's simply no way to make this work with Unity's built in rendering path unless you write your own lighting system from scratch. You can also make your shader use per vertex point lights either by forcing the shader to use per vertex lighting exclusively, or by setting your point lights to be non-important. However this disables all shadows entirely.


If you don't want to change all your sprites to be alpha tested, there is one other solution. Cheat.

By that I mean don't do any lighting on the GPU for sprites. Instead use c# raycasts to trace from the sprite's center (or the ground) to the lights that are in range and set the sprite / material color manually to mimic the desired look.