Need help with matrix algebra in 2D billboard shader

The shader that I am trying to make is a 2D billboard shader that can properly interact with all types of lights. I also want to make the shadows of the sprite face each light individually. (I will explain this later).

To get to the point that I am at right now, I followed a few tutorials and links.

https://catlikecoding.com/unity/tutorials/rendering/part-7/

https://en.wikibooks.org/wiki/Cg_Programming/Unity/Billboards

These links helped me understand the basics of billboarding and shaders.

I tried to take this billboard effect to the next level and added lighting interaction as well. I defined the normal as the direction of the camera direction and defined the tangent as the normal cross multiplied by (0, 1, 0). This, combined with a basic normal map gives me dynamic lighting for 2D sprites.

https://imgur.com/DjhTzVG

Problems start to pop up once I try to get shadows to work. They work fine when the camera is in a fixed position, but a weird effect starts to pop up when the camera starts rotating around the player.

https://imgur.com/EFtXuWJ

As seen in the gif, the shadow is appearing for the non-billboarded sprite. While the shader makes the sprite look billboarded, it doesn't actually change the vertices of the sprite. This is where my limited matrix algebra knowledge slows my progress. I feel like I need to transform the vertex/worldposition of the struct for the shadow macros.

https://imgur.com/DDzR7Z7

Since some of the shadow macros use the worldposition to calculate the shadow attenuation, I believe that it is necessary to perform some transformation on the vertices to find the position that fits the billboarded position.

This is my current progress. Any insight or help is appreciated.

https://pastebin.com/95NUL1Gv

An alternative is to create a MonoBehaviour to do the billboard effect instead, but this doesn't work well with the shadow effect that I am also trying to achieve.

 

Unity's directional light shadows make use of the camera depth texture. Specifically, shadows are cast on the camera depth texture and rendered out to a screen space texture. Then the forward base pass is reading that screen space shadow texture.

Basically you need to also have a custom shadow caster pass that billboards.

However once you do that you'll see something that looks kind of similar to the current bug you have. And may even look worse. What that'll be is the sprite will now be billboarding towards the light, since that'll be the current view the billboard shader will be using when rendering the shadow maps. As a result the sprite will be shadowing itself like it's two intersecting quads.

There are a few ways to fix that.

One is to disable shadow receiving or shadow casting on the sprite renderer. Which you probably don't want since the entire goal is to presumably get both.

One is to use different method of billboarding so that even the shadow map rendering is aimed towards the player's camera instead of the light. The disadvantage is if you're rotated so the sprite is perfectly edge on to the light it won't cast any shadow since it's seeing the infinitely thin edge of the mesh. But it does offer a little more control over how the sprite is billboarded. Basically you'd need use the

unity_WorldToCamera

matrix (which is always the primary camera's transform matrix) to rotate the sprite towards "the camera" rather than using the cheaper trick of pretending the "world" position is in view space that the current shader is.

The "last" way I'll suggest is to push the sprite towards the current view by the sprite's radius in the forward pass(es) and the camera depth texture pass, but leave it where it is in the shadow map pass. This can be tricky to do properly though. Just moving the sprite's pivot towards the camera can cause it to get noticeably larger. You can either offset the depth in the fragment shader, which I don't recommend, or you can move the vertex closer to the camera, calculate the clip space position of that, and reproject the z back onto the original clip space position.

Code (csharp):

1. // picking up from how your current shader is setup
2. float4 view_pos = world_pos - world_origin + view_origin;
3. float4 clip_pos = mul(UNITY_MATRIX_P, view_pos);
4.  
5. // positive values push towards the camera
6. // as -z is away from it
7. view_pos.z += _SpriteCameraOffset;
8.  
9. float4 offset_clip_pos = mul(UNITY_MATRIX_P, view_pos);
10.  
11. // reproject new z onto old clip space
12. clip_pos.z = offset_clip_pos.z / offset_clip_pos,w * clip_pos.w;

You can do this to the shadow map pass too, and to start with you'll "have to" because the camera depth texture and shadow map uses the same shadow caster pass often with exactly the same keywords. (There are some threads on the forum where I discuss how to differentiate between them.)

But you can start by trying to applying a small offset and you'll notice nothing happens. But if you push it too far you'll start to clip into the walls in front, or floor behind (if you push it away). You can kind of work around that if your camera doesn't look up or down too much with some additional tricks. Like in this thread:
https://forum.unity.com/threads/pro...-sprites-clipping-into-3d-environment.680374/

But ultimately you're going to run into the fact Unity's lighting system relies on surfaces being at a certain position in world space, and casting shadows onto it, and you're just rendering a flat sprite. Unless you start going the route of making your flat sprite not actually a flat sprite, potentially building out a whole 3D mesh fo it to wrap around, you'll have this issue.

 

Thank you for your detailed response. I will try to implement your suggestions.
Regarding the individual shadows, I actually want to be able to make them billboard to their individual lights. I only briefly mentioned this in the original post, but I aim to give it a more 3d "feel" by trying to hide the fact that the sprite is infinitely thin from certain angles. The reason why I refrained from mentioning it is that I realized that the effect breaks apart when the directional light is shining directly above the sprite.

Edit: Like this: https://imgur.com/a/zAU19Ud
I guess a solution to this is to make a shadow map and to sample from the map depending on the angle and location of the light.

 

If that’s the kind of lighting you want to get, you may want to explore using vertical billboarding for shadow maps, and augment with an additional blob” shape at the middle of the head that’s set to be a shadow caster only. That doesn’t need any special shader, just have it be flat with the ground.

 

I think I finally understand what you mean by "But ultimately you're going to run into the fact Unity's lighting system relies on surfaces being at a certain position in world space". I followed your advice and got pretty far into the effect. The shadows work well with point lights but I run into a problem when it comes to directional lights.
Point Light: https://imgur.com/UeSjVxL
Directional Lights (I took off the alpha clipping to make it easier to see):
Camera VB: https://imgur.com/kwSOvto
(The shadow becomes invisible when the camera and light direction are perpendiculars)
DirLightVB: https://imgur.com/gfhMheL
(I get my desired effect but it breaks apart when a shadow is being cast on it)

Unless there is a solution to the second option, I think I am done with this shader. I will just use my previous effect.

Edit: If anyone wants to fix the problem here it is: https://pastebin.com/HtP9zBFH