Shader stencil and Sprite Masks not working together

Hi everyone,
I'm working on a game where I have 'shadows' made of semi transparent sprites. The game has a day/night cycle and as such the sprites rotate and scale constantly based on the time of day. Their alpha is also calculated against the time of day and changes constantly too. This is all working really well except if shadows overlap, as you'd expect, both semi transparent sprites are visible. I do not want them to blend like that.

The problem:

The desired result:

(I simply turned the sprite renderers off to demonstrate)

The general idea, which does not work on my shadows, seems to be:
-all shadows need to use the same material and shader.
-said shader is a copy of sprites/default but uses stencil to do something like:

Stencil {
Ref 2
Comp NotEqual
Pass Replace
}

I, however, can make this shader work fine on other objects, just not my shadows.

Eventually, I came to the realisation that it doesn't work because the shadows also use sprite masks. I have two different types of shadows, one for the general time of day and one for reacting to close light sources, and I use masks to make them visible and invisible respectively (i.e. in close proximity to a light source, you cannot see the 'time of day' shadow but can instead see the shadow rotated inversely from the light source, such as a camp fire). It unfortunately means I can't not use sprite masks and mask interaction.

I assume, therefore, that the sprite mask is making use of the stencil buffer, and I'm essentially trying to do two things at once. If I change the mask interaction on the shadows, I essentially turn the shadows into sprite masks. Not what I want.

TL;DR
Is there any way I can use the Stencil buffer and sprite mask interaction of a sprite renderer on the same sprite?

OTHERWISE:
I have seen it suggested I could make a separate camera that only renders the shadows, put them into a render texture, run a shader or some sort on that (I guess test if alpha > x and then if so lower to x?) and re-render that, so they are all one blob? This might even run faster, but I have no idea how to do it and can't find any documentation that explains any of it.

If anyone can shine some light on this (because, you know, shadows..) I'd be more than grateful

 

In case anyone comes across this in the future and has the same problem I had, I found a solution!

It involves four slight variants on the sprites/default shader, which you can download from the usual unity downloads page.

I stopped using sprite masks, and instead wrote my own shaders that do what sprite masks do, but also allow me to have this added stencil functionality. Each shader remains a single pass shader and only needs a few changes.

You need to remove all your sprite masks and set mask interaction for all your sprites to be none. Sprite masks you'll be making are sprites set with a different material, they just use the standard sprite renderer. I suggest for 2D games putting them on a layer underneath everything so they're not visible.

These stencils need to be inside a shader pass before CGPROGRAM.

Sprite Mask:

Code (CSharp):

1. Stencil{
2. Ref 1
3. Comp Always
4. Pass Replace
5. }

This says 'always write the stencil buffer to 1' for the shape that your sprite mask is.

Visible Inside Mask:

Code (CSharp):

1. Stencil{
2. Ref 1
3. Comp Equal
4. }

This says 'is the buffer 1? If yes, draw'. The mask will set the buffer to 1, so this will only be visible inside your mask.

Visible Outside Mask with Sprite Blending:

Code (CSharp):

1. Stencil{
2. Ref 0
3. Comp Equal
4. Pass DecrWrap
5. }

This says 'is the buffer 0? (which it will be outside the mask), if so, decrease to 255'. The reason for this is that if another shadow is in the same place, it'll see a buffer of 255, so the comparison check will fail and it'll not draw.

Visible Outside Mask without Blending:

Code (CSharp):

1. Stencil{
2. Ref 1
3. Comp NotEqual
4. }

This says 'is the buffer 1? If no, draw'. This doesn't care about the merging and simply checks if it's out of a mask. This is important as I had a lot of issues before using this fourth shader where the blending sprites were acting as a mask for sprites visible outside of mask on a different sorting layer to the blending ones. No idea what that's about, but I fixed it so I'm happy.

Finally, another important code is the following:

First, replace the #pragma fragment SpriteFrag to be:

Code (CSharp):

1. #pragma fragment frag

Then underneath the #pragma lines, before ENDCG, add a new fragment section which you've just referenced:

Code (CSharp):

1. fixed4 frag(v2f IN) : SV_Target
2. {
3. fixed4 c = tex2D(_MainTex, IN.texcoord) * IN.color;
4. if (c.a<0.1) clip (-1);
5. c.rgb *= c.a;
6. return c;
7. }

This cuts off the sprites properly to their transparency, so sprites of any shape will work. Otherwise, you'll have weird blank areas surrounding your sprites.

And you're done! Enjoy, hopefully someone somewhere finds this helpful. Last week I knew nothing about shaders, today I managed to solve this

 

Hi, thanks for your instructions, but i just cant get it run, can u upload your shader file as an example. Thanks

 

bump.
Please put the whole shader

 

For anyone wondering how to get this effect with, for example, a standard lit or unlit URP sprite shader. Create a new shader graph of the type you require (lit/unlit). Hook this graph up with a SampleTexture2D (make sure this is named MainTex). Plug the color and alpha into the outputs on shader graph. This is now a completely normal unaffected sprite shader. In editor, click 'view generated code'. Copy the generated code into a new .shader file. Now you can add the aforementioned stencil in and voila you have a graphless URP sprite shader with stencil manips. I use this exact technique to make Tilemaps which act as sprite masks!
TL;DR
-You don't need OP's whole shader, you can add the stencil feature to your own shaders for your own specific needs!