Custom shader breaks Unity built-in masking

Hi everyone,
my custom shader works like a charm for what I wanted it, but if it is referenced in the scene, Unity built-in masking stops working (both the Sprite Mask component and the rect-based Mask stop masking).

What I needed is a 2D unlit shader that would combat overlapping semi-transparency for 2D non-light based shadows. In other words, instead of this:

I want this:


Since I don't know the first thing about shaders, I've copied the shader (like a clueless monkey) from here.
It works but, as I mentioned it breaks other stuff (masking). As soon as I change the shader of the referenced material to any other shader, the masks start working without problems, so I know the shader is the culprit.
My project uses the Universal Render Pipeline and Unity 2019.3. I've tried simply adding in the RendePipeline tag into the shader, but that did not fix the issue.
Please keep in mind that things like "z buffer" and "stencil" are justs words to me.
Thanks for and advice

 

After a bit of fiddling around, I found out the problem has to do with the stencil. Trying different shaders that use the stencil and expose stencil properties shows that stencil ID, Comparison and Operation values that help achieve the required effect always break Unity built-in masking. This leads me to the conclusion stencil is "shared" between shaders and masking effects. I've also tried different Stencil Write/Read Mask values, but that didn't help. Keep in mind my understanding of the stencil topic is super-shallow.

 

Hi YD_JMysior I'm here to help you
can you recreate the problem and send it as a package?
because I haven't enough time to recreate the problem

 

Hi Sayed. Thanks for looking into it!
While trying to recreate the problem on a separate scene I noticed the problem with the mask not working only shows up if the mask object is rendered on top of the sprites that use the custom shader (with stencil). As soon as I move the mask and masked object outside they work ok. Also as soon as I change the material of the "shadows" to one that does not use the stencil it works ok.
See this image. The selected object is the mask. The red object should only be visible inside the mask - but it's not.


Anyway, I'm attaching the repro package.
Thanks in advance

 

Further research shows this has also to do with the sorting layers. If the masked object (and mask setting) is on a lower layer than the custom shader object, than the masking works ok.
Also normally in 2D the Z position of the sprites does not matter - only the rendering layers and order in layer matter. When the stencil-based shader is used, changing the Z position of the masked object yields unexpected results:
- when the masked object is closer to the camera (z = -1) it renders (but is not masked properly)
- when the masked object is further to the camera (z = 1) it becomes invisible (which is not the case if the custom shader is not used on shadow objects).

 

Hi sorry I'm busy now but I will check it out as soon as possible

 

Thanks in advance Sayed

 

Hi dear friend
I tried to solve your problem
but It seems sprite mask override the stencil so I couldn't fix it

but anyway I could fix that with MeshRenderer:



Mask:



Masked Image:


Shadows:



I explained here how to show stencil enums in inspector:
https://gamedev.stackexchange.com/q...hat-ignores-transparent-sprites/158132#158132

if you couldn't fix this problem you can use world space shader to solve overlapping problem

 

Sayed,
thanks a lot for looking into this! Does this stencil overriding problem sound like something that should be sent to Unity team for investigation? From what you did I can see using different stencil ID for each shader group should work as expected, but it seems the SpriteMask is working on all stencils...
Is my understanding correct - you have created a separate material instance (each of off the same shader) for each effect I need (mask, masked object and shadow). In other words, you used a custom mask, correct?

 

you're welcome
yes I mean sprite will override the stencil
when I changed the stencil Id in sprite shader nothing happens
I tried to decompile the Sprite class but I couldn't remove this part



yeah I used a custom mask in 3d
because mesh renderer won't override stencils
I attached a unitypackage you can review it