How to make stencil mask for a stencil mask

I have made a stencil shader mask and object. The stencil object behind the stencil mask can only be seen when looking through the stencil mask. (code for the stencil object and mask below)

Does anyone know how I can make a stencil mask for my current stencil mask?

Code (CSharp):

1. Shader "Custom/Stencil/UI/Default-Mask"
2. {
3.     Properties
4.     {
5.         [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
6.         _Color ("Tint", Color) = (1,1,1,1)
7.    
8.         _StencilComp ("Stencil Comparison", Float) = 8
9.         _Stencil ("Stencil ID", Float) = 1
10.         _StencilOp ("Stencil Operation", Float) = 0
11.         _StencilWriteMask ("Stencil Write Mask", Float) = 255
12.         _StencilReadMask ("Stencil Read Mask", Float) = 255
13.  
14.         _ColorMask ("Color Mask", Float) = 15
15.  
16.         [Toggle(UNITY_UI_ALPHACLIP)] _UseUIAlphaClip ("Use Alpha Clip", Float) = 0
17.     }
18.  
19.     SubShader
20.     {
21.         Tags
22.         {
23.             //"Queue"="Transparent"
24.             "Queue" = "Geometry-100"
25.             "IgnoreProjector"="True"
26.             "RenderType"="Transparent"
27.             "PreviewType"="Plane"
28.             "CanUseSpriteAtlas"="True"
29.         }
30.    
31.         Stencil
32.         {
33.             Ref [_Stencil]
34.             Comp [_StencilComp]
35.             Pass replace
36.             ReadMask [_StencilReadMask]
37.             WriteMask [_StencilWriteMask]
38.         }

Code (CSharp):

1. Shader "Custom/Stencil/UI/Default-Object"
2. {
3.     Properties
4.     {
5.         [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
6.         _Color ("Tint", Color) = (1,1,1,1)
7.      
8.         _StencilComp ("Stencil Comparison", Float) = 8
9.         _Stencil ("Stencil ID", Float) = 1
10.         _StencilOp ("Stencil Operation", Float) = 0
11.         _StencilWriteMask ("Stencil Write Mask", Float) = 255
12.         _StencilReadMask ("Stencil Read Mask", Float) = 255
13.  
14.         _ColorMask ("Color Mask", Float) = 15
15.  
16.         [Toggle(UNITY_UI_ALPHACLIP)] _UseUIAlphaClip ("Use Alpha Clip", Float) = 0
17.     }
18.  
19.     SubShader
20.     {
21.         Tags
22.         {
23.             "Queue"="Transparent"
24.             "IgnoreProjector"="True"
25.             "RenderType"="Transparent"
26.             "PreviewType"="Plane"
27.             "CanUseSpriteAtlas"="True"
28.         }
29.      
30.         Stencil
31.         {
32.             Ref [_Stencil]
33.             Comp equal
34.             Pass [_StencilOp]
35.             ReadMask [_StencilReadMask]
36.             WriteMask [_StencilWriteMask]
37.         }
38.  

 

Like I have said more often on the forum here. Don't put your reference to 1 and your masks to 255. If you want to only use one bit, set your masks to 1 too.

It's quite possible to mask the mask by using a second bit in the stencil buffer.

Step 1: (Write to the first bit)
Ref 1
Comp always
Pass replace
WriteMask 1

Step 2: (If the first bit is set, write to the second bit)
Ref 3
Comp equals
Pass replace
ReadMask 1
WriteMask 2

Step 3: (If the second bit is set, render)
Ref 2
Comp equals
ReadMask 2

Another way would be to just write to bits 1 and 2 in the first two steps and directly compare with both in the third step.

 

I am really new to shader, there for sorry for the stupid questions that now follows:

"How should I implement this shader? And does this work for masks in masks in masks?"

As far as I understand, I need do the following three steps:
Step 1 - Create a shader file for a stencil object
Step 2 - Implement my code for a stencil object, but change the (read/write) masks to 1?
Step 3 - Create a shader file for a stencil mask
Step 4 - Implement your code for a shader mask (that can also see read other shader masks) within the "Stencil{...}" of my code?

Or do I need to create a new shader file specificly for a shader mask mask?

Also, where would I use my "Queue" = "Geometry-100"?

Again, so sorry for my lack of knowledge.

 

The way those shaders are set up, you don't need any extra shader. It can be controlled from the settings in the material. My first 2 steps can be done with the first shader and the last step with the second shader.

The render order is important, but you can control that through the material too. (The queue in the shader is just the default value for the material.)

 

Where can I define the if statement?

You said:

Step 2: (If the first bit is set, write to the second bit)
Step 3: (If the second bit is set, render)

Because I suppose this doesn't work:

Code (CSharp):

1.  
2.  
3.     SubShader
4.     {
5.         Tags
6.         {
7.             //"Queue"="Transparent"
8.             "Queue" = "Geometry-100"
9.             "IgnoreProjector"="True"
10.             "RenderType"="Transparent"
11.             "PreviewType"="Plane"
12.             "CanUseSpriteAtlas"="True"
13.         }
14.         Stencil
15.         {
16.             // old code:
17.             //Ref [_Stencil]
18.             //Comp [_StencilComp]
19.             //Pass replace
20.             //ReadMask [_StencilReadMask]
21.             //WriteMask [_StencilWriteMask]
22.          
23.            //Step 1: (Write to the first bit)
24.            Ref 1
25.            Comp always
26.            Pass replace
27.            WriteMask 1
28.  
29.            //Step 2: (If the first bit is set, write to the second bit)
30.            Ref 3
31.            Comp equals
32.            Pass replace
33.            ReadMask 1
34.            WriteMask 2
35.  
36.         }

 

Well, that's almost it. But the two different steps should be two different shaders/passes. How would you mask your mask in the same step?

 

I like your way of explaining, but I am not sure where I made the mistake in the code, which applies (as far as I understood) your feedback.

Step 1 and 2 are added as subshaders, to the "mask" shader.
Step 3 is for the "object/non-mask" shader.

But it seems something isn't going as I planned, now all are invisible.
A short overview of the changes are demonstrated in the code below:

Stencil mask

Code (CSharp):

1.  
2. Shader "Custom/Stencil/UI/Default-Mask"
3. {
4.     // Subshader to define the base shader for the object
5.     SubShader
6.         {
7.             Tags
8.             {
9.                 "Queue" = "Geometry-100"
10.                 "IgnoreProjector" = "True"
11.                 "RenderType" = "Transparent"
12.                 "PreviewType" = "Plane"
13.                 "CanUseSpriteAtlas" = "True"
14.             }
15.  
16.             //Step 1: (Write to the first bit)
17.             Stencil
18.             {
19.                 Ref 1
20.                 Comp always
21.                 Pass replace
22.                 ReadMask 1
23.                 WriteMask 1
24.             }
25.  
26.         } // End subshader stencil for masking mask
27.  
28.         // Subshader to define the base shader for the object
29.         SubShader
30.         {
31.             Tags
32.             {
33.                 "Queue" = "Geometry-100"
34.                 "IgnoreProjector" = "True"
35.                 "RenderType" = "Transparent"
36.                 "PreviewType" = "Plane"
37.                 "CanUseSpriteAtlas" = "True"
38.             }
39.  
40.             //Step 2: (If the first bit is set, write to the second bit)
41.             Stencil
42.             {
43.                 Ref 3
44.                 Comp equal
45.                 Pass replace
46.                 ReadMask 1
47.                 WriteMask 2
48.             }
49.  
50.     } // End subshader stencil for masking mask
51. }
52.  

Stencil object

Code (CSharp):

1. Shader "Custom/Stencil/UI/Default-Object"
2. {
3.     SubShader
4.     {
5.         Tags {
6.             "Queue" = "Transparent"
7.             "IgnoreProjector" = "True"
8.             "RenderType" = "Transparent"
9.             "PreviewType" = "Plane"
10.             "CanUseSpriteAtlas" = "True"
11.         }
12.  
13.         //Step 3: (If the second bit is set, render)
14.         Stencil {
15.  
16.         Ref 2
17.         Comp equal
18.         ReadMask 2
19.  
20.  
21.         }
22.     } // End subshader
23. }
24.  

I really wan't to make this work.

It may be better of sending the whole code, so anyone can have a closer look.
There for, I have put a test project and have added it to this reply.
The current code works for Meshes and UI, it is a little more advance, but works as told above.

Also here is a image that helps explaining the situation:

 

Don't know if you're still working on this, but you're setting the wrong ref values. The ref value is the one used for the comparison, so now you're setting it to 1, then checking if it's equal to 3, then checking if it's equal to 2. Of course they are all invisible, the comparison is always false. Try using Ref 1 for all of them, for example.

Also, it seems you can do two stencils in the same pass by doing two separate Stencil { }.

Big thanks to jvo3dc for the explanation, it was very helpful.

 

I don't think that's going to work. The hardware can only do one at a time, so with a double block it probably just does one.

 

I deleted that because I was not sure, but I am currently using this in a shader and it's working:

Code (CSharp):

1.             Stencil
2.         {
3.             Ref[_DrawInIndex]
4.             Comp Equal
5.             ReadMask 1
6.         }
7.  
8.             Stencil
9.         {
10.             Ref[_MaskIndex]
11.             Comp NotEqual
12.             Pass replace
13.             //WriteMask 255
14.         }
15.  

Basically I am using a masked object to mask another object, using two different ref values for two different writemasks.

 

Right, well, it's probably working by accident. What are the _DrawInIndex and _MaskIndex set to?

 

Yes, that's very much the case. I think it works because the bitwise operations just happen to work out (those indices were 100 and 101, respectively). However, that means it may interfere with other stencil masks, so even if it works I scrapped that approach, and ended up using a single pass with Pass IncrSat. That way I can feed the increased value to the second masked object.

Maybe other people needing to mask a mask can use the same approach.

 

Well, that's kind of why I went for this:

Code (csharp):

1.  
2. Ref 3
3. Comp equals
4. Pass replace
5. ReadMask 1
6. WriteMask 2
7.  

So, if the first bit is set, then set the second bit. So this mask is writing in the second bit, but it itself is masked by the first bit.

Ref is 3 in this case, but could be anything with the last two bits sets. So 7, 15 or 255 would do exactly the same. It's the masks that really control it here.

 

Hey, so I'm trying to do something similar with nested masks, but in my case I have multiple potential masks inside my outer mask that each could be showing a different underlying image/scene. Trying to understand what's being suggested above and whether that could apply to this situation? Or perhaps this is the same exact situation and I'm not realizing it?

Any guidance would be appreciated!

 

Think I figured it out. Might be a better way to do this, but here's my setup:

I use 3 shaders. One for the Outer Stencil, one for Inner Stencils (looks at first 4 bits), and one for Geometry:

Outer Stencil:

        Stencil

        {

            Ref[_StencilMask]

            Comp always

            Pass replace

        }

Inner Stencil:

        Stencil

        {

            Ref[_StencilMask]

            Comp equal

            ReadMask 240

            Pass replace

            Fail keep

        }

Geometry:

        Stencil

        {

            Ref[_StencilMask]

            Comp equal

        }

So an example setup would be the outer stencil ref id = 16 / inner stencil and geo #1 ref id = 17 / inner stencil and geo #2 ref id = 18

 

Just for completeness: You don't need to hardcode the stencil in the shader. You can pass them as properties as shown here: https://github.com/supyrb/ConfigurableShaders/wiki/Stencil
This makes it way easier to iterate and experiment with stencil shaders

 

Hey noethis, don't know if you're still interested, but I ended up figuring out a setup with the new Sprite Mask component. If you play with the custom range you can selectively mask only certain layers or sortingOrder intervals in the same layer. You could use an outer stencil that applies to everything, and inner stencils that only mask a given sortingOrder. I found it's way easier to maintain than using multiple shaders and materials.

 

I'm interested! How do custom ranges change the stencil behavior? does it change the ref or what?

 

I don't know how Unity handles it behind the scenes. You don't need any custom shader code nor to modify the stencil buffer yourself, you just set it in the editor component and it works with the default sprite shader.

 

Oh, but I'm looking into using it with tilemap, which does not work with the editor component as it's meant for the sprite renderer.
So I can't use the editor component.

 

Do you know (or anyone here) of a good tutorial to properly understand ReadMask/WriteMask? I thought I knew how to use Stencil shaders, but I was only making use of "Ref" value, and then calling to "IncrSat/DecrSat" depending on if it passed or failed; the fact that you can add a Mask, no matter what the Ref value is, made me realize I don't even know what a mask is or how to use it. How does it affect to the whole thing?

For example, in your Step2, I don't get why using Ref3 allows you to test that; I would have used Ref1.
I'm sure it might be a basic Stencil/Bit stuff, but I just don't get it.

I guess I would need an ELI5 of Masks and Stencil Ref.

 

ReadMask and WriteMask are binary masks that are anded with the values.

In my step 2, I want to check whether the first bit (value 1, 00000001) is set and then set the second bit (value 2, 00000010). With replace, the reference value is written, which is the same value as the value that is checked against. So 1 would only work for the test part and 2 would only work for the replace part. But 3 works for both parts by using the right masks.

So I set my reference to 3 (00000011) to have the first two bits sets.
Then the check becomes:
- if ((Stencil & ReadMask) Comp (Ref & ReadMask))
- if ((Stencil & 00000001) == (00000011 & 00000001))
- if ((Stencil & 00000001) == 00000001)

And the replace operator becomes:
- Stencil = Ref & WriteMask
- Stencil = 00000011 & 00000010
- Stencil = 00000010

 

Thank you so much for taking the time to answer, I really appreciate! I think I'm starting to get it; it adds flexibility among other things.

I still have some questions though, probably due to lack of basic knowledge on low level shaders; wouldn't be the same (at least in this specific Step 2-situation), if you had used Ref1, but instead of replace, had used IncrSat? Wouldn't that produce the same result, writing the value 2 where value 1 is read? (if you however want to write a totally different value, much higher, then I guess WriteMask is your only choice, am I right?)

Again, thank you.

 

Yes, in this case Incr/IncrSat with a Ref of 1 should also work. I usually treat the stencil buffer as a set of 8 binary values instead of an integer value that can be increased and decreased.

For stencil volumes you could use Incr and Decr to count how many times you've entered and exited the volume. (And if the final result is 0, you're outside.) But I usually just render double sided and apply Invert to a single bit. Depending on the geometry of the volume, this generally also works fine.

 

Got it. Thank you so much!