Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  2. Ever participated in one our Game Jams? Want pointers on your project? Our Evangelists will be available on Friday to give feedback. Come share your games with us!
    Dismiss Notice

Trying to get a mask effect with Stencil Buffer

Discussion in 'Shaders' started by petey, Jan 3, 2018.

  1. petey


    May 20, 2009
    Hi all!

    I'm just getting my head around the whole stencil buffer thing because it looks like it could be really handy!
    I'm really close to getting this effect working but now I'm not really sure if it's possible. I was hoping to mask the first object (purple) only if it is behind the mask (quad). I want everything else in the scene to remain unaffected.

    You can see the quad is fighting it out with background objects here and I'm not sure how to get around that.
    Any ideas?

  2. Balikk


    Oct 23, 2014

    Can we see your shaders with the stencil please?
    The grey cube is a standard shader?
  3. petey


    May 20, 2009
    Yeah the grey one is a standard shader with no stencil settings but damn! I broke the original scene and can't even get the shaders to do that again :(
    This whole stencil thing looks really interesting but it's like a crazy puzzle to work out. It would be ace if there was a nice tute on it or something because you could do some really interesting stuff with it.
  4. Balikk


    Oct 23, 2014
    I think the thing you have in the scene view is more like a bug. You will have the same effect with this shader :
    Code (CSharp):
    1. Shader "Custom/Test" {
    2.     Properties {
    3.         _Color ("Color", Color) = (1,1,1,1)
    4.     }
    5.     SubShader {
    6.         Tags { "RenderType"="Opaque" "Queue"="Geometry"}
    7.         LOD 200
    9.         ZWrite On
    10.         ColorMask 0
    12.         Pass{}
    13.     }
    14.     FallBack "Diffuse"
    15. }
    In scene view :

    The way stencil works, your front top corner should be masked.

    But maybe I'm wrong, I've seen topics where " bgolus " answered about stencil, he could answer this I think.
    Last edited: Jan 6, 2018
    petey likes this.
  5. petey


    May 20, 2009
    Hey thanks man!
    Balikk likes this.
  6. tomekkie2


    Jul 6, 2012
    Did you mean something alike?

    That's done with two shaders:

    Red material shader:
    Code (CSharp):
    1. ...........................................
    2.         _StencilMask("Stencil Mask", Range(0, 255)) = 1
    3.     }
    4.     SubShader {
    5.         Tags { "RenderType"="Opaque" }
    6.         LOD 200
    8.         Stencil{
    9.             Ref [_StencilMask]
    10.             Comp NotEqual
    11.             Pass Zero
    13.         }
    14. ...........................................
    Quad material shader code:
    Code (CSharp):
    1.     Shader "Custom/Test" {
    2.         Properties {
    3.             _Color ("Color", Color) = (1,1,1,1)
    4.             _StencilMask("Stencil Mask", Range(0, 255)) = 1
    5.         }
    6.         SubShader {
    7.             Tags { "RenderType"="Opaque" "Queue"="Geometry-1"}
    8.             LOD 200
    10.             //ColorMask 0
    11.              Stencil{
    12.             Ref [_StencilMask]
    13.             Comp Never
    14.             Fail Replace
    16.         }
    17.             Pass{}
    18.         }
    20.     }
  7. bgolus


    Dec 7, 2012
    It's all about the rendering order. If the stencil plane has ZWrite On, and it renders first, then the objects that render afterward and behind it won't get rendered, even without using stencils.

    In @Balikk 's example you're seeing what happens when the objects get sorted differently as the camera moves around. That first gif is almost entirely just the effects of the depth buffer. The second gif is more what I would expect from the stencil taking effect.

    @tomekkie2 's example is the stencil working, just like @Balikk 's second gif.

    If by "behind" you actually want that intersection between the plane and the purple box like in your original gif and @Balikk 's first gif, then you just need to render the scene first, then the plane, then the purple box, which you can do by setting the queue and no stencil at all. However if you need it to work with, say, transparent effects that are in the scene then you might have some problems as there's only that one depth buffer and there's now a big solid quad floating invisibly in the scene that'll block effects behind it just as well as that purple box. Stencil has the benefit of only affecting the things you want it to, but you can't really do that intersection effect with those as easily and you get @tomekkie2 's look. My suggestion for if you want both is to not using depth buffers or stencil, but use something like the cross section shaders or potentially a render texture in which you render out the mask in and then the shader on the purple box reads from and that drives the alpha for a cutout / alpha test.
    tomekkie2 and Balikk like this.
  8. petey


    May 20, 2009
    Hey thanks @bgolus
    Sorry that was a pretty bad gif I originally posted, it does show my amount of knowledge when it comes to shaders though :)

    Here's what I have so far,

    This is working pretty cool really but I was thinking it would be nice if could have one of the teeth potentially poking out of the mouth when it closes.

    I think in order to get that I'd have to mask the outside area and use a Zdepth mask to allow the tooth to come in front...

    Not too sure, but it's moving along, thanks for all the help, I had no idea about cross section shaders, they look super handy.