Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Stencil Buffer bug or bad coding?

Discussion in 'Shaders' started by Drowning-Monkeys, Feb 13, 2016.

  1. Drowning-Monkeys

    Drowning-Monkeys

    Joined:
    Mar 6, 2013
    Posts:
    328
    Hi there,

    I'm using a stencil buffer shader found here:
    http://forum.unity3d.com/threads/unity-4-2-stencils-for-portal-rendering.191890/

    The gist of which is quite simple. You have a stencil mask:
    Code (CSharp):
    1. Stencil
    2.         {
    3.             Ref 2
    4.             Comp always
    5.             Pass replace
    6.         }
    and a stencil object:
    Code (CSharp):
    1. Stencil {
    2.         Ref 2
    3.         Comp Equal
    4.         Pass Keep
    5.         Fail Keep
    6.     }
    But when I place an object in outside the bounds of the stencil, it will still draw the object if the camera intersects the buffer:


    What am i doing wrong here?
     
  2. Teravisor

    Teravisor

    Joined:
    Dec 29, 2014
    Posts:
    654
    Imagine stencil buffer as per-pixel byte with data you test/write.

    That wall always writes into stencil buffer. Then when you try to render cube you test those pixels' stencil and get that stencil was written so you do need to render it if wall was previously drawn in that pixel. No matter what distance it has to that plane - it's per-pixel buffer, same as depth. So it's expected behaviour in that case actually.

    One possible solution would be to render invisible walls with ZWrite then only write cube if its ZTest greater. Another is to check world position in fragment shader and clip if it's out of bounds (needs coords for box that's masking in shader though).

    But I suggest to first think: if you want to mask objects outside of cube you first want to think about edge-case when half of cube is inside and half is outside: will this case ever happen and what would you render then? It won't pretty if you just cut cube in middle as you'll see insides.
     
  3. Drowning-Monkeys

    Drowning-Monkeys

    Joined:
    Mar 6, 2013
    Posts:
    328
    ok great, will explore more. I have some other working ideas but i wanted to get a grasp of the stencil buffer as well.