Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Stencil Buffer Overlapping

Discussion in 'Shaders' started by yayoni, May 12, 2019.

  1. yayoni


    Apr 12, 2018

    I have 2 shaders that use stencil buffer. The first shader is a mask with these settings (a plane)

    - Reference = 1
    - Comparaison = Always
    - Pass Front = Replace
    - Zwrite = Off

    And the second shader is for masked objects (red cubes)

    - Reference = 1
    - Comparaison = Not Equals

    Basically I just want to replace the content of my stencil mask with the content of the buffer which works but as you see on this picture, the mask is overlapping objects, the 3 cubes that are in front of it. What I would like to have since the the mask plane is behind the cube (the second cube in front) is just show the plane behind it ?

    And that's what I want to achieve (Imagine the blue plane is the mask), I just want to have my masked showed behind.

    Last edited: May 12, 2019
  2. Julian_Nementic


    Jan 10, 2014
    Could be a problem with within the rendering order. On with render queue are the red Cubes? If they are on transparency (normaly 3000) as well as the mask object, then because the mask object has z-write off the unity renderer cant decide with object is in front.

    Try to apply a opaque material with a renderqueue below 2500 to the red cubes to test this.

    You can read more about render queues here:
    Last edited: May 12, 2019
  3. yayoni


    Apr 12, 2018
    The red cubes are on the 2000 render queue and the mask is using the 1999 render queue. I tried what you said but it doesnt seem to change anything
  4. bgolus


    Dec 7, 2012
    You need to think about the rendering order more individually. One queue for all of the red boxes will not work when using stencils. Try using the frame debugger to get a better sense of how stuff is being rendered.

    Stencils are a 2D screen space thing that do not care about depth. Any pixel your mask object rendered to will have the stencil written to, even if the object is not "visible" in the final render. In the frame debugger you'll see when the mask object is rendered there's nothing to obscure it, so all of those pixels have been marked.

    If you want the mask to not write to the pixels behind the red boxes, you'll need to render those first so they write to the depth buffer. Then when the mask renders, those pixels further away than the red boxes are rejected, and it does not write to the stencil.

    Of course now the mask doesn't "do" anything since the when the boxes rendered, the stencil hadn't yet been written to, and thus were not masked.

    TDLR: For the stencil to mask the boxes you need the mask object to render before the boxes. For the boxes to occlude the mask, you need to render the boxes before the mask. You can't do both!

    So ... options.

    Manual sorting:
    Sort the boxes and the mask so the boxes in front of the mask render first, then the mask, then the boxes behind. If you disable batching on the shaders and set them all to the same queue, Unity may do this for you, but it won't be guaranteed. You can use the material queue, or a script to set the renderer components’ sortingOrder setting. This does take a bit more work as you have to manage the sorting yourself either before hand or while the game is running via scripts, depending on how dynamic your camera / scene is. This is also the only real option if you plan on having multiple masks that affect different sets of objects.

    Use only depth masking:
    One option is to not use stencils at all, but instead render your mask as a depth only shader before the boxes. This doesn’t require any special sorting beyond setting the material queue, it just works! ... except the sky won’t render behind where the mask rendered to, nor will any transparent effects, or anything else with a queue after the mask. This is because Unity’s skybox renders after all opaque objects anywhere the depth wasn’t written to, as are all transparent objects. You can fix the problem of the skybox by rendering it using a giant sphere in the level, or by using an extra camera that only renders the skybox and your main camera is set to only clear depth. You can also use the last queues of the opaque range (which ends at 2500), but the transparent queue problem isn’t really solvable.

    Use a pre-depth:
    The idea here is to get that impossible goal of drawing both the boxes and the mask “first” by splitting up the boxes into a depth only pass and the color pass. The idea would be to render the boxes first using a depth only rendering pass. This will fill in the depth buffer, but not actually show the boxes. Then the stencil mask will render, filling in the stencil buffer, but also get occluded by the depth of the cubes in front. Then draw the cubes again normally. This doesn’t require any special sorting, everything just works! ... except just like with the previous option, the skybox and transparencies won’t render, just now where the hidden boxes were instead of where the entire mask is. Skybox can be solved like above, but there’s another option here which is to render the boxes again, but this time only render them with a depth only shader that clears the depth to the far plane, after the mask renders. This also requires you render out your cubes and mask before any other geometry in the scene, but it fixes the issue with the skybox, transparencies, or any later queues.

    Use a second camera and render texture:
    This is another way of skipping using stencils. The idea here is to render your mask not actually as a mask, but instead just with the scene as you want to see it without the offending objects present slapped onto its surface. Have one camera with the scene with the boxes, and a second with the scene without the boxes (hidden via layers). The mask object uses screen space UVs to display the render texture of the box free scene, and is rendered at the very end of the frame using a very late transparency queue, like 3999. This is the most expensive option, but it's one of the easier to setup, and there's a ton of tutorials on how to do this technique for portals and the like.