Stencil Buffer Overlapping

Hello,

I have 2 shaders that use stencil buffer. The first shader is a mask with these settings (a plane)

- Reference = 1
- Comparaison = Always
- Pass Front = Replace
- Zwrite = Off

And the second shader is for masked objects (red cubes)

- Reference = 1
- Comparaison = Not Equals

Basically I just want to replace the content of my stencil mask with the content of the buffer which works but as you see on this picture, the mask is overlapping objects, the 3 cubes that are in front of it. What I would like to have since the the mask plane is behind the cube (the second cube in front) is just show the plane behind it ?



And that's what I want to achieve (Imagine the blue plane is the mask), I just want to have my masked showed behind.

 

Could be a problem with within the rendering order. On with render queue are the red Cubes? If they are on transparency (normaly 3000) as well as the mask object, then because the mask object has z-write off the unity renderer cant decide with object is in front.

Try to apply a opaque material with a renderqueue below 2500 to the red cubes to test this.

You can read more about render queues here: https://docs.unity3d.com/Manual/SL-SubShaderTags.html

 

The red cubes are on the 2000 render queue and the mask is using the 1999 render queue. I tried what you said but it doesnt seem to change anything

 

You need to think about the rendering order more individually. One queue for all of the red boxes will not work when using stencils. Try using the frame debugger to get a better sense of how stuff is being rendered.

Stencils are a 2D screen space thing that do not care about depth. Any pixel your mask object rendered to will have the stencil written to, even if the object is not "visible" in the final render. In the frame debugger you'll see when the mask object is rendered there's nothing to obscure it, so all of those pixels have been marked.

If you want the mask to not write to the pixels behind the red boxes, you'll need to render those first so they write to the depth buffer. Then when the mask renders, those pixels further away than the red boxes are rejected, and it does not write to the stencil.

Of course now the mask doesn't "do" anything since the when the boxes rendered, the stencil hadn't yet been written to, and thus were not masked.

TDLR: For the stencil to mask the boxes you need the mask object to render before the boxes. For the boxes to occlude the mask, you need to render the boxes before the mask. You can't do both!


So ... options.

Manual sorting:
Sort the boxes and the mask so the boxes in front of the mask render first, then the mask, then the boxes behind. If you disable batching on the shaders and set them all to the same queue, Unity may do this for you, but it won't be guaranteed. You can use the material queue, or a script to set the renderer components’ sortingOrder setting. This does take a bit more work as you have to manage the sorting yourself either before hand or while the game is running via scripts, depending on how dynamic your camera / scene is. This is also the only real option if you plan on having multiple masks that affect different sets of objects.

Use only depth masking:
One option is to not use stencils at all, but instead render your mask as a depth only shader before the boxes. This doesn’t require any special sorting beyond setting the material queue, it just works! ... except the sky won’t render behind where the mask rendered to, nor will any transparent effects, or anything else with a queue after the mask. This is because Unity’s skybox renders after all opaque objects anywhere the depth wasn’t written to, as are all transparent objects. You can fix the problem of the skybox by rendering it using a giant sphere in the level, or by using an extra camera that only renders the skybox and your main camera is set to only clear depth. You can also use the last queues of the opaque range (which ends at 2500), but the transparent queue problem isn’t really solvable.

Use a pre-depth:
The idea here is to get that impossible goal of drawing both the boxes and the mask “first” by splitting up the boxes into a depth only pass and the color pass. The idea would be to render the boxes first using a depth only rendering pass. This will fill in the depth buffer, but not actually show the boxes. Then the stencil mask will render, filling in the stencil buffer, but also get occluded by the depth of the cubes in front. Then draw the cubes again normally. This doesn’t require any special sorting, everything just works! ... except just like with the previous option, the skybox and transparencies won’t render, just now where the hidden boxes were instead of where the entire mask is. Skybox can be solved like above, but there’s another option here which is to render the boxes again, but this time only render them with a depth only shader that clears the depth to the far plane, after the mask renders. This also requires you render out your cubes and mask before any other geometry in the scene, but it fixes the issue with the skybox, transparencies, or any later queues.

Use a second camera and render texture:
This is another way of skipping using stencils. The idea here is to render your mask not actually as a mask, but instead just with the scene as you want to see it without the offending objects present slapped onto its surface. Have one camera with the scene with the boxes, and a second with the scene without the boxes (hidden via layers). The mask object uses screen space UVs to display the render texture of the box free scene, and is rendered at the very end of the frame using a very late transparency queue, like 3999. This is the most expensive option, but it's one of the easier to setup, and there's a ton of tutorials on how to do this technique for portals and the like.

 

Hi Ben,

I was trying to write a good scheme of stencil mask that can handle multiple layer of walls and windows.

Imagine a level builder, where wall are solid but users can add window, which mask out wall with stencil.

I had such a system using pre-depth approach: it works by rendering out wall depth, then window stencil with depth test, then wall depth again but masked by stencil, at last the wall color using depth test.

it works for simple cases, but it failed to handle more complex requirement like "a window should only mask 1 layer of wall, not more".

I fully understand the complexity involved, and was trying to use stencil incr / decr / masking creatively to solve it. But I am stuck solving a few edge cases.

So I am wondering, are you aware any literature / tool that do smart stencil math? Something like this:

https://tech.metail.com/the-stencil-buffer-and-how-to-use-it-to-visualize-volume-intersections/

And my goal is to avoid manual sorting like this thread:

https://forum.unity.com/threads/res...-in-walls-with-stencil-buffer-shaders.429746/

Thx, appreciate any help!

 

The “simple” solution is to render the window again to clear the stencil after rendering the wall.

 

I don't fully follow, I find a scenario like this very hard to handle (3 windowed wall stack against each other)



Assume I can only *render all walls* or *render all windows*, and I can render them multiple times if needed for stencil or depth test.

And assume I cannot use manual sorting (aka *render some walls* or *some windows*).

With stencil it's trivial to know where wall should or shouldn't be rendered, but I must be able to know *which wall* should be rendered, so that my wall color is correct behind windows.

This is a math and bitwise operation challenge that I am trying to solve

 

Then you’re boned. There’s no solution because what you’re trying to do. If you’re trying to do this with stencils, it requires having an explicit render order to accomplish. That’s kind of the “thing” with stencils and complex setups is they work only when you can control the order of operations, i.e. the render order.

If you render all of the walls, it’s too late to put holes in them. With one wall by itself you can use techniques like rendering a wall depth and then a stencil and then “punch through” with a

ZTest Always/GEqual

pass because you don’t care about the depth values behind it. But as soon as you need to render stuff behind that wall through the window, it all falls down unless you’re able to render each single wall, then punch through that one wall, in the exact perfect order.

If you can render each window “surface” with their own unique stencil ref, just the area that’s coplanar with the wall. Then render all of the walls with their matching stencil ref to skip rendering both depth and color wherever the window “surface” was, you can get what you want. You can render the window interior edge whenever you want, you might even be able to do that first before the window surface or walls. But if you can set unique stencil values per window & wall, you can also sort them individually just as easily.

 

I did the impossible.



I am absolutely sure there are edge cases I didn't catch, but I made it work with stencil without manual sorting... In the end it took me 8 passes.

(I am sure it would be cheaper to just manually sort, which would likely also take 8 passes; But that was the main point: I cannot know in advance where and how walls/windows are placed)

 

Cool thing is I can support arbitrary number of walls/windows without custom sort (well, up to a certain number, there are only so many bits), it will always take 8 passes.

Pretty proud of myself even though I know edge cases are going to bite me

 

What happens if there’s another surface visible through the windows that is just a regular wall / floor?

 

it breaks if it's a wall that can have a window, it's fine for anything else