Render a mesh in front of another

I'm trying to always render the player in front of specific meshes, even if the player is inside of the mesh.

For example:
Imagine a 3D environment with a side camera perceptive, there is a rock mesh with a flat surface on top that acts like a one-way platform when the player jumps from under it. Normally, the player is half way clipped inside and half way hanging out while jumping thru the rock. What I want is for the player to always render in front of the rock mesh, while jumping thru it.

I would like to avoid the multiple camera approach and handle it at the shader level with a stencil buffer. But I can't seem to get the settings right.

On the player's shader:

Code (CSharp):

1.  
2. Stencil {
3.      Ref [_StencilMask]
4. }
5.  

And on the rock's shader:

Code (CSharp):

1.  
2. Tags { "RenderType"="Opaque" "Queue" = "Geometry-1"}
3. Stencil {
4.     Ref [_StencilMask]
5.     Comp GEqual
6.     Pass Replace
7.     Fail Keep
8. }
9.  

I set both stencil mask refs to the same index, but I'm not sure about what my Pass and Fail operations should be. How I understand it should work is that if the rock finds the player intersecting, it should replace its pixels with the players. Any help? Save me @bgolus

 

For the rock's shader, Pass and Fail should be left as their defaults (Keep for both), and thus can be completely removed. Those two are "what to do with the stencil value" settings. You don't want the rock to modify the stencil at all, only read from it, so just remove those settings. All you care about is if the stencil has already been written to by the player, don't render the rock on those pixels, which is what the Comp is for. You probably want NotEqual and not GEqual as you only want to render the rock where the player has not rendered.

You do however want the player to write to the stencil, and right now it does not. With the default settings, just setting a Ref value alone will do nothing. So the player does need Pass Replace for it to write to the stencil buffer.

Also, for any of this to work, you need the player to render before the rock, so for any rocks you want to disappear when the player is behind them you'll want to make sure their queue is higher.

 

Oh I was totally getting this backwards! Thank you so much for explaining! Its working great now! The render queue was the key in getting stencils to work correctly.

 

Returning to this post, but in a different project. Now I'm looking to do the opposite, render the rock in front of the player, even if it is behind the player and without a second camera.

Here is what I have working for rendering behind with _StencilMask property set to 1:

BEHIND:
On the player's shader:

Code (CSharp):

1. Tags { "Queue" = "Transparent"}
2. Stencil {
3.      Ref [_StencilMask]
4.     Comp Always
5.     Pass Replace
6.     Fail Keep
7.     ZFail Keep
8. }

On the rock's shader:

Code (CSharp):

1. Tags { "Queue" = "Transparent+1"}
2. Stencil {
3.     Ref [_StencilMask]
4.     Comp  NotEqual
5.     Pass Keep
6.     Fail Keep
7.     ZFail Keep
8. }

But for rendering in front of the player, I can't find the right combination of settings. When I use ZTest Always, it turns my rock geo inside out and does not sort correctly with itself. Any ideas?

IN FRONT:
On the player's shader:

Code (CSharp):

1. Tags { "Queue" = "Transparent"}
2. Stencil {
3.      Ref [_StencilMask]
4.     Comp Always
5.     Pass Replace
6.     Fail Keep
7.     ZFail Keep
8. }

On the rock's shader:

Code (CSharp):

1. Tags { "Queue" = "Overlay+1"}
2. ZWrite On
3. ZTest Always
4. Stencil {
5.     Ref [_StencilMask]
6.     Comp GreaterEqual
7.     Pass Keep
8.     Fail Keep
9.     ZFail Keep
10. }

 

You've got it backwards again. Make sure the rock renders first (via render queue) and writes to the stencil. When the character renders check the stencil and don't render where the rock was.

Once the depth buffer has been written to, you're kind of too late.

 

Ok, that works by switching the player/rock comparisons & pass operations and rendering rock Transparent-1.
But, in order to support both rendering behind AND in front of the player, I would need a second pass on the player's shader with another stencil check? So, one for ref 1 for things that want to render behind and one for ref 2 for things that want to render in front. Or is it possible to have multiple stencil comparisons in the same pass, but with different refs?

 

So, really it sounds like you're trying to emulate 2D sorting with 3D meshes. Is that accurate? For that you need to carefully manage the order objects render in, just like you would for 2D games. If you want the rock to render "in front", render it first. If you want the rock to render "behind", render it after. Kind of inverse of what you'd do for 2D.

 

Yup. The player is always at a fixed z position, moving only x/y. And the stencil masking is just for several specific environment meshes that come very close to the player, but need to always render in the foreground and have no clipping from the player. And also for some meshes that need to always render behind, like one-way platforms.

I think the player will need to have two stencil comparisons:
Render queue 3000
Ref1/Always/Replace for behind
Ref2/NotEqual/Keep for in front

the behind mesh:
Render queue 3001
Ref1/NotEqual/Keep

the front mesh:
Render queue 2999
Ref2/Always/Replace

Does this seem correct? Its working individually but I need to get the player to handle both at the same time.

Thanks for all your help, its slowly starting to sink in. Shaders sometimes feel like pandora's box.

 

Here is a rough illustration, blue is player:

 

Well, not exactly.

You want to render the green "rock" first, have it write to the stencil buffer a 1.
You want to render the blue "player" next, have it read 1 and skip if equal, write 2 if not. We'll come back to how in a moment.
You want to render the red "rock" last, and have it read 2 and skip if equal.

The thing with stencils is you only get one ref value, so how do you read value and write another? With masks.

The stencil buffer has 8 bits. That can hold numerical values between 0 and 255, or 00000000 and 11111111 in binary. You can set a read and write mask to set which bits you're looking at.

So, for the green rock you use:

Code (csharp):

1. Stencil {
2.     Ref 1
3.     Pass Replace
4. }

That sets the stencil value to 1 anywhere the mesh renders.

For the player you use

Code (csharp):

1. Stencil {
2.     Ref 3       // 00000011
3.     ReadMask 1  // 00000001
4.     WriteMask 2 // 00000010
5.     Comp NotEqual
6.     Pass Replace
7. }

And for the red rock:

Code (csharp):

1. Stencil {
2.     Ref 2
3.     Comp NotEqual
4. }

Even through the ref for the player is 3, the ReadMask is only going to look at the first bit of the stencil buffer for the comparison. When writing it'll only write to the second bit, so a value of 2.

Note, 1 and 2 are masks, not the values that will be written. The magic is in the binary masking of the ref value of 3, which is both the bits of 1 and 2. A ref of 0 would write nothing.