Hiding geometry behind clipped fragments "With Pictures!"tm

Hi, I'm working on our facial system which by default looks like this:


I tried hiding the beige areas using a texture mask and clip(tex2D(_MaskTex, IN.uv_MainTex).r - 0.5);



All this did is turn the beige transparent. I need the parts of the eyes and mouth behind the beige to dissapear too.

My last attempt to fix my problem involved a multi pass shader and the stencil buffer.

Need to know:

1. The entire face (brows, eyes and mouth) is a gameobject composed of a single mesh and uses a single material.
2. The skin and dark chin you see behind are a separate gameobject and mesh

My first pass only has this:

Code (CSharp):

1. Stencil {
2.     Ref 1
3.     Comp Always
4.     Pass Replace
5.     ZFail Zero
6. }
7. ColorMask 0
8.  

My second pass is a simple diffuse shader with this stencil operation:

Code (CSharp):

1. Stencil {
2.     Ref 1
3.     Comp Equal
4. }

Since it's a single mesh, I was hoping the first pass would write 1 to the stencil except for parts behind the beige which should fail ZTest and be set to 0. The 2nd pass is suposed to draw everything marked 1 and drop the rest.

All I managed to do was to turn everything white, can't seem to get anything to work as I'd expect with that stencil buffer.

Can someone help me please?
Thank you.

 

This fact right here means what you're trying to do won't work. Any parts you want to be able to hide must be a separate object from those you don't if you're going to use stencils.

The order of operations needs to be something like this:

1. Draw face's skin & eyebrows, and any other parts of the character you don't want to be masked. No stencil operations, but must write to depth.
2. Draw mask geometry with stencil write to some non-zero ref and ColorMask 0.
   
3. Draw eyes and mouth with stencil compare against that ref.

Any other order will not work for this case. You cannot have the skin and eyebrows draw with the eyes and mouth as then there's no way to mask one but not the other as Stencils are something that happen outside of the shader functions themselves so you can't add any additional logic there.


Some alternatives would be to use destination alpha, or potentially abuse the combination of ZWrite and Blend.

Destination alpha is a trick where you use the alpha of the frame buffer within a shader's blend to mask things without stencils. Honestly, it's not worth discussing. ZWrite and Blend abuse however may be useful, and could even allow for this to work in a single object! This would require merging the mask geometry into the face itself and have all components use the same material. The key will be the polygon order needs to be tightly controlled to ensure they draw in the same order as for the stencil case above. However the trick is to use Blend SrcAlpha OneMinusSrcAlpha and to have the mask geometry have an alpha of 0, either by having a small area with alpha in the texture, using vertex colors, or some other shader trick (like testing for a certain UV region). If the polygons draw in exactly the order described above you will have the effect you want with out the need for stencils.

Here's why:
Eyebrows draw first.
Skin draws, anywhere the eyebrows where aren't drawn to as the depth buffer is already filled there and the skin is further away.
Masks draw, these write to the depth buffer in front of the skin, but behind the eyebrows, but also don't overwrite the color of the skin.
Eyes & Mouth draw, these are behind the mask, so anywhere the mask was drawn will be skipped effectively masking them!

One other thing to think about. If those masks ever extend beyond the bounds of the face, either by being animated there or if the faces can turn in 3d space, then the masks may also hide objects behind the character. You can avoid this issue by rendering the faces with a queue after the default 2000 for geometry, though transparent objects will always be a problem. Stencils don't have this issue.

 

As usual your answers are amazing @bgolus

I will have to research some of this stuff before being able to apply it so please let clarify a few things to make sure I'm chasing the right goose.

When you mention skin; you are talking about the "beige areas" around the eyes and mouth, not the character's skin right?

Here, I moved the facial system away from our model (the beige is clipped in that pic):


Here is the mesh we are using for the face:


Few extra question:

1. You mentionned splitting the skin(beige)/brows/eyes and mouth to use the stencil buffer, do I have to split the mesh into separate game objects or would using different materials for each parts work? Even if its still a single Mesh and gameobject?
2. Is my use of the stencil buffer correct? (Beside the order)
3. Does the clip operation intefere with the stencil buffer?

Thank you so much!
Fred.

 

Here is what the final result should look like:

 

I mean the character's skin (with the scar and stubble). When I say "masks" I'm referring to the beige areas in the first image, as that's what they area, they're masks for the eyes and mouth.

One mesh with multiple materials is fine. You can set the render order by adjusting the shader's queue.

The first "pass" (what I would use as the entire shader for the beige masks) is correct. The second "pass" (what I would use for the eye & mouth shader) should be Comp NotEqual as you want to draw where the beige masks did not, where Comp Equal will only render where they were, which would end up looking like this once you got the order correct.

Absolutely. Using clip (or discard) is saying "don't render at this pixel", which causes it to skip writing to the depth buffer and stencil buffer, which is actually just part of the depth buffer! Traditionally depth buffers are referred to as being 16 bit or 24 bit, but a 24 bit depth buffer uses 32 bits as the last 8 are the stencil. (Though some GPUs actually use all 32 bits for the "24 bit" depth and just have an additional 8 bit stencil on the side.)

 

Now thats a happy face!


Got it to work in forward rendering but the queue is:

1-Mask (beige)
2-Eyebrow, Eyes and mouth
3-Character's skin

Is this effect possible in defered? I never really understood how to use the stencil in that mode.

For now I have resorted to using a 2nd camera in forward to render the face over the rest of the game which runs in defered. But somehow that makes unity's fps counter drop 30 fps...

Thank you very much for your help!

 

Nope, not with stencils at least.

You don’t, because they don’t work. Unity uses the stencils for light culling masks. In earlier versions of Unity 5.0 they only ignored stencil ops on Surface Shaders when rendering using deferred, and technically they only make use of the top few bits of the stencil mask leaving most of it usable. I’m not even sure why since they limit it to a max of 4 light masks, yet they could have allowed as many masks as they have and still have had room left in the stencil buffer for other stuff for people like me who like to push things. However they’ve completely locked it down more recently and all stencil related stuff in any shader is completely stomped on now during the gbuffer passes.

The depth write method I described should work in deferred though as that doesn’t rely on stencils at all.

With the stencil method really only the order of the mask and eyes / mouth mattered. I am a little surprised the eyebrows still work though.


Another solution you might want to look at if deferred is important to you is to give each character a render texture for a face texture. Basically have an orthographic camera that renders to a render texture render out only the face and parts with unlit shaders, and use the resulting render texture as the Albedo used during deferred. Should be far less expensive than rendering a full screen secondary forward shaded camera.

 

Now that you mention it, the brows probably work because they have depth and rise above the mask. So far there doesnt appear to be any artifacts where they overlap the mask. Worst case I'll fix the order.

As for the forward camera it's rendering only one layer and the only thing on it is the face but you're saying its still rendering the whole scene?

Using an ortho cam is a great idea but I will try the depth write method first.

Just wondering, do I absolutely need to merge the mask geometry with the character's model? And Blend SrcAlpha OneMinusSrcAlpha should be set only on the mask right?

BTW Thank you so much for taking the time to share all that knowledge.

Had been stuck on this for a few days... with a headache...

 

Got the mask working in defered!



Strangly the mask causes a slight discoloration, i checked its not the metallic or smoothness setting. I wonder if it has something to do with the light or shadows. The same shader in Forward rendering doesnt show this discoloration.

Since the face's parts have all been split to different materials I set up the render order this way:

2000 Brows and Character
2005 Mask
2010 Eyes & Mouth

The mask uses this:

Code (CSharp):

1.  
2.         Zwrite On
3.         Blend SrcAlpha OneMinusSrcAlpha
4.  
5.         Pass {
6.             ColorMask 0
7.         }
8.  

And the rest are a regular surface shader with Blend SrcAlpha OneMinusSrcAlpha added.

Thanks again for all the help!

 

In some ways, yes. The key thing is to make sure none of the objects the forward camera can see do any lighting and are all explicitly unlit shaders with no fallback. Otherwise I believe the additional camera will re-render the shadow maps for each camera.

You only need to use the blend if you're using the single mesh & material option. If they're separate materials ColorMask 0 does everything you need for the mask, and the other shaders shouldn't use a blend mode at all.

In deferred the depth buffer is used as is for the later deferred shading to calculate the world position and shadow receiving. It may be the tiny difference in depth is enough to show as some minor discoloration.

 

As usual every single one of your posts teaches me something new, not surprising that im following you

You really went above and beyond thank you so much!

Have a happy 4th!

 

For completeness' sake; setting the render queue of the masked objects at 2501 and above fixes the discoloration. Yay!


I have this post to thank for the hint: https://forum.unity.com/threads/problem-with-depth-mask-in-deferred-lighting.142402/#post-1579120

Cheers!

 

A queue of 2501 puts the objects into the transparent queue, which when using the deferred renderer forces it to render using the forward rendering path. However as it’s now transparent it also means it cannot receive shadows.

 

That could become a problem but if its carefully managed we may be able to get away with it.

The problem I had is that the postprocessing stack (v2) is also generating AO for the masked geometry. Not for the whole mask, just the section intersecting with the parts being masked.

I will try fiddling with the zwrite option.

 

If I was doing this I would probably try the render texture method I described earlier. An orthographic forward camera with a far clip range of like 0.1 units (just big enough to cover the face).

 

Your right, I will give it a try. Anyways I've been curious about the performance vs the non ortho setup I tried.

Thank you!