Search Unity

[SOLVED] Mask shader with ZTest Always

Discussion in 'Shaders' started by theFongz, Feb 24, 2018.

  1. theFongz


    Jul 9, 2014
    Hi everyone, I'm having a shader problem.

    I want to create an object which works like a depth mask for transparent objects. However I want it to mask objects even if they are in front of it. I thought that was what ZTest Always is for, but it does not appear to work for me.

    For example, I took the depth mask shader from here:

    Code (CSharp):
    1. Shader "Masked/Mask" {
    2.     SubShader {
    3.         // Render the mask after regular geometry, but before masked geometry and
    4.         // transparent things.
    5.         Tags {"Queue" = "Geometry+10" }
    6.         // Don't draw in the RGBA channels; just the depth buffer
    7.         ColorMask 0
    8.         ZWrite On
    9.         // Do nothing specific in the pass:
    10.         Pass {}
    11.     }
    12. }
    And it gives me this when I apply it to a sphere:


    I want the full circle to be masked from the blue transparent cube. If I add ZTest Always to the shader I get the same result. If I add ZTest GEqual then the shader does nothing at all and nothing is masked from the blue transparent cube.

    Can someone please help?
  2. bgolus


    Dec 7, 2012
    ZTest controls how the object being rendered with that shader is rendered in respect to the existing depth buffer (aka z buffer). It has no direct affect on objects rendered afterward.

    A quick explanation of what the depth buffer is and how it’s generally used.

    The depth buffer is a per pixel depth value used to speed up rendering. This works by having each new object being rendered test against the depth buffer to see if it will be closer to the camera than the value stored in the depth buffer. If it’s going to be further away, it skips rendering those pixels, otherwise it renders the pixel as usual. That is ZTest LEqual, the default value for ZTest. If it’s less than (closer) or equal to the distance in the depth buffer it will render. If the shader being rendered is additionally set to ZWrite On (the default) it will update the depth buffer with new depth values.

    Using ZTest Always just says ignore what’s in the depth buffer and always render. ZTest GEqual will only render if it’s further away that what’s in the depth buffer.

    So, if you were to move that invisible sphere while using ZTest Always or ZTest GEqual behind the red cube, then move your blue cube behind the red cube but in front of the sphere, you’ll see the blue cube “through” the red cube as the sphere will have written its depth value to the depth buffer replacing the depth of the red cube. The blue cube is just using whatever values are in the depth buffer and has no knowledge of what or why.

    To do what you’re looking to do there are two main options. If you want to prevent everything transparent from rendering then the depth buffer manipulation is still a good option. However you’d need to have a slightly more complicated shader for that as you’d need to have the shader move the object’s depth to as close to the camera’s near plane as possible. The alternative and more traditional approach would be to use stencils.
    leadenyume, robot-ink and theFongz like this.
  3. theFongz


    Jul 9, 2014
    Thanks bgolus! That's great information. I thought I might have been barking up the wrong tree. Stencils were the way to go, I didn't know about them before.

    In the end I added this to my mask shader:

    Code (CSharp):
    1. Stencil {
    2.     Ref 2
    3.     Comp Greater
    4.     Pass Replace
    5. }
    As well as setting colormask 0 to make it hollow.

    And also I added this to my transparent shader:

    Code (CSharp):
    1. Stencil {
    2.     Ref 2
    3.     Comp NotEqual
    4.     Pass Zero
    5. }
    And wound up with this (note it's a different transparent shader to the one from before, which was the standard shader):

    mask working.png

    100% success!

    Also thanks to this guy whose instructions were a bit easier to follow than the official stencil manual.

    Full masking shader is below:

    Code (CSharp):
    1. Shader "Custom/MaskShader" {
    2.     SubShader {
    3.         Tags { "RenderType"="Opaque" "Queue"="Transparent-1"}
    4.         colormask 0
    5.         Pass {
    6.             Stencil {
    7.                 Ref 2
    8.                 Comp Greater
    9.                 Pass Replace
    10.             }
    12.             CGPROGRAM
    13.             #pragma vertex vert
    14.             #pragma fragment frag
    15.             struct appdata {
    16.                 float4 vertex : POSITION;
    17.             };
    18.             struct v2f {
    19.                 float4 pos : SV_POSITION;
    20.             };
    21.             v2f vert(appdata v) {
    22.                 v2f o;
    23.                 o.pos = UnityObjectToClipPos(v.vertex);
    24.                 return o;
    25.             }
    26.             half4 frag(v2f i) : SV_Target {
    27.                 return half4(1,0,0,1);
    28.             }
    29.             ENDCG
    30.         }
    31.     }
    32. }