Question Stencil and depth buffer - Selectively hiding or showing verts based on depth?

Hello,

I'm using the following shader code to create a mask that can cut out of object (e.g. terrain). Since I'm cutting out of terrain, there are hills, mountains, rivers, etc.

I want to use this stencil to cause any terrain that is LOWER than the stencil plane to not be rendered (invisible). But I want everything ABOVE the stencil plane to be rendered.

So far I have this code for the mask, which works great.

Code (CSharp):

1. // Unity built-in shader source. Copyright (c) 2016 Unity Technologies. MIT license (see license.txt)
2.  
3. Shader "Stencil/Mask 101ZLess" {
4. Properties {
5.     _Color ("Main Color", Color) = (1,1,1,1)
6.     _MainTex ("Base (RGB)", 2D) = "white" {}
7. }
8. SubShader {
9.     ColorMask 0
10.     ZWrite off
11.  
12.     Tags { "RenderType"="Opaque" "Queue"="Geometry-101" }
13.     LOD 200
14.  
15.     Stencil
16.     {
17.         Ref 1
18.         Comp always
19.         Pass replace
20.     }
21.  
22.     Pass { }
23.  
24. }
25. }
26.  

And I've modified the default nature terrain diffuse shader to react to the stencil.

Code (CSharp):

1. // Unity built-in shader source. Copyright (c) 2016 Unity Technologies. MIT license (see license.txt)
2.  
3. Shader "Stencil/Terrain" {
4.     Properties {
5.         // used in fallback on old cards & base map
6.         [HideInInspector] _MainTex ("BaseMap (RGB)", 2D) = "white" {}
7.         [HideInInspector] _Color ("Main Color", Color) = (1,1,1,1)
8.     }
9.  
10.     CGINCLUDE
11.         #pragma surface surf Lambert vertex:SplatmapVert finalcolor:SplatmapFinalColor finalprepass:SplatmapFinalPrepass finalgbuffer:SplatmapFinalGBuffer addshadow fullforwardshadows
12.         #pragma instancing_options assumeuniformscaling nomatrices nolightprobe nolightmap forwardadd
13.         #pragma multi_compile_fog
14.         #include "TerrainSplatmapCommon.cginc"
15.  
16.         void surf(Input IN, inout SurfaceOutput o)
17.         {
18.             half4 splat_control;
19.             half weight;
20.             fixed4 mixedDiffuse;
21.             SplatmapMix(IN, splat_control, weight, mixedDiffuse, o.Normal);
22.             o.Albedo = mixedDiffuse.rgb;
23.             o.Alpha = weight;
24.         }
25.     ENDCG
26.  
27.     Category {
28.         Tags {
29.             "Queue" = "Geometry-99"
30.             "RenderType" = "Opaque"
31.         }
32.    
33.         Stencil
34.         {
35.             Ref 1
36.             Comp notequal
37.             Pass keep
38.         }
39.    
40.         // TODO: Seems like "#pragma target 3.0 _NORMALMAP" can't fallback correctly on less capable devices?
41.         // Use two sub-shaders to simulate different features for different targets and still fallback correctly.
42.         SubShader { // for sm3.0+ targets
43.             CGPROGRAM
44.                 #pragma target 3.0
45.                 #pragma multi_compile __ _NORMALMAP
46.             ENDCG
47.  
48.             UsePass "Hidden/Nature/Terrain/Utilities/PICKING"
49.             UsePass "Hidden/Nature/Terrain/Utilities/SELECTION"
50.         }
51.         SubShader { // for sm2.0 targets
52.             CGPROGRAM
53.             ENDCG
54.         }
55.     }
56.  
57.     Dependency "AddPassShader"    = "Hidden/TerrainEngine/Splatmap/Diffuse-AddPass"
58.     Dependency "BaseMapShader"    = "Hidden/TerrainEngine/Splatmap/Diffuse-Base"
59.     Dependency "BaseMapGenShader" = "Hidden/TerrainEngine/Splatmap/Diffuse-BaseGen"
60.     Dependency "Details0"         = "Hidden/TerrainEngine/Details/Vertexlit"
61.     Dependency "Details1"         = "Hidden/TerrainEngine/Details/WavingDoublePass"
62.     Dependency "Details2"         = "Hidden/TerrainEngine/Details/BillboardWavingDoublePass"
63.     Dependency "Tree0"            = "Hidden/TerrainEngine/BillboardTree"
64.  
65.     Fallback "Diffuse"
66. }
67.  

So most of the shader code is the same, except the stencil section of course. This code will create a cutout wherever the mask is located on screen (good). But as it is, it renders everything in the mask's viewpoint invisible, no matter the height.

I've been thinking how I can solve this, and I thought that if I use the depth buffer to render the first pass (everything outside of the non-masked region), and use ZFail on that to increase the ref by one, and go for a second pass for ZTest Greater, equal ref 2, I thought it should work, but it doesn't.

Code (CSharp):

1.         SubShader { // for sm3.0+ targets
2.             ZTest LEqual
3.             Stencil
4.             {
5.                 Ref 1
6.                 Comp notequal
7.                 Pass keep
8.                 ZFail incrWrap
9.             }
10.    
11.             CGPROGRAM
12.                 #pragma target 3.0
13.                 #pragma multi_compile __ _NORMALMAP
14.             ENDCG
15.  
16.         ZTest greater
17.         Stencil
18.         {
19.             Ref 2
20.             Comp equal
21.             Pass keep
22.         }
23.  
24.             CGPROGRAM
25.                 #pragma target 3.0
26.                 #pragma multi_compile __ _NORMALMAP
27.             ENDCG
28.  
29.             UsePass "Hidden/Nature/Terrain/Utilities/PICKING"
30.             UsePass "Hidden/Nature/Terrain/Utilities/SELECTION"
31.         }

I am aware that I have the depth buffer disabled on the mask. So I put a transparent plane over it (standard shader) which should have the depth buffer enabled, so it should work right?

Here is the result I get (which I can understand based on the shader code I wrote). (Ignore my test objects, just focus on the rainbow terrain below)


Here is what I would like to achieve:


What am I doing wrong? How should I solve this? I am completely new to shaders, I've never written on in my life, so I need help.