Can you change the impacted depth for stencil shader?

I'm having a problem with a stencil mask sander drawing both objects in front of and behind the object I apply it to. My scene is currently set up like so:

There is simply a (blue) cube with two cylinders on top of it with a window plane in between them.
All blue/purple material are using this shader:

Spoiler

Code (CSharp):

1. Shader "Custom/Past"
2. {
3.     Properties
4.     {
5.         _Color ("Main Color", Color) = (1,1,1,1)
6.         _MainTex ("Base {RGB}", 2D) = "white" {}
7.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
8.         _Metallic ("Metallic", Range(0,1)) = 0.0
9.     }
10.     SubShader
11.     {
12.         Tags {"RenderType" = "Opaque" "Queue"="Geometry-2"}
13.         LOD 200
14.  
15.         Stencil
16.         {
17.             Ref 1
18.             Comp Equal
19.             Pass Keep
20.         }
21.  
22.         CGPROGRAM
23.         // Physically based Standard lighting model, and enable shadows on all light types
24.         #pragma surface surf Standard fullforwardshadows
25.  
26.         // Use shader model 3.0 target, to get nicer looking lighting
27.         #pragma target 3.0
28.  
29.         sampler2D _MainTex;
30.  
31.         struct Input
32.         {
33.             float2 uv_MainTex;
34.         };
35.  
36.         half _Glossiness;
37.         half _Metallic;
38.         fixed4 _Color;
39.  
40.         // Add instancing support for this shader. You need to check 'Enable Instancing' on materials that use the shader.
41.         // See https://docs.unity3d.com/Manual/GPUInstancing.html for more information about instancing.
42.         // #pragma instancing_options assumeuniformscaling
43.         UNITY_INSTANCING_BUFFER_START(Props)
44.             // put more per-instance properties here
45.         UNITY_INSTANCING_BUFFER_END(Props)
46.  
47.         void surf (Input IN, inout SurfaceOutputStandard o)
48.         {
49.             // Albedo comes from a texture tinted by color
50.             fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
51.             o.Albedo = c.rgb;
52.             // Metallic and smoothness come from slider variables
53.             o.Metallic = _Metallic;
54.             o.Smoothness = _Glossiness;
55.             o.Alpha = c.a;
56.         }
57.         ENDCG
58.     }
59.     //Fallback "VertexLit"
60. }

And the window plane in the middle is using this shader:

Spoiler

Code (CSharp):

1. Shader "Custom/PastStencilMask"
2. {
3.     SubShader
4.     {
5.         Tags { "RenderType"="Opaque" "Queue"="Geometry-3" }
6.         ColorMask 0
7.         ZWrite off
8.  
9.         Stencil {
10.             Ref 1
11.             Comp Always
12.             Pass Replace
13.         }
14.         LOD 200
15.  
16.         Pass
17.         {
18.             Cull Back
19.             Ztest Less
20.  
21.             CGPROGRAM
22.             #pragma vertex vert
23.             #pragma fragment frag
24.  
25.             struct appdata
26.             {
27.                 float4 vertex : POSITION;
28.             };
29.             struct v2f
30.             {
31.                 float4 pos : SV_POSITION;
32.             };
33.             v2f vert(appdata v)
34.             {
35.                 v2f o;
36.                 o.pos = UnityObjectToClipPos(v.vertex);
37.                 return o;
38.             }
39.             half4 frag(v2f i) : COLOR
40.             {
41.                 return half4(1,1,1,1);
42.             }
43.             ENDCG
44.         }
45.     }
46.     Fallback "VertexLit"
47. }
48.  

I want the blue/purple objects to only be visible through the window plane.
I think the half cylinder in the foreground is being rendered because the stencil value at the window plane is always 1 regardless of depth correct? If that's the case how would I make the stencil only impact objects behind it so the cylinder in the foreground would not render? I've tried messing around with a depth mask but don't really know enough about them to make them do what I want.

 

Correct. Stencils are a purely 2D effect. The cylinder in front of the stencil geometry has no way to know what depth the stencil was rendered at unless the stencil also writes to depth. But if the stencil writes to the depth that doesn't really help since that'll block the objects behind it, and still won't really help with the object in front as it'd have to use ZTest Greater which will cause other issues for anything more complex than a single cylinder.

The real answer is this isn't something stencils, or even the depth buffer, can really handle alone. If you're very careful about manually sorting objects you can place the objects "in front" of the stencil to render before the stencil does, but at that point you've already figured out it shouldn't be visible to the stencil, so it'd be better to just disable the object in that case.

The more generalized solution is to use a second camera with its frustum setup to match where the "portal" is. That or to actually pass either an SDF or mathematical form of the stencil geometry to the shaders you want to clip and do the coverage & depth comparison against it in those objects' shader. In the case of the above example of a basic quad geometry being used, you could do both the stencil and the in-shader clipping. All you'd need to do is pass the data for a plane (a center point and a normal) to the shader used by the cylinders and floor plane and test to see what side of the plane your objects are on. If they're in front of the plane, discard it.

 

Sorry, I'm still relatively new to shaders so I gotta inquire more about the process of clipping.
Would I first define the planes center position and normal as static inside the shader?
And then convert the clipping plane position to clip space to test which side of the plane any given fragment resides (using the dot product of the plane normal and a vector coming from the planes center to the fragment)?
I'm really not sure, I might be totally wrong. This whole subject may be a bit over my head.

 

I’d put a script on the quad you’re using and have it set global shader properties:
https://docs.unity3d.com/ScriptReference/Shader.SetGlobalVector.html
For the quad’s world space position:
https://docs.unity3d.com/ScriptReference/Transform-position.html
And forward vector:
https://docs.unity3d.com/ScriptReference/Transform-forward.html

Then in your shader you’d have to take the world position, subtract the quad’s position, then do a dot product between that vector and the quad’s forward vector. If the value is positive, it’s in front of the quad. If it’s negative it’s behind. Then you can use the clip() function to hide pixels that are in front of the plane. Any negative value passed to clip() will cause that pixel of the object to be skipped.

Code (CSharp):

1. // add to shader inside CGPROGRAM, but outside a function
2. float3 _MyPlanePos;
3. float3 _MyPlaneNormal;
4.  
5. // need to add this to Input struct
6. float3 worldPos;
7.  
8. // inside surf function
9. float3 planeRelativePos = IN.worldPos - _MyPlanePos.xyz;
10. float planeDist = dot(planeRelativePos, _MyPlaneNormal.xyz);
11. clip(-planeDist) // skips pixels in front of plane

Assuming you only have one “window” at a time this should work well.

 

Ah, so I can gather the quad properties from a traditional script and modify the shader variables there. That makes much more sense. Seeing the syntax and layout of the shader code was super helpful. Thank you very much bgolus, you've been super awesome in helping me understand shaders better, not just here but the many other places you've posted.

 

[QUOTE="

clip(-planeDist) // skips pixels in front of plane[/code]

[/QUOTE]

This is intended a good solution but it's quite expensive on mobile android devices like Quest. Do you know of any alternative to discard, clip or alphablending (heard this was costly too)?