[RESOLVED] Trying to make holes in walls with stencil buffer shaders

Hey,

I'm developing a project where users can build a room and be able to put a hole on a wall.

I'm using stencil buffer shaders to do it :

• one used for the plane making a hole in the wall :
  
  Code (CSharp):

  1. Shader "Custom/Stencil/Mask OneZLess"
  2. {
  3.     SubShader
  4.     {
  5.         Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
  6.         ColorMask 0
  7.         ZWrite off
  8.        
  9.         Stencil
  10.         {
  11.             Ref 1
  12.             Comp always
  13.             Pass replace
  14.         }
  15.        
  16.         Pass
  17.         {
  18.             Cull Back
  19.             ZTest Less
  20.        
  21.             CGPROGRAM
  22.             #pragma vertex vert
  23.             #pragma fragment frag
  24.            
  25.             struct appdata
  26.             {
  27.                 float4 vertex : POSITION;
  28.             };
  29.             struct v2f
  30.             {
  31.                 float4 pos : SV_POSITION;
  32.             };
  33.             v2f vert(appdata v)
  34.             {
  35.                 v2f o;
  36.                 o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
  37.                 return o;
  38.             }
  39.             half4 frag(v2f i) : COLOR
  40.             {
  41.                 return half4(1,1,0,1);
  42.             }
  43.            
  44.             ENDCG
  45.         }
  46.     }
  47. }

• one for the wall that needs to be cut by the plane :
  
  Code (CSharp):

  1. Shader "Custom/Stencil/Diffuse NotEqualOne"
  2. {
  3.  
  4. Properties
  5. {
  6.     _Color ("Main Color", Color) = (1,1,1,1)
  7.     _MainTex ("Base (RGB)", 2D) = "white" {}
  8. }
  9.  
  10. SubShader
  11. {
  12.     Tags { "RenderType"="Opaque" "Queue"="Geometry" }
  13.     LOD 200
  14.  
  15.     Stencil
  16.     {
  17.         Ref 1
  18.         Comp notequal
  19.         Pass keep
  20.     }
  21.  
  22.     CGPROGRAM
  23.     #pragma surface surf Lambert
  24.  
  25.     sampler2D _MainTex;
  26.     fixed4 _Color;
  27.  
  28.     struct Input
  29.     {
  30.         float2 uv_MainTex;
  31.     };
  32.  
  33.     void surf (Input IN, inout SurfaceOutput o)
  34.     {
  35.         fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
  36.         o.Albedo = c.rgb;
  37.         o.Alpha = c.a;
  38.     }
  39.    
  40.     ENDCG
  41. }
  42.  
  43. Fallback "VertexLit"
  44. }
  45.  

The walls are like this :

• The red wall uses the wall shader with Ref = 1
• The 2 blue planes use the mask shader with Ref = 1
• The red green uses the wall shader with Ref = 2
• The 2 yellow planes use the mask shader with Ref = 2

In game it give me this :


Here, no problem, I can see through both holes.


Same thing if I look from the other side, no problem.

The problem is when I'm looking through one hole and there's another behind it. In this case the second hole doesn't render properly, as you can see :


Has anyone ever tried to do something like this ?

 

It looks like you're rendering in this order: mask1, mask2, wall1, wall2
try rendering in this order: mask1, wall1, mask2, wall2

You can do this without making additional shaders; in the upper-right of your inspector, there's a dropdown where you can set the inspector to "debug mode". Do this, and you will be able to manually set the "Render Queue" of selected materials to force materials to draw in a certain order. Any material with a render queue of "-1" will use the "Queue" assigned in it's shader.


*Side note*
It looks like there's some shadow acne on your white wall. Try changing the shadow near/far planes in the quality settings or change the "bias" and "normal bias" settings on your light that casts the shadow to fix it

 

Thanks for your answer, you put me on the right track !

The last thing I had to do was setting the render queues of my objects at runtime.

So now we have :

• Red wall : stencil = 1, render queue = 4000
• Blue planes : stencil = 1, render queue = 3999
• Green wall : stencil = 2, render queue = 3998
• Yellow planes : stencil = 2, render queue 3997

I also had to write a custom standard shader using stencil buffer for my walls, so I could put a normal map on their material.

With this I can now see through multiple holes. I can even properly see through a window behind a hole.

Here's what it looks like :

Spoiler: Pictures

 

good job, it looks great! Those door frames really hide the stencil imperfections

 

Sorry for this late comment, but your wall recive shadows?

 

Shadows still work, but you can't stencil away the portion that casts or receives the shadow. That means the empty door frame in this instance will cast a shadow as if it was a solid wall, and any shadow cast on the door frame will appear as a shadow floating in mid-air.

 

Does anyone have a standard shader version of the 2nd shader?

 

Could you please share your shader ?? I am finding a solution same with this.

Thanks!