Transparent Occlusion

I'm trying to achieve an effect where I have a completely transparent object (so totally invisible, as seen using Fade or Cutout, rather than Transparent or Opaque), and anything behind that object is rendered differently to those out in the open, similar to this:

(but with a transparent cube at the front - and what I really want is to change the alpha of the objects at the back when they are behind the invisible cube).

The bunnies have a script on them to change how they look when they are behind the object with the shader. The shader looks like this:

Code (CSharp):

1. Shader "Occlusion FX/Occlusion Mask" {
2.     Properties {
3.         _MainTex    ("Main", 2D) = "white" {}
4.         _Glossiness ("Smoothness", Range(0, 1)) = 0
5.         _Metallic   ("Metallic", Range(0, 1)) = 0
6.     }
7.     SubShader {
8.         Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
9.         Stencil {
10.             Ref 128
11.             Comp Always
12.             Pass Replace
13.         }
14.         CGPROGRAM
15.         #pragma surface surf Standard
16.         #include "Utils.cginc"
17.         ENDCG
18.     }
19.     FallBack Off
20. }  

Here is what Utils.cginc looks like:

Code (CSharp):

1. #ifndef UTILS_INCLUDED
2. #define UTILS_INCLUDED
3.  
4. sampler2D _MainTex;
5. half _Metallic, _Glossiness;
6. struct Input
7. {
8.     float2 uv_MainTex;
9. };
10. void surf (Input IN, inout SurfaceOutputStandard o)
11. {
12.     half4 c = tex2D (_MainTex, IN.uv_MainTex);
13.     o.Albedo = c.rgb;
14.     o.Alpha = c.a;
15.     o.Metallic = _Metallic;
16.     o.Smoothness = _Glossiness;
17. }
18.  
19. #endif

The problem is that the cube at the front is not totally invisible, even though I have assigned it a transparent texture. I have another Shader which succeeds in completely hiding the front cube by adding ZWrite Off and ZTest Greater to the SubShader like this:

Code (CSharp):

1. Shader "Occlusion FX/Occlusion Glow Source" {
2.     Properties {}
3.     SubShader {
4.         Tags { "RenderType"="Opaque" "Queue"="Geometry" }
5.         ZTest Greater
6.         ZWrite Off
7.  
8.         CGPROGRAM
9.         #pragma surface surf Unlit
10.  
11.         struct Input
12.         {
13.             float2 uv_MainTex;
14.             float3 viewDir;
15.         };
16.         void surf (Input IN, inout SurfaceOutput o)
17.         {
18.             half rim = 1.0 - saturate(dot(normalize(IN.viewDir), o.Normal));
19.             rim = pow(rim, 1.0);
20.             o.Albedo = fixed3(1, 1, 1) * rim;
21.             o.Alpha = 1.0;
22.             o.Emission = fixed3(0.2, 0.2, 0.2);
23.         }
24.         inline fixed4 LightingUnlit (SurfaceOutput s, fixed3 lightDir, fixed atten)
25.         {
26.             return fixed4(s.Albedo, s.Alpha);
27.         }
28.         ENDCG
29.     }
30.     FallBack Off
31. }

but then the bunnies at the back never pick up the effects. If i use a standard shader which I make invisible using Fade or Cutout, there is no effect on the bunnies either.

I read this post by @bgolus https://forum.unity.com/threads/render-mode-transparent-doesnt-work-see-video.357853/#post-2315934

but it seems like my 2nd shader is turning ZWrite Off to make the cube invisible, but I understood his post to say I should be adding:

 Pass {         ZWrite On         ColorMask 0     } 

if I use this shader he linked to, it completely hides the bunnies when they are behind the box (though it does succeed in making the box totally invisible):

Code (CSharp):

1. Shader "Transparent/Diffuse ZWrite" {
2. Properties {
3.     _Color ("Main Color", Color) = (1,1,1,1)
4.     _MainTex ("Base (RGB) Trans (A)", 2D) = "white" {}
5. }
6. SubShader {
7.     Tags {"Queue"="Transparent" "IgnoreProjector"="True" "RenderType"="Transparent"}
8.     LOD 200
9.  
10.     // extra pass that renders to depth buffer only
11.     Pass {
12.         ZWrite On
13.         ColorMask 0
14.     }
15.  
16.     // paste in forward rendering passes from Transparent/Diffuse
17.     UsePass "Transparent/Diffuse/FORWARD"
18. }
19. Fallback "Transparent/VertexLit"
20. }

Obviously I don't really know anything about shaders, but I'm hoping someone can point me in the right direction to achieve this effect. For completeness, here is the script on the bunnies (though I dont think it is important):

Code (CSharp):

1. using UnityEngine;
2.  
3. public class OcclusionFX : MonoBehaviour
4. {
5.     public enum EFX { EFX_Translucency, EFX_Pattern, EFX_Wireframe, EFX_XRay, EFX_Stripe, EFX_Outline };
6.     public EFX m_fx = EFX.EFX_XRay;
7.     private EFX m_currFx = EFX.EFX_XRay;
8.  
9.     [Header("Translucency")]
10.     [Range(0.1f, 0.5f)] public float m_Alpha = 0.3f;
11.     public Color m_Tint = Color.white;
12.     public float m_TintIntensity = 1f;
13.  
14.     [Header("Pattern")]
15.     public Texture2D m_PatternTex;
16.     public float m_PatternScale = 64;
17.     public Color m_PatternColor = Color.yellow;
18.  
19.     [Header("Wireframe")]
20.     [Range(1f, 2000f)] public float m_WireThickness = 800f;
21.     [Range(0f, 1f)] public float m_WireClipThreshold = 0.5f;
22.     public Color m_WireColor = Color.green;
23.  
24.     [Header("XRay")]
25.     public Color m_EmissionColor = Color.white;
26.     public Color m_XrayColor = Color.green;
27.     [Range(0.01f, 2f)] public float m_XrayPower = 0.1f;
28.  
29.     [Header("Stripe")]
30.     public Texture2D m_StripeTex;
31.     public Color m_StripeColor = new Color (1f, 0.8f, 0f, 1f);
32.     [Range(0.05f, 0.5f)] public float m_StripeWidth = 0.1f;
33.     [Range(0.1f, 10f)] public float m_StripeDensity = 5f;
34.  
35.     [Header("Outline")]
36.     [Range(0.01f, 0.1f)] public float m_OutlineWidth = 0.03f;
37.     public Color m_OutlineColor = new Color (0f, 1f, 0f, 1f);
38.  
39.     Shader m_SdrTranslucency;
40.     Shader m_SdrPattern;
41.     Shader m_SdrWireframe;
42.     Shader m_SdrXRay;
43.     Shader m_SdrStripe;
44.     Shader m_SdrOutline;
45.     Renderer m_Renderer;
46.  
47.     Material[] GetMaterials ()
48.     {
49.         if (Application.isPlaying)
50.             return m_Renderer.materials;
51.         else
52.             return m_Renderer.sharedMaterials;
53.     }
54.     void SetMaterialsFloat (string name, float f)
55.     {
56.         Material[] mats = GetMaterials ();
57.         for (int i = 0; i < mats.Length; i++)
58.             mats[i].SetFloat (name, f);
59.     }
60.     void SetMaterialsColor (string name, Color c)
61.     {
62.         Material[] mats = GetMaterials ();
63.         for (int i = 0; i < mats.Length; i++)
64.             mats[i].SetColor (name, c);
65.     }
66.     void SetMaterialsTexture (string name, Texture t)
67.     {
68.         Material[] mats = GetMaterials ();
69.         for (int i = 0; i < mats.Length; i++)
70.             mats[i].SetTexture (name, t);
71.     }
72.     void SetMaterialsShader (Shader sdr)
73.     {
74.         Material[] mats = GetMaterials ();
75.         for (int i = 0; i < mats.Length; i++)
76.             mats[i].shader = sdr;
77.     }
78.     ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
79.     void Start ()
80.     {
81.         m_Renderer = GetComponent<Renderer> ();
82.  
83.         m_SdrTranslucency = Shader.Find ("Occlusion FX/Translucency");
84.         m_SdrPattern = Shader.Find ("Occlusion FX/Pattern");
85.         m_SdrWireframe = Shader.Find ("Occlusion FX/Wireframe");
86.         m_SdrXRay = Shader.Find ("Occlusion FX/XRay");
87.         m_SdrStripe = Shader.Find ("Occlusion FX/Stripe");
88.         m_SdrOutline = Shader.Find ("Occlusion FX/Outline");
89.         ChangeFXShader ();
90.         UpdateFXParameters ();
91.     }
92.     void Update ()
93.     {
94.         UpdateFXParameters ();
95.         if (m_currFx != m_fx)
96.         {
97.             m_currFx = m_fx;
98.             ChangeFXShader ();
99.         }
100.     }
101.     void UpdateFXParameters ()
102.     {
103.         if (m_currFx == EFX.EFX_Translucency)
104.         {
105.             SetMaterialsFloat ("_Alpha", m_Alpha);
106.             SetMaterialsColor ("_Tint", m_Tint);
107.             SetMaterialsFloat ("_TintIntensity", m_TintIntensity);
108.         }
109.         else if (m_currFx == EFX.EFX_Pattern)
110.         {
111.             SetMaterialsTexture ("_PatternTex", m_PatternTex);
112.             SetMaterialsFloat ("_PatternScale", m_PatternScale);
113.             SetMaterialsColor ("_PatternColor", m_PatternColor);
114.         }
115.         else if (m_currFx == EFX.EFX_Wireframe)
116.         {
117.             SetMaterialsFloat ("_WireThickness", m_WireThickness);
118.             SetMaterialsFloat ("_ClipThreshold", m_WireClipThreshold);
119.             SetMaterialsColor ("_WireColor", m_WireColor);
120.         }
121.         else if (m_currFx == EFX.EFX_XRay)
122.         {
123.             SetMaterialsFloat ("_XrayPower", m_XrayPower);
124.             SetMaterialsColor ("_EmissionColor", m_EmissionColor);
125.             SetMaterialsColor ("_XrayColor", m_XrayColor);
126.         }
127.         else if (m_currFx == EFX.EFX_Stripe)
128.         {
129.             SetMaterialsFloat ("_StripeWidth", m_StripeWidth);
130.             SetMaterialsFloat ("_StripeDensity", m_StripeDensity);
131.             SetMaterialsColor ("_StripeColor", m_StripeColor);
132.             SetMaterialsTexture ("_StripeTex", m_StripeTex);
133.         }
134.         else if (m_currFx == EFX.EFX_Outline)
135.         {
136.             SetMaterialsColor ("_OutlineColor", m_OutlineColor);
137.             SetMaterialsFloat ("_Outline", m_OutlineWidth);
138.         }
139.     }
140.     void ChangeFXShader ()
141.     {
142.         if (m_currFx == EFX.EFX_Translucency)
143.             SetMaterialsShader (m_SdrTranslucency);
144.         else if (m_currFx == EFX.EFX_Pattern)
145.             SetMaterialsShader (m_SdrPattern);
146.         else if (m_currFx == EFX.EFX_Wireframe)
147.             SetMaterialsShader (m_SdrWireframe);
148.         else if (m_currFx == EFX.EFX_XRay)
149.             SetMaterialsShader (m_SdrXRay);
150.         else if (m_currFx == EFX.EFX_Stripe)
151.             SetMaterialsShader (m_SdrStripe);
152.         else if (m_currFx == EFX.EFX_Outline)
153.             SetMaterialsShader (m_SdrOutline);
154.     }
155. }
156.  

 

Because your shader is an opaque shader. You never set the surface shader to use “alpha” in the #pragma surface line, so by default it’ll completely ignore whatever o.Alpha is set to and won’t do any kind of blend or fade. That’s not the real problem though.

The real problem is your mask shader should not be a surface shader. Neither should your glow shader, or really any shader that you don’t want to interact with Unity’s lighting system. Surface shaders are vertex fragment shader generators explicitly designed to let users use Unity’s lighting systems easily. If you don’t want lighting, use a vertex fragment shader. Even if you use the hack “Unlit” custom lighting function with a Surface Shader, it will still have weird problems when you have multiple lights in the scene and you're paying some additional unnecessary costs as the shader is doing more work than you need.

The "Diffuse ZWrite" shader you posted is closer to what you want, but you explicitly do not want the UsePass or the Fallback as you really just want a shader that does as little as possible.

Code (CSharp):

1. Shader "Occlusion FX/Occlusion Mask" {
2.     SubShader {
3.         Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
4.         Stencil {
5.             Ref 128
6.             Comp Always
7.             Pass Replace
8.         }
9.         Pass {
10.             Fog { Mode Off }
11.             Color (0,0,0,0)
12.             ColorMask 0
13.         }
14.     }
15. }

Note that the Pass {} with no CGPROGRAM block inside is also vertex fragment shader generator. This used to be for fixed function shaders, but those no longer really exist, so instead that generates a simple vertex fragment shader. The above code will construct a very simple shader, though curiously a hand written one would still be slightly faster.

The main issue you’re having is you’re not thinking about the rendering order. Be it stencil based or depth buffer based, the "mask" has to render before the objects you want to be hidden / shown are rendered. A combination of stencil and ZWrite is likely what you'll need in the end. There are a number of threads on the forum for how to go about this. It all depends on exactly how you want this to behave.

For example:
The invisible cube mask, is it only hiding the bunnies or specific objects in the scene, or everything in the scene?
Is the mask hiding everything that appears in that part of the screen, or only things that are behind it; ie: if a bunny is closer to the camera than the box does it render normally?
If there's an object in front of the mask, do the "glows" show on top of that object too, or should it only show where the cube is "visible".

This thread (and the one linked to within it) go into these topics.
https://forum.unity.com/threads/sorting-issues-with-stencil-masks.526740/

 

Actually the script you provided does pretty much exactly what I want - it masks the bunnies behind the cube but not those closer to the camera than the cube. Thank you so much!

One final question - is it possible to change the opacity of the cube through script, for example attaching it to a slider?
I tried changing all 4 values in the Color inside the Pass and nothing seemed to happen.