Search Unity

  1. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  2. Improved Prefab workflow (includes Nested Prefabs!), 2D isometric Tilemap and more! Get the 2018.3 Beta now.
    Dismiss Notice
  3. Want more efficiency in your development work? Sign up to receive weekly tech and creative know-how from Unity experts.
    Dismiss Notice
  4. Participate with students all over the world and build projects to teach people. Join now!
    Dismiss Notice
  5. Build games and experiences that can load instantly and without install. Explore the Project Tiny Preview today!
    Dismiss Notice
  6. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
  7. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Cull rendering between camera and stencil mask

Discussion in 'Shaders' started by theDawckta, Aug 8, 2018.

  1. theDawckta

    theDawckta

    Joined:
    Jun 16, 2014
    Posts:
    43
    Hello community,

    I could use some help to see if this is even possible.

    So I am trying to make a "snow globe (more like a cylinder)" effect to be used in ar where the snow globe stays in place but you can move what is inside the snow globe. The environments bounds go outside the snow globe but any environment that falls outside of the snow globes internal bounds is culled so you don't see it.

    Using stencil buffer I have come close but there is one problem remaining. I don't know how to tell the rendering engine to not draw objects that are between the camera and the front stencil mask.

    If I am going about this all wrong and there is an easier way please let me know. This was just what I came up with to achieve the effect off the top of my head after messing around with depth buffer and stencil buffer. My shader knowledge is beginner if that.

    Picture time.



    So the problem is #4 being rendered when it is in between #2 and the camera. You can see in the camera view on the left side of the picture the column is out of the cylinder but is still getting rendered.

    This is the makeup of the scene.

    1. Rear half cylinder
    Half cylinder with normals pointing toward the camera (normals pointing away from the inside)

    I have this stencil shader on that geo
    Code (CSharp):
    1. Shader "Custom/Stencil/Mask OneZLessZon"
    2. {
    3.     SubShader
    4.     {
    5.         Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
    6.         ColorMask 0
    7.         ZWrite on
    8.        
    9.         Stencil
    10.         {
    11.             Ref 1
    12.             Comp always
    13.             Pass replace
    14.         }
    15.  
    16.         CGINCLUDE
    17.             struct appdata
    18.             {
    19.                 float4 vertex : POSITION;
    20.             };
    21.             struct v2f
    22.             {
    23.                 float4 pos : SV_POSITION;
    24.             };
    25.             v2f vert(appdata v)
    26.             {
    27.                 v2f o;
    28.                 o.pos = UnityObjectToClipPos(v.vertex);
    29.                 return o;
    30.             }
    31.             half4 frag(v2f i) : COLOR
    32.             {
    33.                 return half4(1,1,0,1);
    34.             }
    35.         ENDCG
    36.        
    37.         Pass
    38.         {
    39.             Cull Back
    40.             ZTest Less
    41.        
    42.             CGPROGRAM
    43.             #pragma vertex vert
    44.             #pragma fragment frag
    45.             ENDCG
    46.         }
    47.     }
    48. }
    2. Front half cylinder
    Half cylinder with normals pointing toward the camera (normals pointing away from the outside)

    I have this stencil shader on that geo
    Code (CSharp):
    1. Shader "Custom/Stencil/Mask OneZLess"
    2. {
    3.     SubShader
    4.     {
    5.         Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
    6.         ColorMask 0
    7.         ZWrite off
    8.        
    9.         Stencil
    10.         {
    11.             Ref 1
    12.             Comp always
    13.             Pass replace
    14.         }
    15.  
    16.         CGINCLUDE
    17.             struct appdata
    18.             {
    19.                 float4 vertex : POSITION;
    20.             };
    21.             struct v2f
    22.             {
    23.                 float4 pos : SV_POSITION;
    24.             };
    25.             v2f vert(appdata v)
    26.             {
    27.                 v2f o;
    28.                 o.pos = UnityObjectToClipPos(v.vertex);
    29.                 return o;
    30.             }
    31.             half4 frag(v2f i) : COLOR
    32.             {
    33.                 return half4(1,1,0,1);
    34.             }
    35.         ENDCG
    36.        
    37.         Pass
    38.         {
    39.             Cull Back
    40.             ZTest Less
    41.        
    42.             CGPROGRAM
    43.             #pragma vertex vert
    44.             #pragma fragment frag
    45.             ENDCG
    46.         }
    47.     }
    48. }
    3. Environment
    Everything in the environment has this stencil shader.
    Code (CSharp):
    1. Shader "Custom/Stencil/Diffuse RefEqualOne"
    2. {
    3.     Properties
    4.     {
    5.         _Color ("Main Color", Color) = (1,1,1,1)
    6.         _MainTex ("Base (RGB)", 2D) = "white" {}
    7.     }
    8.  
    9.     SubShader
    10.     {
    11.         Tags { "RenderType"="Opaque" "Queue"="Geometry" }
    12.         LOD 200
    13.  
    14.         Stencil
    15.         {
    16.             Ref 1
    17.             Comp equal
    18.             Pass keep
    19.         }
    20.  
    21.         CGPROGRAM
    22.         #pragma surface surf Lambert
    23.  
    24.         sampler2D _MainTex;
    25.         fixed4 _Color;
    26.  
    27.         struct Input
    28.         {
    29.             float2 uv_MainTex;
    30.         };
    31.  
    32.         void surf (Input IN, inout SurfaceOutput o)
    33.         {
    34.             fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
    35.             o.Albedo = c.rgb;
    36.             o.Alpha = c.a;
    37.         }
    38.        
    39.         ENDCG
    40.     }
    41.  
    42.     Fallback "VertexLit"
    43. }
    #1 and #2 rotate to face the camera in update to maintain the illusion.

    The back part seems to be working pretty well. You can see in the picture as the rear tower is going beyond the mask it is getting cut out and if pulled back further wouldn't be rendered. This was pretty much found by accident by toggling ZWrite to on in the #1 shader.

    The bad part is the front cylinder getting rendered when it is outside of the view cylinder. I feel like if I can get this to not render I have my problem solved.

    Thanks for looking, please let me know if there is an easier way or if I am going about this all wrong.
     
  2. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    533
    Hey there,

    interesting problem. I think stencil could be really tricky to use here (I think you would need a defined render order and a bit of magic). My idea would be to discard the position through the shader itself. Here is the result:

    upload_2018-8-10_9-39-1.png

    and here is the shader code:
    Code (CSharp):
    1.  
    2. // --------------------------------------------------------------------------------------------------------------------
    3. // <copyright file="CylinderArea.shader" company="Supyrb">
    4. //   Copyright (c) 2018 Supyrb. All rights reserved.
    5. // </copyright>
    6. // <author>
    7. //   Johannes Deml
    8. //   send@johannesdeml.com
    9. // </author>
    10. // --------------------------------------------------------------------------------------------------------------------
    11.  
    12. // From https://forum.unity.com/threads/cull-rendering-between-camera-and-stencil-mask.544280/
    13. Shader "Supyrb/CylinderArea"
    14. {
    15.    Properties
    16.    {
    17.        [HDR] _Color("Main Color", Color) = (1,1,1,1)
    18.        _CylinderCenter("Cylinder Center", Vector) = (0,0,0,0)
    19.        _CylinderRadius("Cylinder Radius", Float) = 3
    20.        
    21.        [Header(Stencil)]
    22.        _Stencil ("Stencil ID [0;255]", Float) = 0
    23.        _ReadMask ("ReadMask [0;255]", Int) = 255
    24.        _WriteMask ("WriteMask [0;255]", Int) = 255
    25.        [Enum(UnityEngine.Rendering.CompareFunction)] _StencilComp ("Stencil Comparison", Int) = 0
    26.        [Enum(UnityEngine.Rendering.StencilOp)] _StencilOp ("Stencil Operation", Int) = 0
    27.        [Enum(UnityEngine.Rendering.StencilOp)] _StencilFail ("Stencil Fail", Int) = 0
    28.        [Enum(UnityEngine.Rendering.StencilOp)] _StencilZFail ("Stencil ZFail", Int) = 0
    29.        
    30.        [Header(Rendering)]
    31.        [Enum(UnityEngine.Rendering.CullMode)] _Culling ("Culling", Int) = 2
    32.        [Enum(Off,0,On,1)] _ZWrite("ZWrite", Int) = 1
    33.        [Enum(UnityEngine.Rendering.CompareFunction)] _ZTest ("ZTest", Int) = 4
    34.        _Offset("Offset", Float) = 0
    35.        [Enum(None,0,Alpha,1,Red,8,Green,4,Blue,2,RGB,14,RGBA,15)] _ColorMask("Writing Color Mask", Int) = 15
    36.    }
    37.    
    38.    SubShader
    39.    {
    40.        Stencil
    41.        {
    42.            Ref [_Stencil]
    43.            ReadMask [_ReadMask]
    44.            WriteMask [_WriteMask]
    45.            Comp [_StencilComp]
    46.            Pass [_StencilOp]
    47.            Fail [_StencilFail]
    48.            ZFail [_StencilZFail]
    49.        }
    50.        
    51.        Tags{ "Queue" = "Geometry" "RenderType" = "Opaque" }
    52.        LOD 200
    53.        Cull [_Culling]
    54.        Offset [_Offset], [_Offset]
    55.        ZWrite [_ZWrite]
    56.        ZTest [_ZTest]
    57.        ColorMask [_ColorMask]
    58.  
    59.        CGPROGRAM
    60.        #pragma surface surf Standard
    61.        
    62.        struct Input {
    63.             float3 worldPos;
    64.         };
    65.         half4 _Color;
    66.         float4 _CylinderCenter;
    67.        float _CylinderRadius;
    68.        float squaredHorizontalDistance(float3 a, float3 b) {
    69.            float3 ab = b -a;
    70.            return ab.x * ab.x + ab.z * ab.z;
    71.        }
    72.         void surf(Input IN, inout SurfaceOutputStandard o)
    73.         {
    74.             float squaredDistance = squaredHorizontalDistance(_CylinderCenter.xyz, IN.worldPos);
    75.            if(squaredDistance > _CylinderRadius)
    76.            {
    77.                discard;
    78.            }
    79.            
    80.             o.Albedo = _Color;
    81.         }
    82.        
    83.        ENDCG
    84.  
    85.        // shadow caster pass
    86.        Pass{
    87.            Name "Caster"
    88.            Tags{ "LightMode" = "ShadowCaster" }
    89.            Cull [_Culling]
    90.            Offset [_Offset], [_Offset]
    91.            ZWrite [_ZWrite]
    92.            ZTest [_ZTest]
    93.            ColorMask [_ColorMask]
    94.            
    95.            CGPROGRAM
    96.            #pragma vertex vert
    97.            #pragma fragment frag
    98.            #pragma target 2.0
    99.            #pragma multi_compile_shadowcaster
    100.            #pragma multi_compile_instancing // allow instanced shadow pass for most of the shaders
    101.            #include "UnityCG.cginc"
    102.            
    103.            float4 _CylinderCenter;
    104.            float _CylinderRadius;
    105.    
    106.            float squaredHorizontalDistance(float3 a, float3 b) {
    107.                float3 ab = b -a;
    108.                return ab.x * ab.x + ab.z * ab.z;
    109.            }
    110.  
    111.            
    112.            struct v2f {
    113.                V2F_SHADOW_CASTER;
    114.                float3 worldPos : TEXCOORD0;
    115.                UNITY_VERTEX_OUTPUT_STEREO
    116.            };
    117.  
    118.            v2f vert(appdata_base v)
    119.            {
    120.                v2f o;
    121.                UNITY_SETUP_INSTANCE_ID(v);
    122.                o.worldPos = mul(unity_ObjectToWorld, v.vertex);
    123.                UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
    124.                TRANSFER_SHADOW_CASTER_NORMALOFFSET(o);
    125.                return o;
    126.            }
    127.  
    128.            uniform sampler2D _MainTex;
    129.            uniform half _Cutoff;
    130.  
    131.            float4 frag(v2f IN) : SV_Target
    132.            {
    133.                float squaredDistance = squaredHorizontalDistance(_CylinderCenter.xyz, IN.worldPos);
    134.                if(squaredDistance > _CylinderRadius)
    135.                {
    136.                    discard;
    137.                }
    138.                SHADOW_CASTER_FRAGMENT(IN)
    139.            }
    140.            ENDCG
    141.        }
    142.    }
    143. }
    144.  
    (Using quite a lot of configurations, as seen here)

    The special logic is more or less in those lines:
    Code (CSharp):
    1.         float4 _CylinderCenter;
    2.         float _CylinderRadius;
    3.         float squaredHorizontalDistance(float3 a, float3 b) {
    4.             float3 ab = b -a;
    5.             return ab.x * ab.x + ab.z * ab.z;
    6.         }
    7.         void surf(Input IN, inout SurfaceOutputStandard o)
    8.         {
    9.             float squaredDistance = squaredHorizontalDistance(_CylinderCenter.xyz, IN.worldPos);
    10.             if(squaredDistance > _CylinderRadius)
    11.             {
    12.                 discard;
    13.             }
    You might want to consider to do something special with the areas that are cut through the shader. An idea for how to tackle that might be here: http://www.shaderslab.com/demo-45---section.html

    Edit: Oh yeah, just as a side notice. If you have just one "Snow Globe" you could set the center and radius through a global setter (Shader.SetGlobalVector()) in order to avoid keeping track of all materials that are inside the globe. If you're fancy you can also store the radius in the 4 value of the vector, I didn't do that for better readability :)
     
    Last edited: Aug 10, 2018
  3. theDawckta

    theDawckta

    Joined:
    Jun 16, 2014
    Posts:
    43
    Thank you for the help!

    Can you tell me how to set that up in a scene? I tried putting the shader on a GameObject using a mesh renderer and material I made to see if it would effect some primitives I added. It doesn't seem to cut them though.
     
  4. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    533
    In the shader you have the two values Cylinder Center and Cylinder Radius, which control where the meshes are discarded. For the start try putting a plane to 0,0,0 and put a material with that shader on it. The default values of 0,0,0 and a radius of 3 should already cut the plane.

    Of course, it does not necessarily make sense to have those values set by the material, but rather through a script. That's why I was pointing you to Shader.SetGlobalVector() :)
     
  5. theDawckta

    theDawckta

    Joined:
    Jun 16, 2014
    Posts:
    43
    Ahhh yes, didn't have my objects zero'd, I also didn't have the shader on every object, it was late :|

    This seems a lot like using clip in shaders which appears to have a similar effect. Having the end parts capped with even a single color would really be great. I have seen capping effects like the one you linked but I have only seen them done with a flat plane. Is there a way to get the same effect but on a cylinder, where the cut into the geo has a shape to it? The cut would really improve the effect but it will take me a bit of study to get that part done,

    I see what you mean about Shader.SetGlobalVector. So to move the environment I would offset the center position in the shader using Shader.SetGlobalVector with the world position of the parent that holds all my environment items.

    Thanks for the help, this has potential of being a cool little effect. Just need to figure out how to cap the ends.
     
  6. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    533
    Thought about the caps problem a bit, and I think I found a pretty good solution:
    Doing a second pass in which we only render the bakcfaces which fill the stencil buffer with a value.
    This value then masks a cylinder around the area you want to render. Here is my result:
    upload_2018-8-12_12-18-54.png

    The project is in the result, to create no confusions with the setup.
    The cylinder of course should match the radius you defined in the shader (with unity's cyliner I think you need to add +0.5 scaling, not quite sure why though. I also put the render queue of the cylinder to 3000 in order to get the shadows of the other objects working properly. If you write your own specific shader with ZWrite completly disabled this should not be necessary.

    As for the SetGlobalVector: The value should be the position of your cylinder, not of the objects you want to move through. You should move their transforms as you're used to.

    Looking forward to seeing the result :)

    Edit: Oh btw. I also thought about other ways of getting the objects smooth into the globe. You could also just scale the object when they reach the globe through the shader. This would probably look smoother :)
     

    Attached Files:

  7. theDawckta

    theDawckta

    Joined:
    Jun 16, 2014
    Posts:
    43
    Thanks for the reply! I got tied up with a bunch of UI work but finally got back to trying this out again.

    So I grabbed a little geo off the app store to get a demo rolling but it exposed a flaw :(



    As you can see above, when a piece of geo gets cut you can see any geo inside of it that is inside the masking cylinder. This doesn't look like an easy fix, is there any way to prevent artifacts from the inside geo from rendering so you still get a smooth blue when something is cut.

    Thanks again for replying, this technique is really cool, it's so close!
     
  8. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    533
    Hey there,

    I was sadly not able to reproduce the problem with simple geometry, but I took a quick look at the shader again. I guess the problem of other meshes showing up in the blue area that are behind the rendered mesh is easily fiaxble by setting ZWrite to true. There are still two cases for which I don't really have an aswer to: Meshes that clip in one another and complex concave meshes. I did make a new shader version with enabled ZWrite. If you would use this one with only convex meshes it should render without problems.
    upload_2018-9-29_10-37-30.png
    (Yellow cube is clipping through the mesh hull of the capsule and is therefore visible)

    Did you find a solution to the problem? Or did you try scaling the objects through a shader. I think that would make an even smoother effect that is way easier achievable.
     

    Attached Files: