Search Unity

  1. Get all the Unite Berlin 2018 news on the blog.
    Dismiss Notice
  2. Unity 2018.2 has arrived! Read about it here.
    Dismiss Notice
  3. We're looking for your feedback on the platforms you use and how you use them. Let us know!
    Dismiss Notice
  4. Improve your Unity skills with a certified instructor in a private, interactive classroom. Learn more.
    Dismiss Notice
  5. ARCore is out of developer preview! Read about it here.
    Dismiss Notice
  6. Magic Leap’s Lumin SDK Technical Preview for Unity lets you get started creating content for Magic Leap One™. Find more information on our blog!
    Dismiss Notice
  7. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Cull rendering between camera and stencil mask

Discussion in 'Shaders' started by theDawckta, Aug 8, 2018 at 12:07 AM.

  1. theDawckta

    theDawckta

    Joined:
    Jun 16, 2014
    Posts:
    39
    Hello community,

    I could use some help to see if this is even possible.

    So I am trying to make a "snow globe (more like a cylinder)" effect to be used in ar where the snow globe stays in place but you can move what is inside the snow globe. The environments bounds go outside the snow globe but any environment that falls outside of the snow globes internal bounds is culled so you don't see it.

    Using stencil buffer I have come close but there is one problem remaining. I don't know how to tell the rendering engine to not draw objects that are between the camera and the front stencil mask.

    If I am going about this all wrong and there is an easier way please let me know. This was just what I came up with to achieve the effect off the top of my head after messing around with depth buffer and stencil buffer. My shader knowledge is beginner if that.

    Picture time.



    So the problem is #4 being rendered when it is in between #2 and the camera. You can see in the camera view on the left side of the picture the column is out of the cylinder but is still getting rendered.

    This is the makeup of the scene.

    1. Rear half cylinder
    Half cylinder with normals pointing toward the camera (normals pointing away from the inside)

    I have this stencil shader on that geo
    Code (CSharp):
    1. Shader "Custom/Stencil/Mask OneZLessZon"
    2. {
    3.     SubShader
    4.     {
    5.         Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
    6.         ColorMask 0
    7.         ZWrite on
    8.        
    9.         Stencil
    10.         {
    11.             Ref 1
    12.             Comp always
    13.             Pass replace
    14.         }
    15.  
    16.         CGINCLUDE
    17.             struct appdata
    18.             {
    19.                 float4 vertex : POSITION;
    20.             };
    21.             struct v2f
    22.             {
    23.                 float4 pos : SV_POSITION;
    24.             };
    25.             v2f vert(appdata v)
    26.             {
    27.                 v2f o;
    28.                 o.pos = UnityObjectToClipPos(v.vertex);
    29.                 return o;
    30.             }
    31.             half4 frag(v2f i) : COLOR
    32.             {
    33.                 return half4(1,1,0,1);
    34.             }
    35.         ENDCG
    36.        
    37.         Pass
    38.         {
    39.             Cull Back
    40.             ZTest Less
    41.        
    42.             CGPROGRAM
    43.             #pragma vertex vert
    44.             #pragma fragment frag
    45.             ENDCG
    46.         }
    47.     }
    48. }
    2. Front half cylinder
    Half cylinder with normals pointing toward the camera (normals pointing away from the outside)

    I have this stencil shader on that geo
    Code (CSharp):
    1. Shader "Custom/Stencil/Mask OneZLess"
    2. {
    3.     SubShader
    4.     {
    5.         Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
    6.         ColorMask 0
    7.         ZWrite off
    8.        
    9.         Stencil
    10.         {
    11.             Ref 1
    12.             Comp always
    13.             Pass replace
    14.         }
    15.  
    16.         CGINCLUDE
    17.             struct appdata
    18.             {
    19.                 float4 vertex : POSITION;
    20.             };
    21.             struct v2f
    22.             {
    23.                 float4 pos : SV_POSITION;
    24.             };
    25.             v2f vert(appdata v)
    26.             {
    27.                 v2f o;
    28.                 o.pos = UnityObjectToClipPos(v.vertex);
    29.                 return o;
    30.             }
    31.             half4 frag(v2f i) : COLOR
    32.             {
    33.                 return half4(1,1,0,1);
    34.             }
    35.         ENDCG
    36.        
    37.         Pass
    38.         {
    39.             Cull Back
    40.             ZTest Less
    41.        
    42.             CGPROGRAM
    43.             #pragma vertex vert
    44.             #pragma fragment frag
    45.             ENDCG
    46.         }
    47.     }
    48. }
    3. Environment
    Everything in the environment has this stencil shader.
    Code (CSharp):
    1. Shader "Custom/Stencil/Diffuse RefEqualOne"
    2. {
    3.     Properties
    4.     {
    5.         _Color ("Main Color", Color) = (1,1,1,1)
    6.         _MainTex ("Base (RGB)", 2D) = "white" {}
    7.     }
    8.  
    9.     SubShader
    10.     {
    11.         Tags { "RenderType"="Opaque" "Queue"="Geometry" }
    12.         LOD 200
    13.  
    14.         Stencil
    15.         {
    16.             Ref 1
    17.             Comp equal
    18.             Pass keep
    19.         }
    20.  
    21.         CGPROGRAM
    22.         #pragma surface surf Lambert
    23.  
    24.         sampler2D _MainTex;
    25.         fixed4 _Color;
    26.  
    27.         struct Input
    28.         {
    29.             float2 uv_MainTex;
    30.         };
    31.  
    32.         void surf (Input IN, inout SurfaceOutput o)
    33.         {
    34.             fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
    35.             o.Albedo = c.rgb;
    36.             o.Alpha = c.a;
    37.         }
    38.        
    39.         ENDCG
    40.     }
    41.  
    42.     Fallback "VertexLit"
    43. }
    #1 and #2 rotate to face the camera in update to maintain the illusion.

    The back part seems to be working pretty well. You can see in the picture as the rear tower is going beyond the mask it is getting cut out and if pulled back further wouldn't be rendered. This was pretty much found by accident by toggling ZWrite to on in the #1 shader.

    The bad part is the front cylinder getting rendered when it is outside of the view cylinder. I feel like if I can get this to not render I have my problem solved.

    Thanks for looking, please let me know if there is an easier way or if I am going about this all wrong.
     
  2. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    497
    Hey there,

    interesting problem. I think stencil could be really tricky to use here (I think you would need a defined render order and a bit of magic). My idea would be to discard the position through the shader itself. Here is the result:

    upload_2018-8-10_9-39-1.png

    and here is the shader code:
    Code (CSharp):
    1.  
    2. // --------------------------------------------------------------------------------------------------------------------
    3. // <copyright file="CylinderArea.shader" company="Supyrb">
    4. //   Copyright (c) 2018 Supyrb. All rights reserved.
    5. // </copyright>
    6. // <author>
    7. //   Johannes Deml
    8. //   send@johannesdeml.com
    9. // </author>
    10. // --------------------------------------------------------------------------------------------------------------------
    11.  
    12. // From https://forum.unity.com/threads/cull-rendering-between-camera-and-stencil-mask.544280/
    13. Shader "Supyrb/CylinderArea"
    14. {
    15.    Properties
    16.    {
    17.        [HDR] _Color("Main Color", Color) = (1,1,1,1)
    18.        _CylinderCenter("Cylinder Center", Vector) = (0,0,0,0)
    19.        _CylinderRadius("Cylinder Radius", Float) = 3
    20.        
    21.        [Header(Stencil)]
    22.        _Stencil ("Stencil ID [0;255]", Float) = 0
    23.        _ReadMask ("ReadMask [0;255]", Int) = 255
    24.        _WriteMask ("WriteMask [0;255]", Int) = 255
    25.        [Enum(UnityEngine.Rendering.CompareFunction)] _StencilComp ("Stencil Comparison", Int) = 0
    26.        [Enum(UnityEngine.Rendering.StencilOp)] _StencilOp ("Stencil Operation", Int) = 0
    27.        [Enum(UnityEngine.Rendering.StencilOp)] _StencilFail ("Stencil Fail", Int) = 0
    28.        [Enum(UnityEngine.Rendering.StencilOp)] _StencilZFail ("Stencil ZFail", Int) = 0
    29.        
    30.        [Header(Rendering)]
    31.        [Enum(UnityEngine.Rendering.CullMode)] _Culling ("Culling", Int) = 2
    32.        [Enum(Off,0,On,1)] _ZWrite("ZWrite", Int) = 1
    33.        [Enum(UnityEngine.Rendering.CompareFunction)] _ZTest ("ZTest", Int) = 4
    34.        _Offset("Offset", Float) = 0
    35.        [Enum(None,0,Alpha,1,Red,8,Green,4,Blue,2,RGB,14,RGBA,15)] _ColorMask("Writing Color Mask", Int) = 15
    36.    }
    37.    
    38.    SubShader
    39.    {
    40.        Stencil
    41.        {
    42.            Ref [_Stencil]
    43.            ReadMask [_ReadMask]
    44.            WriteMask [_WriteMask]
    45.            Comp [_StencilComp]
    46.            Pass [_StencilOp]
    47.            Fail [_StencilFail]
    48.            ZFail [_StencilZFail]
    49.        }
    50.        
    51.        Tags{ "Queue" = "Geometry" "RenderType" = "Opaque" }
    52.        LOD 200
    53.        Cull [_Culling]
    54.        Offset [_Offset], [_Offset]
    55.        ZWrite [_ZWrite]
    56.        ZTest [_ZTest]
    57.        ColorMask [_ColorMask]
    58.  
    59.        CGPROGRAM
    60.        #pragma surface surf Standard
    61.        
    62.        struct Input {
    63.             float3 worldPos;
    64.         };
    65.         half4 _Color;
    66.         float4 _CylinderCenter;
    67.        float _CylinderRadius;
    68.        float squaredHorizontalDistance(float3 a, float3 b) {
    69.            float3 ab = b -a;
    70.            return ab.x * ab.x + ab.z * ab.z;
    71.        }
    72.         void surf(Input IN, inout SurfaceOutputStandard o)
    73.         {
    74.             float squaredDistance = squaredHorizontalDistance(_CylinderCenter.xyz, IN.worldPos);
    75.            if(squaredDistance > _CylinderRadius)
    76.            {
    77.                discard;
    78.            }
    79.            
    80.             o.Albedo = _Color;
    81.         }
    82.        
    83.        ENDCG
    84.  
    85.        // shadow caster pass
    86.        Pass{
    87.            Name "Caster"
    88.            Tags{ "LightMode" = "ShadowCaster" }
    89.            Cull [_Culling]
    90.            Offset [_Offset], [_Offset]
    91.            ZWrite [_ZWrite]
    92.            ZTest [_ZTest]
    93.            ColorMask [_ColorMask]
    94.            
    95.            CGPROGRAM
    96.            #pragma vertex vert
    97.            #pragma fragment frag
    98.            #pragma target 2.0
    99.            #pragma multi_compile_shadowcaster
    100.            #pragma multi_compile_instancing // allow instanced shadow pass for most of the shaders
    101.            #include "UnityCG.cginc"
    102.            
    103.            float4 _CylinderCenter;
    104.            float _CylinderRadius;
    105.    
    106.            float squaredHorizontalDistance(float3 a, float3 b) {
    107.                float3 ab = b -a;
    108.                return ab.x * ab.x + ab.z * ab.z;
    109.            }
    110.  
    111.            
    112.            struct v2f {
    113.                V2F_SHADOW_CASTER;
    114.                float3 worldPos : TEXCOORD0;
    115.                UNITY_VERTEX_OUTPUT_STEREO
    116.            };
    117.  
    118.            v2f vert(appdata_base v)
    119.            {
    120.                v2f o;
    121.                UNITY_SETUP_INSTANCE_ID(v);
    122.                o.worldPos = mul(unity_ObjectToWorld, v.vertex);
    123.                UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
    124.                TRANSFER_SHADOW_CASTER_NORMALOFFSET(o);
    125.                return o;
    126.            }
    127.  
    128.            uniform sampler2D _MainTex;
    129.            uniform half _Cutoff;
    130.  
    131.            float4 frag(v2f IN) : SV_Target
    132.            {
    133.                float squaredDistance = squaredHorizontalDistance(_CylinderCenter.xyz, IN.worldPos);
    134.                if(squaredDistance > _CylinderRadius)
    135.                {
    136.                    discard;
    137.                }
    138.                SHADOW_CASTER_FRAGMENT(IN)
    139.            }
    140.            ENDCG
    141.        }
    142.    }
    143. }
    144.  
    (Using quite a lot of configurations, as seen here)

    The special logic is more or less in those lines:
    Code (CSharp):
    1.         float4 _CylinderCenter;
    2.         float _CylinderRadius;
    3.         float squaredHorizontalDistance(float3 a, float3 b) {
    4.             float3 ab = b -a;
    5.             return ab.x * ab.x + ab.z * ab.z;
    6.         }
    7.         void surf(Input IN, inout SurfaceOutputStandard o)
    8.         {
    9.             float squaredDistance = squaredHorizontalDistance(_CylinderCenter.xyz, IN.worldPos);
    10.             if(squaredDistance > _CylinderRadius)
    11.             {
    12.                 discard;
    13.             }
    You might want to consider to do something special with the areas that are cut through the shader. An idea for how to tackle that might be here: http://www.shaderslab.com/demo-45---section.html

    Edit: Oh yeah, just as a side notice. If you have just one "Snow Globe" you could set the center and radius through a global setter (Shader.SetGlobalVector()) in order to avoid keeping track of all materials that are inside the globe. If you're fancy you can also store the radius in the 4 value of the vector, I didn't do that for better readability :)
     
    Last edited: Aug 10, 2018 at 9:36 AM
  3. theDawckta

    theDawckta

    Joined:
    Jun 16, 2014
    Posts:
    39
    Thank you for the help!

    Can you tell me how to set that up in a scene? I tried putting the shader on a GameObject using a mesh renderer and material I made to see if it would effect some primitives I added. It doesn't seem to cut them though.
     
  4. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    497
    In the shader you have the two values Cylinder Center and Cylinder Radius, which control where the meshes are discarded. For the start try putting a plane to 0,0,0 and put a material with that shader on it. The default values of 0,0,0 and a radius of 3 should already cut the plane.

    Of course, it does not necessarily make sense to have those values set by the material, but rather through a script. That's why I was pointing you to Shader.SetGlobalVector() :)
     
  5. theDawckta

    theDawckta

    Joined:
    Jun 16, 2014
    Posts:
    39
    Ahhh yes, didn't have my objects zero'd, I also didn't have the shader on every object, it was late :|

    This seems a lot like using clip in shaders which appears to have a similar effect. Having the end parts capped with even a single color would really be great. I have seen capping effects like the one you linked but I have only seen them done with a flat plane. Is there a way to get the same effect but on a cylinder, where the cut into the geo has a shape to it? The cut would really improve the effect but it will take me a bit of study to get that part done,

    I see what you mean about Shader.SetGlobalVector. So to move the environment I would offset the center position in the shader using Shader.SetGlobalVector with the world position of the parent that holds all my environment items.

    Thanks for the help, this has potential of being a cool little effect. Just need to figure out how to cap the ends.
     
  6. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    497
    Thought about the caps problem a bit, and I think I found a pretty good solution:
    Doing a second pass in which we only render the bakcfaces which fill the stencil buffer with a value.
    This value then masks a cylinder around the area you want to render. Here is my result:
    upload_2018-8-12_12-18-54.png

    The project is in the result, to create no confusions with the setup.
    The cylinder of course should match the radius you defined in the shader (with unity's cyliner I think you need to add +0.5 scaling, not quite sure why though. I also put the render queue of the cylinder to 3000 in order to get the shadows of the other objects working properly. If you write your own specific shader with ZWrite completly disabled this should not be necessary.

    As for the SetGlobalVector: The value should be the position of your cylinder, not of the objects you want to move through. You should move their transforms as you're used to.

    Looking forward to seeing the result :)

    Edit: Oh btw. I also thought about other ways of getting the objects smooth into the globe. You could also just scale the object when they reach the globe through the shader. This would probably look smoother :)
     

    Attached Files: