Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Cut-Out Shader casts wrong shadows

Discussion in 'Shaders' started by launzone, Apr 13, 2021.

  1. launzone

    launzone

    Joined:
    Dec 2, 2015
    Posts:
    57
    Hey,
    I am working on a custom cut-out shader. (i am using custom toon shaders for all elements in my game that receive "noisy" shadows) I got the part working that the albedo gets cut off based on the alpha of a texture. The shader doesn't cast and receive shadows in the right way yet though.
    It seems like it casts and receives the shadow using the full geometry and not with the cut out texture.

    upload_2021-4-13_14-43-22.png upload_2021-4-13_14-44-47.png

    i don't really have an idea how to go about this, i tried looking at the standard and other cut out shaders but they are black boxes for me. any help would be appreciated!

    this are the tags and pragma and include things:


    Code (CSharp):
    1. Tags {
    2.                 "LightMode" = "ForwardBase"
    3.                 "PassFlags" = "OnlyDirectional"
    4.                 "Queue" = "AlphaTest"
    5.                 "IgnoreProjector" = "True"
    6.                 "RenderType" = "TransparentCutout"
    7.             }
    8.  
    9.             Cull off
    10.  
    11.             CGPROGRAM
    12.             #pragma vertex vert addshadow fullforwardshadows
    13.             #pragma fragment frag
    14.             #pragma multi_compile_fwdbase
    15.          
    16.             #include "UnityCG.cginc"
    17.             #include "Lighting.cginc"
    18.             #include "AutoLight.cginc"
    this are the parts where i get the shadows:

    Code (CSharp):
    1. v2f vert (appdata v)
    2.             {
    3.                 v2f o;
    4.                 o.pos = UnityObjectToClipPos(v.vertex);
    5.                 o.worldPos = mul(unity_ObjectToWorld, v.vertex);
    6.                 o.uv = TRANSFORM_TEX(v.uv, _MainTex);
    7.                 o.worldNormal = UnityObjectToWorldNormal(v.normal);
    8.                 o.viewDir = WorldSpaceViewDir(v.vertex);
    9.                 o.color = v.color;
    10.                 TRANSFER_SHADOW(o)
    11.                 return o;
    12.             }
    here is where i do the cut out part:

    Code (CSharp):
    1. float4 albedo = tex2D(_MainTex, i.uv);
    2.                 clip(tex2D(_MainTex, i.uv).a * albedo.a - _Cutoff);
     
  2. Invertex

    Invertex

    Joined:
    Nov 7, 2013
    Posts:
    1,539
    launzone likes this.
  3. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    I just wanted to point out this line.
    addshadow
    and
    fullforwardshadows
    are optional parameters for the
    #pragma surface
    used by Surface Shaders. They have no affect on vertex fragment shaders. Surface Shaders are vertex fragment shader generators, and can generate multiple passes including the
    "LightMode"="ForwardBase"
    pass you have above, as well as several others. If you use the
    addshadow
    it generates a
    "LightMode"="ShaderCaster"
    pass.

    If you're writing the vertex fragment shader manually, you'll need to also write the shadow caster pass manually, like @Invertex mentioned above.
     
    ElliotB and launzone like this.
  4. launzone

    launzone

    Joined:
    Dec 2, 2015
    Posts:
    57
    okay thank you both i will give this a try and report back when i figure it out!
     
  5. launzone

    launzone

    Joined:
    Dec 2, 2015
    Posts:
    57
    I understand by following this documentation to implement a "standard" shadow casting pass but I don't see how to change it in any way? It looks exactly the same. The doc mentions : "... object has custom vertex shader based deformations, or has alpha cutout / semitransparent parts... " which is actually both true of my shader but I just don't see what to do about it. Maybe I just don't understand the basics of making a shadow casting pass, are there some basic tutorials on this? All I see are just black-boxes like "SHADOW_CASTER_FRAGMENT(i)" and I have no idea how I can affect the shadow that is being cast.
    Also, can you tell me what you mean when you say the ShadowCaster pass outputs depth? I don't see the fragment shader outputting color or anything like I would be used to in the example in the documentation.
    Sorry for all the questions! I thought I grasped shaders a little bit better by now but I keep getting confused by all these unity functions and included stuff that is predefined.
     
  6. launzone

    launzone

    Joined:
    Dec 2, 2015
    Posts:
    57
    i guess i can see how it will be easy to do the same transformations to the vertex in the vert of the shadow caster pass but how do i use the texture of cutout?
     
  7. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    See this built in shader:
    https://github.com/TwoTailsGames/Un...esourcesExtra/AlphaTest-VertexLit.shader#L129

    The short version is any code you're doing in your forward base pass that would affect the shape / shadow of the object need to also be done in the shadow caster pass too.

    On a side note, your example shader code above is:
    Code (csharp):
    1. float4 albedo = tex2D(_MainTex, i.uv);
    2. clip(tex2D(_MainTex, i.uv).a * albedo.a - _Cutoff);
    Why are you multiplying the alpha value of _MainTex by itself?
     
    launzone likes this.
  8. launzone

    launzone

    Joined:
    Dec 2, 2015
    Posts:
    57
    okay, i think i figured it out, this post was very helpful.

    in my shadowcaster pass i discarded based on texture alpha:

    Code (CSharp):
    1. float4 frag(v2f i) : SV_Target
    2.             {
    3.                 float a = tex2D(_MainTex, i.uv).a;
    4.                 if (a < _Cutoff) {
    5.                     discard;
    6.                 }
    7.                 SHADOW_CASTER_FRAGMENT(i)
    8.             }
    thank to bgolus and invertex for pointing me in the right direction!
    upload_2021-4-15_16-1-5.png
     
    Invertex likes this.
  9. launzone

    launzone

    Joined:
    Dec 2, 2015
    Posts:
    57
    yes upon closer inspection that didn't make much sense