Search Unity

Beginner in Graphics - On a Quest to do the foliage from the Witness

Discussion in 'General Graphics' started by hardcorebadger, Feb 22, 2018.

  1. hardcorebadger


    Nov 28, 2016
    Hey everyone!

    I'm going to preface this with the fact that I have scoured the world wide web in search of answers on this one, and I've actually found some. But the issue is I either can't seem to get them to work correctly, or I don't fully understand them. I'm new to graphics relatively, I have a background in CS, and I've been learning shaders and CG for a couple months now. I'm also not a bad modeler and I'm super familiar with unity on the editor and C# side.

    So anyway, I just cannot get this effect to work. First of all, a picture of the witness:

    So these look super cool, I've seen this look in a few other games, etc. They give this feel of almost an impressionist painting - it's awesome. So now into what I have heard about how to do this:

    1. You gotta correct your normals with something like normal theif in 3dsmax or normal edit in blender
    2. Write a shader that essentially drops the alpha in relation to the view direction (normal dot view dir) to fade out quads with their edges facing the camera
    3. Add to that shader AlphaToMask with 4x MSAA

    So here's what I've got:

    1. Corrected normals with standard surface shader (This looks kinda good but trust me, this is a perfect angle, the whole seeing the quad edges is a huge issue here, plus some other thing I'm not stoked on)

    So first off, you can still totally see the quad edges, each seems to basically have their shadows, making it rather obvious we have a bunch of quads on a stick. Right off the bat - should the normal transfer (from an elliptic uv sphere) be fixing that? did I do that part wrong somehow? This is clearly way better than the quads without doing the normal correction I did. It's like if we could just blend/smooth this all out somehow it would work pretty well... But still In the witness and other games like it, you seriously can barely tell they are using quads with leaf textures - how do they get that? Possibly the AlphaToMask thing? Let's continue...

    Okay lets's try that shader idea...

    Code (CSharp):
    1. Shader "Nature/Witness/leaves"
    2. {
    3.     Properties
    4.     {
    5.         [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
    6.         _Color ("Tint", Color) = (1,1,1,1)
    7.         _Cutoff ("Shadow alpha cutoff", Range(0,1)) = 0.5
    8.         _EdgeTransparency ("Edge Transparency Factor", Range(0,1)) = 3
    9.     }
    10.     SubShader
    11.     {
    12.         Tags
    13.         {
    14.             "Queue"="AlphaTest"
    15.             "IgnoreProjector"="True"
    16.             "RenderType"="TransparentCutout"
    17.             "PreviewType"="Plane"
    18.             "CanUseSpriteAtlas"="True"
    19.         }
    20.         LOD 200
    21.         Cull Off
    22.         Lighting On
    23.         AlphaToMask On
    24.         CGPROGRAM
    25.         #pragma surface surf Lambert addshadow alphatest:_Cutoff
    26.         sampler2D _MainTex;
    27.         fixed4 _Color;
    28.         float _EdgeTransparency;
    29.         struct Input
    30.         {
    31.             float2 uv_MainTex;
    32.             float3 viewDir;
    33.         };
    34.         void surf (Input IN, inout SurfaceOutput o)
    35.         {
    36.             fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
    38.             // normal shader
    39.             o.Albedo = c.rgb;
    40.             o.Alpha = c.a;
    41.             o.Alpha *= 1.0 -  (dot (normalize(IN.viewDir), o.Normal)*_EdgeTransparency);
    43.             // this is what I used in the test video with the white/black showing normal dot view
    44.             //o.Albedo = 1.0 -  (dot (normalize(IN.viewDir), o.Normal)*_EdgeTransparency);;
    45.             //o.Alpha = 1;
    46.         }
    47.         ENDCG
    49.     }
    50.     Fallback "Legacy Shaders/Transparent/Cutout/VertexLit"
    51.     Dependency "OptimizedShader" = "Hidden/Nature/Tree Creator Leaves Fast Optimized"
    52. }
    So here's my shader - pretty straightforward, using a surface shader, gunna drop the alpha based on the normal-dot-view like was proposed across the internet I think first appearing on the witness dev blog.

    But here's the issue. I corrected all the normals. So it's not going to fade out the weird edges of the quads anymore, it's just gunna fade out whatever is right in front of me. With uncorrected normals, this technique actually works BETTER. Here's a test I did to show my math is working, this is just passing that dot calculation line straight to albedo:

    So this is basically what I just said would happen - it's just fading out based on the corrected normals, which is not really the reason we were doing this whole fade out edges quads. Basically this is just ends up cutting out a piece of the tree in front of the cam it looks exactly like we have the camera's clipping distance set super high. AlphaToMask is in there, from what I can tell that was basically inconsequential.

    So from what I can tell, the standard shader is working the best - is there something I'm not getting about this technique? How do you get that billboard-esque effect without actually billboarding and having all the terrible artifacts that would come with it? It looks so simple, hence why I thought I might try it, but it's turning out to be quite a mystery...

    Any help would be greatly appreciated! Thanks in advance!
  2. Mauri


    Dec 9, 2010
    theANMATOR2b likes this.
  3. AcidArrow


    May 20, 2010
    As far as I remember normal bending is a big part of the look of the witness trees ( also look in the vertex normals section in this link : ), so you would have to manually edit your trees, or create your own, to do that.
    theANMATOR2b likes this.
  4. hardcorebadger


    Nov 28, 2016
    Thanks for the response!

    Yeah I've looked through those - Basically thats where I'm getting #1 and #2 from -

    #1 (Correcting Normals)

    #2 (Normal Fadeout Shader)

    The issue I'm having is that doesn't seem to make sense - if your normals are corrected to a sphere, they aren't going to be very helpful in telling when a plane is at an unflattering angle because they no longer represent the way that plane is facing, they represent the way the sphere is facing, which is always out towards the viewer (not what we want).

    Hence, I'm assuming I'm just not fully up to speed with what might be going on here - or do you think this is weird too?
  5. AcidArrow


    May 20, 2010
    Hmm fair point. Maybe they had two sets of normals? One for lighting and one for fades? You could potentially pack your normals into vertex colors Unity and then have two sets to work with as well?
  6. hardcorebadger


    Nov 28, 2016
  7. hardcorebadger


    Nov 28, 2016
    That's not a bad idea - I'll try that out - but honestly I'm pretty sure there's some type of blending thing I'm not doing right - I've heard tales of Alpha to Coverage being used... Not really sure how that works or what it does, but simply turning on AlphaToMask in my shader and turning on 4x MSAA didn't really do anything - basically there' some way they are softening the edges between the planes post vert (not sure if thats a term but basically I mean after the triangulation part of the pipeline) - My thought is that's more important than the normal based fade outs.
  8. AcidArrow


    May 20, 2010
    (for the record, I just tried the normal fading thing on a normal bent tree I have and it looks ridiculous, as we all guessed it would, just confirming)

    About the rest. some thoughts:

    Maybe self shadowing should be turned off? (maybe turn shadows off in general for testing). If we want to avoid any harshness, shadows from direct sunlight are probably not a good idea.

    About the edges: Maybe he is using alpha blending instead of alpha testing? There's even some technique where you alpha test for z-testing and then at the very edges you alpha blend. I think you can simply do it in two passes.

    (old link but I couldn't find anything better right now : )
  9. hardcorebadger


    Nov 28, 2016
    Another thing I heard was possibly turning off casting shadow and using an invisible object to cast the shadow onto the leaves - I'm gunna try a couple of these things and get back with the results
  10. hardcorebadger


    Nov 28, 2016
    With receive Shadows

    Without Receive Shadows

    So that's progress
  11. AcidArrow


    May 20, 2010
    I did a super hacky implementation of the view angle fading and it seems to help a bit:

    With fading
    Without Fading

    (maybe open them in new tabs and switch back and forth so it's more obvious what changed ;) )
  12. hardcorebadger


    Nov 28, 2016
    wait do you mean the double normals idea or the edge blending 2 pass thing
  13. AcidArrow


    May 20, 2010
    The double normals, without really using double normals though. I use baked lighting and no real time lights at all, so after baking (with my somewhat bent normals), I just recalculated the normals in the mesh importer and tweaked my shader :p (this is why I said super hacky). It's by no means a proper solution.

    I just wanted to see how it looks. It works okay. Although I'm not sure if it's worth it in my case. The leaves are all alpha blended, so there's no proper z-order, and when the camera is moving, it looks like a mess.

    So fading the leaves that are at an angle seems like polishing something that has otherwise very fundamental problems. Plus it's for a mobile game and I don't want to spend performance if I don't have to.
  14. bgolus


    Dec 7, 2012
    Yep, The Witness is using two normals. It's possible to encode both of these into the mesh in an external program by storing the normals in the vertex colors like @AcidArrow suggested, but you can also calculate these in the editor and store them in an extra UV channel, or calculate them in the fragment shader with out anything else!

    Code (csharp):
    1. float3 positionToNormal(float3 position)
    2. {
    3.     float3 dpx = ddx( position );
    4.     float3 dpy = ddy( position ) * _ProjectionParams.x;
    5.     return normalize(cross(dpx, dpy));
    6. }
    The normal will be whatever space the position is in. The normal might be flipped vs the actual surface normal, but that's not an issue for this use case since you just need the absolute value of the dot product. However there's one issue with using that code in a Surface Shader. Unity's Surface Shader is overzealous in it's optimizations and won't pass the worldPos to the surf function if only used by that function! Basically Surface Shaders will fight you every step of the way here and you'll have to pass the data you need yourself with a custom vertex function. I'd recommend encoding the normal in the vertex color instead if you're going to stick with surface shaders.

    Alternatively you could encode one of the normals in a UV channel using an external program using some form of 2 channel normal encoding, like spheremap or octahedron encoding. Basically Unity will only use the first two components of the UV when importing a mesh so you can't just store the full vector even though most modelling programs do actually support 3 component UVs.

    I wrote a lot about Alpha to Coverage here:

    I mention Surface Shaders briefly near the end. I've got a second part that's been in the works for the last few months I may never finish, but it goes a bit more deeply into the issues of using alpha to coverage in a Surface Shader. Again, they'll be actively fighting you as Unity never expected AlphaToMask to be used with a Surface Shader. The short version is you can't use the addshadow or alphatest keywords as they both prevent AlphaToMask from actually doing what you expect. Specifically the alphatest keyword does the clipping, then forces the output alpha value (which Alpha to Coverage needs) to 1.0 instead of the alpha you set, and the default shadow pass always outputs an alpha of 0.0 and the shadowcaster pass addshadow generates doesn't know to turn AlphaToMask off.
    AcidArrow likes this.
  15. bgolus


    Dec 7, 2012
    For Wayward Sky I disabled shadows casting of bushes, but kept shadow receiving, which fit with the style of that game. It also meant the bushes would still get shadows cast on them. Because Wayward Sky is a VR game the lack of shadow casting for grounding them is less important, but having them be fully lit all the time was important.

    An alternate solution I investigated was to modify the shadowcaster pass (which is used for both shadow casting and receiving by Unity) to push the surfaces towards the camera when calculating shadow receiving. This can lead to some minor swimming of the shadows as you move around the object, and may cause you problems if you use the camera depth texture for other purposes. That swimming was also way too noticeable for VR to be usable.

    Another hack would be to have your bushes use entirely separate geometry for shadow casting. Basically you'd need to have a blob mesh that surrounds the bush and has its faces inverted with some kind of bush texture on it that roughly matches your bush's shape. Won't be exact, but will give you ground shadows with out the issues of self shadowing.

    The Witness sidesteps the issue entirely by changing how they handle shadow maps on foliage. There are two main ways to do shadowmaps, one is to sample the shadow map using a hardware compare sampler which gives you bilinearly softened shadow edges. When you sample the shadowmap in the shader you pass in the position in the texture for both the UV and the depth and the hardware does the comparison for you. This is what Unity does, as well as doing multiple samples to soften it further if soft shadows are enabled. The alternative is to just get back the raw depth value from the shadow map and do the distance comparison on your own. This is a little slower and you don't get the bilinear softening, but you can do stuff like fade in the shadow rather than be a binary on/off in the depth compare. The Witness appears to use this second method on foliage which gives an additional volumetric appearance to the shadows too, but they use the other method for everything else.

    It is not trivial to modify Unity to do this. And the most straight forward ways are an all or nothing change.
  16. hardcorebadger


    Nov 28, 2016
    Thanks for the response - I've read a bunch of your posts across the web in relation to this topic (that's why I asked about Alpha to coverage and the invisible blob shadow) so thanks for taking the time to respond yet again about this! haha! I'm just starting to get into this stuff so It'll take me a while to unpack all this new info - probably end up with some more questions lol.
  17. hardcorebadger


    Nov 28, 2016
    Well, I went and learned how the z buffer works, painters algorithm, and all this unsorted transparency stuff - it is pretty annoying. After that I just decided to start from scratch. Wrote basic vert-frag (from bgolus' advice so there's no more dealing with all this under-the-hood unity stuff). Started with an unlit shader, added diffuse shading and a hard coded an alpha test (so I could use blending if I needed to)...

    That... Um, well surprisingly, that looks pretty good? I'm kind of astounded I didn't try this yet - literally have like 20 shaders in this project and none of them was just a simple hand-made diffuse shader. I think it's ready AcidArrow's normal hack now... going to use bgolus' math to calculate the old normals and see what happens.
    Nexer8, a436t4ataf and AcidArrow like this.
  18. AcidArrow


    May 20, 2010
    That looks pretty great!

    Just in case you need it, (although you probably won't, since you seem to know your stuff) my hack was doing this in vert:

    Code (csharp):
    1.                 half3 viewDir = normalize(ObjSpaceViewDir(v.vertex));
    2.                 o.rim = abs(dot(normalize(v.normal), viewDir));
    3.                 o.rim = smoothstep(0.2,0.25,o.rim);
    And then c.a *= i.rim; in frag.

    I used actual normals, so switch v.normal with your decoded ones.

    It's pretty basic, maybe there are better approaches to it. Also those smoothstep values are pretty important and probably need a lot of tweaking and testing to see what works best.
  19. hardcorebadger


    Nov 28, 2016
    So I got this working - the issue is while it looks fine in a still shot, when you move the cam it's pretty obvious these edges are disappearing - basically because I'm using a cutoff, it's all or nothing - so we finally come to a need for blending. The only reasonable method I saw of order independent transparency was stochastic blending with MSAA... So I'm gunna try to get that working.

    Note, as far as performance goes, it may be worth it to just stick this the last thing I posted - doing this normal recalculation has to happen on a frag level for some weird reason (not sure, I just got an error if I called it in the vert) and then now we're turning on 4x MSAA just for this blending thing - my guess is for most purposes the above picture was pretty good.

    Regardless, I'm gunna see what happens just to see

    To clarify - we need the bending working for the normal fading to actually fade, not just cut.
  20. bgolus


    Dec 7, 2012
    Derivatives (what those ddx and ddy functions are) only work in the fragment shader. See the alpha to coverage article I linked to above. I go into what they are and why they only work in the fragment shader.

    An alternative that a lot of AAA games use (including the witness when the quality setting is lowered!) is a dithered fade. You can either use some kind of bayer dither or a noise texture / function. Unity actually has two built in bayer dither textures it uses for things like LOD fading or dithered shadows.
  21. AcidArrow


    May 20, 2010
    Really? Can we access that from a shader?
  22. bgolus


    Dec 7, 2012
    Yep, there’s the 3D texture used for the shadows called _DitherMaskLOD, and a 2D one used for the LOD crossfade which I don’t remember the name of.
  23. hardcorebadger


    Nov 28, 2016

    So I can use dither sampling for a stochastic blend? I tried to sample the 3D one but I wasn't sure which input coordinates to use and even then what exactly do you do? Just clip based on the dither? These are looking pretty good though - here's the update.
    a436t4ataf likes this.
  24. hardcorebadger


    Nov 28, 2016
    So I got MSAA / Alpha to Coverage working! Transparency that actually works!


    The one thing I think would be better is if we could do it in terms of cam world position rather than literal view direction, the next video shows why I think it may be better.


    Basically, your peripherals are at maybe up to 45 degrees different angle than the actual view direction, so you end up actually seeing the edge blending happening. If it was based solely on camera position, this would be fixed.


    I tried that, it looked super weird.

    Anyway, I saw bgolus on your medium post in the sharpening section that you were talking about how to fix the fact that layered semitransparent pixels in alpha to coverage don't add up their opacities like you would get in a blend - is there some way to just set all the leaves to say a 0.5 alpha and get the A2C to add the alphas so that the heavily layered areas end up fully opaque and random 1-layer leaves would have a very low final alpha?
    Last edited: Feb 25, 2018
  25. bgolus


    Dec 7, 2012
    What you'd want to do is base it on the direction from the camera to the center of each quad / polygon shape. That would give you the best of both worlds, but that would require storing the center position in the vertex data, or using complicated geometry shaders (and using some features of geometry shaders that don't work in Unity), or some really complicated fragment shader magic.

    It's a ton of work, and really no one is going to notice either way. :)


    You can randomize the coverage samples used. I posted an example that allows you to use different samples per material:

    For what you'd want you could change between coverage samples via noise and / or primitive id. Inigo Quilez (best known for posted something of an example for that here:

    That's written for OpenGL, but it works the same for Direct3D / Unity ShaderLab with some minor tweaks to naming. The "primitive id" comes from a uint primitiveID : SV_PrimitiveID fragment input, the "pixel" comes from int2 vpos : VPOS fragment input. The frame id and blue noise functions are custom things and would either need to be replaced with something similar or can be skipped entirely.
  26. YossiMH_BleatingSheep


    Nov 6, 2015
    I'd be very grateful to hear any advice you have to offer in this area. :)
  27. bgolus


    Dec 7, 2012
    My advice is don't use a surface shader. :p

    More realistically, the solution is to not use addshadow in your surface shader, and instead write a custom shadowcaster pass manually that just uses your basic alpha test cutoff.
  28. MadeFromPolygons


    Oct 5, 2013
    Ben do you know is it possible to recreate the effects listed in this thread via shadergraph?
  29. bgolus


    Dec 7, 2012
    Nope, it is not. There's no option for enabling alpha to coverage with Shader Graph. You're stuck with alpha blend or alpha test, and that's it.

    The HD pipeline is setup assuming deferred rendering w/ temporal anti-aliasing, which can't use alpha to coverage since deferred excludes the use of MSAA. Since the direction Unity has gone with Shader Graph has been to limit it to only features that both the HD and Lightweight pipelines have, it makes sense they would not include alpha to coverage. I think that direction is totally wrong, but that's where we are.
  30. MadeFromPolygons


    Oct 5, 2013
    Okay thanks, I guess for now we will have to keep writing shaders by hand but hope they add this to shadergraph at some point!
  31. YossiMH_BleatingSheep


    Nov 6, 2015
    In case anyone else follows in my path, what I was missing was the
    parameter for the
    #pragma surface
  32. MadAboutPandas3


    Jul 3, 2017

    You (hardcorebadger, AcidArrow) combined two features:
    One is to fade out away the face when almost perpedicular to the camera view.
    The other is to use a normal vector which roughly point away from the center of model.
    So we need two normals per vertex.

    I want to combine these features in my tree. My idea is to simply interpret the local position of each vertex as its second normal! The origin of the fluffy tree mesh top will be somewhere inside the trunk, it think on the ground. I tried it in Unlit and Surface Shader, but it did not work.

    Can somebody help?

    Kind Regards,
  33. bgolus


    Dec 7, 2012
    The idea isn't a bad option, and it's how it is sometimes accomplished, but you should show the shader code from your attempts.
    AcidArrow likes this.
  34. MadAboutPandas3


    Jul 3, 2017
    Code (CSharp):
    2. Shader "Custom/Double Sided Alpha Cutout Fresnet Sphere Normals"
    3. {
    4.     //
    6.     Properties
    7.     {
    8.         _MainTex("Texture", 2D) = "white" {}
    9.         _Cutoff("Alpha cutoff", Range(0.0, 1.0)) = 0.4
    10.         _AlphaOffset("Alpha offset", Range(-1.0, 1.0)) = 0
    11.         _NormalsOffset("Normals offset", Vector) = (0,0,0)
    12.     }
    13.     SubShader
    14.     {
    15.         Tags
    16.         {
    17.             //"RenderType"="Transparent"
    18.             "RenderType"="TransparentCutout"
    19.             "Queue"="AlphaTest"
    20.             "IgnoreProjector"="True"
    21.         }
    22.         LOD 100
    23.         Cull Off
    25.         Pass
    26.         {
    27.             // try use Alpha To Coverage if multisample anti-aliasing is set to 4 (MSAA)
    28.             //AlphaToMask On
    30.             CGPROGRAM
    31.             #pragma vertex vert
    32.             #pragma fragment frag
    33.             // make fog work
    34.             //#pragma multi_compile_fog
    36.             #include "UnityCG.cginc"
    37.             #include "Lighting.cginc"    // for _LightColor0
    39.             struct appdata
    40.             {
    41.                 float4 vertex : POSITION;
    42.                 float2 uv : TEXCOORD0;
    43.                 half3 normal : NORMAL;
    44.             };
    46.             struct v2f
    47.             {
    48.                 float2 uv : TEXCOORD0;
    49.                 //UNITY_FOG_COORDS(1)
    50.                 float4 vertex : SV_POSITION;
    51.                 half3 worldNormal : NORMAL;
    52.                 fixed4 diff : COLOR0;
    53.                 float fresnel : TEXCOORD1;
    54.             };
    56.             sampler2D _MainTex;
    57.             float4 _MainTex_ST;
    58.             float _Cutoff;
    59.             float _AlphaOffset;
    60.             float3 _NormalsOffset;
    62.             v2f vert (appdata v)
    63.             {
    64.                 v2f o;
    66.                 // transform from local to world space
    67.                 o.vertex = UnityObjectToClipPos(v.vertex);
    68.                 o.uv = TRANSFORM_TEX(v.uv, _MainTex);
    69.                 o.worldNormal = UnityObjectToWorldNormal(v.normal);
    71.                 // precalc camera-perpedicular alpha
    72. // this works in test meshes but not good with real tree model
    73.                 float3 view = normalize(ObjSpaceViewDir(v.vertex));
    74.                 float vn = dot(view, v.normal);
    75.                 // double sided, flip normal if necessary
    76.                 //o.worldNormal *= sign(o.worldNormal);
    77.                 vn *= sign(vn);              
    78.                 //o.fresnel = _FresnelBias + _FresnelScale * pow(1 + vn, _FresnelPower);
    79.                 o.fresnel = saturate(_AlphaOffset + vn);
    81.                 // overwrite normal with the normals on a sphere
    82. // this should work too but not good, sometimes light source direction flips
    83.                 // attention: verctor 0,0,0 can result to NAN
    84.                 o.worldNormal = UnityObjectToWorldNormal(normalize(_NormalsOffset +;
    86.                 // dot product between normal and light direction for standard diffuse (Lambert) lighting
    87.                 half nl = max(0, dot(o.worldNormal,;
    88.                 // factor in the light color
    89.                 o.diff = nl * _LightColor0;
    90.                 // in addition to the diffuse lighting from the main light, add illumination from ambient or light probes
    91.                 // ShadeSH9 function from UnityCG.cginc evaluates it, using world space normal
    92.                 o.diff.rgb += ShadeSH9(half4(o.worldNormal, 1));
    93.                 o.diff.a = 1;
    95.                 //UNITY_TRANSFER_FOG(o, o.vertex);
    96.                 return o;
    97.             }
    99.             fixed4 frag (v2f i /*, fixed facing : VFACE*/) : SV_Target
    100.             {
    101.                 // sample the texture
    102.                 fixed4 col = tex2D(_MainTex, i.uv);
    104.                  // multiply by diffuse lighting
    105.                 col *= saturate(i.diff);
    107.                 // calc alpha
    108.                 col.a *= i.fresnel;
    110.                 // test show normals
    111. // if enabled the normals look correct
    112.                 //col.rgb = i.worldNormal * 0.5 + 0.5;
    114.                 // apply fog
    115.                 //UNITY_APPLY_FOG(i.fogCoord, col);
    117.                 clip(col.a - _Cutoff);
    119.                 return col;
    120.             }
    121.             ENDCG
    122.         }
    123.     }
    124. }
  35. MadAboutPandas3


    Jul 3, 2017
    My shader is kind of working in general but not from all view directions.

    I would also be happy to use a third-party shader. I already tried and bought a some tree shader assets from the Unity Asset Store, but there were looking bad.

    Do you have any links?
  36. bgolus


    Dec 7, 2012
    You may want to move many of the dot products into the fragment shader rather than the vertex shader as they'll yield slightly nicer looking results, though that does mean needing to pass the two normals and the (un-normalized!) view direction from the vertex to the fragment.

    The main issue you're having is you're trying to use lighting on a shader not using a LightMode tag. Without setting a light mode your directional light information will be randomly flipping, and you may be missing ambient lighting data (ShadeSH9 may return black).

    Add this at line 27
    Tags { "LightMode" = "ForwardBase" }
  37. MadAboutPandas3


    Jul 3, 2017
    Hey bgolus,
    That line was the key! Thanks!
    Most of the calculation is intentionally in the vertex shader. It think it matches the wanted look better and is a bit faster. I will make a more pixel shader heavy version and compare them.
    Kind Regards,
  38. MadAboutPandas3


    Jul 3, 2017
    This is the current version of my shader. Instead of a shadow pass rendering the tree mesh again I place one quad with a tree shaped texture on it in the middle of the tree. The shadow looks very similar to the "real" shadow. The variables DARKEN can be ignored, i tried to make the tree darker in the inside of the tree. The shader did not make it into our current game, but maybe the code it can help someone.

    I will add pictures soon.

    Code (CSharp):
    1. Shader "Custom/Double Sided Alpha Cutout Fresnel Sphere Normals Shadows"
    2. {
    3.     //
    4.     //
    5.     //
    6.     //
    7.     //
    8.     //
    10.     // moved most possible calculation from fragment to vertex shader for perfomance
    12.     // DOUBLESIDES rendering and lightning
    13.     // LIGHTNING diffuse Lambert shading including lightning/reflection probes and lightmaps
    14.     // ALPHA uses cutout for transparency, full screen anti-aliasing effect recommended
    15.     // FRESNEL hides faces almost perdendicular to camera using simple fresnel rim
    16.     // FLUFFYNORMALS calculates normal vectors pointing from mesh origin away for fluffy effect
    17.     // FOG disabled for performance
    18.     // SHADOWS casting disabled for performance
    19.     // INNERDARKEN allows darker pixels closer to origin of the mesh
    20.     // DITHER order independent transparency (OIT) alpha two pass or dithering
    22.     // TODO
    23.     // SOFT_PARTICLES soft particles, but needs a costly extra grab pass
    24.     // LOD_FADE_CROSSFADE support LOD cross fading
    26.     Properties
    27.     {
    28.         _MainTex("Texture", 2D) = "white" {}
    29.         // ALPHA
    30.         _Cutoff("Alpha cutoff", Range(0.0, 1.0)) = 0.4
    31.         // FRESNEL
    32.         _AlphaOffset("Alpha offset", Range(-1.0, 1.0)) = 0
    33.         // DITHER
    34.         _AlphaDither("Alpha dither", Range(0.0, 1.0)) = 0.2
    35.         // FLUFFYNORMALS
    36.         _NormalsOffset("Normals offset", Vector) = (0,0,0)
    37.         // INNERDARKEN
    38.         _DarkenRadius("Inner Darken Radius", Range(0.0, 10.0)) = 1.0
    39.         _DarkenHeight("Inner Darken Height", Range(-10.0, 10.0)) = 0.0
    40.         _DarkenPower("Inner Darken Power", Range(0.0, 1.0)) = 0.0
    41.     }
    42.     SubShader
    43.     {
    44.         Tags
    45.         {
    46.             //"RenderType"="Transparent"
    47.             "RenderType"="TransparentCutout"
    48.             "Queue"="AlphaTest"
    49.             "IgnoreProjector"="True"
    50.         }
    51.         LOD 100
    52.         Cull Off
    54.         Pass
    55.         {
    56.             // try use Alpha To Coverage if multisample anti-aliasing is set to 4 (MSAA)
    57.             //AlphaToMask On
    59.             Tags
    60.             {
    61.                 "LightMode" = "ForwardBase"     // for ShadeSH9
    62.                 //"LightMode" = "ShadowCaster"
    63.             }
    65.             CGPROGRAM
    67.             // Upgrade NOTE: excluded shader from DX11; has structs without semantics (struct v2f members vertex)
    68.             //#pragma exclude_renderers d3d11
    70.             #pragma vertex vert
    71.             #pragma fragment frag
    72.             #pragma target 3.0  // for VPOS
    74.             // FOG make fog work
    75.             //#pragma multi_compile_fog
    77.             // LOD
    78.             #pragma multi_compile _ LOD_FADE_CROSSFADE
    79.             //#pragma multi_compile _ LOD_FADE_CROSSFADE DITHER_ALPA
    80.             //#pragma shader_feature _ _RENDERING_CUTOUT _RENDERING_FADE _RENDERING_TRANSPARENT
    83.             #include "UnityCG.cginc"     // for UnityObjectToWorldNormal
    84.             #include "Lighting.cginc"    // for _LightColor0
    86.             struct appdata
    87.             {
    88.                 float4 vertex : POSITION;
    89.                 float2 uv : TEXCOORD0;
    90.                 half3 normal : NORMAL;
    91.             };
    93.             // note: no SV_POSITION in this struct
    94.             struct v2f
    95.             {
    96.                 float2 uv : TEXCOORD0;
    97.                 fixed4 diff : COLOR0;
    99.                 // FOG
    100.                 //UNITY_FOG_COORDS(1)
    102.                 // FRESNEL LOD
    104. //                UNITY_VPOS_TYPE vpos : VPOS;
    105. //#else
    106. //                float4 vertex : POSITION;    // deprecated? SV_POSITION to ensure compatibility with xbox
    107. //#endif
    108.                 half3 worldNormal : NORMAL;
    109.                 float fresnel : TEXCOORD1;
    110.             };
    112.             // LIGHTING
    113.             sampler2D _MainTex;
    114.             float4 _MainTex_ST;
    116.             // FRESNEL
    117.             float _Cutoff;
    118.             float _AlphaOffset;
    119.             float _AlphaDither;
    120.             float3 _NormalsOffset;
    122.             // INNER DARKEN
    123.             float _DarkenRadius;
    124.             float _DarkenHeight;
    125.             float _DarkenPower;
    127.             // FRESNEL LOD automatically assigned by unity
    128.             sampler3D _DitherMaskLOD;
    130.             v2f vert (
    131.                 appdata v,
    132.                 out float4 outpos : SV_POSITION // clip space position output
    133.                 )
    134.             {
    135.                 v2f o;
    137.                 // transform from local to world space
    138.                 //o.vertex = UnityObjectToClipPos(v.vertex);
    139.                 float4 worldpos = mul(unity_ObjectToWorld, v.vertex);
    140.                 o.uv = TRANSFORM_TEX(v.uv, _MainTex);
    141.                 //o.worldNormal = UnityObjectToWorldNormal(v.normal);
    142.                 o.worldNormal = normalize(mul(v.normal, (float3x3)unity_WorldToObject));
    144.                 // FRESNEL precalc camera-perpedicular alpha
    145.                 //float3 view = normalize(ObjSpaceViewDir(v.vertex));
    146.                 //float vn = dot(view, v.normal);
    147.                 float vn = dot(o.worldNormal, normalize( -;
    149.                 // DOUBLESIDED flip normal if necessary
    150.                 //o.worldNormal *= sign(o.worldNormal);
    151.                 vn *= sign(vn);
    152.                 //o.fresnel = _FresnelBias + _FresnelScale * pow(1 + vn, _FresnelPower);
    153.                 o.fresnel = saturate(_AlphaOffset + vn);
    155.                 // FLUFFY NORMALS overwrite normal with the normals on a sphere
    156.                 // attention: vector 0,0,0 can result to NAN
    157.                 o.worldNormal = UnityObjectToWorldNormal(normalize(_NormalsOffset +;
    158.                 //o.worldnormal = normalize(mul(_NormalsOffset +, (float3x3)unity_WorldToObject));
    160.                 // LIGHTING dot product between normal and light direction for standard diffuse (Lambert) lighting
    161.                 half nl = max(0, dot(o.worldNormal,;
    162.                 // factor in the light color
    163.                 o.diff = nl * _LightColor0;
    164.                 // in addition to the diffuse lighting from the main light, add illumination from ambient or light probes
    165.                 // ShadeSH9 function from UnityCG.cginc evaluates it, using world space normal
    166.                 o.diff.rgb += ShadeSH9(half4(o.worldNormal, 1));
    167.                 o.diff.a = 1;
    169.                 // INNER DARKER make darker in the middle, attention does not calculate scale
    170.                 float dist = length(v.vertex);
    171.                 //float grad = saturate(dist * (1.0 / _DarkenRadius)) * (_DarkenPower - 1.0);
    172.                 float grad = saturate(((dist -_DarkenRadius) * _DarkenPower) + 1.0);
    173.                 o.diff.rgb *= grad;
    175.                 // HEIGHT DARKER make darker in the bottom, attention does not calculate scale
    176.                 grad = saturate(((v.vertex.y - _DarkenHeight) * _DarkenPower) + 1.0);
    177.                 o.diff.rgb *= grad;
    179.                 // this would be more performant, but darkens the result a bit. why?
    180.                 //o.diff = saturate(o.diff);
    182.                 // DITHER pass coordinates to calculate screen pixel position
    183.                 //outpos = UnityObjectToClipPos(v.vertex);
    184.                 outpos = mul(UNITY_MATRIX_VP, worldpos);    // performance optimization
    186.                 // FOG
    187.                 //UNITY_TRANSFER_FOG(o, outpos);
    189.                 return o;
    190.             }
    192.             fixed4 frag (
    193.                 v2f i,
    194.                 /*fixed facing : VFACE,*/
    195.                 UNITY_VPOS_TYPE screenPos : VPOS
    196.                 ) : SV_Target
    197.             {
    198.                 // LIGHTING sample the texture, diffuse lighting
    199.                 fixed4 col = tex2D(_MainTex, i.uv);
    200.                 //col *= i.diff;
    201.                 col *= saturate(i.diff);
    203.                 // FRESNEL calc alpha
    204.                 col.a *= i.fresnel;
    205.                 // test show normals
    206.                 //col.rgb = i.worldNormal * 0.5 + 0.5;
    208.                 // APLHA apply alpha offset
    209.                 col.a -= _Cutoff;
    211.                 // ALPHA LOD
    212. //#if defined(LOD_FADE_CROSSFADE)
    213. //                UnityApplyDitherCrossFade(i.vpos);
    214. //#else
    215.                  // note: clip HLSL instruction stops rendering a pixel if value is negative
    216.                 // FRESNEL DITHER if pixel is clipped by fresnel effect replace it by dither
    217.                 if (i.fresnel < _AlphaDither)
    218.                 {
    219.                     // test with big checkerboard pattern
    220.                     //screenPos.xy = floor(screenPos.xy * 0.25) * 0.5;
    221.                     // test with pixel checkerboard pattern
    222.                     //screenPos.xy = floor(screenPos.xy) * 0.5;
    224.                     //float checker = -frac(screenPos.r + screenPos.g);
    225.                     //clip(checker);
    227.                     // query a texture with 16 dither pattern from full transparent (0) to full opaque (0.9375) in steps of 0.625
    228.                     // the border to the non-dithered area is mapped to reach full opaque
    229.                     float dither = tex3D(_DitherMaskLOD, float3(screenPos.xy * 0.25, i.fresnel / _AlphaDither * 0.9375)).a;
    230.                     clip(dither - 0.01);
    231.                 }
    233.                 // ALPHA cutout discard pixel when low alpha
    234.                 //if defined(_RENDERING_CUTOUT)
    235.                 clip(col.a);
    236.                 //#endif
    238. //#endif
    239.                 // FOG apply
    240.                 //UNITY_APPLY_FOG(i.fogCoord, col);
    242.                 return col;
    243.             }
    244.             ENDCG
    245.         }
    247.         // pull in shadow caster from VertexLit built-in shader
    248.         // SHADOWS enable casting here, might look wrong, depends on tree
    249.         //UsePass "Legacy Shaders/VertexLit/SHADOWCASTER"
    251.         /*
    252.         // using macros from UnityCG.cginc
    253.         Pass
    254.         {
    255.             Tags {"LightMode" = "ShadowCaster"}
    257.             CGPROGRAM
    258.             #pragma vertex vert
    259.             #pragma fragment frag
    260.             #pragma multi_compile_shadowcaster
    261.             #include "UnityCG.cginc"
    263.             struct v2f {
    264.                 V2F_SHADOW_CASTER;
    265.             };
    267.             v2f vert(appdata_base v)
    268.             {
    269.                 v2f o;
    271.                 return o;
    272.             }
    274.             float4 frag(v2f i) : SV_Target
    275.             {
    276.                 SHADOW_CASTER_FRAGMENT(i)
    277.             }
    278.             ENDCG
    279.         }
    280.         */
    281.     }
    283.     //Fallback "Standard"
    284. }
    Dawesign likes this.
  39. Dawesign


    Jan 26, 2019
    Hey guys, I have just read through this and the work you guys did is incredible. I'm completely new to shaders and am wondering if it's possible to achieve this effect with the Amplify Shader Editor?
  40. bgolus


    Dec 7, 2012
    Dawesign likes this.
  41. creat327


    Mar 19, 2009
    Anyone got this to work with built-in trees?
    Unity built-in trees look pretty bad unless you add the mip maps preserver coverage on the generated texture. This is usually ok, but the default shader is ULTRA SLOW in the Oculus Quest 2.
    Im trying to write a custom shader from the ideas on this thread and replace Hidden/TerrainEngine/BillboardTree with it. So far no luck, whatever I try it breaks. Any help would be awesome otherwise I would need to remove trees from Oculus Quest because performance sucks.