Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[SOLVED] Fresnel Edge Shader - Zelda BotW

Discussion in 'Shaders' started by flogelz, Oct 13, 2018.

  1. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    So i was looking into getting this nice edge effect, that you see in the Breath of the Wild. After reading about it (about sobel and other methods), it seemed that fresnel is the cheapest but also a non-consistent method of doing it, because its based on the angle to the camera.

    Link_Outline.jpg

    After thinking about it tho, wouldn't it be possible to make it more consistent? I'm no code master but I just wanted to share my thoughts here.

    In most fresnel implementations, you can have a value influence the spread of the fresnel (at which angle to start showing fresnel, using a hard falloff or a smooth falloff). So what would happen if we, instead of having a static number in there, would feed in the angle towards the surface normalized (im not sure about the exact math opereations), a value between 0 and 1. Wouldn't this basically change the fresnel spread dynamically, so that you can have a outline, which is kind of consistent??

    Frensel_Outline_Concept.png


    Would be awesome, if some of you could help me out here!! It probably isnt even that comlicated to do, but like i said before, my shader knowledge is pretty limited...

    Btw I also think, Nintendo did something similar with fresnel (a look at this bear model shows, that the outline effect indeed is fresnel. They just stabilized it somehow in most situations)

    Bear.jpg
     
    Last edited: Oct 13, 2018
  2. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    You could normalize it. This is a range between 0 and 1 and then scale it back up using another float, to create what is basically a gradient outline of constant thickness.

    Don't forget to modify that scale float by resolution or whatever personal preference.

    Edit: read that you considered it. Why not try it?
     
  3. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    Ah I tried it several times, but i guess i'm just using the wrong varibales(?)

    Heres the code for the sharp fresnel (the 0.7 is the value which should be dynamic and changes the spread):
    Code (CSharp):
    1. half rim = step(0.7, 1 - saturate(dot (normalize(IN.viewDir), o.Normal)));
    So could I use the viewDir normalized in there? Or how would i calculate this number?
     
  4. kepesh

    kepesh

    Joined:
    Dec 29, 2017
    Posts:
    92
    Did you find any solution?
     
  5. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    6,492
    ndotv elevate to a power then passed to a ramp texture (or a step function) used as a mask to a ndotl passed through a ramp/step function
     
  6. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    Sadly i didn't found a Solution yet, but I'm still searching occasionally. And thanks @neoshaman for the answer! But what you describe is just the effect, that the fresnel only is seen in lit parts of the model, right? I'm looking for this rather weirdly stable fresnel though. Im just checking ingame again right now and it is infact fresnel, thats for sure (small inconsistencies with the width and so on) But its way more stable overall, then what normal fresnel could supply.
    Also whats not happening so often are these big fresnel areas, when looking at it from a really low angle.
     
  7. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    One thing for sure tho, i could be linked to the lighting model, like you said @neoshaman. Because if the camera gets too close to Link, they turn him transparent to fade him out, which seems to change the shader. Interestingly, the lighting overall changes a bit and the fresnel outlines dissappear, until you get again far away from him and the solid shader is used again.
     
  8. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    Ok, so I'm just analyzing the outline, writing notes and stuff, but I'm pretty sure now, that it uses the depth buffer in some way. I talked about the transparent shader earlier and am now seeing, that in some cases only the outlines disappear, once Link turns transparent. If it would be only Fresnel, without anything else to it, why would they remove this effect then? So it has to be about the depth buffer.
     
  9. kepesh

    kepesh

    Joined:
    Dec 29, 2017
    Posts:
    92
  10. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    As far as i can tell, it's still Fresnel in the end, not the vertex displacement like minionsArt is doing it. The outlines are still on the model and not separate. And that's also what confuses me i guess- Other solutions like edge detection from a sobel filter would produce much more consistent Outlines, but also couldnt be masked by the lit areas. So it has to be Fresnel somehow. Thanks for the suggestion though ouo b
     
  11. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    I also noticed that the frensel gets scaled in combination to the distance towards the camera.
     
  12. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    Yeah, I agree with this. It's clearly a fresnel effect.

    My guess is the effect is carefully tuned on Link (and other 'hero' characters) to look as clean as it does, but it's not perfect and the outline gets extra thin or extra fat at some angles like you would expect from a fresnel effect, even on Link. A big part of getting these kinds of outlines to look good is having a nice, smooth, high polygon model to work from. This is part of why it looks so good on Link, and other major characters as they have the highest poly counts. There may be an additional bit of hinting baked into either their meshes or a texture that stores the relative curvature of that part of the mesh, much like what pre-integrated skin shaders use.

    One interesting bit about Zelda:BotW, is it appears to be using a deferred renderer, and the cell shading on characters and painterly / sketchy effects on the world are likely applied as a post process effect of sorts. This means that something like a depth based Sobel edge detection filter could have been applied to determine the outlines and get perfect single-pixel width outlines, but they did not choose to go that route for either performance or stylistic reasons. There's also no reason you couldn't calculate the sobel and mask it by shadows. A sobel filter doesn't need to be applied to the entire screen at all times.
     
    flogelz and neoshaman like this.
  13. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    6,492
    BOTW cell shading is def a postprocess effect as highlighted by bugs. It's nicely done, and probably use alot of hack to get the final result. Nintendo is known to be creative with visual effect like mipmap hack to get amazing ocean shading in sunshine and multiple quad duplicating the scene render and applying a max() on pixel intensity to get teh painterly effect on skyward sword.
     
    flogelz likes this.
  14. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    About the postprocess effect, I think i saw something similar in a glitch video, where you can stand under a bridge, and the celshading gets turned off! So it probably was done using it (Never wrote a postprocessshader tho)

    I also looked more into it and actually all models, that have the celshading applied, use this sort of fresnel lines. Probably for performance reasons. Actually most objects, that don't use the celshading seem to use a edge detection filter. Its weird and really subtle, but its has to be one or they use evil magicxD
    EdgeDetection Example.png
     
    Last edited: Apr 2, 2019
  15. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    So one year later now, i guess I found out what was going on and @neoshaman was right all along. I was lead by two reasons too believe that it was some fresnel magic. One being the inconsistencies of the edge effect and the fact that it seemed only to be located on Link.

    As my last post stated, i noticed and edge effect way more subtle on other objects too, I still can't say for sure, where they mask it away, but the effect is there all over the scene. Probably a lower intensity in shadows and maybe some light direction mask mixed in there. They use probably the mask for the toonshading also for this effect and let the outlines on the characters be much more noticable.

    Regarding the inconsistencies, I reread some old tutorials by chance and found this: https://roystan.net/articles/outline-shader.html As you can see in the middle of the page, such artifacts can indeed happen with postprocess effects too.

    Based on these facts, I would say it's pretty clear for me now, what they used here! Thanks for the help along the way~ :)
     
  16. BattleAngelAlita

    BattleAngelAlita

    Joined:
    Nov 20, 2016
    Posts:
    400
    This is mask/stencil effect.
     

    Attached Files:

  17. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    Yes, you could achieve something similar with stencils, but in BotWs case, I don't think thats what they done here. Stencils only can do Outlines by shifting the whole maks into one direction and in the BotW image above, you can see that this isn't the case. It has way more similarities with a outline postfx effect.
     
  18. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    6,492
    I should have provided the source of those conclusion:

    when people discovered the glitch that tells you it's post process


    also I could have save you time with this article
    https://roystan.net/articles/toon-shader.html

    I'm sorry to have let you down so long, yout twitter is fire by the way, my first stop everytime on the platform (though the sdfgi of godot 4.0 is coming close now)
     
    flogelz likes this.
  19. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    That article is just someone attempting to recreate the effect. It doesn't prove anything one way or another since it's just someone's guess. It also completely skips over the most impressive part of the BotW implementation, which is the surprisingly consistent line width on Link. Not perfect mind you, hence why I do agree it's a Fresnel based effect, combined with it being applied as part of either a post process or a deferred lighting system.

    The line width part is really the main topic of this thread. Whether or not it's a post process is almost irrellevant.
     
    flogelz and neoshaman like this.
  20. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    @neoshaman No worries! It's awesome alone that you two help anyways!

    @bgolus Regarding the fresnel or edge detection discussion, the edge detection script I'm using at the moment can produce under specific settings with the depth threshold similar inconsistencies like fresnel. I just tried a direct comparison and tho not exactly the same, I think it's close enough.



    I also looked further into the edge effect on other objects and it is indeed mostly everywhere. Also on big structures like mountains or the skeletons in the desert.

    Outline_Desert.png

    That's why I don't think anymore, that Nintendo used any kind of textures to help support the effect. They just mask by lighting and add it on top of the existing color, except for cel shaded objects, where they make the lines opaque to really highlight the style.
     
    neoshaman likes this.
  21. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    Also when making the effect a bit more extreme, it resembles the bear in the first post a lot.

    Outline - Object.png
     
    neoshaman likes this.
  22. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    6,492
    IF you like nintendo effect, i have another fun effect, I'll describe one and you will tell me in which nintendo game they use it:
    - sample 8 directions as an hexagone (more or less close with depth)
    - compare center with those sample,
    - keep the most bright one

    Fun alteration, use a small (preferably square) array of 0 and 1 with any shape (non negative image, ie the background is 0), sample only the 1

    It should probably be easy and quick to implement if you have already a sobel filter (also known as edge detection), and you could probably mask it with the light direction like you found out above (in fact I think that zelda game did it, so if you guess, you would know since when they experiment with that technique)
    post image if you try it ;)
     
    flogelz likes this.
  23. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    You're describing Skyward Sword's DoF effect. It's not all that different than how several modern DoF systems work to produce bokeh. Nintendo just did it with fewer samples (they only needed it to work at 480p) and using the max sample instead of an averaged HDR value like what you'd see in other titles. It's even how Unity's Universal and HDRP DoF work.
     
  24. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    6,492
    Yeah but it create an amazing paintairly brush effect, especially in emulation, I'm sure @flogel would have been interested in.

    I just wanted himto see for himmself. Also it's DOF like but the intent was to imitate a school of painting, if I'm not mistaken, Seurat's pointillism artstyle, I'm not sure about the depth scaling part, as if basically render the scene on 6 quad that blend down with a max as a post process, and punched out with depth for character.
     
    flogelz likes this.
  25. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    I wanted to come back to this comment, because BotW the "camera" view adds a stronger outline than what's present during normal gameplay. That outline is very clearly post processing base (likely purely depth based). It's still there outside of the camera view, but not as strong.

    I also think they're probably using two different "outlines", both fresnel and post processing based, for different situations. But I think you've convinced me the main one that's visible on the "toon" shaded characters is probably still purely depth post processing based, just with a relatively shallow depth comparison. The non-"toon" shaded surfaces are probably using the same depth outline, but with a wider, softer depth comparison.
     
    flogelz likes this.
  26. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    6,492
    I wonder if the BOTW shader modding community know the full answer, they have been reverse engineering the whole engine.
     
    flogelz likes this.
  27. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    I'm sure they do. I'm not sure where to find them though. Most of the attention is going to be on the Wii U version rather than the Switch version though, which means they'll probably be understanding them in terms of the Wii U's TEV states rather than traditional vertex fragment shaders.
     
    flogelz likes this.
  28. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    6,492
    WII U is traditional shader, wii U has the TEV for back compatibility, but it's cull in its own silicon only in wii mode. The shading itself is state of the art of the time, with a lot of stuff on compute (some physics, animation, etc grass, ...), only the cpu is basically the same but 3 times (3cores and boosted frequency) which was the main weakness. I have seen some decompile code on modding discord, you can tell even the wonky jumbled mess structure when you have a triplanar shader (I looked into xenoblade shader lol), and you can look at the size of texture to spot post processing effect (unsurprisingly the size of screen resolution multiply). But I'm not really invested, so I don't know much more than that, I didn't go there since the early days (they haven't made progress in the modding of navmesh to that day lol).
     
    flogelz likes this.
  29. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    I actually was looking into the SkywardSword DoF effect exactly when you wrote it back then:'DD It's a really sweet effect I want to try at some point! (Hehe so i knew immediately what you were hinting atxDD) I wanted to use it for an project, but we changed our minds, so the effect has to stay on my to-do list for a little longer.
     
    neoshaman likes this.
  30. kepesh

    kepesh

    Joined:
    Dec 29, 2017
    Posts:
    92
    How exactly do you do this as a post effect? I'm clueless when it comes to these kinds of things.
     
  31. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
  32. ianwexl0rz

    ianwexl0rz

    Joined:
    Jul 20, 2017
    Posts:
    4
    I know this thread is quite old and considered solved by the OP but I think there is more to this technique that is worth exploring.

    While I believe there is definitely a sobel-like edge detection filter involved, I do not think it is part of any post-process in the traditional sense. It looks like the edge-detection filter is used to modify the normal gbuffer before lighting is calculated.

    I say this because the "embossed edge" effect seems present on almost all opaque geometry and seems to adhere to the properties of PBR materials.

    I suspect the fresnel calculation in the toon materials doesn't do anything particularly special and the perturbed normals are responsible for generally creating a clean edge, but there is nothing stopping the rim-lighting from spilling over at glancing angles on lower resolution geometry as seen on the bear.

    I do not think the normal modification uses the raw result of the depth-detection filter as a mask because the lighting wraps around sharp corners smoothly while sobel would show obvious artifacts at that thickness.

    Which gets me thinking: Would it be possible to generate a screen space distance field from depth discontinuities using a jump flood algorithm and use that to perturb the normals of the closest edge in the gbuffer?

    Given a distance field, bending the normals away from the camera below a certain threshold should serve to exaggerate the rim lighting. Although you wouldn't be restricted to using a hard-cutoff, it could be a linear falloff or whatever kind of curve you want mapped to the desired edge thickness.

    I'm eager to try this out but it would take me a while to piece together with my current skills and I'm not sure when I'll have the time. I figured I might as well share my idea in case anyone else wants to give it a try or finds it useful!
     
  33. ianwexl0rz

    ianwexl0rz

    Joined:
    Jul 20, 2017
    Posts:
    4
    I still haven't tried generating a distance field but I wanted to see what I could accomplish by perturbing the normals along the camera forward axis before writing to the gbuffer.

    The result is not bad at all for dense organic meshes but it breaks down on low res and hard-edged geometry as expected. I really like how edges pick up environment light "for free" compared to a purely post-process effect.

    edge_normal_test.png

    I didn't change any parameters in my BRDF between renders.
    The edge light is just fresnel with a high exponent masked by half lambert:
    Code (CSharp):
    1.  
    2. float edgeLight = (nl * 0.5 + 0.5) * pow(1 - nv, 12);
    3.  
    The code for the modifying the normals lives in a modified version of UnityStandardCore.cginc, starting at line 256:

    Code (CSharp):
    1.  
    2. FragmentCommonData o = UNITY_SETUP_BRDF_INPUT (i_tex);
    3. float3 normalWorld = PerPixelWorldNormal(i_tex, tangentToWorld);
    4. float3 eyeVec = NormalizePerPixelNormal(i_eyeVec);
    5.  
    6. // Bend edge normals
    7. normalWorld = mul(RotationAlign(half3(0,0,-1), eyeVec), normalWorld);
    8. normalWorld.z *= smoothstep(0.3, 0.5, normalWorld.z);
    9.  normalWorld = mul(RotationAlign(eyeVec, half3(0,0,-1)), normalWorld);
    10.  
    11. o.normalWorld = normalize(normalWorld);
    12. o.eyeVec = eyeVec;
    13. o.posWorld = i_posWorld;
    14.  
    The RotationAlign function is by Inigo Quilez (source)

    Note that if you simply transform the normals to camera space instead of rotating to eyeVec you will get a similar effect in the center of the screen but the outlines will distort based on the FoV the further away they are from the center.
     
    bgolus likes this.
  34. fleity

    fleity

    Joined:
    Oct 13, 2015
    Posts:
    337
    Sorry for digging this up, but can someone elaborate a bit why the above function is better / what exactly is happening? I have been trying to wrap my head around it for a bit now but I seem to be stuck.

    Sure obviously ianwexl0rz is modifiying the normal to point into a different direction. To do that the normal is first transformed / rotated into view space? There we do a cut off using the smoothstep to change the gradient to stay closer to the meshs edges (I am looking at this with a sphere).
    And then we transform it back and 1-value pow12 etc.

    In shadergraph the result looks deceptively like a regular fresnel, but in the viewport quite differently which is hard to put into words, similar to a dot product with the normal there is a bright and a dark side to the sphere, the second vector is view dependend and always points away from view?

    I absolutely see how this can be used as a fresnel mask and that is behaves differently but I struggle to explain in which way it is different that I therefore have a hard time evaluating it.

    It looks similar to offsetting the normal vector before calculating the fresnel but still different with a more pronounced edge.
    Offset as in normalize(NormalWorld + Offset) or normalize(lerp(NormalWorld, Offset, 0.5).

    Why the custom RotationAlign? why not use the built in view matrix. If it returns a matrix anyway isn't that just extra steps? (because that looks similar but with the x-axis flipped)

    upload_2023-3-2_11-33-50.png
     

    Attached Files:

    Last edited: Mar 2, 2023