Search Unity

  1. Get all the Unite Berlin 2018 news on the blog.
    Dismiss Notice
  2. Unity 2018.2 has arrived! Read about it here.
    Dismiss Notice
  3. Improve your Unity skills with a certified instructor in a private, interactive classroom. Learn more.
    Dismiss Notice
  4. ARCore is out of developer preview! Read about it here.
    Dismiss Notice
  5. Magic Leap’s Lumin SDK Technical Preview for Unity lets you get started creating content for Magic Leap One™. Find more information on our blog!
    Dismiss Notice
  6. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Outline shader behavior with thick outlines

Discussion in 'Shaders' started by BottleEngineer, Jul 11, 2018.

  1. BottleEngineer

    BottleEngineer

    Joined:
    Jul 10, 2018
    Posts:
    11
    I'm working on a simple outline shader using vertex normal pushing. It works decently well with thin outlines, but still needs more work. However, I need thick outlines (>10 pixels thick) and I get some funny behavior at those ranges. For my application, I don't need lighting, so everything is unlit solid colors.

    The problem is that with larger outlines, some vertices can be pushed past others, creating what I assume are holes (?). However, I draw the actual geometry in the second pass with solid color with culling turned off, so I would think this wouldn't be an issue. In a perfect world, the outline should only appear around the perimeter of objects.

    gfycat example

    Is there a way I can improve this to avoid that effect? Is there a good explanation of what is happening?



    Here is the shader:

    Code (CSharp):
    1. Shader "Outline"
    2. {
    3.     Properties
    4.     {
    5.         [Enum(UnityEngine.Rendering.CompareFunction)] _ZTest("ZTest", Float) = 0
    6.         _OutlineColor("Outline Color", Color) = (1, 1, 1, 1)
    7.         _FillColor("Fill Color", Color) = (1, 1, 1, 1)
    8.         _OutlineWidth("Outline Width", Range(0, 15)) = 2
    9.     }
    10.  
    11.     SubShader
    12.     {
    13.         Tags
    14.         {
    15.             "RenderType" = "Opaque"
    16.         }
    17.  
    18.         Pass // First pass: vertex normal push and single color fill
    19.         {
    20.             Name "VertNormalPush"
    21.             Cull Front
    22.             ZTest [_ZTest]
    23.  
    24.             CGPROGRAM
    25.             #pragma vertex vert
    26.             #pragma fragment frag
    27.             #include "UnityCG.cginc"
    28.  
    29.             struct appdata
    30.             {
    31.                 float4 vertex : POSITION;
    32.                 float3 normal : NORMAL;
    33.                 float3 smoothNormal : TEXCOORD3;
    34.             };
    35.  
    36.             struct v2f
    37.             {
    38.                 float4 position : SV_POSITION;
    39.                 fixed4 color : COLOR;
    40.             };
    41.  
    42.             uniform fixed4 _OutlineColor;
    43.             uniform float _OutlineWidth;
    44.  
    45.             v2f vert(appdata input)
    46.             {
    47.                 v2f output;
    48.                 float3 normal = any(input.smoothNormal) ? input.smoothNormal : input.normal;
    49.                 float3 viewPosition = UnityObjectToViewPos(input.vertex);
    50.                 float3 viewNormal = normalize(mul((float3x3)UNITY_MATRIX_IT_MV, normal));
    51.                 output.position = UnityViewToClipPos(viewPosition + viewNormal * _OutlineWidth / 1000.0);
    52.                 output.color = _OutlineColor;
    53.                 return output;
    54.             }
    55.  
    56.             fixed4 frag(v2f input) : SV_Target
    57.             {
    58.                 return input.color;
    59.             }
    60.  
    61.             ENDCG
    62.         }
    63.  
    64.         Pass // Second pass: solid color geometry fill
    65.         {
    66.             Name "SolidColorFill"
    67.             Cull Off
    68.  
    69.             CGPROGRAM
    70.             #pragma vertex vert
    71.             #pragma fragment frag
    72.  
    73.             uniform fixed4 _FillColor;
    74.  
    75.             float4 vert(float4 vertexPos : POSITION) : SV_POSITION
    76.             {
    77.                 return UnityObjectToClipPos(vertexPos);
    78.             }
    79.  
    80.             float4 frag(void) : COLOR
    81.             {
    82.                 return _FillColor;
    83.             }
    84.  
    85.             ENDCG
    86.         }
    87.     }
    88. }
     
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    4,521
    Not holes, back faces. Since the vertex normals are pointing in different directions, some faces which would normally be hidden by front face culling get turned towards the camera and become visible.

    There are a couple of different methods, depending on your needs and your scene setup, to ensure the outline really is just an outline.

    Method 1: Use ZWrite Off on the outline pass. Works well if you don’t need / want later objects to intersect with the outline, but likely also requires you set your object’s render queue to be fairly late (like 2451, or “Queue”=“AlphaTest+1”) to prevent other normal objects in the scene from rendering on top of the outlines.

    Method 2: Use a camera depth offset on the outline pass. Push the outline vertices away from the camera by some amount so those back faces are pushed behind the object. This can be done by adding a view dir offset, or by directly modifying the clip space z. In some cases the value needed to remove all interior lines may cause the outline to disappear “behind” other objects when too close.

    Method 3: Use a stencil or destination alpha. Render your object and write to the stencil. Then render the outline and compare against the stencil value to exclude rendering where the object was. When using this technique I use Cull Back even for the outline.

    One thing to note, for method 1 and 2 the draw order of both the passes and the objects are important. If you want an outline around all of the objects and not individual objects you need to draw all of the objects with pass 1, then all of the objects with pass 2. If the objects are dynamically / statically batched or instanced, this may sometimes happen automatically. Otherwise you may need have the outline and main object rendering be different shaders with different queues, and have two separate renderer components with those materials.
     
    Kumo-Kairo likes this.
  3. BottleEngineer

    BottleEngineer

    Joined:
    Jul 10, 2018
    Posts:
    11
    I'm hoping to get each individual object to have it's own outline, as opposed to only outlining around a group objects. Courtesy of Paint, I drew what I want to achieve: OutlineHelp.png

    This works great for getting an overall edge around objects, but individual outlines are lost on duplicate objects (image above).

    I'm looking into your other suggestions now.
     
    Last edited: Jul 18, 2018 at 3:58 PM
  4. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    4,521
    If you want per object outlines, then you need to ensure each object renders both passes one after another before rendering the next. To ensure that you'll want to disable batching / instancing for the shader for methods 1 and 3, but that might not be enough.

    For method 1 you'll additionally want a back to front draw order, which means you'll need to change the render queue to >2500 to put it into the transparency queues. Assuming you can each object to render both passes before rendering the next object, this should work pretty well. In this case I would also swap to using Cull Back as well.

    For method 3 you'll want to keep front to back draw order, so the usual geometry queue (2000) is fine.

    Unity's draw order sorting is a complex beast. It'll sometimes choose to draw a single pass of multiple objects that share a material, even with out batching or instancing, before drawing the second pass of the material. But sometimes it won't. This messes with the two methods that rely on the draw order as they really require consistency. To work around this myself I've used command buffers to explicitly control the order, or assigned a Renderer.sortingOrder value via script at runtime which does work on mesh renderers even though it's not exposed in the inspector like it is for sprites.

    Method 2 doesn't care as it's purely taking advantage of the depth buffer which will correct sorting no matter the draw order, but for objects that are close to each other the outlines may penetrate into the other objects, or get hidden behind them because they are just being pushed behind the object.
     
    SugoiDev likes this.
  5. BottleEngineer

    BottleEngineer

    Joined:
    Jul 10, 2018
    Posts:
    11
    Sticking with Method 1 first -- I tried this and it's a large improvement. So close! However, there are some breaks in the outlines. I'm not sure why. With all ZWrites turned off, the effect is perfect, except for depth sorting of course.

    There is also some funny behavior of (only some) objects apparently changing depth, because their outlines will change to be over/under other objects as the camera pans over them. I haven't really troubleshooted this yet.

    My next idea is to implement 3 passes using Method 1. (1) Vertex push and draw "2" onto the stencil buffer, (2) Invisibly draw the normal size mesh, solely for the purpose of decrementing the stencil to 1, which should leave a value of 2 for the outlines, and (3) only draw interior color where stencil is 1. This should make sure I'm never drawing over outlines.

    My question is, for the first pass of this, will I be drawing 2's onto the stencil for the pushed vertex geometry like I want to, or will it really only be for the original geometry?
     
    Last edited: Jul 18, 2018 at 3:55 PM
  6. BottleEngineer

    BottleEngineer

    Joined:
    Jul 10, 2018
    Posts:
    11
    Here is an example of what I mean (gfycat). Any idea why this might happen, especially causing such large changes? I moved the far clip plane to be as close as possible for maximum precision, in case that might be related, but that didn't seem make a difference.
     
  7. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    3,078
    Method 4: (secret sauce)

    Push your vertex only in screen space, ie perpendicularly to the screen, that is when dot product with view direction is 0. That's how it is done traditionally/

    Likely to be combine with other method.
     
  8. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    4,521
    Nothing to do with precision here. This is simply Unity's default opaque depth sorting in action. It's roughly* front to back based on the closest bounding box center ... and it appears to be sorted by distance and not depth. This means the sorting order is unlikely to change when the camera rotates, but can when the camera pans (like what you're seeing).
    * Additionally the default opaque sort takes into account more than just the distance.
    https://docs.unity3d.com/ScriptReference/Rendering.OpaqueSortMode.html

    You can switch to force front to back (ignore materials or sorting layer, but batching & instancing may still interfere), or off which will draw in the order the objects exist in the scene (which again, batching & instancing may mess with). But you're better off leaving it the default and being explicit with the sortingOrder.
    https://docs.unity3d.com/ScriptReference/Renderer-sortingOrder.html
     
    Last edited: Jul 16, 2018 at 7:51 PM
  9. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    4,521
    I would call this a refinement rather than anything else. It's useful for keeping screen-space constant outline sizes. Doing the push in "2D" XY only and scaling by Z in view space can give identical results. Apart from the constant line width, for methods 1 and 3 it will make no tangible difference apart from some differences at the near clipping plane, simply because the outline isn't being pushed towards the camera.

    The view / screen space xy only normal push can be useful for method 2 as it can mean less push away from the camera is required to remove artifacts on the front faces as it's not first getting pushed towards the camera via the normal, and in turn less artifacts from depth intersections with other objects. Ultimately though you will still have those artifacts unless you keep a line-width (or greater) sized distance buffer between objects.
     
  10. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    4,521
    One more thing. If you're wondering "how do professional games handle this?" The short answer is they don't and just leave the artifacts in (see Super Smash Bros 3DS), or they use a screen space outline effect (see Borderlands), or they render out elements to sprites in real time and use sprite sorting (see Brawl Stars).
     
  11. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    3,078
    Or they squash the entire model, in z in camera space, like in guilty gears.
     
  12. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    4,521
    Per character, or during cinematic they just run with the artifacts of the effect. As far as I know they're just doing the basic normal push technique in screen space, with some additional control for per-vertex width and offsets. I'm not sure if they're squashing the models in clip space or not during the normal gameplay camera, though that would give them the sprite like sorting they get (last character to attack is always "on top").
     
  13. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    3,078
    They have a gdc talk available in youtube, and a few doc and interview everywhere, they explain it all.
    Anyway just complementing the information you are providing.

    There will always be artifact and any solution is a mix of all mentioned shader trick, artistic hack and art direction.

     
  14. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    4,521
    Yep, that's a great gdc talk, even if they're a little hand wavy about the outlines. They also don't mention the z squishing in that talk that I remember (maybe it is, but I never noticed?). There are some Japanese posts and articles that have been partially translated on other sites that go into a little more detail.

    This post in particular does a great job of translating an in depth Japanese language article on the topic that covers a lot of the topics in that video, and more.
    https://polycount.com/discussion/comment/2099538#Comment_2099538

    Normally characters in Guilty Gear do not intersect at all and overlap like sprites. Like here:

    Each characters lines are perfectly clean, and the characters have zero intersecting geometry. They're effectively sprites. Clip space Z scaling is a cheap way of getting this, but you have to be very careful as it can mess with linear interpolation and cause odd artifacts, even with a style as simple as this. It also will cause issues with anything else you want to intersect with, like effects or the scene geometry itself.

    However you can see in this image (taken at a moment when the camera moves) that the sprite like sorting isn't always the case. Here they intersect, and you can see some of the outline mesh getting a bit messy.

    See Baiken's sleaves and sword intersecting with Ky's coattails.

    Curiously, for Arc's latest game, Dragon Ball Fighter Z, it looks like they've completely given up on the sprite-like sorting and just let characters intersect all of the time, possibly because the characters interact with environments far more.

    Android 18's (the girl) forearm is clearly intersecting Cell's (big green dude) hand, but the outline holds up. Not sure if this is luck or they've changed their technique. Also curious is that Android 17 (the guy in the background) is sorting more sprite-like, but it's more likely just the luck of there poses than any flattening. There are also screenshots where one character is showing clear signs of their feet's outlines penetrating into the ground and being occluded, but the other character isn't, and is drawing on top of objects in the scene. And others where their feet are intersecting with the ground objects, but still showing proper outlines(!) around the occluded area, like here:

    Look closely at the feet of Trunks (blond guy) and Zamasu (green skin). They intersect the ground, but there are nice outlines still. Conversely Black's feet (pink hair) don't intersect at all?


    Lots of tricky stuff.
     
    Last edited: Jul 17, 2018 at 9:14 PM
    jRocket and neoshaman like this.
  15. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    4,521
    neoshaman likes this.