Outline shader behavior with thick outlines

I'm working on a simple outline shader using vertex normal pushing. It works decently well with thin outlines, but still needs more work. However, I need thick outlines (>10 pixels thick) and I get some funny behavior at those ranges. For my application, I don't need lighting, so everything is unlit solid colors.

The problem is that with larger outlines, some vertices can be pushed past others, creating what I assume are holes (?). However, I draw the actual geometry in the second pass with solid color with culling turned off, so I would think this wouldn't be an issue. In a perfect world, the outline should only appear around the perimeter of objects.

gfycat example

Is there a way I can improve this to avoid that effect? Is there a good explanation of what is happening?



Here is the shader:

Code (CSharp):

1. Shader "Outline"
2. {
3.     Properties
4.     {
5.         [Enum(UnityEngine.Rendering.CompareFunction)] _ZTest("ZTest", Float) = 0
6.         _OutlineColor("Outline Color", Color) = (1, 1, 1, 1)
7.         _FillColor("Fill Color", Color) = (1, 1, 1, 1)
8.         _OutlineWidth("Outline Width", Range(0, 15)) = 2
9.     }
10.  
11.     SubShader
12.     {
13.         Tags
14.         {
15.             "RenderType" = "Opaque"
16.         }
17.  
18.         Pass // First pass: vertex normal push and single color fill
19.         {
20.             Name "VertNormalPush"
21.             Cull Front
22.             ZTest [_ZTest]
23.  
24.             CGPROGRAM
25.             #pragma vertex vert
26.             #pragma fragment frag
27.             #include "UnityCG.cginc"
28.  
29.             struct appdata
30.             {
31.                 float4 vertex : POSITION;
32.                 float3 normal : NORMAL;
33.                 float3 smoothNormal : TEXCOORD3;
34.             };
35.  
36.             struct v2f
37.             {
38.                 float4 position : SV_POSITION;
39.                 fixed4 color : COLOR;
40.             };
41.  
42.             uniform fixed4 _OutlineColor;
43.             uniform float _OutlineWidth;
44.  
45.             v2f vert(appdata input)
46.             {
47.                 v2f output;
48.                 float3 normal = any(input.smoothNormal) ? input.smoothNormal : input.normal;
49.                 float3 viewPosition = UnityObjectToViewPos(input.vertex);
50.                 float3 viewNormal = normalize(mul((float3x3)UNITY_MATRIX_IT_MV, normal));
51.                 output.position = UnityViewToClipPos(viewPosition + viewNormal * _OutlineWidth / 1000.0);
52.                 output.color = _OutlineColor;
53.                 return output;
54.             }
55.  
56.             fixed4 frag(v2f input) : SV_Target
57.             {
58.                 return input.color;
59.             }
60.  
61.             ENDCG
62.         }
63.  
64.         Pass // Second pass: solid color geometry fill
65.         {
66.             Name "SolidColorFill"
67.             Cull Off
68.  
69.             CGPROGRAM
70.             #pragma vertex vert
71.             #pragma fragment frag
72.  
73.             uniform fixed4 _FillColor;
74.  
75.             float4 vert(float4 vertexPos : POSITION) : SV_POSITION
76.             {
77.                 return UnityObjectToClipPos(vertexPos);
78.             }
79.  
80.             float4 frag(void) : COLOR
81.             {
82.                 return _FillColor;
83.             }
84.  
85.             ENDCG
86.         }
87.     }
88. }

 

Not holes, back faces. Since the vertex normals are pointing in different directions, some faces which would normally be hidden by front face culling get turned towards the camera and become visible.

There are a couple of different methods, depending on your needs and your scene setup, to ensure the outline really is just an outline.

Method 1: Use ZWrite Off on the outline pass. Works well if you don’t need / want later objects to intersect with the outline, but likely also requires you set your object’s render queue to be fairly late (like 2451, or “Queue”=“AlphaTest+1”) to prevent other normal objects in the scene from rendering on top of the outlines.

Method 2: Use a camera depth offset on the outline pass. Push the outline vertices away from the camera by some amount so those back faces are pushed behind the object. This can be done by adding a view dir offset, or by directly modifying the clip space z. In some cases the value needed to remove all interior lines may cause the outline to disappear “behind” other objects when too close.

Method 3: Use a stencil or destination alpha. Render your object and write to the stencil. Then render the outline and compare against the stencil value to exclude rendering where the object was. When using this technique I use Cull Back even for the outline.

One thing to note, for method 1 and 2 the draw order of both the passes and the objects are important. If you want an outline around all of the objects and not individual objects you need to draw all of the objects with pass 1, then all of the objects with pass 2. If the objects are dynamically / statically batched or instanced, this may sometimes happen automatically. Otherwise you may need have the outline and main object rendering be different shaders with different queues, and have two separate renderer components with those materials.

 

I'm hoping to get each individual object to have it's own outline, as opposed to only outlining around a group objects. Courtesy of Paint, I drew what I want to achieve:

This works great for getting an overall edge around objects, but individual outlines are lost on duplicate objects (image above).

I'm looking into your other suggestions now.

 

If you want per object outlines, then you need to ensure each object renders both passes one after another before rendering the next. To ensure that you'll want to disable batching / instancing for the shader for methods 1 and 3, but that might not be enough.

For method 1 you'll additionally want a back to front draw order, which means you'll need to change the render queue to >2500 to put it into the transparency queues. Assuming you can each object to render both passes before rendering the next object, this should work pretty well. In this case I would also swap to using Cull Back as well.

For method 3 you'll want to keep front to back draw order, so the usual geometry queue (2000) is fine.

Unity's draw order sorting is a complex beast. It'll sometimes choose to draw a single pass of multiple objects that share a material, even with out batching or instancing, before drawing the second pass of the material. But sometimes it won't. This messes with the two methods that rely on the draw order as they really require consistency. To work around this myself I've used command buffers to explicitly control the order, or assigned a Renderer.sortingOrder value via script at runtime which does work on mesh renderers even though it's not exposed in the inspector like it is for sprites.

Method 2 doesn't care as it's purely taking advantage of the depth buffer which will correct sorting no matter the draw order, but for objects that are close to each other the outlines may penetrate into the other objects, or get hidden behind them because they are just being pushed behind the object.

 

Sticking with Method 1 first -- I tried this and it's a large improvement. So close! However, there are some breaks in the outlines. I'm not sure why. With all ZWrites turned off, the effect is perfect, except for depth sorting of course.

There is also some funny behavior of (only some) objects apparently changing depth, because their outlines will change to be over/under other objects as the camera pans over them. I haven't really troubleshooted this yet.

My next idea is to implement 3 passes using Method 1. (1) Vertex push and draw "2" onto the stencil buffer, (2) Invisibly draw the normal size mesh, solely for the purpose of decrementing the stencil to 1, which should leave a value of 2 for the outlines, and (3) only draw interior color where stencil is 1. This should make sure I'm never drawing over outlines.

My question is, for the first pass of this, will I be drawing 2's onto the stencil for the pushed vertex geometry like I want to, or will it really only be for the original geometry?

 

Here is an example of what I mean (gfycat). Any idea why this might happen, especially causing such large changes? I moved the far clip plane to be as close as possible for maximum precision, in case that might be related, but that didn't seem make a difference.

 

Method 4: (secret sauce)

Push your vertex only in screen space, ie perpendicularly to the screen, that is when dot product with view direction is 0. That's how it is done traditionally/

Likely to be combine with other method.

 

Nothing to do with precision here. This is simply Unity's default opaque depth sorting in action. It's roughly* front to back based on the closest bounding box center ... and it appears to be sorted by distance and not depth. This means the sorting order is unlikely to change when the camera rotates, but can when the camera pans (like what you're seeing).
* Additionally the default opaque sort takes into account more than just the distance.
https://docs.unity3d.com/ScriptReference/Rendering.OpaqueSortMode.html

You can switch to force front to back (ignore materials or sorting layer, but batching & instancing may still interfere), or off which will draw in the order the objects exist in the scene (which again, batching & instancing may mess with). But you're better off leaving it the default and being explicit with the sortingOrder.
https://docs.unity3d.com/ScriptReference/Renderer-sortingOrder.html

 

I would call this a refinement rather than anything else. It's useful for keeping screen-space constant outline sizes. Doing the push in "2D" XY only and scaling by Z in view space can give identical results. Apart from the constant line width, for methods 1 and 3 it will make no tangible difference apart from some differences at the near clipping plane, simply because the outline isn't being pushed towards the camera.

The view / screen space xy only normal push can be useful for method 2 as it can mean less push away from the camera is required to remove artifacts on the front faces as it's not first getting pushed towards the camera via the normal, and in turn less artifacts from depth intersections with other objects. Ultimately though you will still have those artifacts unless you keep a line-width (or greater) sized distance buffer between objects.

 

One more thing. If you're wondering "how do professional games handle this?" The short answer is they don't and just leave the artifacts in (see Super Smash Bros 3DS), or they use a screen space outline effect (see Borderlands), or they render out elements to sprites in real time and use sprite sorting (see Brawl Stars).

 

Or they squash the entire model, in z in camera space, like in guilty gears.

 

Per character, or during cinematic they just run with the artifacts of the effect. As far as I know they're just doing the basic normal push technique in screen space, with some additional control for per-vertex width and offsets. I'm not sure if they're squashing the models in clip space or not during the normal gameplay camera, though that would give them the sprite like sorting they get (last character to attack is always "on top").

 

They have a gdc talk available in youtube, and a few doc and interview everywhere, they explain it all.
Anyway just complementing the information you are providing.

There will always be artifact and any solution is a mix of all mentioned shader trick, artistic hack and art direction.

 

Yep, that's a great gdc talk, even if they're a little hand wavy about the outlines. They also don't mention the z squishing in that talk that I remember (maybe it is, but I never noticed?). There are some Japanese posts and articles that have been partially translated on other sites that go into a little more detail.

This post in particular does a great job of translating an in depth Japanese language article on the topic that covers a lot of the topics in that video, and more.
https://polycount.com/discussion/comment/2099538#Comment_2099538

Normally characters in Guilty Gear do not intersect at all and overlap like sprites. Like here:

Each characters lines are perfectly clean, and the characters have zero intersecting geometry. They're effectively sprites. Clip space Z scaling is a cheap way of getting this, but you have to be very careful as it can mess with linear interpolation and cause odd artifacts, even with a style as simple as this. It also will cause issues with anything else you want to intersect with, like effects or the scene geometry itself.

However you can see in this image (taken at a moment when the camera moves) that the sprite like sorting isn't always the case. Here they intersect, and you can see some of the outline mesh getting a bit messy.

See Baiken's sleaves and sword intersecting with Ky's coattails.

Curiously, for Arc's latest game, Dragon Ball Fighter Z, it looks like they've completely given up on the sprite-like sorting and just let characters intersect all of the time, possibly because the characters interact with environments far more.

Android 18's (the girl) forearm is clearly intersecting Cell's (big green dude) hand, but the outline holds up. Not sure if this is luck or they've changed their technique. Also curious is that Android 17 (the guy in the background) is sorting more sprite-like, but it's more likely just the luck of there poses than any flattening. There are also screenshots where one character is showing clear signs of their feet's outlines penetrating into the ground and being occluded, but the other character isn't, and is drawing on top of objects in the scene. And others where their feet are intersecting with the ground objects, but still showing proper outlines(!) around the occluded area, like here:

Look closely at the feet of Trunks (blond guy) and Zamasu (green skin). They intersect the ground, but there are nice outlines still. Conversely Black's feet (pink hair) don't intersect at all?


Lots of tricky stuff.

edit: They're definitely using the ~1 meter offset technique still. The logic for when they switch seems not totally straight forward, but it's basically when ever someone starts an attack they push toward the camera and everyone else goes "back" to the place they look like they should be. With assist characters it appears it does not push them toward the camera, but does pull everyone back, which explains the Cell, 17 and 18 image. 17 was the last one to attack, so Cell and 18 are in their "real" position and intersecting. Same with Trunks and Zamasu, Black is attacking and thus floats over the rocks (and his effects are noticeably pushed toward the camera as well).

 

@neoshaman btw, no "squishing", just offsetting by ~1 meter in view space.
https://www.4gamer.net/games/216/G021678/20140714079/