(2020) Any *good* outline shaders that don't require messing with rendering pipelines?

I've tried the old toon outline one: (thanks Mauri)

https://forum.unity.com/threads/whe...der-go-the-one-with-the-outline.972540/unread

And a few others such as Aubergine's,
https://assetstore.unity.com/packages/vfx/shaders/aubergines-toon-shaders-8705

But they all suffer from this sort of BS around the edges...
It wasn't cute in 2013 and it's getting a bit old seeing the same "solutions" over and over.



Are there any good toon shaders
-that will do a clean outline?
-which can also pick out the odd edge based on depth?
-which don't require taking a leap of faith with a new rendering pipeline?
-which will preferably work on U2018/2019?

Thank you so much in advance.
J

 

Because the technique they're using hasn't fundamentally changed since 2000, when it was used in Jet Set Radio, and still has the same limitations. Most of these are taking the base mesh geometry, and rendering it a second time inside out and pushed outward along the vertex normal. For it to work requires special handling of the mesh. ie: All meshes must be water tight and have smoothed normals. Any hard edges w/o a filled seam will result in holes in the outline. If you don't want to fix the meshes yourself, you can use something like Toony Colors Pro 2 which includes utilities for processing meshes in-editor. Even then, there are meshes that can't be fixed by those tools alone, and because it is just an inverted mesh may end up with odd artifacts wherever two surfaces are close to each other.

The "problem" is there are no other techniques that come anywhere close to the flexibility and speed of that inverted hull mesh outline technique.

Also, that "screwing with the render pipelines" comment... the outline effect you're probably thinking of, the one that Unity showed off for the URP, is exactly the same basic technique as the one you're using above, and has the same limitations. This article even calls out the fact that Unity's implementation has unfortunate artifacts.
https://medium.com/@porter.johnross...in-the-universal-render-pipeline-506d5f3ce496


Then there are the more complicated geometry shader based techniques you can find in academic papers. Those are all limited to academic papers because they use rendering API features not a single real time game engine actually implements. This is because to implement the features needed wold make literally everything render much slower. Also, if you look at any of the benchmarks for them you'll see they're basically uselessly slow even when rendering a single object. It's good for offline rendering where it's plausibly faster than other visually similar approaches, but not for real time. Simplified approaches exist, but they have very similar limitations as the inverted hull technique in that they require special mesh processing, so realistically don't offer any obvious benefits over it outside of some very specific use cases.

The next most common outline technique would be a post process based approach, either depth based, or better yet screen space normal and depth based. The Borderlands series is a good example of this technique. It doesn't require any special handling of the scene geometry, only a post process. But realistically it's limited to relatively narrow outlines; most implementations max out at 2 pixel wide outlines. Any wider than maybe 4-5 pixels becomes implausibly slow even for modern GPUs. Borderlands 3 maxes out at around 3 pixels, and is using some fairly advanced techniques to make that fast.

Unity used to ship with this as a "built in" image effect.
https://docs.unity3d.com/550/Documentation/Manual/script-EdgeDetectEffectNormals.html

However today you'd probably want to download one that works with Unity's Post Processing Stack v2. Like this one:
https://github.com/jean-moreno/EdgeDetect-PostProcessingUnity

I also explore a bunch of this in my article on wide outlines, thought it's explicitly about doing wide silhouettes, like a selection outline rather than something akin to a toon shaded outline.
https://medium.com/@bgolus/the-quest-for-very-wide-outlines-ba82ed442cd9

 

TLDR version: That outline shader isn't the problem, your meshes are. And depending on what you need, there's probably not any other solution.

 

Thanks bgolus; I really appreciate the time taken and your detailed response there.

Would you mind elaborating here? is the implication that my meshes aren't closed? Or that just, in general you have to screw around with them in some way (adding extra seams, flipped faces, etc) Or that they should be split into constituent parts for cleaner outlines? Or that just in general, things like faces are too complex to offload onto a shader alone without the depth buffer?
E.g. without the issues you see in that guy's beard from your screenie.

It was mostly this bit here that was bothering me:


Example: (Cubes from Blender, cylinder from Unity)



I've found a couple of slightly better options, hopefully they'll be useful to other people wanting to learn too:
With all the documentation, they should be great study aids!

In the comments of all places, lol:

This absolute legend has made a nice smooth one:
https://forum.unity.com/threads/outline-shader-with-smoothness.533488/

Shrimpey's excellent git repo
https://github.com/Shrimpey/Outlined-Diffuse-Shader-Fixed

Primitives example:


The first one is pure outline, the second and third exhibit the same issues on stuff like the blender monkey here:
This may or may not work for your needs.

 

Because those meshes have hard edges. You cannot use any mesh with hard edges*. The Unity Sphere and Capsule default meshes are the only ones that come with Unity that work with this technique, otherwise you need to provide your own smoothed meshes. If you use Blender boxes, try adding a very thin chamfered edge and smooth the normals. Maybe use the Normal Edit Modifier with directional > parallel per box face.

* The caveat here being you can use hard edged meshes if you encode smoothed normal data in some form, like encoded in the vertex colors or as mesh tangents, which is what Toony Colors Pro 2 does for you.

The first post and most of the shaders in that repo are doing the outline by just scaling up the mesh by some arbitrary amount. Works okay on uniformly sized convex objects. Fails on anything more complex, like a torus, or even the cylinder there (notice how much thinner the outlines are on the sides). Similarly, the blender monkey outlines are seemingly nonexistent on some parts of the mesh, like the left side of the face or the top of the ears, because those surfaces are parallel to the direction of the scaling. There are also fun issues like if you have meshes of different relative scale the same "outline width" material property will translate to wildly different outline widths. A sphere that is 1 unit in diameter will have outlines half the width of a sphere that is 2 units in diameter.

That video tutorial shows the same issues too, with lots of the edges of the "ground" missing outlines entirely.

Those artifacts are why the normal based method is so much more common as it can properly handle almost any mesh shape as long as the mesh has been properly prepared, and also why Toony Colors Pro 2 is so popular because it can do most of that work for you without you needing to do it by hand.

 

Once more, thanks for the detailed response!

 

Thanks again bgolus, didn't take long to patch something together with the info your provided:

It may need a bit of work for more general use, but for my purposes (generating stills), it does the job perfectly:




Component to store the averaged normal data in the mesh colours:

Code (CSharp):

1.  
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5.  
6. // Stores modified vert normals in the color data.
7. // Skipping a lot of error checking for the sake of brevity.
8.  
9. public class Meshticles : MonoBehaviour
10. {
11.    
12.     public Mesh originalMesh;
13.    
14.     [ContextMenu("DoTheThing")]
15.     public void DoTheThing(){
16.  
17.         MeshFilter mf = GetComponent<MeshFilter>();
18.         MeshRenderer mr = GetComponent<MeshRenderer>();
19.  
20.         if ( originalMesh == null )
21.             originalMesh = mf.sharedMesh;
22.  
23.         Mesh newMesh = (Mesh)Instantiate( originalMesh );
24.        
25.         newMesh.name += "_baked";
26.        
27.         List<Color> vertColors = new List<Color>( newMesh.vertexCount );
28.        
29.         // so it's not cloning the array on every iteration      
30.         Vector3[] verts = newMesh.vertices;
31.         Vector3[] norms = newMesh.normals;
32.  
33.         for( int i = 0; i < newMesh.vertexCount; i++ ){
34.            
35.  
36.             // Faces associated with this vert
37.             List<Vector3> activeFaces = new List<Vector3>();
38.  
39.             // If a nother vert shares the same space as this one
40.             // then we'll average in its normal
41.             for( int a = 0; a < newMesh.vertexCount; a++ ){
42.                 if ( a == i ) continue;
43.                 if ( verts[i] == verts[a] )
44.                     activeFaces.Add( norms[a] );
45.             }
46.  
47.             activeFaces.Add( norms[i] );
48.  
49.             // average the faces (normals)
50.             Vector3 vertColor = Vector3.zero;
51.             for( int n = 0; n < activeFaces.Count; n++ ){
52.                 vertColor += activeFaces[n];
53.             }
54.             vertColor.Normalize();
55.            
56.             // Even if it's orphaned, etc
57.             vertColors.Add( V3ToColor( vertColor ) );
58.            
59.         }
60.  
61.         newMesh.SetColors( vertColors );
62.        
63.         mf.sharedMesh = newMesh;
64.        
65.     }
66.  
67.     Color V3ToColor( Vector3 inVector ){
68.         return new Color( inVector.x, inVector.y, inVector.z, 0 );
69.     }
70.    
71.  
72.  
73.     [ContextMenu("RestoreTheThing")]
74.     public void RestoreTheThing(){
75.        
76.         if ( originalMesh != null )
77.             GetComponent<MeshFilter>().sharedMesh = originalMesh;
78.  
79.     }
80.  
81.  
82. }
83.  

And a modified version of Shrimpey's shader to handle it.

Code (CSharp):

1.  
2. // MIT license:
3. // https://github.com/Shrimpey/Outlined-Diffuse-Shader-Fixed
4.  
5. // Upgrade NOTE: replaced 'mul(UNITY_MATRIX_MVP,*)' with 'UnityObjectToClipPos(*)'
6.  
7. Shader "Outlined/Custom_Tweaks" {
8.     Properties {
9.         _Color ("Main Color", Color) = (.5,.5,.5,1)
10.         _OutlineColor ("Outline Color", Color) = (0,0,0,1)
11.         _Outline ("Outline width", Range (0, 1)) = .1
12.         _MainTex ("Base (RGB)", 2D) = "white" { }
13.         _DepthFudge ("Depth Fudge", Range (-0.1, 0.1)) = -0.01
14.  
15.     }
16. CGINCLUDE
17. #include "UnityCG.cginc"
18. struct appdata {
19.     float4 vertex : POSITION;
20.     float3 normal : NORMAL;
21. };
22. struct v2f {
23.     float4 pos : POSITION;
24.     float4 color : COLOR;
25. };
26. uniform float _Outline;
27. uniform float4 _OutlineColor;
28. uniform float _DepthFudge;
29.  
30. v2f vert(appdata_full v) {
31.  
32.     // just make a copy of incoming vertex data but scaled according to normal direction
33.     v2f o;
34.    
35.     // Less pretty
36.     //v.vertex += ( v.color ) * _Outline;
37.  
38.     o.pos = UnityObjectToClipPos(v.vertex);
39.    
40.     fixed3 norm = mul( (float3x3)UNITY_MATRIX_MV, v.color );
41.     norm.x *= UNITY_MATRIX_P[0][0];
42.     norm.y *= UNITY_MATRIX_P[1][1];
43.     o.pos.xy += norm.xy * _Outline;
44.  
45.     o.pos.z += _DepthFudge;
46.    
47.     o.color = _OutlineColor;
48.     return o;
49. }
50. ENDCG
51.     SubShader {
52.         //Tags {"Queue" = "Geometry+100" }
53. CGPROGRAM
54. #pragma surface surf Lambert
55. sampler2D _MainTex;
56. fixed4 _Color;
57. struct Input {
58.     float2 uv_MainTex;
59. };
60. void surf (Input IN, inout SurfaceOutput o) {
61.     fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
62.     o.Albedo = c.rgb;
63.     o.Alpha = c.a;
64. }
65. ENDCG
66.         // note that a vertex shader is specified here but its using the one above
67.         Pass {
68.             Name "OUTLINE"
69.             Tags { "LightMode" = "Always" }
70.             Cull Front
71.             ZWrite On
72.             ColorMask RGB
73.             Blend SrcAlpha OneMinusSrcAlpha
74.             //Offset 50,50
75.             CGPROGRAM
76.             #pragma vertex vert
77.             #pragma fragment frag
78.             half4 frag(v2f i) :COLOR { return i.color; }
79.             ENDCG
80.         }
81.     }
82.     SubShader {
83. CGPROGRAM
84. #pragma surface surf Lambert
85. sampler2D _MainTex;
86. fixed4 _Color;
87. struct Input {
88.     float2 uv_MainTex;
89. };
90. void surf (Input IN, inout SurfaceOutput o) {
91.     fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
92.     o.Albedo = c.rgb;
93.     o.Alpha = c.a;
94. }
95. ENDCG
96.         Pass {
97.             Name "OUTLINE"
98.             Tags { "LightMode" = "Always" }
99.             Cull Front
100.             ZWrite On
101.             ColorMask RGB
102.             Blend SrcAlpha OneMinusSrcAlpha
103.             CGPROGRAM
104.             #pragma vertex vert
105.             #pragma exclude_renderers gles xbox360 ps3
106.             ENDCG
107.             SetTexture [_MainTex] { combine primary }
108.         }
109.     }
110.     Fallback "Diffuse"
111. }

 

I'm ... actually a little confused how that's working. Unity's vertex colors are stored as a byte, an integer value between 0 and 255 which the GPU uses to represent a value between 0.0 and 1.0. It shouldn't be able to store negative values. You should need to remap the -1.0 to 1.0 range of the smoothed vector to a 0.0 to 1.0 range of a Color, or 0 to 255 range of a Color32, and then do

v.color.xyz * 2.0 - 1.0

in the shader to extract. Does the outline work from all sides of the object?

 

Hmm, this rings a bell. I vaguely remember this issue.
But yeah it's working from all sides. The eyes perhaps have a tiny bias, but I've not had a lot of time to fiddle.

In fact, if I store a negative version of the color

Code (CSharp):

1. return new Color( inVector.x, inVector.y, inVector.z, 0 ) * -1;

and negate the outline range

Code (CSharp):

1. _Outline ("Outline width", Range (-1, 1)) = .1

It's business as usual.

That said, this also works:

Code (CSharp):

1.     byte Remap( float inVal ){
2.         return (byte)((( inVal * 0.5f ) + 0.5f)*255);
3.     }
4.  
5.     Color V3ToColor( Vector3 inVector ){
6.         //return new Color( inVector.x, inVector.y, inVector.z, 0 ) * -1;
7.         return new Color32( Remap(inVector.x), Remap(inVector.y), Remap(inVector.z), 0 );
8.     }

Code (CSharp):

1. fixed3 norm = mul( (float3x3)UNITY_MATRIX_MV, (v.color.xyz * 2.0) - 1.0 );

(With*out* a 1/256 multiplier on width)


Edit:

It did becomes stuck to one side briefly, as if only using the positive part of each axis, after reverting to the first shader/script I posted, but returning the range here to 0,1 fixed it. _Outline ("Outline width", Range (0, 1)) = .1
I don't have a lot of time to test today, but it definitely sounds like there's something going on behind the curtains.

 

What version of Unity are you using?

 

2018.4 -> 2019.2
That's the range I have on this machine right now.

 

I'm not sure when they made the change, but when assigning a Color collection, they write the VertexAttributeFormat as Float32, preserving the negative values in shader, but if you feed it a Color32, it will of course be set to UNorm8 for obvious reasons.

The change has been back-ported to all Unity versions it seems, even Unity 5.

 

I too struggled with the outline artifacts, mostly the outline being cutted in sharp corners. Recently I realized the problem is not the shader but the mesh itself as blogus pointed out already. Using "smooth shading" should fix most of them.
Unity3D Art Tips and Tricks feat. Blender 3D [Synersteel]

 

Would a post-processing effect similar to this solve your problem: https://bgolus.medium.com/the-quest-for-very-wide-outlines-ba82ed442cd9

 

The problem is that you are trying to move the vertex along his normal and when we have hard edges we get artifacts. Hard edges are splitted vertex.
To solve this you can simply change the smoothgroups/hard edges in a 3d software, but the model appearence will be strange.
But we can do this same process inside Unity, and store this new smoothing information in a another mesh property, in my case I am saving it in mesh.tangent.

*This is a AssetPostprocessor, and it will working when you set a label "Highlightable" to your model.

Code (CSharp):

1. using UnityEngine;
2. using UnityEditor;
3. using System.Collections.Generic;
4. using System;
5. using System.IO;
6. using System.Linq;
7.  
8. public class CustomModelImporter : AssetPostprocessor
9. {
10.     private void OnPreprocessModel()
11.     {
12.         var labelList = AssetDatabase.GetLabels(assetImporter);
13.         if(labelList.Contains("Highlightable"))
14.         {
15.             ModelImporter modelImporter = assetImporter as ModelImporter;
16.             modelImporter.isReadable = true;
17.         }
18.     }
19.     void OnPostprocessModel (GameObject gameObject)
20.     {
21.         var labelList = AssetDatabase.GetLabels(assetImporter);
22.         if(labelList.Contains("Highlightable"))
23.         {
24.             SmoothNormals(gameObject);
25.             ModelImporter modelImporter = assetImporter as ModelImporter;
26.             modelImporter.isReadable = false;
27.         }
28.     }
29.  
30.     void SmoothNormals(GameObject gameObject)
31.     {
32.         var meshFilters = gameObject.GetComponentsInChildren<MeshFilter>();
33.  
34.         foreach (var meshFilter in meshFilters)
35.         {
36.             SmoothNormals(meshFilter.sharedMesh);
37.         }
38.  
39.         var skinnedMeshRenderers = gameObject.GetComponentsInChildren<SkinnedMeshRenderer>();
40.  
41.         foreach (var skinnedMeshRenderer in skinnedMeshRenderers)
42.         {
43.             SmoothNormals(skinnedMeshRenderer.sharedMesh);
44.         }
45.     }
46.  
47.     void SmoothNormals(Mesh sharedMesh)
48.     {
49.         Vector3[] smoothedNormals = RecalculateNormals(sharedMesh, 180);
50.  
51.         Vector4[] smoothedTangents = new Vector4[smoothedNormals.Length];
52.  
53.         for(int i = 0; i < smoothedNormals.Length; i++)
54.         {
55.             Vector3 normal = smoothedNormals[i];
56.             smoothedTangents[i] = new Vector4(normal.x, normal.y, normal.z, 0f);
57.         }
58.         sharedMesh.tangents = smoothedTangents;
59.     }
60. }

After that, in outline shader you will need read this new information.

uniform float4 _OutlineColor;
uniform float _OutlineWidth;
sampler2D _MainTex;

struct appdata
{
float4 vertex : POSITION;
float3 normal : NORMAL;
float3 tangent : TANGENT;
float3 texCoord : TEXCOORD0;
float4 color : TEXCOORD1;
};

struct v2f
{
float4 pos : SV_POSITION;
float4 color : TEXCOORD0;
};

v2f vert(appdata v)
{
v2f o;

float3 normal = normalize(v.tangent);
o.pos = v.vertex + float4(normal, 0.0) * _OutlineWidth;
o.pos = UnityObjectToClipPos(o.pos);
o.color = _OutlineColor;

return o;
}

float4 frag(v2f i) : COLOR
{
return i.color;
}

 

I have written countless asset post processor to do all kinds of weird tricks on meshes and other assets but I never thought of just using the label system to store if those operations should be applied. I messed around with jsons in userdata and other additionally created files, heck I have already looked into how the PSD importer creates an overwrite importer + inspector just for that specific use case, but labels is sooooo much easier. Thanks you so much for just mentioning that tiny detail ^^