Shader for Adding an Outline to the Inside of the mesh

That title might seem a tad weird, I'll try to elaborate. I've been trying for two days to get this working, but I can't figure it out. Essentially, I'm looking for a shader that will add an outline to an object, but not around the object per se, but inside of the object. Shrinking the object in vertex shader and using multiple passes to fill in the back with the border color would be perfectly fine, but I can't figure out how to shrink the object in the shader without screwing up the other transformations as well. The shader is a simple unlit non-textured shader that supports transparency. Any help would be greatly appreciated

 

Shrink it the same way the Standard Assets Toon shader does, by pushing the mesh by the mesh’s vertex normals.

However to get “inlines” is definitely going to take a multi-pass approach. You don’t want to shrink the base textured pass because it can have an effect on the look. So you need a pass that renders an inside mask, a pass that renders the outline, and a pass to rendered the textured pass. You’ll also need to choose between using stencils or destination alpha for the mask. Both are effective, but stencils may not be available on mobile platforms, but destination alpha can be a little more difficult to use depending on the situation.

Here’s the high level idea.

Pass 1: Mask
Draw mesh with vertices pushed in by their normal in projection space. Again, check out the Standard Assets Toon Outline shader for this. This pass should use ZWrite Off, and either write to the stencil with some value, or fill the alpha with black.

For stencil masking:
ZWrite Off
ColorMask 0 // do not render any color
Stencil {
Ref 1
Comp Always
Pass Replace
}

fixed4 frag () : SV_Target { return fixed4(0,0,0,0); }

Or for destination alpha:
ZWrite Off
ColorMask A // render only to alpha

fixed4 frag () : SV_Target { return fixed4(0,0,0,0); }


Pass 2: Texture
Just draw this pass normally; texture, lighting, etc. We just need to make sure not to undo the first pass. With the stencil mask we could potentially save a little perf by skipping the pixels that will eventually be an outline, but this is only useful if you’re outline is really thick, so we’ll skip that.

For stencil masking:
ZWrite On
// that’s it

Or for destination alpha:
ZWrite On
ColorMask RGB

Pass 3: Outline
This is where the magic happen. Now you render the outline pass skipping the areas where the mask was rendered. Don’t do anything special in the vertex shader, it should match the textured pass.

For stencil masking:
ZWrite Off
Stencil {
Ref 1
Comp NotEqual
}

fixed4 frag () : SV_Target { fixed4(_OutlineColor.rgb, 1); }

Or for destination alpha:
ZWrite Off
Blend DstAlpha OneMinusDstAlpha, One Zero

fixed4 frag () : SV_Target { fixed4(_OutlineColor.rgb, 1); }


And we’re done!


Well, sort of. There are two potential issues. For the stencil buffer we’re leaving it a little dirty, and when two things with outlines overlap on screen, the outlines may not appear on one of them where we would expect them to. This can be solved by adding a fourth pass that’ll clear the stencil, or modifying the outline pass to use:

Stencil {
Ref 0
Comp Equal
Fail Replace
}

For the destination alpha option, we’re relying on the scene’s alpha to already be 1. Most of Unity’s built in opaque shaders force their output alpha to 1, but the built in skybox shaders don’t, and the camera clear color I believe defaults to an alpha of zero. That means if the object has the sky behind it it may not show the outline. This can be solved with an additional pass before the others to force the alpha to one. It would be nice to do this with the textured pass so we don’t have to do 4 passes, but that pass needs to write to the depth buffer, and we don’t want that before we’ve rendered the inset mask pass as it will likely be pushed “behind” the textured pass.

 

Wow, thank you so much