Question How to make a custom stencil mask with a MeshRenderer that works like SpriteMask?

I'm trying to make a version of SpriteMask with a custom procedural mesh instead of a Sprite, but I don't know how SpriteMask fully works in order to replicate it.

To my knowledge, SpriteMask internally renders 2 sprites at 2 different sorting layers with 2 different materials, one material to write stencil=1 at the front sorting layer, and one rendering the same sprite to clear the stencil buffer at the back sorting layer.

But they do it internally so the renderer/shader/material are all hidden. Even if I look at the SpriteMask shader, there's no reference to stencil, so I don't even know how they do it.

So how would I even do it?

I thought I could use Graphics.DrawMesh instead of duplicating the same mesh&meshrenderer into a child gameobject, but I don't know how to include Sorting Layer into Graphics.DrawMesh.

The shader itself should be pretty simple. It just needs to write to stencil, nothing special since it's supposed to be invisible.

I could potentially have done it with the same meshrenderer that just uses 2 materials, but then I wouldn't be able to leverage different Sorting Layers on the meshrenderer, so I figured I would need to do it programmatically with Graphics.DrawMesh or with a 2nd meshrenderer, but both seem awkward.

Any ideas?

 

Unity's sprite rendering forcibly overrides the stencil render state when rendering sprites, which is why the shader doesn't have anything related in their shaders.

While you could write a shader that's basically identical to the Sprite-Mask.shader with the stencil settings exposed, there's an easier option. You can replace the sprite's mesh.

The Sprite class includes an array of triangle indices, and vertex positions. You can override those using the

OverrideGeometry()

function (the UVs will be auto-generated, though they can be overridden as well). Then use that as your sprite in the sprite renderer.
https://docs.unity3d.com/2019.1/Documentation/ScriptReference/Sprite-vertices.html

 

Oh interesting so I can make a spline script that triangulates a fill mesh inside the spline, and then OverrideGeometry the SpriteMask sprite instance with that generated mesh? That's actually really clever, thanks!

It's too bad SpriteShape doesn't seem to allow access to the verts and triangles of its fill mesh, because that would be a convenient way to generate the mesh to override the SpriteMask mesh instead of needing to make my own spline mesh generator.