Mask 3D Mesh using 2D Image

Hello,

I have been trying to mask a 3D object using a 2D sprite rendered on a Canvas using Screen Space - Camera with an Ortographic camera. This will result in the following (not masking).

However what I want to achieve is that only the part of the sphere overlapping with the 2D UI image will be rendered and visible. Something like this (this is done in photoshop to show an example what I want to achieve):


I have tried looking into Stencelling, but the problem I'm facing with that is that the mesh that will be the "mask" must be 3D and not able to scale based on the canvas.

Is this something that can be done or is this impossible to do within Unity UI and 3D meshes?

Kind regards,

Ramon

 

The problem you're having is the order of rendering. 3D mesh objects render based on their material's queue, and always before the UI components, regardless of if you're using a Screen Space - Overlay / Camera or a World Space Canvas. For stencils, or really any kind of masking to work, the UI in this case needs to render first.

I can think of two ways to handle this problem.

Option 1: Don't use Canvas & UI Components. Or maybe more explicitly don't try to set the stencil to mask the 3D objects using canvas elements. The trick is to have an mesh renderer that's using a custom shader that only writes to the stencil, and uses an earlier render queue than object you need masked. If you want this to line up with a UI element, you'd need to write a custom script to keep the mesh renderer's game object aligned with the screen space position of the UI. For an orthographic camera you might be able to get away with the mesh renderer mask being a child of the UI element, but it's not really going to be guaranteed to line up.

Option 2: Use a render texture. This is what I'm currently doing for a project. I render the object I want to be masked by the UI with a second camera that renders to a render texture that is displayed on a Raw Image in the Canvas. That second camera can only see the "hidden" objects, and is using a solid color background of (0,0,0,0) so it's transparent anywhere the hidden objects aren't. And now because it's just another image element in the UI it can be masked and manipulated any way I need it to be. You may still need some custom script work for this, as Unity's UI mask system will only mask child elements. So if the mask needs to move around on screen you'll need to counter-move the raw image showing the render texture.