Inverse Mask

Short term, you will get the ability to create a hole in a geometry generated with the upcoming vector API (which can be used as a mask). But there are no plans regarding cancelling the masking provided by an ancestor. I'll bring this idea to the attention of our product manager though. Thanks.

 

So if I understand correctly, as your circle shrinks, it reveals what's behind it. In this case, why not just draw the circle after the rectangle instead of using masking?

 

I am going to start from the beginning. Please let me know if any assumption are wrong.

I understand that you want to hide some elements when they are outside the circle. Do these elements are UI, meshes? both? This may affect the overall strategy. The UI may be rendered last, on top of the regular camera rendering for most default configuration.

If you want to fill the outside of a circle with a color, then there is multiple way to do it. You already found one, and you can use the same sprite in UI toolkit. You could also use a mesh in game with a sorting order telling unity to draw it after the other 3d elements. Another strategy would be to use the UI Toolkit Vector API to draw a large rectangle (larger than the screen) with a hole in it and scale that visual element. The mesh would be generated once, but it may be harder to find what size is visible and where it will be placed. Check the transform-origin if you are using this as you could set the origin of the scaling to be at the center of your circle. You could also regenerate the mesh at each frame with an exact shape. Overall, the render an object with a hole on top approach would work without masking: you are just filling pixels outside the circle after that something else rendered previously. This is not masking, as you are not preventing the rendering in some section.

So if you want to prevent the rendering on some pixels, (aka use real masking) then the mask is applied first and the shader use to render after will consider that information and ignore some areas. UI Tookit uses that when the overflow is hidden. This prevents text in a child element, for example, to appear outside an element, even if the element has rounded corners. But for this to work, the elements that need to be masked need to have that specific shader. You will not be able to mask the camera rendering, as UI toolkit will be rendered after that camera.

One third option, that would leave even more control, would be to use an intermediate render texture and do a copy, pixel by pixel, like you want it to happen. This has the disadvantage of being slower than any other method described above.