Search Unity

Question Stencil Rendering Question

Discussion in 'Shaders' started by lukamoon, May 13, 2023.

  1. lukamoon

    lukamoon

    Joined:
    Nov 15, 2018
    Posts:
    15
    Hello! First off, apologies for how bad I am with stencils, this is my "learning" project with them.. anyways,

    I'm writing a custom set of stencil shaders, where the main idea is that I have a shader that "filters" out everything except my chosen meshes (without using camera layers) - such as having my "filtered" shader only be able to read a player's character, a mesh that needs focused for a mission, etc... Right now it mostly works, except for when an object is intersecting or in front of the focused mesh. Here is the current setup, with the right-hand side of the screen/GIF being an output of the filtered shader for debugging. (https://i.imgur.com/uS2ZtPz; forums won't let me embed an Imgur video).

    As you can see, the cube behind the desired, focused mesh is filtered out successfully. The red sphere, which is the desired mesh, is all that the filtering shader can see - and that's what I want. However, as I move the red sphere either into the undesired cube, or behind it, you can see that the shape of the desired mesh is kept, but it's weirdly blended with whatever is obstructing its view to the camera. I'm also afraid of when I change rendering settings, there may be some issues with the mesh normally (ex. outside of the filtered shader, the sphere is on top of the cube, such as with ZTest Always.).

    The current setup:
    Cube for testing is just standard shader at 2000 render queue with no settings modified.
    The sphere (desired mesh) is using these stencil settings: Comp Always, Pass Replace, Fail Replace, and ZFail Replace. ZTest is LessEqual, and the Queue is at 4000.
    The filtering shader is using these stencil settings: Comp NotEqual, Pass Keep, Fail Replace, ZFail Replace. No ZTest is specified, and a rendering Queue of 5000.


    Ideally, I'd like to keep this all a shader-only setup and use rendering techniques - if it is even possible. Thanks!