Search Unity

Accessing the stencil buffer?

Discussion in 'Shaders' started by SunnySunshine, Sep 13, 2019.

  1. SunnySunshine

    SunnySunshine

    Joined:
    May 18, 2009
    Posts:
    673
    Is there any way to access the stencil buffer in Unity?

    I have this idea of using the stencil buffer as an ID map for applying certain effects on objects as a post process step, rather than having to draw the objects again into another buffer. So far I haven't seen any way to accomplish this, even though it should be possible in theory.
     
  2. SunnySunshine

    SunnySunshine

    Joined:
    May 18, 2009
    Posts:
    673
    After some research, it seems like this isn't possible to do.

    Here's another topic that's similar to mine:
    https://forum.unity.com/threads/_cameradepthtexture-g-to-access-the-stencil-values.432925/

    There are some topics that cover the use of stencils in post processing, but don't be fooled - they're not about actually accessing the stencil buffer. Rather, just about using the stencil buffer (ref) to decide where post fx is applied.

    It is possible to create an ID map from stencil buffer by drawing a quad (DrawMesh) with a material that uses the stencil buffer to decide what pixels to render (ref). That shader could then just write a color in the fragment shader. However, using this technique, you'd have to issue 1 draw call per color you'd want to write. Hardly ideal.

    Hope this saves somebody some time.
     
  3. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    7,160
    So, there's two problems.

    One is that the depth texture isn't the depth buffer (like I talk about in that thread). However the stencil buffer is still active during post processing, which is why it can be used to mask elements during post processing.

    The other bigger issue is that until Shader Model 5.1 (Direct3D 11.3 & Direct3D 12) it wasn't possible to get the stencil value directly in the fragment shader. Like you said, it could only be used as a mask to exclude pixels from being rendered. If you're using Direct3D 12, then you could actually do what you're looking to do.


    However the better solution would be to use custom shaders that output to multiple render targets.
     
    SunnySunshine likes this.
  4. SunnySunshine

    SunnySunshine

    Joined:
    May 18, 2009
    Posts:
    673
    Any more info on this? Is it possible to do in Unity?
     
  5. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    7,160
    Use Graphics.SetRenderTarget() with an array of color buffers, and have your shader write out to both SV_Target0 and SV_Target1.
     
    SunnySunshine likes this.
  6. SunnySunshine

    SunnySunshine

    Joined:
    May 18, 2009
    Posts:
    673
    omg, this may potentially change everything. Thank you so much for sharing that!
     
  7. SunnySunshine

    SunnySunshine

    Joined:
    May 18, 2009
    Posts:
    673