Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[SOLVED kinda] The 5th bit Stencil Buffer - Deferred

Discussion in 'Shaders' started by flogelz, Jan 4, 2020.

  1. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    I have a postprocess effect, that should only be visible on specific objects. To achieve this, I read about the stencil buffer and how it is handled in deferred. The documentation states, that it would be possible to use the 5th bit of the buffer for this purpose, since the others are used already.

    I have these settings on my shader and think, I'm writing to the 5th bit already, but it doesn't mask anything.

    The object that writes to the buffer:
    Code (CSharp):
    1.         Stencil
    2.         {
    3.             Ref 32
    4.             WriteMask 32
    5.             Comp Always
    6.             Pass Replace
    7.         }
    The postProcess effect, that reads the mask:
    Code (CSharp):
    1.         Stencil
    2.         {
    3.             Ref 32
    4.             ReadMask 32
    5.             Comp Equal
    6.             Pass Keep
    7.         }
    The whole thing itself doesn't look that complicated, but maybe I'm missing something?
     
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,321
    It’s not complicated, and it would work ... if Unity hadn’t disabled all stencil writes during the gbuffer pass. It used to work just fine, then at some point they disabled stencils on deferred objects rendered normally, but you could still do stencils on stuff drawn via command buffers, but I think that got disabled too.
     
    FM-Productions and flogelz like this.
  3. flogelz

    flogelz

    Joined:
    Aug 10, 2018
    Posts:
    141
    @bgolus Ah, what a bummer. But thanks for clarifying this! (I wished they would mention it in the documentation tho.)

    I'm now using the the final g-buffer function on the objects I wanna mask and let them write a value to the unused alpha channel of the normal gBuffer. Not as sweet as stencils, but it does its job :)
     
  4. Waffle1434

    Waffle1434

    Joined:
    Jun 1, 2016
    Posts:
    5
    Unity giveth and Unity taketh away, truly tragic. I can understand preventing custom shaders unknowingly wreaking havoc on the stencil buffer, but at the very least they could have had a #pragma override or something.

    Deferred rendering already takes up enough memory, last thing I want to do is need an extra Render Texture, being able to squeeze everything out of what's already there is a must.
     
  5. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    714
    Wait.. when? We're on 2019.4 and stencil in deferred during gbuffer writes seems to work for me? We're using it for a custom decal system. Is that going to break when we update to 2020????
     
  6. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,321
    I assume you're using command buffers? It doesn't work for in scene mesh renderers, that's for sure. And it was broken as of 2019.2 or so for me. And I couldn't get command buffers to do it either in 2019.4. How exactly are you rendering your decals, and is the shader pass set to
    "LightMode"="Deferred"
    ? If it's working for you, then there might be a work around still.
     
  7. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    714
    Almost all materials in our game are using some variant of our custom standard shader, which is just a variant of Unity's standard shader. We're using code that looks like this:

    Code (CSharp):
    1.  
    2. Properties
    3. {
    4. ...
    5.     [IntRange] _StencilRef("Stencil Reference Value", Range(0,255)) = 0
    6. ...
    7. }
    8.  
    9. SubShader
    10. {
    11.     Tags { "RenderType" = "Opaque"  }
    12.  
    13.     Stencil
    14.     {
    15.     Ref[_StencilRef]
    16.     WriteMask[_StencilRef]
    17.     Comp Always
    18.     Pass Replace
    19.     }
    20. ...
    21. }
    Then, we build a CommandBuffer to render boxes with a special shader that actually renders decals (just samples against the gbuffers to figure it out, and uses stencil to compare if its allowed to draw at that pixel)

    We're using the value of 8 for "allowed" and anything else for "not allowed." Maybe there just happens to be something about that specific bit (0000 1000) that is okay?

    edit: note, it turned out this is still working only if you have the cull mask for lights set to "everything," once you start removing layer from lights things stop working as you'd expect. According to the unity docs, this is because the last four bits are reserved for those light culling masks. This is also why you can only mask out up to 4 layers!
     
    Last edited: Mar 28, 2021
    FM-Productions likes this.