Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

HDRP: Rendering to stencil buffer?

Discussion in 'Graphics Experimental Previews' started by GlaireDaggers, Feb 27, 2019.

  1. GlaireDaggers

    GlaireDaggers

    Joined:
    Feb 18, 2018
    Posts:
    13
    Hi,
    I'm trying to experiment with visual styles in Unity's HDRP pipeline. One thing I'm looking at implementing is a stylized outline shader, probably as a post effect edge detection shader using depth+normals.
    Thing is, I'd like to see if I can keep the outline restricted to just characters and important objects, leaving background scenery alone. My initial thought was to use a stencil buffer mask for this so that the objects that need an outline just render a particular stencil bit and then the outline post effect shader masks on that bit.
    Is there a way to change what value an object renders into the stencil buffer, perhaps via the shader graph? Or will I have to write my own shader for this?
     
  2. GlaireDaggers

    GlaireDaggers

    Joined:
    Feb 18, 2018
    Posts:
    13
    Also I'm using the HDRP's normal buffers and the camera depth texture for this effect. It works for the most part aside from the fact that I haven't found a way to mask out outlines on the environment, BUT
    Another issue is that outlines just render right on top of transparent things if I have my post effect class marked as "Before Stack", but if I change it to "Before Transparent" it just vanishes entirely? :/
     
  3. GlaireDaggers

    GlaireDaggers

    Joined:
    Feb 18, 2018
    Posts:
    13
    So in the end I did sorta get this working by modifying the HDRP...

    I changed the type of the lighting buffer to ARGBHalf to give myself an alpha channel (takes up more memory though, couldn't find a float type with a smaller alpha :/)

    Then I changed the Lit material to have an _OUTLINE_ON keyword and an inspector toggle to match, and then changed the gbuffer encode/decode functions to pack 0 or 1 into the alpha of the lighting buffer depending on whether the material has this keyword enabled or not.
    Then I changed the deferred compute shader to pass this through to the final alpha channel of the main buffer (since I noticed nothing in my scene actually wrote to this alpha channel, it was pure white), and had my outline shader operate on that.

    I'm sure there's a better way to do this, but in the meantime it works and I can selectively enable outlines on a per-material basis.
     
    customphase and hippocoder like this.
  4. raduBenea

    raduBenea

    Joined:
    Mar 6, 2017
    Posts:
    2
    I could use your help with a similar problem. I want to preserve the alpha channel throughout the render process, and I don't know where to start. Your mention of changing the lighting buffer sounds like the way to go but how did you do it ?