Question URP Visualize Stencil in Texture

Hi all!

In Unity 2020.3.17 and URP 10.6.0 my goal is to visualize the stencil buffer in a render texture to use for post processing.

I have two very simple shaders:
The first one just writes to the stencil buffer and the second one displays a solid color when the stencil test succeeds.

When testing it with a sphere it works as it should (see screenshot). There is a plane behind the visible geometry that uses the stencil shader to set the stencil value if it passes LEqual ZTest. The sphere is only visible for areas outside the geometry which is intended.

Now when I try to use a custom renderer feature to render a fullscreen mesh with the same material to a render texture and use the Frame Debugger to look at the texture, it does not seem to work at all and just displays black. The Feature is being executed after all other Rendering is done which is also intended.
Since I don't know much about RenderTextures or even Shaders in general the problem could be literally anything, from RenderTexture settings to lots of other stuff.

Here's my Renderer Feature code:

Code (CSharp):

1. using System;
2. using UnityEngine;
3. using UnityEngine.Rendering;
4. using UnityEngine.Rendering.Universal;
5.  
6. namespace MM.Rendering
7. {
8.     [Serializable]
9.     internal class BlackBoxSettings
10.     {
11.         [SerializeField] internal RenderPassEvent m_eRenderPassEvent = RenderPassEvent.AfterRenderingOpaques;
12.         [SerializeField] internal Material m_stencilColorMaterial;
13.  
14.         public RenderTexture m_renderTexture;
15.     }
16.  
17.     internal class BlackBoxPass : ScriptableRenderPass
18.     {
19.         private BlackBoxSettings m_settings;
20.         private RenderTextureDescriptor m_renderTextureDescriptor;
21.         private RenderTargetIdentifier m_stencilTextureTarget = new RenderTargetIdentifier(s_stencilTextureID, 0, CubemapFace.Unknown, RenderTargetIdentifier.AllDepthSlices);
22.  
23.         private static readonly int s_stencilTextureID = Shader.PropertyToID("_StencilTexture");
24.  
25.         public BlackBoxPass(BlackBoxSettings _settings)
26.         {
27.             m_settings = _settings;
28.             renderPassEvent = m_settings.m_eRenderPassEvent;
29.             ConfigureInput(ScriptableRenderPassInput.Depth);
30.         }
31.  
32.         public override void OnCameraSetup(CommandBuffer cmd, ref RenderingData renderingData)
33.         {
34.             m_renderTextureDescriptor = renderingData.cameraData.cameraTargetDescriptor;
35.             //m_renderTextureDescriptor.msaaSamples = 1;
36.             m_renderTextureDescriptor.depthBufferBits = 0;
37.             m_renderTextureDescriptor.colorFormat = RenderTextureFormat.ARGB32;
38.  
39.             cmd.GetTemporaryRT(s_stencilTextureID, m_renderTextureDescriptor, FilterMode.Bilinear);
40.  
41.             ConfigureTarget(s_stencilTextureID);
42.             ConfigureClear(ClearFlag.None, Color.white);
43.         }
44.  
45.         public override void Execute(ScriptableRenderContext context, ref RenderingData renderingData)
46.         {
47.             CommandBuffer cmd = CommandBufferPool.Get("BlackBox");
48.  
49.             cmd.SetRenderTarget(
50.                 m_stencilTextureTarget,
51.                 RenderBufferLoadAction.Load,
52.                 RenderBufferStoreAction.Store,
53.                 m_stencilTextureTarget,
54.                 RenderBufferLoadAction.Load,
55.                 RenderBufferStoreAction.Store
56.                 );
57.             cmd.DrawMesh(RenderingUtils.fullscreenMesh, Matrix4x4.identity, m_settings.m_stencilColorMaterial, 0, 0);
58.  
59.             context.ExecuteCommandBuffer(cmd);
60.             CommandBufferPool.Release(cmd);
61.         }
62.  
63.         public override void OnCameraCleanup(CommandBuffer cmd)
64.         {
65.             cmd.ReleaseTemporaryRT(s_stencilTextureID);
66.         }
67.     }
68.  
69.     internal class BlackBoxRenderFeature : ScriptableRendererFeature
70.     {
71.         [SerializeField] private BlackBoxSettings m_settings;
72.  
73.         private BlackBoxPass m_blackBoxPass;
74.  
75.         public override void AddRenderPasses(ScriptableRenderer renderer, ref RenderingData renderingData)
76.         {
77.             renderer.EnqueuePass(m_blackBoxPass);
78.         }
79.  
80.         public override void Create()
81.         {
82.             m_blackBoxPass ??= new BlackBoxPass(m_settings);
83.         }
84.     }
85. }

I would greatly appreciate any help on this!

Cheers

 

You need a depth buffer to use stencils, because the stencil buffer is part of the depth buffer. Using 0 for the depth buffer bits disables the depth buffer, that needs to be 24.

 

Hi and thank you for your reply!

I realized my code wasn't actually showing the texture on screen and also took your advice into account and modified my code as follows:

Code (CSharp):

1. public override void OnCameraSetup(CommandBuffer cmd, ref RenderingData renderingData)
2.         {
3.             m_renderTextureDescriptor = renderingData.cameraData.cameraTargetDescriptor;
4.             m_renderTextureDescriptor.depthBufferBits = 24;
5.             m_renderTextureDescriptor.colorFormat = RenderTextureFormat.ARGB32;
6.  
7.             cmd.GetTemporaryRT(s_stencilTextureID, m_renderTextureDescriptor, FilterMode.Bilinear);
8.  
9.             ConfigureTarget(s_stencilTextureID);
10.             ConfigureClear(ClearFlag.None, Color.white);
11.         }
12.  
13.         public override void Execute(ScriptableRenderContext context, ref RenderingData renderingData)
14.         {
15.             CommandBuffer cmd = CommandBufferPool.Get("BlackBox");
16.  
17.             cmd.SetRenderTarget(
18.                 m_stencilTextureTarget,
19.                 RenderBufferLoadAction.Load,
20.                 RenderBufferStoreAction.Store,
21.                 m_stencilTextureTarget,
22.                 RenderBufferLoadAction.Load,
23.                 RenderBufferStoreAction.Store
24.                 );
25.  
26.             cmd.DrawMesh(RenderingUtils.fullscreenMesh, Matrix4x4.identity, m_settings.m_stencilColorMaterial, 0, 0);
27.             Blit(cmd, m_stencilTextureTarget, m_sourceID);
28.             context.ExecuteCommandBuffer(cmd);
29.  
30.             CommandBufferPool.Release(cmd);
31.         }

m_sourceID is the renderer.cameraColorTarget.

Now it actually display a black screen and the Frame Debugger shows the RenderTexture as pure black so something else must be amiss here. I am very unsure about the ConfigureTarget(). ConfigureClear() and cmd.SetRenderTarget() calls because these are pretty much copy pasted from the SSAO feature since I barely know what I am doing here.

 

Unfortunately that I can't help you with as I haven't touched the URP at all.

 

That's unfortunate, thank you for your input though!

 

Not sure how helpful this is, but this is what I use to draw it to an RT for shadows:

Code (CSharp):

1.         Pass
2.         {
3.             Name "CopyStencilBuffer"
4.             Tags { "LightMode" = "SRPDefaultUnlit" }
5.             Stencil { Comp NotEqual Ref 0 }
6.             Cull Off
7.             ZTest Always
8.             ZWrite Off
9.             Blend One One
10.             HLSLPROGRAM
11.                 #pragma vertex FullscreenVert
12.                 #pragma fragment copyStencilBuffer_PS
13.                 #include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Core.hlsl"
14.                 #include "Packages/com.unity.render-pipelines.universal/Shaders/PostProcessing/Common.hlsl"
15.                 uniform float4 _burningmime_lightChannel; // this needs to be set as a global for some rasin
16.                 float4 copyStencilBuffer_PS() : SV_Target { return _burningmime_lightChannel; }
17.             ENDHLSL
18.         }

For debugging these types of things, generally you need to either use the frame debugger or RenderDoc and see exactly what's happening at each step of the way.

 

Thank you for your input! Can you explain to me how you use this?
When I expose the color as a Property and use this in a material in my current setup, it still displays a black screen.
If I use it on my test sphere it works as it should though!

I think debugging something stencil related is difficult even with the Frame Debugger because you can't really check if the stencil has the values you expect it to have. My custom render pass renders last and that's about everything I can take away from the Frame Debugger.

 

Replace this part:

Code (CSharp):

1.                 uniform float4 _burningmime_lightChannel; // this needs to be set as a global for some rasin
2.                 float4 copyStencilBuffer_PS() : SV_Target { return _burningmime_lightChannel; }

With this:

Code (CSharp):

1.                 float4 copyStencilBuffer_PS() : SV_Target { return float4(1, 1, 1, 1); }

In a RenderPass I believe that material properties aren't updated. So you have to set them as globals in the command buffer eg

cmd.SetGlobalVector

. But that code is just an example, you can customize it how you need.

I use RenderDoc (free, and integrated with Unity). To see stencil buffer...



It's a bit of a PITA especially since all the Unity Editor stuff also appears in RenderDoc, but it works to show the Stencil.

 

RenderDoc seems great, I will keep using it. Thanks for the recommendation!

I can see that in my Depth Texture the Stencil values are as expected and I am now setting the Color as a global vector. It is still drawing a black screen and now I am convinced my C# pass code is doing something wrong. A wrong blit, clear, target, anything. I feel like I am close now.
Do you have any input on my C# pass? I am setting the Color before setting the Render Target now.

The screenshot shows the depth texture with correct stencil values. I just feel like it's still not used correctly.

 

I notice in your pass you don't have Configure() and are doing stuff in OnCameraSetup() (and then clearing/setting the RT again in draw). I'm not sure it's totally needed, but generally I try to structure my passes like this:

Code (CSharp):

1.  
2. private readonly ProfilingSampler _profilingSampler;
3. private Material _objectsMaterial;
4. private RenderTargetHandle _objectsRT;
5.  
6. public HighlightObjectsPass(HighlightService owner)
7. {
8.     // load your resources and init rendertarget handles
9.     _objectsRT.Init(NAME_OBJECTS_TEX);
10.     _objectsMaterial = new Material(Inject.get<GraphicsResources>().HighlightObjects);
11.     _profilingSampler = new ProfilingSampler(GetType().Name);
12.     renderPassEvent = RenderPassEvent.AfterRenderingOpaques;
13. }
14.  
15. public override void Configure(CommandBuffer cmd, RenderTextureDescriptor cameraTextureDescriptor)
16. {
17.     // configure your render targets and set the target/clear flags with these methods
18.     RenderTextureDescriptor objectsRTDescriptor = GraphicsUtil.describeRT(cameraTextureDescriptor, RenderTextureFormat.R8);
19.     cmd.GetTemporaryRT(_owner._objectsRT.id, objectsRTDescriptor, FilterMode.Point);
20.     ConfigureTarget(_owner._objectsRT.Identifier());
21.     ConfigureClear(ClearFlag.Color, Color.black);
22. }
23.  
24. public override void Execute(ScriptableRenderContext ctx, ref RenderingData rd)
25. {
26.     // managing profiling and command buffer here in one place
27.     CommandBuffer cmd = CommandBufferPool.Get();
28.     using(new ProfilingScope(cmd, _profilingSampler))
29.         executeImpl(ctx, cmd, ref rd);
30.     ctx.ExecuteCommandBuffer(cmd);
31.     CommandBufferPool.Release(cmd);
32. }
33.  
34. private void executeImpl(ScriptableRenderContext ctx, CommandBuffer cmd, ref RenderingData rd)
35. {
36.     // actually drawing stuff -- in your case it would just be this part:
37.     cmd.DrawMesh(RenderingUtils.fullscreenMesh, Matrix4x4.identity, m_settings.m_stencilColorMaterial, 0, 0);
38. }
39.  
40. public override void FrameCleanup(CommandBuffer cmd)
41. {
42.     cmd.ReleaseTemporaryRT(_owner._objectsRT.id);
43. }

(that won't compile as-is, since it's cobbled together from several files and modified, but it should give you the idea).

AFAIK, OnCameraSetup() is called very early in the pipeline. The ConfigureXXX() methods are done in the Configure() override, and then you can assume the render target is already set and cleared when it gets to Execute().

 

I really appreciate your help and patience on this. I modified my renderpass code to the following:

Code (CSharp):

1. [Serializable]
2.     internal class BlackBoxSettings
3.     {
4.         [SerializeField] internal RenderPassEvent m_eRenderPassEvent = RenderPassEvent.AfterRenderingOpaques;
5.         [SerializeField] internal Material m_stencilColorMaterial;
6.         [SerializeField] internal Color m_color = Color.red;
7.     }
8.  
9.     internal class BlackBoxPass : ScriptableRenderPass
10.     {
11.         private readonly ProfilingSampler m_profilingSampler;
12.         private BlackBoxSettings m_settings;
13.         private RenderTextureDescriptor m_renderTextureDescriptor;
14.         private RenderTargetIdentifier m_sourceID;
15.         private RenderTargetHandle m_renderTargetHandle;
16.  
17.         private static readonly int s_stencilColorID = Shader.PropertyToID("_StencilColor");
18.  
19.         public BlackBoxPass(BlackBoxSettings _settings)
20.         {
21.             m_profilingSampler = new ProfilingSampler(GetType().Name);
22.             m_settings = _settings;
23.             m_renderTargetHandle.Init("_StencilTexture");
24.             renderPassEvent = _settings.m_eRenderPassEvent;
25.         }
26.  
27.         public void SetSource(RenderTargetIdentifier _source)
28.         {
29.             m_sourceID = _source;
30.         }
31.  
32.         public override void Configure(CommandBuffer cmd, RenderTextureDescriptor cameraTextureDescriptor)
33.         {
34.             m_renderTextureDescriptor = cameraTextureDescriptor;
35.             m_renderTextureDescriptor.depthBufferBits = 24;
36.             m_renderTextureDescriptor.colorFormat = RenderTextureFormat.ARGB32;
37.             cmd.GetTemporaryRT(m_renderTargetHandle.id, m_renderTextureDescriptor, FilterMode.Point);
38.  
39.             ConfigureInput(ScriptableRenderPassInput.Depth);
40.             ConfigureTarget(m_renderTargetHandle.Identifier());
41.             ConfigureClear(ClearFlag.Color, Color.white);          
42.         }
43.  
44.         public override void Execute(ScriptableRenderContext context, ref RenderingData renderingData)
45.         {
46.             CommandBuffer cmd = CommandBufferPool.Get("BlackBox");
47.  
48.             using (new ProfilingScope(cmd, m_profilingSampler))
49.             {
50.                 cmd.SetGlobalVector(s_stencilColorID, m_settings.m_color);
51.  
52.                 cmd.SetRenderTarget(
53.                     m_renderTargetHandle.Identifier(),
54.                     RenderBufferLoadAction.DontCare,
55.                     RenderBufferStoreAction.Store,
56.                     m_renderTargetHandle.Identifier(),
57.                     RenderBufferLoadAction.Load,
58.                     RenderBufferStoreAction.Store
59.                     );
60.  
61.                 cmd.DrawMesh(RenderingUtils.fullscreenMesh, Matrix4x4.identity, m_settings.m_stencilColorMaterial, 0, 0);
62.                 //Blit(cmd, m_renderTargetHandle.Identifier(), m_sourceID);
63.             }
64.  
65.             context.ExecuteCommandBuffer(cmd);
66.             CommandBufferPool.Release(cmd);
67.         }
68.  
69.         public override void FrameCleanup(CommandBuffer cmd)
70.         {
71.             cmd.ReleaseTemporaryRT(m_renderTargetHandle.id);
72.         }
73.     }
74.  
75.     internal class BlackBoxRenderFeature : ScriptableRendererFeature
76.     {
77.         [SerializeField] private BlackBoxSettings m_settings;
78.  
79.         private BlackBoxPass m_blackBoxPass;
80.  
81.         public override void AddRenderPasses(ScriptableRenderer renderer, ref RenderingData renderingData)
82.         {
83.             m_blackBoxPass.SetSource(renderer.cameraColorTarget);
84.             renderer.EnqueuePass(m_blackBoxPass);
85.         }
86.  
87.         public override void Create()
88.         {
89.             m_blackBoxPass = new BlackBoxPass(m_settings);
90.         }
91.     }

I cleaned up my test scene to only include two objects:
The visible plane rendering with a Lit shader in queue 2000.
And a huge mesh behind the plane rendering with the stencil mask shader on 2001.
Thus, the expected result in my _StencilTexture would be to have everything red (color set in settings) except where the foreground plane is, which should be white due to the clear settings.

In RenderDoc I can now see though that the Stencils are correct until my own pass starts and then I somehow seem to lose the depth information.

 

First, you can remove this part entirely; the ConfigureTarget() part already does it for you:

Code (CSharp):

1.  
2.                 cmd.SetRenderTarget(
3.                     m_renderTargetHandle.Identifier(),
4.                     RenderBufferLoadAction.DontCare,
5.                     RenderBufferStoreAction.Store,
6.                     m_renderTargetHandle.Identifier(),
7.                     RenderBufferLoadAction.Load,
8.                     RenderBufferStoreAction.Store
9.                     );

Other than that, it looks like you're binding to a brand new target but haven't drawn anything to its stencil buffer. The stencil is not global; every depth buffer has its own stencil (or not depending on how it's configured). So when you set the render target to m_renderTargetHandle, you're looking at that stencil buffer (which is missing because you only have a 24-bit depth buffer; if you wanted a new stencil you'd need a 32-bit depth buffer and a different depth format I think -- but that doesn't seem like what you want, it seems like you want to read stencil values already written to the main depth buffer).

I'm not 100% sure this works on all platforms (I've only tested it on Windows DX11, on my own laptop, with Unity 2021.2, URP forward rendering, MSAA off, and a cat sitting right in front of the fan), but I was able to bind to the scene depth stencil like this:

Code (CSharp):

1. // In class...
2. private const string UNITY_DEPTH_PREPASS_TEXTURE_NAME = "_CameraDepthTexture";
3. private RenderTargetHandle _rtDepthPrepass;
4.  
5. // In constructor...
6. _rtDepthPrepass.Init(UNITY_DEPTH_PREPASS_TEXTURE_NAME);
7.  
8. // In Configure()...
9. ConfigureTarget(m_renderTargetHandle.Identifier(), _rtDepthPrepass.Identifier());
10. ConfigureInput(ScriptableRenderPassInput.Depth);

That's just magically assuming the _CameraDepthTexture exists and will have a valid stencil buffer, which I'm not sure is offficial for all platforms.

However, some things are going to be topsy-turvy-upside down but not all of it (and that's going to change whether MSAA is on or not. If you get the other stuff working, but it looks like the scene is upside down, I might be able to give you a couple pointers, although my solution was basically "do

1-uv.y

or

positionCS.y*=-1

on random things in the shaders, test, see if it works, if it doesn't try it on a different set of things, rinse and repeat for 4 hours".

 

Getting closer and closer, gaining understanding bit by bit.

You are correct. I want to read stencil values from the main depth buffer and for now just draw a solid color into my own RenderTexture wherever the stencil is not 0 or has the exact value I need using my stencil color material.
I will worry about other platforms or MSAA later.

Using your suggested changes the depth texture is now successfully preserved, but it somehow seems to lose all stencil information once my own pass starts (see screenshots).

This is my code now:

Code (CSharp):

1. public override void Configure(CommandBuffer cmd, RenderTextureDescriptor cameraTextureDescriptor)
2.         {
3.             m_rtDescriptorStencilColor = cameraTextureDescriptor;
4.             m_rtDescriptorStencilColor.depthBufferBits = 32;
5.             m_rtDescriptorStencilColor.msaaSamples = 1;
6.             m_rtDescriptorStencilColor.colorFormat = RenderTextureFormat.ARGB32;
7.             cmd.GetTemporaryRT(m_rtStencilColor.id, m_rtDescriptorStencilColor, FilterMode.Point);
8.  
9.             ConfigureClear(ClearFlag.Color, Color.white);
10.             ConfigureInput(ScriptableRenderPassInput.Depth);
11.             ConfigureTarget(m_rtStencilColor.Identifier(), m_rtDepthPrepass.Identifier());
12.         }
13.  
14.         public override void Execute(ScriptableRenderContext context, ref RenderingData renderingData)
15.         {
16.             CommandBuffer cmd = CommandBufferPool.Get("BlackBox");
17.  
18.             using (new ProfilingScope(cmd, m_profilingSampler))
19.             {
20.                 cmd.SetGlobalVector(s_stencilColorID, m_settings.m_color);
21.                 cmd.DrawMesh(RenderingUtils.fullscreenMesh, Matrix4x4.identity, m_settings.m_stencilColorMaterial, 0, 0);
22.  
23.                 //Blit(cmd, m_rtDepthPrepass.Identifier(), m_rtStencilColor.Identifier(), m_settings.m_stencilColorMaterial, 0);
24.                 //Blit(cmd, m_rtStencilColor.Identifier(), m_sourceID);
25.             }
26.  
27.             context.ExecuteCommandBuffer(cmd);
28.             CommandBufferPool.Release(cmd);
29.         }

Also, as you can see I tried just blitting the depth texture into my own rendertexture using the same material instead of drawing a fullscreen mesh. Is there a notable difference between the two methods?

 

At this point I'm a little out of my depth, since I only needed the depth values and not the stencil itself. Are you seeing the depth and it's just the stencil being cleared or are both missing when your pass begins? You may need to ping someone from the URP team or post over on that forum.

One way to get it work is just to draw those renderers again in your custom pass with a replacement shader. Obviously, this would be slow if there are many, but if there are only a few then it's not a problem.

Blitting won't work; you need to actually draw something with the stencil buffer bound. That

 cmd.DrawMesh(RenderingUtils.fullscreenMesh, ...)

*should* work fine if the stencil has been filled.

 

Ha, nice one!

Yeah, the depth is still there, it's just the stencil that's missing.

I guess I will post over on the URP forum.
Huge thanks to you for helping me get this far!

My post in the URP forum:
https://forum.unity.com/threads/visualize-stencil-in-rendertexture.1199779/#post-7667200

 

I'm not familiar with URP but maybe I can offer some thoughts (I deal with srp a lot

In srp, you can draw stencil buffer with a custom Blit (like what they have in Blitter.cs in HDRP, you can just copy the code there)

Then, when you set the stencil texture, the typical command buffer set up requires you to add a command "
cmd.SetGlobalTexture(ShaderKeywordManager.MAIN_TEXTURE, src, RenderTextureSubElement.Stencil);
"
The most important thing is the last param, which you explicitly tell unity I want the stencil component of the RT.

From your code snippet, I suspect there may be a way for you to declare that you want to use the Stencil component when binding the resources in the render graph.

Hope this helps! I know dealing with stencil/srp is painful bruh

 

Hi and thank you for your reply!
My main problem was even before drawing the stencil, I needed to access it and I didn't know where to take it from.

If I copy the _CameraColorTexture into my own RT and then render the fullscreen mesh with my stencil shader I almost get what I need. The area around the plane is white (stencil matches) but I need everything else to be black.
As described in my URP Forum Post here: https://forum.unity.com/threads/visualize-stencil-in-rendertexture.1199779/#post-7673140