Question How to read value from the stencil buffer in hlsl?

Hi!

I'm pretty new to shaders/urp (so I'm still wrapping my head around a lot of concepts). I'm creating a custom outline shader in a full screen pass renderer feature (using Unity 2023.1.0) and was looking into ways of filtering what objects/pixels the outline is applied to/on top of. I've managed to do some filtering using the world height (world position y) of a the screen UV but it's not enough for what I want (you can see examples of the effect in the attached images, basically, I want the "fuzzy" outline to only be applied to the "cubes" in the center of the screen and nothing else).

After investigating for a bit I realized that the stencil buffer could be the perfect solution for deciding what objects to apply the outline to, ie:
1. create a material that sets stencil buffer to 1.
2. apply said material to all objects I want the outline applied to.
3. while running my sobel filter, check the stencil buffer for the UV I'm currently sampling. If buffer == 1, we run the sobel filter calculation; if not we just ignore it.

Thing is - like I said at the start - I'm pretty new to all of this so I'm unsure of how to go about it, if I'm doing anything wrong or if it's even possible. Documentation for `.hlsl` functions/functionality is pretty lackluster too so I'm struggling quite a bit. Here's my sobel filter code:

Code (CSharp):

1. void DepthSobel_float(float2 UV, float maxUVWidth, float widthSamples, float2 screenSize, float normalizedCameraZ, float depthThreshold, float depthThickness, float depthStrength, out float2 Out) {
2.     float res = 0;
3.     float noiseStep = 0;
4.     float minWidthSample = maxUVWidth / widthSamples;
5.     float width = screenSize.x;
6.     float height = screenSize.y;
7.  
8.     [unroll(10)] for (int i = 1; i <= widthSamples; i++) {
9.         float currentWidthSample = minWidthSample * i;
10.         float screenWidthUVSample = (currentWidthSample / width) / normalizedCameraZ;
11.         float screenHeightUVSample = (currentWidthSample / height) / normalizedCameraZ;
12.         float2 sobel = 0;
13.  
14.         [unroll] for (int j = 0; j < 9; j++)
15.         {
16.             float2 sobelSamplePoint = UV + GenSobelSamplePoint(j, screenWidthUVSample, screenHeightUVSample);
17.             // NOTE: thought I had something with this but it does not seem to work.
18.             float stencilValue = GetStencilValue(sobelSamplePoint);
19.             if (stencilValue != 0) continue;
20.  
21.             float depth = LinearEyeDepth(SHADERGRAPH_SAMPLE_SCENE_DEPTH(sobelSamplePoint),_ZBufferParams) + normalizedCameraZ;
22.             float smoothDepth = smoothstep(-100, 100, depth);
23.             sobel += smoothDepth * sobelMatrix[j];
24.         }
25.  
26.         float normalizedSobel = pow(smoothstep(0, depthThreshold, length(sobel)), depthThickness) * depthStrength;
27.  
28.         if (res < normalizedSobel)
29.         {
30.             res = normalizedSobel;
31.             noiseStep = i / widthSamples;
32.         }
33.     }
34.  
35.     Out = float2(res, noiseStep);
36. }

Would REALLY appreciate any help/pointers anyone can offer . Huge thanks in advance

 

Managed to sort this out using a custom render feature to create a global texture out of a single layer, using that inside the hlsl file as a filter on what to render.