Multiple stencil operations per pass?

Hello, I suddenly have a need for performing multiple stencil operations in one pass, is this at all possible in Unity ShaderLab?

I know it is possible in opengl, but opengl in shaderlab is wonky at best.
I have already tried adding multiple stencil operations to the same pass, but the compiler only uses the last one.
Using the -Front and -Back options are not completely equivalent, and leads to some unacceptable artefacts.

 

Is it possible in opengl? Afaik, only the latest call to glStencilOp/Func/Mask takes effect. Maybe you mean how you can set different operations on pass and fail? You can do that in Unity as well...

Maybe it would help if you elaborated on what you're trying to achieve?

 

I will elaborate:
What I am attempting to achieve is a stencil-based light effect where the light only shows up when the geometry of the light intersects something else.
I do this by first running a back culled stencil pass, where (Stencil pass) puts a 1 into the stencil buffer, and the other keep the current value.
Next I run a front culled stencil pass on stencil values equal to 1, where a zfail leads to increasing the value to 2.
I then render a 2 in the stencil buffer as a light area to achieve the wanted effect.
This looks great most of the time, but fails if one of the meshes is too far under ground/inside a wall and one of the other meshes can "see" that part. (Illustrated in the photo).

The marked part consists of 1's generated by the bottom mesh, elevated to 2's by the zFail of the top mesh.

What I want to do is finish rendering one of these before moving on to the next.
They require two stencil passes to complete.

EDIT: Formatting

 

Exactly as you say, marking a volume in stencil requires two passes, even in openGL. Unity does support multiple passes per shader just fine... every Pass block defined inside a subshader is executed sequentially, so as your first pass, you could have a depth/stencil only pass without a fragment shader that just marks areas where front sides z-test fails and then a full pass where you apply your effect only on back facing polygons, when the z-test fails and when the stencil bit from the previous pass isn't on. The reason to stencil check "if previous test didn't fail" instead of "if previous test passed" is to support the case when camera is inside the mesh and the front face is not rendered at all.

 

Actually, that is exactly what I am doing, I have two shader passes.


I think the problem is that the first pass happens for all light objects, then the second pass happens.
I need both passes to happen for one light, then both passes to happen for the next one.

I did try adding a third pass that zeroed out anything from the first pass that was left over (still 1), but that did nothing.
This leads me to believe that the passes are done once for all objects using the shader before the next pass is started.
Of course it might just be me that's bad at shaders.

 

Hmm.. where and how are you reading the stencil and applying the effect?

 

I'm just applying it (Equal 2) as an additive alpha to a quad in front of the camera atm, but the plan is to use Graphics.Blit to get it as a Screen Effect.

Now here is the fun part:
I can get close to what I want using this:

These lights do not get any strange underground interactions.
But they get artefacts/errors at the edges. I do not think turning culling off works properly the way I am using it.

 

Hmm, it seems to me you're trying to use Comp in stencil to set the operation of the depth test.. that is done instead in the pass block with ZTest.

Unity also uses the stencil for some stuff, especially in deferred mode, so that might give you the wrong results. Try to use ReadMask and WriteMask to limit the operations only to the one bit you're interested in.

I assume you don't want any additive effects where two "light meshes" intersect and that's why you are applying it as a post processing effect... I don't think you need to actually. A faster (and easier way) would be to apply the light directly in the second pass and then making sure no other light renders at that point.. like this:

Code (CSharp):

1. Pass {
2.     // Render front facing geometry and mark occluded areas in stencil
3.     ZTest LEqual
4.     ZWrite Off
5.     Cull Back
6.     ColorMask 0
7.     Lighting Off
8.  
9.     Stencil {
10.         Ref 1
11.         Comp Always
12.         WriteMask 1
13.         ZFail Replace
14.     }
15. }
16.  
17. Pass {
18.     // Render back facing geometry and apply light only when it's occluded
19.     // and the previous front facing pass wasn't occluded, thus guaranteeing
20.     // that the light contains a mesh.
21.     // Then, if everything passes, mark the stencil so that no other light
22.     // gets applied to this pixel.
23.     ZTest Greater
24.     ZWrite Off
25.     Cull Front
26.  
27.     Stencil {
28.         Ref 1
29.         Comp NotEqual
30.         ReadMask 1
31.         WriteMask 1
32.         Pass Replace
33.         Fail Zero
34.     }
35.  
36.     CGPROGRAM
37.     // Your lighting shader here
38.     ENDCG
39. }

 

I already accounted for the deffered problem, I manually clear the stencil buffer after the geometry renderqueue using my screen-quad.

I tested your solution, it still has the same problem:


Changing the ZTest is only useful for flipping the meaning of Pass and ZFail.

You would not happen to know a trick that makes a shader perform all passes on an object before moving on to the next?

 

I'm pretty sure the default behavior is to render all passes of a single object before moving on to the next... Oh! I think I know whats happening! Try turning off dynamic batching

Edit: If that doesn't work for some reason, I suppose you could instead draw the spheres in OnPostRender of the camera using DrawMeshNow()... it's a bit of work, but should be a bit faster than unity's default pipeline either way and gives you more control.

 

I tried turning off dynamic batching. I got one extra drawcall, but nothing else happened.

But I did figure out a solution that seems to work:
I have two types of light, static and dynamic.

Dynamic lights area drawn first, in the render queue area Geometry+2 to Geometry+59
They have one number each, so a max number of 57 dynamic lights per scene.
These render queue numbers are set by [ExecuteInEditMode] scripts, on the materials. (One material instance per light)
The dynamic lights can be as far into the geometry as we want, so they are perfect for moving around.

After dynamic lights, at Geometry+60 we draw all static lights. We just need to be careful when placing these, making sure the meshes are not too far into the surrounding geometry.
At Geometry+61 we draw regual shadows from directional lights, and leave out the spots marked in the stencil buffer.

 

Unity 5's command buffers seem a perfect fit for this sort of thing. It's on the blog if interested, otherwise I'm wondering why you'd do this approach - is there a specific reason - might be there's a better way.

 

What you mean is having different stencil settings for the front and back side of the triangles. That's called two sided stencil states. (It's very specifically two sided, not multiple operations.)

This is possible in Unity:

 

Command buffers seem like the exact thing I want. I am doing things this way to get "light areas" that interact with terrain in an interesting way.

I already tried doing this, for some reason I get strange artifacts at the edges: