Standard Shader with stencil mask.

So I was trying out stencil masks with the Standard shader. I made a slightly altered version of the Standard shader, just adding

Code (CSharp):

1.  
2.     Stencil {
3.             Ref [_StencilMask]
4.             Comp Equal
5.             Pass Keep
6.             Fail Keep
7.     }
8.  

to both sub-shaders (with '_StencilMask' a new Int-porperty).

This works, however shadows aren't being rendered properly. The object using this stencil mask shows the shadows from objects behind it.

The weird shadows:


(The cube in the center has my altered standard shader with the stencil mask. The see-through side of the outer box has a shader that sets the stencil mask.)

The shadows I would expect:

(Here I just changed the shader back to the normal standard shader.)

Why is this happening? Is it something I can fix?

I added the altered standard shader, as well as the shader that sets the stencil mask.

Thanks!

 

Bumb. Anyone from Unity that could take a look at this?

 

Did you ever figure this out?

 

Sadly no. Though thanks for the bump! Hopefully someone from Unity or so sees it now.

 

I gave it a try.
It seems ShadowCaster pass does not have proper access to the stencil buffer.
You may try adding another Stencil section with "Comp Always" inside the ShadowCaster pass and see what happens.
This will fix the shadow on the faces of the center cube where it is drawn, but gives weird shadow where it is not drawn.


You may also try changing the Platform to Android or iOS in the Build Settings window.
On these two platforms, it seems Unity does not use ShadowCaster pass, thus no weird shadow.

 

Hmm, those are interesting results. What does the ShadowCaster pass do? Since it seems there are still shadows on iOS and Android...

 

OK. I got it right.
It is wrong where I said "Unity does not use ShadowCaster pass (for Android and iOS)".
Unity is using ShadowCaster pass in two different ways for PC (and maybe for Mac and Linux aswell).

What matters is that the see-through side also has to be involved in the separate pass which creates DepthTexture.
Hence it also needs ShadowCaster pass.
And with tricky Stencil operation.

Code (ShaderLab):

1.         Stencil
2.         {
3.             Ref [_StencilMask]
4.             Comp Never
5.             Fail Replace
6.         }
7.  

To fully understand whats going on you should take a look inside Unity's FrameDebugger.
Following captures are taken after I fixed the shader (shader_StencilMask.shader).


Note: Unity version 5.3.3p1

 

I tried your updates shaders, but something is still wrong. I made a little project to show it. Just look around the cube a little from different angles, shadows are ok sometimes, but from other angles it's still broken.

 

anything new to add here? I'm looking to get masked postpro running on my project, but i have to rely on the standard shaders to maintain the look we got so far. Thus i figured, expanding the standard shader would be a good idea. It seems to still give the described errors though? Or are they gone by now?

 

I haven't played with this since then anymore, so as far as I know the bug is still there. This was several versions ago though, so it might have been fixed, but I never heard anything from Unity about this. (I send them a bug report through the editor)

 

i have tried your shader and it work properly now

 

Cool! Thanks for letting me know