Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Using Shader.SetGlobal fails for setting Stencil States

Discussion in 'Shaders' started by Noisecrime, Jun 19, 2017.

  1. Noisecrime

    Noisecrime

    Joined:
    Apr 7, 2010
    Posts:
    2,051
    Found a frustrating bug in Unity recently where I discovered that setting stencil states via Shader.SetGlobalInt() would fail to have any effect. It was strange as some others reported it had worked for them in the past, but then broken.

    Looking into the issue again today I happily stumbled upon the cause and got a workaround in the process.

    So first thing to remember when using Shader.SetGlobal to change values in a shader, is that the parameter name cannot be a property in the shaders property block. If the parameter is also a property, the property will override or cause the SetGlobalcommand to fail.

    More importantly due to Unity caching Shader Properties in the asset ( you can see this in the inspector if you switch to debug mode when viewing a material) it means that if you start with a property in the property block, then remove it to make it a global parameter, setGlobal will still fail as Unity thinks the property still exists!

    This caught me out for a quite a while, though the solution is simple, either create a new material or to use an editor script to remove unused properties.


    Setting Stencil Ref via SetGlobal is doomed to failure.

    So the problem is setting Stencil Ref state via SetGlobal tends to fail in most cases. In fact from testing I discovered that it will always fail unless you happen to update any other shader state at the same time! e.g

    Code (CSharp):
    1. SubShader
    2.     {
    3.         Tags { "RenderType"="Opaque" }
    4.         LOD 200
    5.  
    6.         Stencil
    7.         {
    8.             Ref [_GlobalDiffuseStencilRef]
    9.             Comp equal
    10.             Pass keep
    11.             Fail keep
    12.         }
    13. ...
    14. }
    will fail to update the Stencil Ref value via Shader.SetGlobalInt("_GlobalDiffuseStencilRef", value)

    whilst the following and adding Shader.SetGlobalInt("_GlobalZWrite", value) will work fine.
    Code (CSharp):
    1. SubShader
    2.     {
    3.         Tags { "RenderType"="Opaque" }
    4.         LOD 200
    5.  
    6.         // BugFix:
    7.         // Requires (any?) state setting to ensure that Stencil state updates via Shader.SetGlobal.
    8.         ZWrite [_GlobalZWrite]
    9.  
    10.         Stencil
    11.         {
    12.             Ref [_GlobalDiffuseStencilRef]
    13.             Comp equal
    14.             Pass keep
    15.             Fail keep
    16.         }
    17.  
    This suggests to me something wrong in Unity's code that for some reason fails to update stencil states unless another unrelated state has/is also updated.

    At least this is true for all the tests I've made so far and since its a frustrating bug I really hope the workaround here will work across the board.

    From testing this bug is present in 4.7.1, 5.4.2, and 2017.1.0b6.

    Unsure if the unrelated state has to be 'ZWrite', would be very surprised if it is, so I suspect any other state like cull, colorMask, ZTest etc will work, But I don't think setting anyo other stencil state would work, I think the entire stencil block gets ignored until you change another state.

    It may be you don't even need to set the other state, just have it present in the shader, though I'm not sure how that would work.

    I tested changing the other state using a property from the shader material block, instead of using Shader.SetGlobal on it, and that also failed, so the key seems to be setting a parameter that is unrelated to stencil states first.


    Edit: After further testing I've determined that

    Using ColorMask as the 'other' unrelated state does not work. So not all other states will work to force an update of Stencil Ref State. However even better I've discovered it doesn't have to be an unrelated state, in fact using any other stencil state will work. e.g.

    Code (CSharp):
    1. Stencil
    2.         {
    3.             Ref[_GlobalDiffuseStencilRef]
    4.             Comp[_GlobalDiffuseStencilCompare] // equal
    5.             Pass keep
    6.             Fail keep
    7.         }
    This will now work fine, no need for ZWrite global setting. I had assumed that since stencil ref state wasn't being updated that all the stencil states would fail, but that doesn't seem to be the case. It increasingly appears that SetGlobal only fails with stencil ref state and not any other stencil states and that the ref value can be forced to update as long as you also update another stencil state.

    Indeed I tested setting Ref in combination of one of Comp, Pass and Fail and in all cases setting at least one of those worked to ensure that the stencil ref was also updated. Yet stencil Ref on its own will always fail. I didn't test other stencil states ( Zfail, ReadMask, WriteMask ).

    This is pretty good news as it became apparent having to set another state such as ZWrite was going to be a problem as it interfered with other requirements of various shaders. Knowing now that you only have to set one other stencil state ( or at least one of Comp, Pass, Fail ) is much easier to work with, since if you are setting stencil ref at the global level chances are very high that the other stencil states will all be the same as well.
     
    Last edited: Jun 20, 2017
    Kronnect likes this.
  2. Kronnect

    Kronnect

    Joined:
    Nov 16, 2014
    Posts:
    2,894
    Your workaround is indeed still applicable to Unity 2019.2.
     
    Noisecrime likes this.