Search Unity

  1. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice

Sprite Mask Sprite Mask inner workings?

Discussion in '2D Experimental Preview' started by ZimM, Sep 1, 2017.

  1. ZimM


    Dec 24, 2012
    The new Sprite Mask feature is very nice, but it is a black box. How exactly does it work? I can only guess that it makes heavy use of stencil buffer internally, but this isn't documented anywhere. Frame Debugger shows nothing related to the stencil when rendering masks or masked sprites, and sprite shaders have no mention of stencil as well.
    How safe is it to use this feature if I'm already using stencil extensively? It seems to be causing some troubles (read the first 2 paragraphs):

    Also, is there any good way to use Sprite Mask with MeshRenderer? I mean, I can write a shader that will use the stencil mask, but for that, I must at least know what values are being written into stencil.

  2. Sergi_Valls


    Unity Technologies

    Dec 2, 2016
    Hi, let me help clarify this:

    SpriteMask component will render the masking sprite twice. The first time it will increment the stencil buffer. The second time it will decrement (FrameDebug does show that information, check the Stencil Pass for IncrementSaturate/DecrementSaturate).
    SpriteRenderer offers an easy way to configure the interaction with the StencilBuffer (internally we are setting stencil states for you):
    - VisibleInsideMask will set StencilRef value to 1 and a CompareFunc to LessEqual. That means will render pixels where 1 is less or equal to the value of the StencilBuffer.
    - VisibleOutsideMask will set StencilRef value to 1 and a CompareFunc to Greater. That means will render pixels where 1 is greater to the value of the StencilBuffer.
    - None (no interaction) will use the stencil states defined by the material.

    You can create your own Shader in order to interact with SpriteMasks. All you need to do is to set the right stencil states:

    Code (CSharp):
    1. Shader "Custom/SpriteMaskInteraction"
    2. {
    3.     Properties
    4.     {
    5.         [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
    6.         _Color ("Tint", Color) = (1,1,1,1)
    7.         [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0
    8.         [HideInInspector] _RendererColor ("RendererColor", Color) = (1,1,1,1)
    9.         [HideInInspector] _Flip ("Flip", Vector) = (1,1,1,1)
    10.         [PerRendererData] _AlphaTex ("External Alpha", 2D) = "white" {}
    11.         [PerRendererData] _EnableExternalAlpha ("Enable External Alpha", Float) = 0
    12.     }
    14.     SubShader
    15.     {
    16.         Tags
    17.         {
    18.             "Queue"="Transparent"
    19.             "IgnoreProjector"="True"
    20.             "RenderType"="Transparent"
    21.             "PreviewType"="Plane"
    22.             "CanUseSpriteAtlas"="True"
    23.         }
    25.         Cull Off
    26.         Lighting Off
    27.         ZWrite Off
    28.         Blend One OneMinusSrcAlpha
    30.         Pass
    31.         {
    32.             Stencil {
    33.                 Ref 1  //Customize this value
    34.                 Comp Equal //Customize the compare function
    35.                 Pass Keep
    36.             }
    38.         CGPROGRAM
    39.             #pragma vertex SpriteVert
    40.             #pragma fragment SpriteFrag
    41.             #pragma target 2.0
    42.             #pragma multi_compile_instancing
    43.             #pragma multi_compile _ PIXELSNAP_ON
    44.             #pragma multi_compile _ ETC1_EXTERNAL_ALPHA
    45.             #include "UnitySprites.cginc"
    46.         ENDCG
    47.         }
    48.     }
    49. }
    lassade, mitaywalle, Djayp and 2 others like this.
  3. ZimM


    Dec 24, 2012
    Thanks for the detailed answer!

    However, here's how the Frame Debugger looks like for a simple scene with a mask and an object:
    There is nothing regarding Stencil anywhere. Also, this block is missing from built-in Sprites shaders, so how does it work? Is Unity magically setting the stencil state under the hood for SpriteRenderer only?
    Code (CSharp):
    1.             Stencil {
    2.                 Ref 1  //Customize this value
    3.                 Comp Equal //Customize the compare function
    4.                 Pass Keep
    5.             }
    Also, it seems like the masking system doesn't work at all for opaque sprites. I often use an opaque sprite shader with SpriteRenderer that uses z-test and does z-write to reduce overdraw.
    Code (CSharp):
    2. Shader "Sprites/Opaque"
    3. {
    4.     Properties
    5.     {
    6.         [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
    7.         _Color ("Tint", Color) = (1,1,1,1)
    8.         [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0
    9.     }
    11.     SubShader
    12.     {
    13.         Tags
    14.         {
    15.             "Queue" = "Geometry"
    16.             "IgnoreProjector"="True"
    17.             "RenderType"="Opaque"
    18.             "PreviewType"="Plane"
    19.             "CanUseSpriteAtlas"="True"
    20.         }
    22.         Cull Off
    23.         Lighting Off
    24.         ZWrite On
    25.         ZTest LEqual
    26.         Fog { Mode Off }
    27.         Blend Off
    29.         Pass
    30.         {
    32.         CGPROGRAM
    33.             #pragma vertex vert
    34.             #pragma fragment frag
    35.             #pragma multi_compile DUMMY PIXELSNAP_ON
    36.             #include "UnityCG.cginc"
    38.             struct appdata_t
    39.             {
    40.                 float4 vertex   : POSITION;
    41.                 float4 color    : COLOR;
    42.                 float2 texcoord : TEXCOORD0;
    43.             };
    45.             struct v2f
    46.             {
    47.                 float4 vertex   : SV_POSITION;
    48.                 fixed4 color    : COLOR;
    49.                 half2 texcoord  : TEXCOORD0;
    50.             };
    52.             fixed4 _Color;
    54.             v2f vert(appdata_t IN)
    55.             {
    56.                 v2f OUT;
    57.                 OUT.vertex = UnityObjectToClipPos(IN.vertex);
    58.                 OUT.texcoord = IN.texcoord;
    59.                 OUT.color = IN.color * _Color;
    60.                 #ifdef PIXELSNAP_ON
    61.                 OUT.vertex = UnityPixelSnap (OUT.vertex);
    62.                 #endif
    64.                 return OUT;
    65.             }
    67.             sampler2D _MainTex;
    69.             fixed4 frag(v2f IN) : SV_Target
    70.             {
    71.                 fixed4 c = tex2D(_MainTex, IN.texcoord) * IN.color;
    72.                 return c;
    73.             }
    74.         ENDCG
    75.         }
    76.     }
    77. }
    However, when I use this shader, the masking stops working. It seems to be that way because Sprites-Mask shader uses hardcoded Transparent queue. How can I change the render queue of the mask? Is it possible to use a custom shader for the mask itself?

    I've also tried it with a MeshRenderer and it works fine... if the shader uses a transparent render queue.
    Last edited: Sep 7, 2017
  4. Sergi_Valls


    Unity Technologies

    Dec 2, 2016
    About the FrameDebugger, my bad, 2017.1 does not show the information. It will in next versions.
    Yes, we are setting stencil states for SpriteRenderer and ParticleSystemRenderer only.
    SpriteMask is like any other renderer, you can access its material, change it or change its render queue.
    I will check if we can solve some of the issues. Thanks for your feedback!
    mitaywalle likes this.
  5. amisner2k


    Jan 9, 2017
    Has anything been done about this yet? There is something definitely going on with the SpriteRenderer and custom shaders. I came across this issue by trying to add some multiplicative blending to the built-in Sprites-Default shader.

    I'm on 2017.3.0f3

    I created a new shader asset and copy pasted the Sprites-Default source from the Unity downloads page to it. I then created a new material with that new shader assigned. I then assigned that material to the SpriteRenderer component responsible for rendering the sprite that I wish to be masked. And my sprite-to-be-masked simply disappears. I'm using the SpriteMask component on another game object that represents my masking sprite and everything works when I use the actual built-in default sprite shader, but when I try and make SpriteRenderer use my own custom shader (which I can only assume is the exact same source as the actual built-in shader), it stops working.

    Here are my discoveries:

    1. As long as the Mask Interaction setting is set to None, my custom shader works and my sprite is visible. It's just not masked.

    2. If I set the Mask Interaction setting to "Visible Inside Mask", my sprite disappears when it should be partially visible. At this point, if I simply switch the shader that my material uses from my custom shader to the built-in shader, the sprite appears masked as expected. Switching the shader used by my material back to my custom shader, again causes the sprite disappear.

    3. If I set the Mask Interaction setting to "Visible Outside Mask" (with my material using my custom shader), my sprite appears in full without being affected by the mask.

    4. If I have the Mask Interaction setting set to anything other than None, then it doesn't seem to matter what Stencil states I set in my custom shader, they don't appear to affect sprite remains invisible. If the setting *is* None however, then the stencil value at each pixel appears to remain at 0, since I can get my sprite to appear using my custom shader as long as my Stencil shader logic is the following:

    Code (CSharp):
    1. Stencil {
    2.     Ref 0  //Customize this value
    3.     Comp Equal //Customize the compare function
    4.     Pass Keep
    5. }
    I tried fiddling with what the Ref value is and the Comp method but none of it makes sense. It gives me the impression that the stencil value across all pixels under the sprite is zero, since comparing for equality to Ref 0 causes my sprite to appear, but comparing for equality to any other Ref value below or above 0 causes my sprite not to appear. But what's strangest is that doing Comp Less or LEqual to Ref 1 doesn't cause my sprite to appear, but doing Comp GEqual to 0 does cause it to appear. If GEqual to 0 works, I would've assumed Less and LEqual to 0 to work. But this I'm guessing is due to my lack of understanding of the stencil buffer.

    Either way, the interaction between SpriteRenderer, SpriteMask, and my custom sprite shader (which I'll repeat was copied verbatim from the built-in shader source available from the Unity downloads page) is baffling me to say the least.

    I just wanted to change the blend mode of my sprites while retaining the same functionality inherent to the built-in sprites! What could possibly go wrong?

    Thanks for listening.
  6. amisner2k


    Jan 9, 2017
  7. amisner2k


    Jan 9, 2017
    Ok, so I just started up Unity and was prompted that a new update was available, 2017.3.1f1. My issue appears to be fixed with this new version! I realized one of the things I didn't try yesterday was to simply close and restart Unity and or reimport my assets. I can't say for sure, but it's possible that's all I originally had to do. Either way, I'm happy that things are working as expected again.

    Crisis averted. :)
  8. darreney


    Oct 9, 2018
    Hi @Sergi_Valls , i know this is an old thread but I am having issues with the inner workings too.
    I need to use SpriteMask with RenderTexture but it only takes in a Sprite and not a Texture. And i can't create Sprite with any Texture other than Texture2D.

    So i tried to use a script in hope that i can overwrite the _AlphaTex

    Code (CSharp):
    1. public class RenderTextureMask : MonoBehaviour
    2. {
    3.     public RenderTexture rtexture;
    5.     private Renderer _renderer;
    6.     private MaterialPropertyBlock _propBlock;
    8.     // Start is called before the first frame update
    9.     void Start()
    10.     {
    11.         _propBlock = new MaterialPropertyBlock();
    12.         _renderer = GetComponent<SpriteMask>();
    13.     }
    15.     // Update is called once per frame
    16.     void LateUpdate()
    17.     {
    18.         var hasProperty = _renderer.HasPropertyBlock();
    19.         // Get the current value of the material properties in the renderer.
    20.         _renderer.GetPropertyBlock(_propBlock);
    21.         // Assign our new value.
    22.         _propBlock.SetTexture("_AlphaTex", rtexture);
    23.         // Apply the edited values to the renderer.
    24.         _renderer.SetPropertyBlock(_propBlock);
    25.     }
    26. }
    However, this code does do not affect the SpriteMask any bit and i wonder if i did not understand the workings enough.. Or maybe this alpha texture is being combined with all the other SpriteMasks in the scene before sending it to the shader?
    And what is the best approach to use render texture as mask?
  9. Sergi_Valls


    Unity Technologies

    Dec 2, 2016
    You could try using "_MainTex" instead.
    You can create a Sprite from your RenderTexture. Copy RT's pixels into a Texture2D:
  10. darreney


    Oct 9, 2018
    Thanks for the reply!

    Copying the RT's pixels is definitely not what i'm going for cos the RT takes up half the screen and is changing every frame and i can't keep performing this expensive process.. (especially on mobile)

    Will try out your suggestion with "_MainTex" when i can.
    For now i've already got my own shader to perform what i needed.
    Just thought that it is quite limiting that i can't use the built-in SpriteMask to work with render textures.
  11. marcokaki


    Apr 7, 2020
    Hi @Sergi_Valls, if I am creating my own shader, how is it possible to create a stencil buffer that mask only the given range of sorting order just like the option of "custom range" in the built-in SpriteMask. Thanks in advance!
  12. Sergi_Valls


    Unity Technologies

    Dec 2, 2016
    You will need two renderers, sorted at the beginning and end of your range. The first one will draw to the stencil and the last one will clear it or revert the operation. Renderers sorted in between will be able to interact with the prepared stencil.
    marcokaki likes this.
  13. Jamez0r


    Jul 29, 2019
    @Sergi_Valls Do you know if it would be possible to use a SpriteShape to SET the stencil in the same way the SpriteMask would with a Sprite?

    I'm working on a 2D top-down game, and I'd love to use SpriteShape to make puddles that are on the ground, and have the puddles set the mask so that I can draw reflections of characters on top of them.

    I was hoping I could look at the SpriteMask and then maybe make a customized version of it for the SpriteShape, but I can't see the code for the SpriteMask to try to figure out what to do.

    Thanks for any help!