Resolved Blend colors only for an area of a sprite

I'm trying to create a robust tutorial system for a mobile game where a semitransparent overlay appears on the whole screen and the interactible items are highlighted (not affected by the overlay). I have created a simplified scene to illustrate what I am trying to achieve.

This is how it looks with the overlay off
overlay off.png

And with the overlay on
overlay on.png

If I want to highlight an object that is not obstructed by other objects in the scene, I can just change the sortingLayer of the sprite and it will be on top of the overlay. However, this does not work with objects that are obstructed by other items (see below).

actual behaviour.png

What I would like to happen in this particular example is for the monster and it's arms (blue and green rectangles) to be highlighted, but only the part above the red rectangle (see below - bring in front only the part with the yellow rectangle highlight).

desired behaviour.png

I also want the solution for this to be reusable for other similar situations and to not have to resort to using sprite masks that match perfectly the object which I am trying to highlight. I think this problem can be solved with shaders and alpha blending, but I'm not sure what the solution would be.

My idea is to write a shader that would be put on the monster material that would act something like this "If you have the tutorial transparent overlay on top of you, do not blend the colors, use only the original colors, but if you have other objects in front of you, blend the colors".

Is it possible to solve it this way, or are there better ways of handling this?

 

Found a solution using stencil buffers!

solution.png

What I ended up doing was using 3 shaders with stencil buffers added to them. One for the monster (green and blue rectangles), one for the counter (red rectangle) and one for the overlay. The key here is to set the render queues properly so the monster parts are drawn first, then the counter, and finally the overlay. For the monster I write a value into the buffer (e.g. 2). For the counter I write another value in the buffer (1) and in the overlay I render only the pixels when the stencil buffer is value is different than the one assigned to the monster.

Shader for the object that has to be highlighted (green and blue rectangles in this case):

Code (CSharp):

1. // Upgrade NOTE: replaced 'mul(UNITY_MATRIX_MVP,*)' with 'UnityObjectToClipPos(*)'
2.  
3. Shader "Monster/BlendMonsterTest" {
4.      Properties {
5.          [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
6.          _Color ("Color", Color) = (1,1,1,1)
7.      }
8.      SubShader {
9.           Tags{ "Queue" = "Transparent"
10.              "RenderType" = "Transparent" }
11.             Blend SrcAlpha OneMinusSrcAlpha
12.             Stencil {
13.                 Ref 2
14.                 Comp Always
15.                 Pass replace
16.             }
17.          Pass {
18.          CGPROGRAM
19. // Upgrade NOTE: excluded shader from DX11, OpenGL ES 2.0 because it uses unsized arrays
20. #pragma exclude_renderers d3d11 gles
21.          #pragma vertex vert
22.          #pragma fragment frag
23.          #include "UnityCG.cginc"
24.  
25.      
26.          uniform fixed4 _Color;
27.          uniform sampler2D _MainTex;
28.  
29.          struct appdata_t {
30.          float4 vertex   : POSITION;
31.          float4 color    : COLOR;
32.          float2 texcoord : TEXCOORD0;
33.          };
34.          struct v2f {
35.              float4 pos : SV_POSITION;
36.              float2 uv : TEXCOORD0;
37.              float4 color : COLOR;
38.          };
39.          v2f vert (appdata_t v) {
40.              v2f o;
41.              o.pos = UnityObjectToClipPos (v.vertex);
42.              o.uv = v.texcoord;
43.              o.color = v.color;
44.              return o;
45.          }
46.          fixed4 frag (v2f i) : COLOR {
47.             fixed4 textureColor = tex2D(_MainTex, i.uv) * _Color;
48.             clip(textureColor.a - 0.05);
49.             return textureColor;
50.          }
51.          ENDCG
52.          }
53.      }
54. }

Shader for the object(s) that blocks the highlighted object:

* same as previous one except for the stencil

Code (CSharp):

1.  
2. Stencil {
3.          Ref 1
4.          Comp Always
5.           Pass replace
6. }

Shader for the overlay:

*same as the previous one except for the stencil

Code (CSharp):

1.  
2. Stencil {
3.      Ref 2
4.      Comp NotEqual
5. }