Question Best way to handle circular sorting?

I have a background which is on Sorting Layer 0 (Background), and has order 0. This is an Image/Canvas object folllowing a camera.

I have a bunch of generated Tiles that could have almost any sorting order number. Their SortingLayer however is 1 and is always displayed on top of Background.

I have another Sorting Layer with id 2, OnTopOfTiles. A shadow might be on the bottom of this layer, this would be shown on top of the tile.

However, the shadow should never be shown on the background. And this is where it gets tricky.
I cant just use a mask, as the tiles might be next to each other, and in that case the shadow can enter another tile.

Any idea how to go about this issue?

 

well you should use the tiles as a mask so the shadow only displays on the tiles

how to do it? probably shaders but everytime I ask how to mask with shaders ppl always say dont bother its too hard

 

Yeah I feared I was rearing shader-territory.
I tried conversing with ChatGPT about this and shaders were suggested, but even with ChatGPT's support, I could not find a way that actually worked. So here's to hoping someone may have stumbled upon a similar issue and/or know of an Asset Store purchase that would be nice.

 

you should be careful with the asset store purchase though, it might claim to do something but then it only works on a very narrow scope and youll find yourself breaking the asset just from pushing it a bit over what was shown in the video

just be aware that this is something that has been asked many times with no one ever giving something concrete

 

Ah well, thanks for the heads-up.
Guess I'll have to live with unsatisfying graphics until an epiphany strikes the Unity team or someone more skilled than me with the same issue.

 

Actually managed to fix this issue, had to learn about render textures in order to grasp how, not sure how performance heavy my solution is.

Basically:
* I removed the Layer needing cutout from my MainCamera culling mask (in my case, Shadows layer)
* I added two new cameras, ShadowRenderCamera, CutoutRenderCamera.
* I added two RenderTextures to my project ShadowRenderTexture, CutoutRenderTexture.
* Assigned the render textures to the respectively created cameras.
* Set culling mask on ShadowRenderCamera to the layer I wanted masked (In this case Shadows)
* Set culling mask on CutoutRenderTexture to MainLayer.
* Added a canvas in Screen Space - Camera, follows my MainCamera. Sorting layer is ShadowsOnMainLayer.
* Added an image to this canvas, fully stretching the screen. Not raycastable, and sprite is None.
* Added a new material to this image, with this shader:

Code (CSharp):

1. Shader "Custom/MainLayerMaskShadows"
2. {
3.     Properties
4.     {
5.         _MainTex ("Background Texture", 2D) = "white" {}
6.         _MaskTex ("Mask Texture", 2D) = "white" {}
7.     }
8.     SubShader
9.     {
10.         Pass
11.         {
12.             CGPROGRAM
13.             #pragma vertex vert
14.             #pragma fragment frag
15.  
16.             struct appdata
17.             {
18.                 float4 vertex : POSITION;
19.                 float2 uv : TEXCOORD0;
20.             };
21.  
22.             struct v2f
23.             {
24.                 float2 uv : TEXCOORD0;
25.                 float4 vertex : SV_POSITION;
26.             };
27.  
28.             sampler2D _MainTex;
29.             sampler2D _MaskTex;
30.  
31.             v2f vert (appdata v)
32.             {
33.                 v2f o;
34.                 o.vertex = UnityObjectToClipPos(v.vertex);
35.                 o.uv = v.uv;
36.                 return o;
37.             }
38.  
39.             fixed4 frag (v2f i) : SV_Target
40.             {
41.                 fixed4 color = tex2D(_MainTex, i.uv);
42.                 fixed4 mask = tex2D(_MaskTex, i.uv);
43.                 if (mask.a < 0.5)
44.                     discard;
45.                 return color;
46.             }
47.             ENDCG
48.         }
49.     }
50. }
51.  

And finally, added the shadow to the backgroundTex (slightly bad naming I know) and cutout to the mask texture.


Now whatever is rendered in my Shadows layer, is only within the cutout that matches my MainLayer.

 

good job nice

if you want to check if it affects performance open the stats tab on the playmode window
I suspect the hit will be on increased drawcalls from the extra cameras