Drawing objects based only on layer order with no overdraw. [solved]

Hi everybody, my question is a bit complicated, but I'll do my best to be as clear as possible:

I'm trying to write a shader for a 2.5D game that fulfills the following 2 requirements:

1. Draw opaque objects in front of each other based solely on the order they are rendered in, as set by their order in layer (which basically happens by default if ZTest is Off & you're rendering back to front like in the transparent queue).
2. Produce no overdraw.

My first idea for avoiding overdraw was using ZTest & rendering Front To Back. The problem is: Adding "ZTest Less" gets rid of the overdraw, but the shader no longer draws objects solely based on their rendering order. Objects with lower ZDepth will always be drawn in front of others, no matter their order in layer.

Does anybody have an idea how the 2 requirements could be made to work together in a shader? Basically I'm looking for a way to tell a shader if a pixel has been rendered before and if so (regardless of ZDepth) to skip rendering it again.

One more constraint:
There's going to be >255 objects on screen using this shader, so I think the stencil buffer wouldn't work as a solution.


Here's what the top of my shader-code looks like so far:

Code (CSharp):

1. SubShader
2.     {
3.         Tags
4.         {
5.             "Queue"="Geometry"
6.             "IgnoreProjector"="True"
7.             "RenderType"="Opaque"
8.         }
9.  
10.  
11.         Cull Back
12.         Lighting Off
13.         ZTest Less
14.         ZWrite On
15.         Blend Off
16.  
17.         Pass
18.         {}

Thanks for reading, if anyone has an idea you'd really help me out!

 

The stencil buffer should actually be helpful here. And your idea of drawing front to back is right.

You should essentially determine the entire draw order and then execute it in reversed order to remove overdraw. You can use a single bit in the stencil buffer to mark where you have already drawn something to later skip those pixels.

So it's close to what you already did, but only uses a single bit in the stencil buffer instead of the entire depth value in the depth buffer.

 

Oh god I am such an idiot. Thank you so much, you're totally right. For some reason I thought I'd have to increase the stencil buffer for each object and so it's 255-integer wouldn't have been enough, but of course that's not the case, just need to increase by 1 when the pixel is rendered for the first time and that's it.

Again, thank you, I've been trying to figure this out for quite some time!

 

Hi Friedemann_A, I know I am a bit late but I am having a problem with overdraw that is killing performance for my Android game. Can you share the whole shader code if possible?

 

Sure, here's the full code, not sure how helpful it is in your case though. To my understanding, this will only be relevant if you only render fully opaque objects. You might have to modify the code, most importantly the ZTest stuff. The Stencil Variables are set in the inspector and I'm setting them to:
Mask Mode = Greater
Stencil Pass Operation = IncrementSaturate
Stencil Ref Value = 1

Code (CSharp):

1. Shader "2D Meshes/Animated"
2. {
3.     Properties
4.     {
5.         [Enum(CompareFunction)] _StencilComp("Mask Mode", Int) = 5
6.         [Enum(StencilOp)] _StencilOperation("Stencil Pass Operation", Int) = 6
7.         _StencilRefValue("Stencil Ref Value", Int) = 1
8.     }
9.     SubShader
10.     {
11.         Tags
12.         {
13.             "Queue"="Geometry"
14.             "IgnoreProjector"="True"
15.             "RenderType"="Opaque"
16.             "DisableBatching" = "True"
17.         }
18.  
19.         // No Lighting, cull backsides
20.         Cull Back
21.         Lighting Off
22.         // No ZTest, because we want to sort meshes by their
23.         // sortingOrder only. This sortingORder is assigned
24.         // according to the mesh's position in the Unity-Hierarchy
25.         // by CustomMeshEditor.cs
26.         ZTest Off
27.         // ZWrite on, so objects that use ZTest can still be occluded by this
28.         ZWrite On
29.         // No transparency
30.         Blend Off
31.  
32.         Pass
33.         {
34.             // Stencil is used to make the Meshes occlude each other.
35.             // Only the first one that is drawn at a certain position is shown.
36.             Stencil
37.             {
38.                 Ref [_StencilRefValue]
39.                 Comp [_StencilComp]
40.                 Pass [_StencilOperation]
41.                 Fail Keep
42.             }
43.  
44.             Name "GEOMETRY"
45.  
46.             CGPROGRAM
47.             #pragma vertex vert
48.             #pragma fragment frag
49.             #include "UnityCG.cginc"
50.  
51.             struct appdata
52.             {
53.                 float4 vertex : POSITION;
54.                 fixed4 color : COLOR;
55.             };
56.  
57.             struct v2f
58.             {
59.                 fixed4 color : COLOR;
60.                 float4 vertex : SV_POSITION;
61.             };
62.  
63.             v2f vert (appdata IN)
64.             {
65.                 v2f o;
66.                 o.vertex = UnityObjectToClipPos(animatedVertex);
67.                 o.color = IN.color;
68.                 return o;
69.             }
70.          
71.             fixed4 frag (v2f IN) : SV_Target
72.             {
73.                 return IN.color;
74.             }
75.             ENDCG
76.         }
77.     }
78. }
79.  

Best of luck!