Problem Solving: 2D Billboard Sprites clipping into 3D environment

Hey guys, I've been experimenting with building a tactics engine using 3D geometry but 2D character sprites, similar to the Final Fantasy Tactics and Disgaea series. I'm using a simple shader for billboard sprites that always face the camera, and this works quite well in most cases.

Using one of their sprites as a placeholder, you can see they render fine at a shallow angle:

And are properly occluded by objects in front:


However, at any steeper camera angle, if they're close to a vertical wall, the sprite tilts back to maintain its facing, and is clipped through by the wall.


This is realistic behaviour for a flat plane, but the sprites have their perspective baked in, and I need to avoid this clipping somehow. All of the sprite should render over objects further back than its bottom pixel, but still be occluded by obstacles in front.

Is there some way I can achieve this or avoid the clipping issue another way, short of designing my 3D space around it?

Thanks in advance!

EDIT: Including the shader I used for reference.

Code (CSharp):

1. Shader "Unlit/Sprite_Billboard_Unlit"
2. {
3.     Properties
4.     {
5.         _MainTex("Texture", 2D) = "white" {}
6.     }
7.  
8.     SubShader
9.     {
10.         Tags{ "Queue" = "Transparent" "IgnoreProjector" = "True" "RenderType" = "Transparent" "DisableBatching" = "True" }
11.  
12.         ZWrite Off
13.         Blend SrcAlpha OneMinusSrcAlpha
14.  
15.         Pass
16.         {
17.             CGPROGRAM
18.             #pragma vertex vert
19.             #pragma fragment frag
20.             // make fog work
21.             #pragma multi_compile_fog
22.  
23.             #include "UnityCG.cginc"
24.  
25.             struct appdata
26.             {
27.                 float4 vertex : POSITION;
28.                 float2 uv : TEXCOORD0;
29.             };
30.  
31.             struct v2f
32.             {
33.                 float2 uv : TEXCOORD0;
34.                 UNITY_FOG_COORDS(1)
35.                 float4 pos : SV_POSITION;
36.             };
37.  
38.             sampler2D _MainTex;
39.             float4 _MainTex_ST;
40.  
41.             v2f vert(appdata v)
42.             {
43.                 v2f o;
44.                 o.pos = UnityObjectToClipPos(v.vertex);
45.                 o.uv = v.uv.xy;
46.  
47.                 // billboard mesh towards camera
48.                 float3 vpos = mul((float3x3)unity_ObjectToWorld, v.vertex.xyz);
49.                 float4 worldCoord = float4(unity_ObjectToWorld._m03, unity_ObjectToWorld._m13, unity_ObjectToWorld._m23, 1);
50.                 float4 viewPos = mul(UNITY_MATRIX_V, worldCoord) + float4(vpos, 0);
51.                 float4 outPos = mul(UNITY_MATRIX_P, viewPos);
52.  
53.                 o.pos = outPos;
54.  
55.                 UNITY_TRANSFER_FOG(o,o.vertex);
56.                 return o;
57.             }
58.  
59.             fixed4 frag(v2f i) : SV_Target
60.             {
61.                 // sample the texture
62.                 fixed4 col = tex2D(_MainTex, i.uv);
63.                 // apply fog
64.                 UNITY_APPLY_FOG(i.fogCoord, col);
65.                 return col;
66.             }
67.             ENDCG
68.         }
69.     }
70. }
71.  

 

I would like to ask about possible solutions to this, too.

I'm fairly new to shaders so forgive me if I'm asking something stupid, but is there a possibility to assign a single z-value to all of the pixels of an object?

Or maybe there's another type of solution to this? Maybe something to do with these stencil settings?

 

i'd try with setting sortingorder (based on z coordinate perhaps)
https://docs.unity3d.com/ScriptReference/Renderer-sortingOrder.html

similar to when using sprites,
https://docs.unity3d.com/Manual/2DSorting.html

 

A screen facing sprite already has the same z across the entire surface, that’s in fact kind of the problem.

Stencils are useful for masking whether something should or shouldn’t show in an area of the screen. But this is a 2D screen space thing, you’d have to sort everything manually.

Won’t matter, because the depth buffer is the problem here. The sprite is intersecting with the depth buffer, and the entire purpose of the depth buffer is to produce identical results regardless of the sorting order.

The “simple” solution is don’t put the sprite’s pivot on the ground. Put it at the center of the space they should be standing on and don’t let them get so close to the blocks to intersect.

Alternatively use vertically aligned sprites rather than camera facing sprites, that way they won’t ever intersect with the 3D objects due to vertical camera orientation. I believe this is the solution Octopath Traveler uses.

The more advanced shader only option would be to use a shader that pushes the sprite towards the camera by some world space amount. This would avoid them clipping into geometry they’re in front of, but may cause them to clip into geometry they’re behind.

The even more advanced shader only option would be to output the z as if the sprite is a vertically aligned sprite, but keep the appearance of a screen space sprite.


However, what I think both Disgaea and FFT do is sort all objects & ground tiles as if they were still 2D sprites. The character sprites never intersect with 3D geometry, they’re always either on top or behind geometry, which means they’re not using the depth buffer at all for the sprites (ZTest always). But each 3D object properly sorts with itself and those around it, and vfx clip all over the place, so depth is obviously still being used.

 

Most games I have seen doing this 2d/3d mix will be designed from the start with a tightly constrained camera tilt baked into the geometry. That means modeling that cube "leaning back" so it still feels like a cube but doesn't lean forward into the paper-thin elements.

 

Like this (though this game doesn't use sprites).

It should be noted that both FFT and Disgaea use orthographic cameras to help make stuff easier. Paper Mario uses fully 3D worlds, and camera facing sprites, but they also limit the camera movement significantly, and let sprites clip through objects; a bit of jank between the 2D and 3D elements is part of the game's shtick.

Some other games with supposed mixed 2D and 3D elements cheat, and the 3D elements are flattened or otherwise rendered as sprites.

 

A tradition as old as Donkey Kong Country.

 

Some modern games ship with real animated 3D geometry and render them to sprites in real time. Some recent “2D” fighting games on consoles, or Brawl Stars on mobile.

 

Thanks for the replies everyone! I have seen that method of tilting the levels, but I would like to keep physics on if it's at all possible. That seems like an interesting way to do it, though.

This is actually what I meant in my first reply when I said about the z-value of the pixels, but I guess I explained it wrong? I meant coding a custom shader and in the fragment shader modify the pos.z value. Is that what you meant, too?

Would that be plausible? If so, how can I reference a point in space to the fragment shader?

 

You'd probably want to do it in the vertex shader. Much cheaper and easier to do it there. However regardless of if you do it in the vertex or fragment shader, this hack will cause problems if your camera angle looks down or up too far, as eventually the appropriate z position goes beyond infinity, and before that well outside of the near and far clipping planes. That can be solved by clamping, but it's something to be wary of. Think about looking straight down on a normal camera facing billboard and a really, really big vertical billboard. Any place you can't see the vertical billboard there's no valid mathematical solution to where to put the camera facing billboard's "z" where it'll still be visible, so at some point you just have to clamp the values to something reasonable, or maybe blend to using a different technique when the angle gets too high (like simply offsetting towards the camera).

Now how to do it, you'd have to first calculate the position for each vertex of the screen facing sprite, like you're already doing. Then you'd take the world space view direction and do the math for a ray plane intersection, using the sprite's pivot position as a point on the plane, and the y flattened, negative camera forward vector as the plane's normal. Then you'd calculate the clip space position for both of those values, and override the z of the screen facing with the calculated vertical billboard's "position".

Code (csharp):

1. float4 viewPos = // what you have above.
2. float4 outPos = mul(UNITY_MATRIX_P, viewPos);
3.  
4. float3 planeNormal = -normalize(UNITY_MATRIX_V._m20, 0.0, UNITY_MATRIX_V._m22);
5. float3 planePoint = unity_ObjectToWorld._m03_m13_m23;
6. float3 rayStart = _WorldSpaceCameraPos.xyz;
7. float3 rayDir = normalize(mul(UNITY_MATRIX_I_V, viewPos).xyz - rayStart); // convert view to world, minus camera pos
8. float dist = rayPlaneIntersection(planeNormal, planePoint, rayDir, rayStart);
9.  
10. float4 planeOutPos = mul(UNITY_MATRIX_VP, float4(rayStart + rayDir * dist, 1.0));
11.  
12. outPos.z = planeOutPos.z / planeOutPos.w * outPos.w;

 

Random thing, the rayPlaneIntersection function isn't one that ships with Unity. You'd have to implement it yourself.
Or steal the one I used in this shader, though the order of the ray and plane inputs are backwards from the above code:
https://forum.unity.com/threads/relief-function-shader.549472/#post-3665737

 

Thank you for your reply bgolus. I have to take some time and go through your code with thought to understand it. I'm fairly bad at matrices and vector math so I´ll try to implement your example and post my findings.

 

Actually Unity does ship with one, it's this one https://docs.unity3d.com/ScriptReference/Plane.Raycast.html - you just need to supply a Plane for it, which is easy enough.

 

That's a c# function. This is about doing raytracing in a shader which uses HLSL and can't use c#.

 

Just curious, but have you had any success with that shader method? I was looking for this very thing when I first posted (sort z-depth as if the sprite is aligned to the up axis, but display the sprite as if it's facing the angled camera straight on).

Unfortunately, I'm also not particularly versed in writing shaders - but I can take another crack at this to try it out if nobody else has done so.

In my case, there's no worry that the camera angle will ever become too steep. I've locked it at 45 degrees or shallower, which matches the hand-drawn perspective for the sprites I'm using.

What I find interesting about this is that I've definitely seen games solve the issue in a non-disruptive way, but as far as I can tell it's never been properly documented online. Take Recettear for example - which is an indie game itself:



The camera definitely has perspective, though it's slight. Character sprites can almost hug the walls without clipping - and in the case of the 3D chest model you have to be literally standing inside it before inevitable clipping occurs (which it would on a sprite standing upright as well). That you can stand both in front and behind the chest and clip partially also indicates the z-buffer is still being used, doesn't it?

 

You could rotate the sprite to a normal vertical position. This "compresses" the sprite and it looks shorter, little distorted.
Then, you just scale the sprite on the y axis. This should only work on orthographic cameras. You need to calculate how much the sprite needs to be stretched. For example, if your cameras rotation x-axis looks town at 60 degrees. Your compressed vertical sprite needs to be stretched to exactly 2, on the y axis. Works for me pixel perfectly (if that's a word). You might have to watch out for proper floor contact and 3d depth, but it worked pretty good for me.

Oh and of course, it only works with a fixed camera angle, at least at the x axis. If you want to rotate the cam on the y axis, you need a proper interpolation for the sprites, to rotate with it, like in ragnarok online.

 

When I run this, planeOutPos.w is always 0, so it doesn't work. Am I doing something wrong?

 

Here's a working version of the shader with my coding errors fixed.

Spoiler: BillboardVerticalZDepth.shader

Code (CSharp):

1.  
2. Shader "Unlit/BillboardVerticalZDepth"
3. {
4.     Properties
5.     {
6.         _MainTex("Texture", 2D) = "white" {}
7.     }
8.  
9.     SubShader
10.     {
11.         Tags{ "Queue" = "Transparent" "IgnoreProjector" = "True" "RenderType" = "Transparent" "DisableBatching" = "True" }
12.  
13.         ZWrite Off
14.         Blend SrcAlpha OneMinusSrcAlpha
15.  
16.         Pass
17.         {
18.             CGPROGRAM
19.             #pragma vertex vert
20.             #pragma fragment frag
21.             // make fog work
22.             #pragma multi_compile_fog
23.  
24.             #include "UnityCG.cginc"
25.  
26.             struct appdata
27.             {
28.                 float4 vertex : POSITION;
29.                 float2 uv : TEXCOORD0;
30.             };
31.  
32.             struct v2f
33.             {
34.                 float4 pos : SV_POSITION;
35.                 float2 uv : TEXCOORD0;
36.                 UNITY_FOG_COORDS(1)
37.             };
38.  
39.             sampler2D _MainTex;
40.             float4 _MainTex_ST;
41.  
42.             float rayPlaneIntersection( float3 rayDir, float3 rayPos, float3 planeNormal, float3 planePos)
43.             {
44.                 float denom = dot(planeNormal, rayDir);
45.                 denom = max(denom, 0.000001); // avoid divide by zero
46.                 float3 diff = planePos - rayPos;
47.                 return dot(diff, planeNormal) / denom;
48.             }
49.  
50.             v2f vert(appdata v)
51.             {
52.                 v2f o;
53.                 o.pos = UnityObjectToClipPos(v.vertex);
54.                 o.uv = v.uv.xy;
55.  
56.                 // billboard mesh towards camera
57.                 float3 vpos = mul((float3x3)unity_ObjectToWorld, v.vertex.xyz);
58.                 float4 worldCoord = float4(unity_ObjectToWorld._m03, unity_ObjectToWorld._m13, unity_ObjectToWorld._m23, 1);
59.                 float4 viewPos = mul(UNITY_MATRIX_V, worldCoord) + float4(vpos, 0);
60.  
61.                 o.pos = mul(UNITY_MATRIX_P, viewPos);
62.  
63.                 // calculate distance to vertical billboard plane seen at this vertex's screen position
64.                 float3 planeNormal = normalize(float3(UNITY_MATRIX_V._m20, 0.0, UNITY_MATRIX_V._m22));
65.                 float3 planePoint = unity_ObjectToWorld._m03_m13_m23;
66.                 float3 rayStart = _WorldSpaceCameraPos.xyz;
67.                 float3 rayDir = -normalize(mul(UNITY_MATRIX_I_V, float4(viewPos.xyz, 1.0)).xyz - rayStart); // convert view to world, minus camera pos
68.                 float dist = rayPlaneIntersection(rayDir, rayStart, planeNormal, planePoint);
69.  
70.                 // calculate the clip space z for vertical plane
71.                 float4 planeOutPos = mul(UNITY_MATRIX_VP, float4(rayStart + rayDir * dist, 1.0));
72.                 float newPosZ = planeOutPos.z / planeOutPos.w * o.pos.w;
73.  
74.                 // use the closest clip space z
75.                 #if defined(UNITY_REVERSED_Z)
76.                 o.pos.z = max(o.pos.z, newPosZ);
77.                 #else
78.                 o.pos.z = min(o.pos.z, newPosZ);
79.                 #endif
80.  
81.                 UNITY_TRANSFER_FOG(o,o.vertex);
82.                 return o;
83.             }
84.  
85.             fixed4 frag(v2f i) : SV_Target
86.             {
87.                 fixed4 col = tex2D(_MainTex, i.uv);
88.                 UNITY_APPLY_FOG(i.fogCoord, col);
89.  
90.                 return col;
91.             }
92.             ENDCG
93.         }
94.     }
95. }

Viewed from above


And an example of just how close to that block the quad actually is.

 

In case no one understood my suggestion (works only on orthographic cameras!):

- Rotate your camera to 30,0,0
- Put your sprite into a vertical position, so it can stand close beside a 3d cube without clipping inside
- Scale your sprite to exact 1,1.154251,1 // cos(30rad)

Your sprite now looks exact as he would be rotate by 30 degrees, but now, it won't clip inside the cube.
It might not be an option for everyone, but it's a pretty easy solution.

 

Yeah, there’s a lot of solutions that involve just moving the sprite towards the camera a little.

 

Thanks, it works perfectly!

I couldn't help but notice that you've moved some of the minus signs around from the shader you posted previously, though. I wonder if that's why it works now?

Code (CSharp):

1. return -dot(diff, planeNormal) / denom;
2. return dot(diff, planeNormal) / denom;
3.  
4. float3 planeNormal = -normalize(UNITY_MATRIX_V._m20, 0.0, UNITY_MATRIX_V._m22);
5. float3 planeNormal = normalize(float3(UNITY_MATRIX_V._m20, 0.0, UNITY_MATRIX_V._m22));
6.  
7. float3 rayDir = normalize(mul(UNITY_MATRIX_I_V, viewPos).xyz - rayStart); // convert view to world, minus camera pos
8. float3 rayDir = -normalize(mul(UNITY_MATRIX_I_V, float4(viewPos.xyz, 1.0)).xyz - rayStart); // convert view to world, minus camera pos

But maybe it's just this part:

Code (CSharp):

1. // use the closest clip space z
2. #if defined(UNITY_REVERSED_Z)
3. o.pos.z = max(o.pos.z, newPosZ);
4. #else
5. o.pos.z = min(o.pos.z, newPosZ);
6. #endif

 

Yep, that’s part of it. There’s a graphics programmer idiom of “always have an even number of sign errors.”

That prevents the bottom edge of the sprite from going through the floor. Like the top is being effectively pulled towards the camera by the plane projection, the bottom edge is being pushed away. Works fine when there’s no floor, but with one the above example image had the tongue down under the floor.

 

Yep, this is a simple to understand solution I've seen used in a few places - but the obvious downside is that if anything ever intersects overhead, such an arch, the lengthened sprite can clip through it. I'd have done something like this if I weren't using a perspective camera (Ragnarok Online does this too, but the perspective is slight) or planning on overhead objects.

 

Having tried the shader out, it works great! All the occlusions between sprites and geometry behave in my scenes.

I'm not sure if it's related directly to the billboarding, but I do have one strange issue when using it at the moment: transparency sorting between my sprites where they overlap is... unusual. It seems reversed from one angle, but mostly correct from the opposite side.

This behaviour is also present on the simpler version of the shader I used for billboarding before that only rotated the sprite to face the camera without any changes to its z-depth, so I'm guessing I just missed some broader consideration for sprite alphas.

EDIT: Turns out the issue was that for some reason my project was using a custom sorting axis on its cameras. Problem solved!

 

I've confirmed the above issue has nothing to do with the billboarding method. You can get the same results laying out a bunch of standard 2D sprites and lining them up in a row, using the default sprite shader (Sprites > Default).

From the front, the sprites order correctly. From the back, the sprites can still be seen as they don't cull backfaces, but their sorting is backwards.



I know sorting the depth of transparent pixels has always been a troublesome issue. Is there any solution I might use here?

 

...................................can work

 

It's interesting issue,
but are you sure that stencil could not solve this problem?
It seems 2d sprite in 3D scene.

 

Yes, I’m sure.

Stencils aren’t some magic solve all thing people think they are. The main problem with them is stencils are a 2D screen space technique, not 3D. They know nothing of the scene depth.

You could do something like render the sprites first with a stencil write, then render the scene with a material that doesn’t render where the sprites did, but then walls that should render over the sprite don’t. So you could fix that by detecting which walls are closer to the camera than the sprite and using a material that renders over the sprite. And now you’ve basically started sorting the world geometry against the sprite and no longer need the stencil at all!

 

Hello! This thread has been on our radar for at least a year now. We are very excited to see the proposed solution because it is extremely relevant to our game as well.

If I may, I would like to bring up another challenge in the same vein, one that developers working with sprites in 3D space will undoubtedly encounter.



It turns out that not only walls can be problematic, but floors as well.

The issue is that sprites are never drawn at a 90 degree angle from the side, but rather a very specific angle which best matches the average angle of the camera in-game. This means that if you think of the sprite has a 3D object, the real contact point of the feet is not actually the bottom-most row of pixels, but rather a little above in the middle of the foot.

Using the bottom-most row (or the one right after) will avoid clipping with the floor, but has two problems:

1. The sprite is actually drawn for 8 different directions. Depending on the direction, The distance between the bottom-most row and the feet will fluctuate and simply snapping to the bottom-most row that will avoid clipping will cause the sprite to swim up and down as the sprite changes between directions, which causes visual jerk and is quite irritating.

2. Using the lowest row of all the directions and frames of animation will avoid the visual jerk, but now the sprites appear to hover several pixels above the ground, almost like they are tip-toing.

Furthermore, this gets less and less accurate the more 'horizontal' a character gets. See this next image where the player is belly-sliding on the ground, where the real contact point would be the belly which is way above the bottom-most row:



This is an on-going issue since the start of our project and to this day we have not found a good solution. I have a feeling the solution proposed above for the walls could be adapted to solve this issue as well, but I'd like to know what you think @bgolus, if you don't mind.

Thanks!

 

The above technique is mainly adjusting the top edge of the sprite by projecting the vertex depth onto a plane. That plane happens to be one that's centered on the sprite's pivot and facing the camera. But you could do the same to project onto a plane that's at or above the ground.

If your sprites are setup so that the sprite's pivot is placed at the desired "ground" position, then you can do something like this:

Code (csharp):

1. // same as the original example shader
2. float dist = rayPlaneIntersection(rayDir, rayStart, planeNormal, planePoint);
3.  
4. // added check to get distance to an arbitrary ground plane
5. float groundDist = rayPlaneIntersection(rayDir, rayStart, float3(0,1,0), planePoint);
6. // use "min" distance to either plane (I think the distances are actually negative)
7. dist = max(dist, groundDist);
8.  
9. // then do the rest of the shader normally
10. // calculate the clip space z for vertical plane
11. float4 planeOutPos = mul(UNITY_MATRIX_VP, float4(rayStart + rayDir * dist, 1.0));

You can also add some small y offset to the

planePoint

passed to the second intersection if you want to really make sure it doesn't intersect, especially if your ground isn't perfectly flat. Just be wary that too much of an offset and you might start to see your sprites intersecting with objects in front of the mesh, as this is also pulling the mesh's depth toward the camera.

edit: swapped the

min

for a

max

for future copy/pasters.

 

Incredible, I can't believe we now have a fix to this problem. YES, it works! However note that the correct operation to choose between the two intersections at line 7 is actually

max

. I don't really get it, I worked it out on paper just to see and even then it seems plainly obvious that

min

is what you would want.

 

I've noticed another peculiarity with this method, which is that the sprites face the camera a little too accurately, and so their rotation around the axis between camera and sprite doesn't respect the skew you would expect to see due to the camera perspective.



I know this effect might be desirable for certain sprites (any that you want to always render directly aligned with the screen pane - maybe UI elements like floating damage numbers), but when used for characters it breaks the illusion of depth. Is there some method we could use to correct for this skew by, for instance, rotating the sprite plane so it faces "up" in the world, while still keeping it perfectly perpendicular to the camera?



Rotating the sprites manually in Unity actually achieves this effect perfectly - I'm only at a loss as to how I'd arrive at the right degree of rotation and implement that inside the sprite shader:

 

The original shader is a view facing billboard shader. You probably want a camera facing billboard. A view facing billboard aligns the billboard to the view plane, where as a camera facing billboard points the quad towards the camera position. There are a lot of different types of camera facing billboards, but you probably want one like this:

Code (csharp):

1. // replace lines 56 - 59 with this
2. // billboard mesh towards camera
3. float3 vpos = mul((float3x3)unity_ObjectToWorld, v.vertex.xyz);
4. float4 worldCoord = float4(unity_ObjectToWorld._m03_m13_m23, 1);
5. float4 viewPivot = mul(UNITY_MATRIX_V, worldCoord);
6.  
7. // construct rotation matrix
8. float3 forward = -normalize(viewPivot);
9. float3 up = mul(UNITY_MATRIX_V, float3(0,1,0)).xyz;
10. float3 right = normalize(cross(up,forward));
11. up = cross(forward,right);
12. float3x3 facingRotation = float3x3(right, up, forward);
13.  
14. float4 viewPos = float4(viewPivot + mul(vpos, facingRotation), 1.0);
15.  
16.  
17. // then replace what was line 64 with this
18. float3 planeNormal = normalize((_WorldSpaceCameraPos.xyz - unity_ObjectToWorld._m03_m13_m23) * float3(1,0,1));

 

Incredible! That camera billboard setup works a treat, including the above edit for the ground plane adjustment:


Thanks again!