Issue with ZTest on rotated sprites in 3D world

I'm trying to make a game that uses 3D models and sprites. My camera can move around and change angles, so my sprites also have to look at the camera, through LookAt or biilboarding in a shader.

However this has the issue where in certain camera rotations, the sprite will overlap with a 3D model and get cut-off when I would not want it to. Like this:


I also can't make it always show because then if I turn the camera around, it would ignore ZDepth completely.

My idea was to make it so all the pixels in the sprite would assume the same base Z coordinate. To simulate a completely straight sprite during the ZTest.

The approach was to write a 2 pass vertex stencil shader that would implement the following behavior:

Pass1:

1. Take the input vertex, convert it to world coordinates, change it's Z coordinate to a value taken from a per renderer property and convert it back to object -> camera space
2. Run the depth test with the changed position and write to the stencil buffer. 1 on pass, 0 on ZFail.

Spoiler: Pass1 code

Code (CSharp):

1.  
2. Blend Zero One
3. ZTest LEqual
4. Stencil
5. {
6.     Ref 1
7.     Comp Always
8.     ZFail Zero
9.     Pass replace
10. }
11.  
12. //... structs and pragmas
13.  
14. v2f vert1(appdata v)
15. {
16.     v2f o;              
17.     float4 worldPos = mul(unity_ObjectToWorld, v.vertex);
18.     worldPos.z = _ZCoord;
19.     o.pos = UnityObjectToClipPos(mul(unity_WorldToObject, worldPos));
20.     return o;
21. }
22.  

Pass2:

1. If stencil buffer has 1 then render normally, if has 0 then skip.

Spoiler: Pass1 code

Code (CSharp):

1.  
2. ZTest Always
3. Blend One OneMinusSrcAlpha
4. Stencil
5. {
6.     Ref 1
7.     Comp Equal
8. }
9.  
10. //..structs and pragmas
11.  
12. v2f vert(appdata_t IN)
13. {
14.     v2f OUT;
15.     OUT.vertex = UnityObjectToClipPos(IN.vertex);
16.     return OUT;
17. }

So far this worked but only if I look from a front angle. But if I look from above it cuts off again, but this time not because of ZDepth.



The issue is that the final image is having it's vertices modified.


This seems to be because of the line worldPos.z = _ZCoord; in the first pass, but I don't understand why this affects the second pass.
From what I understand, each Pass should a separate program and each vert function should get it's copy of the original vertex data. So the second pass should only be affected by the stencil buffer and nothing else from the first pass.

Is there something I'm misunderstanding about passes or am I taking the wrong approach for this?

 

Each pass is a unique copy of the vertices, and the sprite is being clipped by the stencil. Everything is working exactly as you’re describing it. There’s nothing going wrong here, and the idea isn’t a bad one, but I think your expectations how how to pull it off are slightly off.

I think the problem you’re running into is your expectations of what setting the Z position does. The sprite is still being rotated to face the camera, you’re just overriding the Z after that. The resulting quad isn’t the same as if it was a vertically aligned billboard, but rather a narrow strip, depending on how far “up” the sprite is facing. Even if that wasn’t a problem, the screen area a vertically aligned sprite covers vs a camera facing sprite is very different, and the vertically aligned billboard will always be in the best case the same size, but more often smaller, and thus the stencil won’t cover the fully visible area of the screen facing sprite.

The basic solution, using the same setup you have, would be to extend the vertical sprite to be very, very tall.

Oddly this topic has come up a couple of times just in the last few days. Here’s a single pass solution I posted recently.
https://forum.unity.com/threads/pro...ping-into-3d-environment.680374/#post-4638199

 

Thanks for your reply bgolus.

It took me a long time but I think I finally understood what you said (and your solution). If you don't mind, correct me if I'm wrong in any of this:

I had two very critical misunderstanding of how shaders work internaly.

• First regarding the stencil / depth buffers. I thought that each pixel in object position had direct indexation to the buffers. However, this is not the case. It seems that the stencil buffer and ZBuffer are in camera space.
• Second was the actual impact of changing the z world coordinate. When I convert a pixel from object to camera space, it takes into account the camera angle. This means that when I change the world z coordinate, when i convert it from world -> object -> camera coordinates, it also ended up changing the final x,y in camera space.

When I merge these two points, I now understand that the first pass is writing to the stencil buffer exactly what I wanted, but not to the index I intended. So when the second pass goes to look, since the pixel had a different z (therefore also different x,y), it was looking a different index of the stencil buffer than I intended. This is also why it worked fine when the camera was completely horizontal, since then the resulting pixel coordinates in both passes would match.
This also means that my approach didn't need two passes at all, since if I maintained the original x,y, then the sprite is already in the correct position.

With this in mind, I made a small change to my code to test it, where I still transform my pixel into world coordinates, but I only use it's z coordinate on my final return value.

Code (CSharp):

1. v2f o;            
2. o.pos = UnityObjectToClipPos(v.vertex);
3. float4 worldPos = mul(unity_ObjectToWorld, v.vertex);
4. worldPos.z = 66; // hardcode for testing
5. o.pos.z = UnityObjectToClipPos(mul(unity_WorldToObject, worldPos)).z;
6. return o;

Spoiler: Result

So now it works exactly as intended!

However, as you said, this still has issues, since this approach does not take the camera orientation into account, it will fail from other angles. This is because when I rotate the camera the axis will also change, and simply flattening the z coordinate would not be enough (I could even have to change the x coordinate too).


All this led me to understand your solution. You take it 2 steps further, with the bonus of doing it in a single pass.

• First you use the sprite pivot point as your reference, this replaces my need to input a z coordinate from outside.
• Then you take into account the camera orientation, with the ray. Solving my issue of only working for the Z coordinate.

From there you apply the same concept I just used. Where you only change the z coordinate of the camera space, while maintaining the original x,y. Which is essentially tricking the ZTest into thinking all the pixels are in the same depth as the pivot point pixel.


Assuming this is mostly correct, thank you very much for explaining and pointing me in the right direction!

 

Mostly correct. The stencil and depth buffer (aka z buffer) are in screen space, sometimes called window or viewport space. This is a little pedantic, but "camera space" refers to something else. Camera space, also called view space, is a world scale position relative to the camera position and orientation. This position is similar to what you'd see on a transform that is a child of the camera's game object, but explicitly ignores any scale from the camera's transform, and z is flipped so -z is forward.

One small tweak to this. You don't need to transform from world back to object. The UnityObjectToClipPos() function's first line is to convert from object space to world space again, then apply the view proj matrix. You can skip those two extra transformations and apply the view proj matrix yourself, either by using mul(UNITY_MATRIX_VP, float4(worldPos.xyz, 1)) or via the UnityWorldToClipPos() function (which itself is just the previous matrix mul).

But otherwise it seems like you’ve got the idea.

 

Sorry to bring this up again, but the issue still plagues me.

I managed to implement the method you described on the other topic and it was working fine. However, this is only for perspective camera, and for my game I would like to keep orthographic camera (unless i can't find a way to fix this).

From what I understand, orthographic camera calculates depths linearly and it does not compute distances from a single center point like the perspective camera. Instead I could imagine the orthographic camera as a plane which calculates point distances as perpendiculars to that plane.

My approach to solving the issue would be to rotate my vertices to make the sprite straight up and facing the camera. So it would essentially render my sprite with actual rotation but calculate depth using a default rotation of (0,0,z). My camera does not change it's y.

Would this approach work or is there a simpler one? This time I think i'd rather know before I waste too much time into something that wouldn't work

Thanks in advance