Problem Solving: 2D Billboard Sprites clipping into 3D environment

Greetings to all! I've been following this topic for a while now and tried to implement it with some success. However, I'm not very shader savvy and use node-based Amplify for all my shader work. Could it be possible to recreate this shader in Amplify or Shader Graph? If someone could help me Out I would be very grateful because for now, I'm depending on extra big colliders to avoid my sprites clipping into some walls. Thank you in advance!

 

Unfortunately not really.

The technique relies heavily on being able to directly modify the clip space position output by the vertex shader, which is something neither Amplify nor Shader Graph lets you do. Both only let you modify the local vertex position. Modifying the local vertex position can get you something similar, but it comes with potentially significant texture distortion artifacts. It doesn’t help that

The alternative would be to do these calculations in the fragment shader and output a new depth there. Unfortunately Shader Graph for the URP doesn’t allow this, and I’m not sure if Amplify does either. It is possible with HDRP though.

 

Thanks for the quick response. That's sad. I actually did what you mentioned regarding the vertex position and it does distort the pixels so that's a no-no. Guess there is no other way around it. The time has come to dive into the strange world of coding shaders. Any good docus in that regard? Thanks in advance

 

Sorry to revive this, but is there absolutely nothing that can be done?
In my game I actually need billboarding sprites, but lit! I always thought to myself "I can just convert that shader into a surf shader later", but today I learned surf shaders are kinda deprecated and don't work with URP/HDRP. So, from what I searched, Shader Graph is the only reliable way to write custom lit shaders in SRP. But then I can't use your algorithm. So I either find a magical way to port this into Shader Graph, write my own HLSL lighting system, or figure out a different algorithm that works with Shader Graph.
Any suggestions? I'm not a shader expert by any means, so I'm struggling to understand why we can't port this to Shader Graph. Can't we manually convert to clip space using the transformation matrices, do all the billboarding logic, and then convert it back into object space?

I'm not sure if I understood what you meant here, but maybe this can be avoided by setting the sprites mesh type to Full Rect?

Thanks a lot for all the help in this thread!! Cheers.

 

Clip space isn’t a float3 3D position. It’s a float4 projective space “4D” position. There are ways you can modify clip space that cannot be replicated in a float3 3D position. So no.

I recently posted a thread on twitter where I show off the problems with replicating the technique only in 3D space. With also shows off the texture distortion issues.
https://twitter.com/bgolus/status/1518994511552466945?s=21&t=iEl6eJ_NdQ6Yy4-Bo3WOvQ

But there is one “solution”. You can make your shader in Shader Graph, then get the generated shader code and paste that into a .shader file to modify by hand.

 

Thanks a lot for the reply! Your tweet was very interesting, I would never expect that sort of distortion to occur. I just couldn't accept it hahah, so I took the time to fully understand the method and replicate it in Blender. I'll leave it here in case it helps someone understand this in the future:

The camera is viewing the quad, but the perspective ruins the sprite.


This is the quad that directly faces the camera that we need to match.


So we manipulate the vertices by stretching its length and reducing the x scale as it goes up.


If done correctly, it perfectly matches the other quad by the camera's perspective. But as you can see, there is a lot of distortion in the texture!


Pretty interesting. It's still mindblowing to me that this happens in local space, but not in clip space. All I know is that all this is way past my current knowledge level, so I'll just roll with it and move on, haha. Thanks again!

Edit: screenshot with more vertices. Now I kind of get it, the distortion seems to be pretty obvious, squares closer to the camera are bigger than the ones further away. We fix that horizontally cause of the x rescaling, but not vertically. This seems like a different distortion from the one in the tweet though, so maybe I misunderstood the process? Or maybe its cause of the weird topology of the sprite "quad" when in Tight mesh type.

 

In a more straightforward test. Take a mesh quad and move the top two vertices toward each other to make a trapezoid. The UVs will be distorted. That's the kind of mesh being created. That inevitably creates some kind of distortion.

Clip space works by letting the x and y positions be interpolated as if it's just a rotated quad, and move the z separately toward the camera such that it only affects the depth.

 

Hiya, I've been playing around with these solutions, this thread is a miracle repository for billboarding. It seems to work fine for higher-resolution sprites. However whenever I use lower resolution sprites with a perspective camera, no matter the billboarding method I use I seem to get these blurry edges to the sprite as seen here: https://imgur.com/XeCNwRq

Changing FOV seems to affect this greatly. I had to turn FOV down to 20 which is fine, it looks better anyway, don't want TOO much perspective for a top-down game.

Just wondering out of curiosity if anyone knows what's causing this. I wouldn't want it to show up in a larger project.

Currently I'm using the bgolus's second solution here: https://forum.unity.com/threads/pro...ping-into-3d-environment.680374/#post-6852500

 

That looks more like a filtering problem. Do you have the image asset Filter Mode set to

Point (no filter)

and Compression set to

None

? It looks like currently it's bilinear filtering.

 

Yep on both of those. It does look that way, though, I agree. Here's the inspector: https://i.imgur.com/F48vZoO.png

 

Thank you so much. I am very impressed with your solution.

There is one thing I want to add. how do I cast a shadow over the sprite?

I solved the shadows cast from the sprite but not the other shadows cast over the sprite.

 

The shader is provided as a proof of concept and is intentionally missing further functionality.

Normally to get shadow receiving working for sprites, the solution is to use an alpha tested Surface Shader. But as Surface Shaders don't allow you to modify the vertex shader's clip space directly they can't be used to do this technique. So instead you'll need to figure out how to write a vertex fragment shader that supports shadow receiving, or at least how to merge an existing vertex fragment shader with that support with this billboard shader.

Unity's own documentation includes a basic example shadow receiving shader.
https://docs.unity3d.com/Manual/SL-VertexFragmentShaderExamples.html

 

Thank you for good advices. Let me try to merge two shaders.

 

If you get solid results I'd be pretty stoked to see the outcome. I've changed to 3D models for rendering characters on my project for other reasons, but this is still maybe one of the only concrete and imitable examples of this shader effect on the internet, so sprites with fake depth that receive shadows and lighting would be an even nicer addition.

Another thing that comes to mind is "attaching" sprites to other sprites, so that they share a pivot or at least temporarily use the same pivot coordinates. I figure this would be pretty necessary if you wanted sprites to carry objects or touch one another for more than a passing interaction (eg. weapon sub-sprites; characters throwing characters).

 

Thanks very much everyone for all this info. I modified this shader to receive shadows. I probably didn't do it the "right" way, but it works for me.



Here's my code if it helps anyone:

Code (CSharp):

1.  
2. Shader "Custom/BillboardSprite"
3. {
4.    Properties
5.    {
6.        _MainTex("Texture", 2D) = "white" {}
7.    }
8.  
9.        SubShader
10.    {
11.        Tags{ "Queue" = "Transparent" "IgnoreProjector" = "True" "RenderType" = "Transparent" "DisableBatching" = "True" }
12.  
13.        ZWrite Off
14.        Blend SrcAlpha OneMinusSrcAlpha
15.  
16.        Pass
17.        {
18.            HLSLPROGRAM
19.            #pragma vertex vert
20.            #pragma fragment frag
21.  
22.            #pragma multi_compile _ _MAIN_LIGHT_SHADOWS _MAIN_LIGHT_SHADOWS_CASCADE _MAIN_LIGHT_SHADOWS_SCREEN
23.            #pragma multi_compile_fragment _ _SHADOWS_SOFT
24.            
25.            #include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Lighting.hlsl"
26.            #include "Packages/com.unity.render-pipelines.universal/Shaders/LitInput.hlsl"
27.            #include "Packages/com.unity.render-pipelines.universal/Shaders/LitForwardPass.hlsl"
28.  
29.            struct appdata
30.            {
31.                float4 vertex : POSITION;
32.                float2 uv : TEXCOORD0;
33.            };
34.  
35.            struct v2f
36.            {
37.                float4 pos : SV_POSITION;
38.                float2 uv : TEXCOORD0;
39.                float4 vert : TEXCOORD1;
40.            };
41.  
42.            sampler2D _MainTex;
43.            float4 _MainTex_ST;
44.  
45.            float rayPlaneIntersection(float3 rayDir, float3 rayPos, float3 planeNormal, float3 planePos)
46.            {
47.                float denom = dot(planeNormal, rayDir);
48.                denom = max(denom, 0.000001); // avoid divide by zero
49.                float3 diff = planePos - rayPos;
50.                return dot(diff, planeNormal) / denom;
51.            }
52.  
53.            v2f vert(appdata v)
54.            {
55.                v2f o;
56.  
57.                o.uv = v.uv.xy;
58.  
59.                // billboard mesh towards camera
60.                float3 vpos = mul((float3x3)unity_ObjectToWorld, v.vertex.xyz);
61.                float4 worldCoord = float4(unity_ObjectToWorld._m03, unity_ObjectToWorld._m13, unity_ObjectToWorld._m23, 1);
62.                float4 viewPos = mul(UNITY_MATRIX_V, worldCoord) + float4(vpos, 0);
63.  
64.                o.pos = mul(UNITY_MATRIX_P, viewPos);
65.  
66.                // calculate distance to vertical billboard plane seen at this vertex's screen position
67.                float3 planeNormal = normalize(float3(UNITY_MATRIX_V._m20, 0.0, UNITY_MATRIX_V._m22));
68.                float3 planePoint = unity_ObjectToWorld._m03_m13_m23;
69.                float3 rayStart = _WorldSpaceCameraPos.xyz;
70.                float3 rayDir = -normalize(mul(UNITY_MATRIX_I_V, float4(viewPos.xyz, 1.0)).xyz - rayStart); // convert view to world, minus camera pos
71.                float dist = rayPlaneIntersection(rayDir, rayStart, planeNormal, planePoint);
72.  
73.                // calculate the clip space z for vertical plane
74.                float4 planeOutPos = mul(UNITY_MATRIX_VP, float4(rayStart + rayDir * dist, 1.0));
75.                float newPosZ = planeOutPos.z / planeOutPos.w * o.pos.w;
76.  
77.                // use the closest clip space z
78.                #if defined(UNITY_REVERSED_Z)
79.                o.pos.z = max(o.pos.z, newPosZ);
80.                #else
81.                o.pos.z = min(o.pos.z, newPosZ);
82.                #endif
83.  
84.                o.vert = v.vertex;
85.  
86.                return o;
87.            }
88.  
89.            half4 frag(v2f i) : SV_Target
90.            {
91.                half4 color = tex2D(_MainTex, i.uv);
92.  
93.                //if (_ReceiveShadows)
94.                {
95.                    float4 worldCoord = float4(unity_ObjectToWorld._m03, unity_ObjectToWorld._m13, unity_ObjectToWorld._m23, 1);
96.                    worldCoord.x -= i.vert.x;
97.                    worldCoord.y += i.vert.y;
98.  
99.                    float4 shadowCoord = TransformWorldToShadowCoord(worldCoord);
100.                    half4 shadowMask = unity_ProbesOcclusion;
101.                    half shadow = MainLightShadow(shadowCoord, worldCoord, shadowMask, _MainLightOcclusionProbes);
102.  
103.                    float _ShadowStrength = 0.3;
104.                    float mul = shadow + _ShadowStrength;
105.                    if (mul > 1)
106.                    {
107.                        mul = 1;
108.                    }
109.                    color.rgb *= mul;
110.                }
111.  
112.                return color;
113.            }
114.            ENDHLSL
115.        }
116.    }
117. }
118.  

 

Sorry for the double post - I also have a question. Does anyone know how I can fix this issue with sprite rendering behind water? @bgolus ?




EDIT: Fixed by using a custom render feature for the water layer, from here: https://forum.unity.com/threads/zwrite-in-shader-graph.719012/#post-5848573

 

This post was also very helpful for me:
https://forum.unity.com/threads/an-issue-with-3d-sprite-billboarding-and-sorting.1256973/

I was able to fix the issue using an incredibly simple 3D shader made in shader graph.

 

Would it be possible to also remove the clip in one of the two axes X or Y??