# [SOLVED] Vertex modifier in surface shader: mesh moves but shadow doesn't update (with addshadow)

Discussion in 'Shaders' started by TheCatProblem, Sep 2, 2018.

1. ### TheCatProblem

Joined:
May 26, 2011
Posts:
22
(Sorry, this is a bit long; in summary, I've tried two methods of doing some math in a vertex modifier function. Both of them seem correct and result in the expected mesh rotation, but the shadow cast by the mesh only updates when using the second method. Details and code below!)

I'm writing a billboard shader (i.e., one in which the model is rotated such that its forward/Z axis is parallel to the normal of the view space's XY plane) that's limited to rotation around the model's up axis (+Y in model space). It requires the view space Z axis (or -Z axis; whichever points out of the screen) expressed in model space. (The angle between the projection of this vector onto the XZ plane in model space and the +Z axis in model space is then calculated; each vertex is rotated about the model space +Y axis by this angle.)

I've tried two methods of obtaining the view space Z axis expressed in model space. Both of them result in identical, correct rotation of the mesh (it rotates to face the view space XY plane to the extent possible by rotation around the model space +Y axis). However, when using the first method, the shadow cast by the mesh doesn't change when the mesh rotates. With the second method, it does.

Method 1 (mesh rotates, shadow doesn't update)
``float3 viewZAxis = UNITY_MATRIX_MV[2].xyz;``

(For the record, the logic behind this is as follows: the modelview matrix transforms from model space to view space and its upper 3x3 is the rotation matrix part of that transformation. Since the columns of a rotation matrix express the basis vectors of the source frame in the destination frame and the transpose of a rotation matrix is equal to its inverse because rotation matrices are orthogonal, the third row of the rotation matrix going from model to view is the third column of the rotation matrix going from view to model and is the +Z axis of the view frame expressed in the model frame.)

Method 2 (mesh rotates and shadow updates)
``````float3 viewZAxis = mul((float3x3)unity_CameraToWorld, float3(0,0,-1));
viewZAxis = mul((float3x3)unity_WorldToObject, viewZAxis);``````

(The matrix unity_CameraToWorld is the inverse of the view matrix. It seems to switch the direction of the view space +Z axis compared to the modelview matrix, hence the -1. Oddly, unity_CameraToWorld is not documented on https://docs.unity3d.com/Manual/SL-UnityShaderVariables.html but does appear in UnityShaderVariables.cginc and is mentioned in lots of forum posts.)

Here's the shader. All I've really done is add a vertex modifier function to the default surface shader. I've also included the addshadow pragma (per https://docs.unity3d.com/Manual/SL-SurfaceShaders.html) and set DisableBatching to True in the tags (per https://docs.unity3d.com/Manual/SL-SubShaderTags.html). Everything I added is called out with an "ADDED" comment.

Code (csharp):
1. Shader "Custom/BillboardYAxis" {
2.     Properties {
3.         _Color ("Color", Color) = (1,1,1,1)
4.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
5.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
6.         _Metallic ("Metallic", Range(0,1)) = 0.0
7.     }
9.         // ADDED DisableBatching
10.         Tags { "RenderType"="Opaque" "DisableBatching"="True" }
11.         LOD 200
12.
13.         CGPROGRAM
14.         // Physically based Standard lighting model, and enable shadows on all light types
17.
18.         // Use shader model 3.0 target, to get nicer looking lighting
19.         #pragma target 3.0
20.
21.         sampler2D _MainTex;
22.
23.         struct Input {
24.             float2 uv_MainTex;
25.         };
26.
27.         half _Glossiness;
28.         half _Metallic;
29.         fixed4 _Color;
30.
31.         // Add instancing support for this shader. You need to check 'Enable Instancing' on materials that use the shader.
33.         // #pragma instancing_options assumeuniformscaling
34.         UNITY_INSTANCING_CBUFFER_START(Props)
35.             // put more per-instance properties here
36.         UNITY_INSTANCING_CBUFFER_END
37.
39.         // Rotates about the Y axis by theta radians.
40.         float4 RotateAroundY (float4 vec, float theta)
41.         {
42.             float sinTheta, cosTheta;
43.             sincos(theta, sinTheta, cosTheta);
44.
45.             // Transformation matrix to apply rotation about the y axis.
46.             float4x4 m = float4x4(
47.                 cosTheta, 0, sinTheta, 0,
48.                 0, 1, 0, 0,
49.                 -sinTheta, 0, cosTheta, 0,
50.                 0, 0, 0, 1);
51.
52.             return mul(m, vec);
53.         }
54.
56.         void vert (inout appdata_full v) {
57.
58.             // METHOD 1: Mesh rotates as expected, but shadow doesn't change.
59.             //float3 viewZAxis = UNITY_MATRIX_MV[2].xyz;
60.
61.             // METHOD 2: Mesh rotates as expected and shadow updates.
62.             float3 viewZAxis = mul((float3x3)unity_CameraToWorld, float3(0,0,-1));
63.             viewZAxis = mul((float3x3)unity_WorldToObject, viewZAxis);
64.
65.             // Remaining code in this function is the same regardless of the method used above.
66.
67.             float theta = atan2(viewZAxis.x, viewZAxis.z);
68.
69.             v.vertex = RotateAroundY(v.vertex, theta);
70.             v.normal = RotateAroundY(float4(v.normal, 0.0), theta).xyz;
71.         }
72.
73.         void surf (Input IN, inout SurfaceOutputStandard o) {
74.             // Albedo comes from a texture tinted by color
75.             fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
76.             o.Albedo = c.rgb;
77.             // Metallic and smoothness come from slider variables
78.             o.Metallic = _Metallic;
79.             o.Smoothness = _Glossiness;
80.             o.Alpha = c.a;
81.         }
82.         ENDCG
83.     }
84.     FallBack "Diffuse"
85. }
I'm stumped as to why the shadows update when using method 2 but not when using method 1, particularly since the mesh rotation is correct for both methods. All I can think of is that accessing UNITY_MATRIX_MV does something under the hood that disables shadow updates, but that seems unlikely.

Any help would be appreciated! Although I could just use method 2 and call it a day, I'm concerned that unity_CameraToWorld may disappear in future versions (since it's undocumented) and frankly this behaviour seems bizarre and I'd like to understand what's going on.

2. ### bgolus

Joined:
Dec 7, 2012
Posts:
11,319
UNITY_MATRIX_MV is the model view matrix for the current view being rendered. In the case of a shadow, it's the view from the light, not the main camera. The unity_CameraToWorld is a transform matrix based on the main camera, no matter what view you're rendering.* This is why it works. When using method 1 it's turning the mesh toward the light when rendering the shadow maps, as that's the data that matrix has.

3. ### TheCatProblem

Joined:
May 26, 2011
Posts:
22
Ah, that makes sense; thanks for your help!

unityunity