Difficulty using normal maps to transform vertices in vertex shader

Discussion in 'Shaders' started by Iron-Warrior, Mar 19, 2017.

1. Iron-Warrior

Joined:
Nov 3, 2009
Posts:
637
Hi, I'm attempting to use a normal map to displace vertices on the ZX plane (in object space). The reason I'm doing this is to eventually be able to pass in a dynamically generated normal map for effects like grass being pushed in a specific direction, or water ripples. Here is what I currently have:

Code (csharp):
1.
3. {
4.     Properties
5.     {
6.         _MainTex ("Texture", 2D) = "white" {}
7.         _Cutoff("Alpha cutoff", Range(0,1)) = 0.5
8.
9.         _ForceMap("Force Map", 2D) = "black" {}
10.     }
12.     {
13.         Tags { "RenderType"="Opaque" }
14.         LOD 100
15.         Cull Off
16.
17.         Pass
18.         {
19.             CGPROGRAM
20.             #pragma vertex vert
21.             #pragma fragment frag
22.
23.             #include "UnityCG.cginc"
24.
25.             struct appdata
26.             {
27.                 float4 vertex : POSITION;
28.                 float2 uv : TEXCOORD0;
29.             };
30.
31.             struct v2f
32.             {
33.                 float2 uv : TEXCOORD0;
34.                 float4 local : TEXCOORD1;
35.                 float4 vertex : SV_POSITION;
36.             };
37.
38.             sampler2D _MainTex;
39.             float4 _MainTex_ST;
40.
41.             sampler2D _ForceMap;
42.             float4 _ForceMap_ST;
43.
44.             v2f vert (appdata v)
45.             {
46.                 v2f o;
47.
48.                 o.local = v.vertex;
49.
50.                 float2 coords = float2(v.vertex.x, v.vertex.z) / 10 + 0.5;
51.                 float3 normal = UnpackNormal(tex2Dlod(_ForceMap, float4(coords, 0, 0)));
52.                 normal.y = 0;
53.
54.                 float4 transformed = v.vertex;
55.                 transformed.xyz += normal;
56.
57.                 o.vertex = UnityObjectToClipPos(transformed);
58.                 o.uv = TRANSFORM_TEX(v.uv, _MainTex);
59.                 return o;
60.             }
61.
62.             float _Cutoff;
63.
64.             fixed4 frag (v2f i) : SV_Target
65.             {
66.                 // sample the texture
67.                 fixed4 col = tex2D(_MainTex, i.uv);
68.                 clip(col.a - _Cutoff);
69.
70.                 float2 coords = float2(i.local.x, i.local.z) / 10 + 0.5;
71.                 fixed4 normalColor = tex2D(_ForceMap, float4(coords, 0, 0));
72.
73.                 return fixed4(coords.x, coords.y, 0, 0);
74.             }
75.             ENDCG
76.         }
77.     }
78. }
79.
I transform the vertex's ZX position into UV coords by first dividing by the mesh's bounds (I am using the default Unity plane to test, and it is 10x10) and then adding 0.5, as Unity's UV coords are in the range of 0-1 (I think), and the mesh is centered around the 0 (being -5 to 5). I then sample the Force map texture at these coords and unpack them into a normal. Here is the normal map I'm using to test:

My assumption is that for tangent space normals, the pastel blue colors are straight up, while the red, dark blue and greens would be pointing towards the edges of the texture. I don't use normal maps much, so please correct any assumptions of mine that are not on the mark!

So I zero out the Y of the normal, add the normal to the vertex and then transform it. My expectation would be to have the mesh stay centered around the 0, the vertices being pushed out into a circle (and the verts on the edge unchanged).

It's kind of there, but two main issues: it's sort of a messy circle, which I don't entirely understand. As well, the entire plane is translated 1 unit up the z axis (I have a copy of the plane with the default material on it below for comparison), which again I'm confused about.

Side note: Does anyone have any general advice on how to debug shaders properly? Outside of coloring normals and UVs, I can't seem to find any handy tools like Debug.Log or stepping the shader.

Thanks for any help,
Erik

2. Iron-Warrior

Joined:
Nov 3, 2009
Posts:
637
Figured it out! When you unpack a normal, the tangent normal is X/Y for left/up, and Z for coming out of the screen...which seems pretty obvious in retrospect, but I guess I was super in Unity mode and didn't think enough about the coordinates system.