[Solved]Object space normal map with wrong behaviour in play mode.

I'm trying to figure out how to implement a standard based surface shader that supports object space normal maps but I'm getting a different shader behaviour while in play mode. Here's some pics:

Spoiler

Scene view:

Play mode:

Here's how I'm doing the normal map:

Code (CSharp):

1. #ifdef IS_OBJ_SPACE
2.                     o.ObjNormal = normalize (mul (normalMap.xyz * 2.0 - 1.0, (float3x3) unity_WorldToObject));
3.                 #else
4.                     o.ObjNormal = normalMap * 2.0 - 1.0;
5.                 #endif

I put the toggle for debbug purpose and I notice that when it's off, the material looks the same, so it's not applying my custom normal map calculation for object space normal maps and just using the one I think is default in Unity and only for tangent space normal maps. I'm using the internal functions for lightning from the UnityPBSLighting.cginc.

Code (CSharp):

1. inline fixed4 LightingCustomStandard (SurfaceOutputCustomStandard s, half3 viewDir, UnityGI gi)
2.             {
3.                 s.ObjNormal = normalize(s.ObjNormal);
4.  
5.                 half oneMinusReflectivity;
6.                 half3 specColor;
7.                 s.Albedo = DiffuseAndSpecularFromMetallic (s.Albedo, s.Metallic, /*out*/ specColor, /*out*/ oneMinusReflectivity);
8.  
9.                 half outputAlpha;
10.                 s.Albedo = PreMultiplyAlpha (s.Albedo, s.Alpha, oneMinusReflectivity, /*out*/ outputAlpha);
11.  
12.                 half4 c = UNITY_BRDF_PBS (s.Albedo, specColor, oneMinusReflectivity, s.Smoothness, s.ObjNormal, viewDir, gi.light, gi.indirect);
13.                 c.a = outputAlpha;
14.  
15.                 return c;
16.             }
17.  
18.             inline void LightingCustomStandard_GI (SurfaceOutputCustomStandard s, UnityGIInput data, inout UnityGI gi)
19.             {
20.                 #if defined(UNITY_PASS_DEFERRED) && UNITY_ENABLE_REFLECTION_BUFFERS
21.                     gi = UnityGlobalIllumination(data, s.Occlusion, s.ObjNormal);
22.                 #else
23.                     Unity_GlossyEnvironmentData g = UnityGlossyEnvironmentSetup(s.Smoothness, data.worldViewDir, s.ObjNormal, lerp(unity_ColorSpaceDielectricSpec.rgb, s.Albedo, s.Metallic));
24.                     gi = UnityGlobalIllumination(data, s.Occlusion, s.ObjNormal, g);
25.                 #endif
26.             }

My two main questions are, why it's displaying different results in scene view and play mode and what can I do to fix this problem?

 

You don't by chance have your mesh's game object set to be static, do you? Object space normals won't work on static or dynamically batched objects as the unity_WorldToObject matrix will be an identity matrix (ie: no scale, offset, or rotation) after the mesh has been batched. Try adding "DisableBatching"="True" to your shader's Tags {} block.

 

It's working now, thank you so much! I didn't know about this information. Looks like Unity just applies the "static thing" in play mode, make sense. Two more questions, if I know enough about batching, it will affect negatively my performance, did you have some advice about it? And why I can't assing a tangent space texture to the same o.objNormal, i'm trying but the light always looks wrong.

 

Yes, it will. My advice is don't use object space normal maps. Use tangent space normal maps instead. They exist for a reason.

Because a tangent space normal is in tangent space, not object space?
https://medium.com/@bgolus/normal-mapping-for-a-triplanar-shader-10bf39dca05a#0576

Unity's surface shaders will handle all of this for you if you use the o.Normal output from a surf function. By the time your lighting function gets it the .Normal will be in world space.

 

@bgolus I was referring about assing it with a proper command, like Unpack Normal or normalMap * 2 - 1. Cause o.objNormal is just a new element of the struct to replace o.Normal in the custom lighting functions, I thought there was a way for that whitout using o.Normal. I'm using object space cause this assets are from BotD, and unfortunately there's nothing I can do about it.

 

If you're using the same rock over and over again, like how BotD seemed to use their assets, then using instancing instead of batching can be quite powerful. So in that case object space normal maps aren't as much an issue. Object space normal maps can also be a little more graceful at hiding LOD swaps, though I'm not sure if BoTD is using mesh LODs or not.

If you're passing object space normals, passing them as a custom variable is a fine way of handling it with how Unity's Surface Shaders are setup. But object and tangent space normals are fundamentally incompatible, and it goes beyond just how to unpack them, though even that is different.

This is going to go deep into tangent space normal maps and how Unity stores / unpacks them.

At the most basic a normal map is a 3D vector packed into a texture, usually by taking a normalized vector and doing normal * 0.5 + 0.5 to pack it, then normalmap * 2.0 - 1.0 to unpack it. This is mostly true for tangent space normal maps too, but there's a key difference between tangent space and object or world space normals that most games take advantage of. In a tangent space normal the Z will always be positive. This, combined with the fact it's a normalized vector, means you can reconstruct the normal from only the unpacked x and y components, so that's all that's stored in the texture. Using a DXT5 texture where one channel of the normal is stored in the high quality alpha channel, and the other is stored in the next highest quality green channel results in a compressed tangent space normal map that is higher quality than using all three components more simply stored in a DXT1, or even often better than storing the z component in one of the other color channels of the DXT5 texture. There are also formats like BC5 which is basically two DXT5 alpha channels in an RG compressed texture which results in even higher quality normals and has been the standard for consoles (for games not using Unity) for the last decade.

So, since Unity only stores two components of the normal, the UnpackNormal function that Unity uses is more than just the normalmap * 2.0 - 1.0, but rather this:

Code (CSharp):

1. // Unpack normal as DXT5nm (1, y, 1, x) or BC5 (x, y, 0, 1)
2. // Note neutral texture like "bump" is (0, 0, 1, 1) to work with both plain RGB normal and DXT5nm/BC5
3. fixed3 UnpackNormalmapRGorAG(fixed4 packednormal)
4. {
5.     // This do the trick
6.    packednormal.x *= packednormal.w;
7.  
8.     fixed3 normal;
9.     normal.xy = packednormal.xy * 2 - 1;
10.     normal.z = sqrt(1 - saturate(dot(normal.xy, normal.xy)));
11.     return normal;
12. }
13. inline fixed3 UnpackNormal(fixed4 packednormal)
14. {
15. #if defined(UNITY_NO_DXT5nm)
16.     return packednormal.xyz * 2 - 1;
17. #else
18.     return UnpackNormalmapRGorAG(packednormal);
19. #endif
20. }

But you can't use that trick for object space normals as the z can be negative. So instead you have to store all 3 components in a texture. Really if you could somehow store two of the components at a higher quality and store just the sign of the last component that would be even better, but there aren't any GPU texture formats I'm aware of that are like that.

The other key difference between object space and tangent space is how you need to transform that from the space it's stored in and world space which is needed for doing lighting. Object space is straight forward, multiply by the transpose inverse object to world rotation matrix, which is what you're doing. Tangent space requires using an interpolated per-vertex tangent to world matrix which requires extra data stored in the mesh (though Unity generates this and has it available to shaders by default for all meshes). That's why you need to use o.Normal for tangent space normals unless you're going to pass & calculate the data needed on your own.

 

@bgolus Oh, I forgot about instancing. You're right, this should hold my performance. Thanks for the explanation, I was not aware of how normal maps work, now I have a starting point. Well, no more questions! haha

Thanks for the help, I really appreciate it.