How to properly capture normals of a model to project them onto a plane

Hello, fellow uniters.
From a couple of weeks ago, I have been struggling with an apparently easy task.
tl;dr: How I can capture a normal map from a model inside Unity, to apply it onto a quad? AKA: normal baking

Context. I am doing a 360 impostor system that renders any 3D model from different angles, and then applies the captured textures to a billboarded quad (cubemaps and all involved).

The system works pretty well, but there's a point of conflict: I can't manage to get normal mapping work.

In this thread I previously posted (thanks to bgolus for his very appreciated help and literature), you can see the issue: lighting is still behaving like it is affecting a plane, and not a volumetric mesh (or a plane with a normal map on it).
I initially blamed texture format for the error (and I had some format issues). But after digging a little bit more, I found that it has more relation to the way I was capturing the normal map.

The method is simple: paint the normals colors onto the model via custom shader, make a render, encode them onto a texture. However, it turns to be a handful of ways to reflect normals onto the model.

In my first attempt, I simply used o.normal = mul(UNITY_MATRIX_IT_MV, v.normal) in a vertex shader to transform the object normals to eye space. This resulted in a tangent-space normal map that derived into the previous mentioned error: lighting behaved like there was no volume, only a plane with some bumping on it.



After searching more, I found that some existing implementations were capturing the normals in world space and then, they transform them to match the position of the quad.
I tried this new approach, and the results were better, but still wrong: I had the volume sensation I was searching for, but the light direction was not the correct.



I tried to transform the normals by the quad position, and to not modify the vertex tangent and normals to make the world-space normal map match. Nothing seems to work.

After another week of frustrating back and forthing, I have decided to reach for help here, hoping for some explanation at mathematical level. I think that if I go back to basics and try to understand how normals and spaces work, I will figure how get the system done.

Hence, my specific questions:
-Why if I use tangent space normals (o.normal = mul(UNITY_MATRIX_IT_MV, v.normal)) I can't achieve the volumetric lighting behaviour? (Could it be because tangent normals are conditioned by vertex normals, which are different from the original model to the quad?)
-How can I transform world-space normal map to object space, to fix the light direction?

 

Fundamentally normal maps are very simple. They’re a record of a direction. Tangent space normal maps are a direction that’s relative to the orientation of a surface’s UVs (xy) and vertex normal (z). World space or object space normal maps are the same, but relative to the orientation of those spaces. And that’s all spaces are too, a definition of what left right, forward back, and up down are.

For Unity, world (and object) space is +x is right, +y is up, and +z is forward.


Unity’s tangent space uses OpenGL’s spec, which is +x right, +y up, +z “out”. That means the orientation doesn’t match world space. Either the Z is inverted, or the X is.

In my last post there I guessed it was the x that was inverted, but it might be the z, and that’s assuming you’re using the default quad. If you’re using your own mesh then I have no idea and it’d be up to you to figure out what the conversion is. If it’s rotated by some constant direction, you probably need to swizzle, if it’s rotating the opposite direction you need to invert one of the axis.

For OpenGL tangent space normal maps, the vertex normal and tangent vectors are the object space direction of the +z and +x axis, with +y being the cross product of the two (multiplied by the w of the tangent, which usually defaults to -1 for UVs with the usual orientation). For a flat axis aligned meshes like a quad, you almost don’t need that data though as it might mostly align to object space (again, with some axis needing to be swizzled or inverted). If you look at a tangent space normal map one channel at a time, the red should look like it’s being lit on the right side, green lit from above, and blue like light is coming from you.

Unity’s view matrix actually aligns to tangent space normals perfectly; +x “right”, +y “up”, +z towards the camera. The normal maps generated from an orthographic camera should match perfectly the orientation of Unity's tangent space normal maps. There's nothing wrong there.


Whatever is wrong with your system comes down to something specific to however you're handling the rotation of each imposter, both in the "object's" rotation and the billboarding. If you actually use IMP's entire system it works flawlessly (even though as I noted it could be more efficient in its shader code). When I replaced IMP's custom meshes with the built in default quad, flipping the x fixed it.

If you're using world space rendered normals (which you then use as object space) removing all modifications to the vertex normal and tangent in the shader should fix the issue and all that's needed is a swizzle and / or inversion on a component or two depending on what mesh you're using.

If you're using view space rendered normals (which you then use as tangent space) then you'll need to update the billboard's vertex normals and tangent with the same rotation you apply to do the billboarding. Btw, you don't need to worry about doing the inverse transpose for the normal, just use the same rotation you're applying to the vertex position and tangent.xyz. Using the inverse transpose for normals only matters when doing scaling on non-axis aligned, and non-uniform scaled objects. You're rotating an axis aligned quad uniformly, so it can be ignored. (The inverse transpose of a uniform rotation matrix is the same as the original matrix.)

 

Thanks for the in-depth response again, bgolus, when I reach work I will try again the swizzle, taking in account how axis change from space to space

 

(EDITED: THIS POST IS WRONG - SEE NEXT POST)
Hello again! Sorry for the necrobump, but these months were strange for everyone, for sure.
Anyways, I have resumed my work with impostors, and I have rewritted the shader for cleaning purposes, and I have re-checked all the theory behind it. I think I have isolated the issue, but the problem is that I don't know why it is working the way it is.

Things to consider:
-I am usign the Unity quad
-I am using world-space normal-map
-I have tried swizzling/inverting X and Z axis (none works)
-I am only transforming vertex position: normal and tangent remain unchanged

The result is the same, but I have noticed something important.
The lighting works as intended if the impostor quad is facing the -z direction:


If i rotate the quads slightly, the light behaves as if the font rotates with them:


This behaviour would make sense if the vertex normals and tangents were rotating as well (since the normal map is worls space).
But the twist is that those vertex normals are not being altered at all, despite following your advices and making sense in theory. Swizzling and inverting x and z axes only result in light-direction being altered, but not fixed.

So, my final guess is that there's only one operation that I need to do to the normals to fix the light, but I still don't know what, or why.

 

Ok, so I am just plain imbecile and I was using a tangent-space normal map instead of the World space one. My bad.
HOWEVER, I have just figured where the error is.
The fact is: I am making all the calculations in world space, asuming the impostor (the full character) has no rotations on it.
As @bgolus said, the normal and tangent should be inaltered, since the normal map is world space; and one of the axis (in this case, X) should be inverted.
What was missing is that the impostor itself could be facing other directions, and I was not adding that transform.

If I change the code like this:

Code (CSharp):

1. o.normal = mul(unity_WorldToObject, float3(0, 0, 1));
2.     o.tangent = mul(unity_WorldToObject, float4(-1, 0, 0, -1));
3.  
4.     //Calculate quad vertex position in world space
5.     int offset = offsetsTransformsBuffer[instanceID / _NumJoints];
6.     float4x4 localTransform = transformsBuffer[offset * _NumJoints + (instanceID % _NumJoints)]; //quad in impostor
7.     float4x4 worldTransform = worldTransformsBuffer[instanceID / _NumJoints]; //impostor in world
8.     float4x4 transform = MatrixMultiplication(localTransform, worldTransform); //quad in world
9.     //
10.  
11.     o.normal = mul(worldTransform, o.normal); //Rotate the normals to face characters direction
12.     o.tangent = mul(worldTransform, o.tangent);

The result turns to:
https://i.gyazo.com/67342b73b831bdb2d9969a9963290bf7.mp4
And it works!

BUT

It still has some issues. If I extend the impostors all over the scene, artifacts appear in those that are far away from the 0, 0, 0


I suspect this is because I apply all the transform to the normal, and I should use only the rotation.

 

Doing a

mul

on a

float4

with a

w

of anything other than 0.0 will treat that as a position, not a direction vector.

 

Thank you for the tip! It sure saved me a lot of time.

Behold, the power of GPU-instanced impostors:
https://i.gyazo.com/a10afa40e9d17be36e9d126217569f71.mp4



Works almost perfectly now. The only thing that is bugging me, Is a strange behaviour light has when the impostor and the lamp are aligned in the Y axis.

See that is not properly lit?
I have no idea what could be causing this, but my bet would be on an improper signing of Y axis in some calculation.
I initially thought about some Bitangent operation I was missing, but as far as I know, bitangent is automatically calculated from normal and tangent by Unity, and there's no bitangent variable on the shader.