Runtime normal map formats

Hi

I'm struggling to load normal textures at runtime (not from the unity editor). I have read a bunch of threads about that, and was able to understand a lot, but now I came to the point where I'm really stuck.

Project settings :
unity 2020.3.14f1
HDRP
I'm using HDRP/Autodesk Interactive shader on my models.

> I want to be able to load textures from a folder on the computer at runtime and use it on my models
> everything works great, except for the normal textures

When I do import texture in the unity editor (not at runtime) I can select the texture type "normal maps" that makes it works. When I do this at runtime, I cannot.

So I'm trying to compute myself the normal texture at the expected format for Unity. So far I failed.

Based on this thread : https://forum.unity.com/threads/runtime-generated-bump-maps-are-not-marked-as-normal-maps.413778/

I tryed this :

Code (CSharp):

1.  
2. Texture2D loadedNormalMap = LoadTextureFromPath(f.FullName);
3. Texture2D convertedNormalMap = new Texture2D(loadedNormalMap.width, loadedNormalMap.height,TextureFormat.RGBA32, true, true);
4.  
5. Color loadedColor;
6. Color convertedColor;
7.  
8. for (int w = 0; w < loadedNormalMap.width; w++)
9. {
10.     yield return null;
11.     for (int h = 0; h < loadedNormalMap.height; h++)
12.     {
13.         loadedColor = loadedNormalMap.GetPixel(w, h);
14.         convertedColor = Color.white;
15.         convertedColor.r = 1;
16.         convertedColor.g = loadedColor.g;
17.         convertedColor.b = 1;
18.         convertedColor.a = loadedColor.r;
19.         convertedNormalMap.SetPixel(w, h, convertedColor);
20.     }
21. }
22. convertedNormalMap.Apply();
23. newMat.SetTexture("_BumpMap", convertedNormalMap);
24. newMat.SetInt("_UseNormalMap", 1);

Set aside the fact that it's very slow, it seems to works !

But looking closely, there is a huge loss of quality between a runtime loaded texture (modified with this code) and the same texture that would have been imported in the Unity editor and marked as "normal map".

Also if I display the texture.graphicsFormat for both texture :
My runtime generated texture says it's "R8G8B8A8_UNorm"
while the unity-editor-imported texture says "RGBA_DXT5_UNorm"

(which explains the quality gap I guess)

> How can I convert my base (.tga / .png ) runtime imported texture to the expected Unity format ?
> Alternatively, What should I ask to my artist coworker, for them to give me a .tga file that would be already all set up ?

PS : I tried to convert my texture to BC5 with another software, to try to import it (without marking it as "normal map" in the editor) and see the result, but it outputed me a DDS file that don't seems to be accepted by unity : "Unsupported Assets/Resources/test_normal_PNG_BC5_1.DDS file. "

PS2:
runtime texture preview (without running the code above):
On the mesh it looks completely wrong, normals looks inverted

runtime texture preview (after having run the code above) :
( texture.graphicsFormat : R8G8B8A8_UNorm)
On the mesh it looks correct

Here I dragged and dropped the Unity texture instead :
( texture.graphicsFormat : RGBA_DXT5_UNorm)
On the mesh it looks way better


@bgolus <3 :help:

 

Eek! Don't do that!

Two reasons:
1) Calling GetPixel() & SetPixels() on each individual pixel of the texture is super duper slow. If you're going to be doing CPU side texture manipulation you want to use GetPixels32() & SetPixels32() to get an array of Color32 values to modify and apply back onto the texture.

Code (csharp):

1. Texture2D loadedNormalMap = LoadTextureFromPath(f.FullName);
2. Texture2D convertedNormalMap = new Texture2D(loadedNormalMap.width, loadedNormalMap.height, TextureFormat.RGBA32, true, true);
3.  
4. Color32[] cols = loadedNormalMap.GetPixels32(0);
5. for (int i=0; i<cols.Length; i++)
6. {
7.     cols[i].a = cols[i].r;
8.     cols[i].r = 255; // Color32 uses a byte per component, so 255 == 1.0
9.     cols[i].b = 255;
10. }
11. convertedNormalMap.SetPixels32(cols, 0);
12. convertedNormalMap.Apply();

2) You don't need to do the swizzle at all anymore! There's very little benefit to swizzling runtime imported normals if you're going to leave the texture uncompressed. You just need to copy the loaded texture into a new one that's set to use linear color values. It's an unfortunate oversight that none of Unity's runtime texture loading systems let you tell it to load into a linear texture, which normal maps need to be, and they all default to sRGB textures. (Note: linear vs sRGB setting on a texture controls how the GPU reads the values when it samples it in the shader. It doesn't change the data in the texture that c# sees.)

You can do just this instead:

Code (csharp):

1. Texture2D loadedNormalMap = LoadTextureFromPath(f.FullName);
2. // note format is now copied from the loaded texture
3. Texture2D convertedNormalMap = new Texture2D(loadedNormalMap.width, loadedNormalMap.height, loadedNormalMap.format, true, true);
4.  
5. // the documentation on this function says it's GPU side only, but this is a lie
6. // this is roughly equivalent to:
7. // convertedNormalMap.SetPixels32(loadedNormalMap.GetPixels32(0), 0);
8. // if both textures are readable on the CPU, which they will be in this case
9. Graphics.CopyTexture(loadedNormalMap, convertedNormalMap);
10.  
11. convertedNormalMap.Apply();

This one is more curious.

R8G8B8A8_UNorm

is a higher quality format than

RGBA_DXT5_UNorm

. Both are used for the same 8 bits per color channel, but DXT5 is a lossy compressed format. So assuming the texture is properly set to be linear it should appear much higher quality than the DXT5 equivalent.

You could compress the texture at runtime using the

texture.Compress()

function, but that'll be even worse quality as it uses a real time compression algorithm that results in significantly worse looking textures than the editor compressed images. And for that you would want to do the swizzle again so there's data in the alpha channel as otherwise it'll compress as a DXT1 which will be much, much worse. (There's a reason why Unity compresses RGB normal maps to an RGBA DXT5 texture instead of an RGB DXT1 texture.)

Hopefully with the above "option 2" this isn't an issue anymore.

Unity's runtime image loaders can only parse .png or .jpg images. You can still technically load a .dds file at runtime, but you have to load the raw bytes of the file, parse those bytes to find the range that hold the image data, and then copy that into a

Texture2D

of the correct format and resolution using

Texture2D.LoadRawTextureData()

. The .dds file has a header that includes information about the format, resolution, mip count, etc. so you have to make sure you don't try to load that into the texture and which Unity itself no longer offers any tools to parse from c#.

 

Thanks a lot for the explanations !

"option 2" works very well indeed.

So isn't there a way to change that setting without copying the texture ?


PS: now it does look better (using option 2) than with the RGBA_DXT5_UNorm.

now

Debug.Log(texture.graphicsFormat)

prints "88" instead of R8G8B8A8_UNorm

 

Nope. The Texture2D has to be created as either linear or sRGB and cannot be changed after the fact.

CopyTexture()

is just one of the fastest way to copy data as it's just doing a copy of the entire data block (per mip level) at once. I think

Get

/

LoadRawTextureData()

might be even faster in some cases, though I've never benchmarked it since I've never actually needed to load textures at runtime like this.

Under the hood there's no reason for this that I'm aware off and most modern APIs you could flip this setting on the texture between draws if you wanted to, but I think there's some legacy stuff Unity is still "supporting" that can't do this.

I don't remember exactly what that "format 88" is. I see it pretty often when debugging stuff with c# managed textures. I think there are a few formats that show something useful when doing

Debug.Log(texture.format)

and show 88 when using

texture.graphicsFormat

.

 

I believe the normal map when used in a shader should NOT be
o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap).rgba);
and should be
o.Normal = tex2D(_BumpMap, IN.uv_BumpMap).rgba;

After using the above code to covert the normal.

Code (CSharp):

1. Texture2D loadedNormalMap = LoadTextureFromPath(f.FullName);
2. // note format is now copied from the loaded texture
3. Texture2D convertedNormalMap = new Texture2D(loadedNormalMap.width, loadedNormalMap.height, loadedNormalMap.format, true, true);
4. // the documentation on this function says it's GPU side only, but this is a lie
5. // this is roughly equivalent to:
6. // convertedNormalMap.SetPixels32(loadedNormalMap.GetPixels32(0), 0);
7. // if both textures are readable on the CPU, which they will be in this case
8. Graphics.CopyTexture(loadedNormalMap, convertedNormalMap);
9. convertedNormalMap.Apply();

Previously our normal code looked like this (before our upgrade to Unity 2021):

Code (CSharp):

1. fixed3 n = tex2D(_BumpMap, IN.uv_BumpMap).rgb;
2. fixed4 normalFixed = float4(n.g, n.g, n.g, n.r);
3. o.Normal = UnpackNormal(normalFixed);

I assume the convertedNormalMap is not stored as DXT5.

Visually it looks ok with the changes above. Though I'd drop a note here to help future people, and be corrected if I'm horribly wrong.