Load and apply normal maps at runtime

Hello everyone,
My game downloads textures (albedo, metallic and normal maps) from internet at runtime and applies them to my model through a dedicated material.

The problem is that my normal map looks wrong when loaded on runtime. The normal map is of type OpenGL.

When you apply normal maps from the Editor, Unity asks for a fix in order to change the texture type from "default" to "normal". When this fix is used, my normal map looks correct. The problem is that at runtime I don't see any option to apply this fix via code.

This is how it looks at runtime (which is WRONG):


This is how it looks from Editor when the fix is applied (which is CORRECT):


I've spent 2 weeks on this with no luck, from my research Unity uses a normal map compression called DXTnm but I don't know how to apply it at runtime via code.

I'm using the universal rendering pipeline with Unity 2020.3.7f1.

I'm going crazy, how can I fix this?
Thanks!

 

Do you control all parts of this or are you expecting your users to be able to add any ad-hoc textures?

I ask because if you control all parts of this process, just use the Unity addressables system to deliver content that is pre-made and properly set-up and fully testable in editor.

Otherwise, I'm not sure how OpenGL normal maps correlate to what Unity is expecting, but I guess at the end of the day it is just data, and some type of transformation must be done. I just don't know where the endpoints for such a thing would be exposed, except at the Texture2D class.

 

Hi Kurt, thanks for the answer.

I cannot use addressables as the content must be downloadable cross-platform.

I can indeed apply a transformation of the texture before applying it to the material, but I need to know which transformation must be done.

I did try to apply the transformation of RGBA suggested by CWolf here ( https://forum.unity.com/threads/runtime-normal-map-created.296795/ ) but it didn't work as the result is still wrong (and Unity Editor still asks for the "fix" when I load it manually).

I really don't know what conversion is needed but it's getting ridicolous!

Any help is very welcome,
thanks!

 

Still no luck, there's not a single resource for the conversion that Unity does under the hood!
@bgolus do you have any suggestion?

Thank you!

 

this doesn't work?

https://docs.unity3d.com/ScriptReference/TextureImporterSettings.html

it has convertToNormalMap or textureType which let you set the texture as normal map

https://docs.unity3d.com/ScriptReference/TextureImporterType.html

 

No, it doesn't work because this is scripting for the Unity Editor, while I'm loading textures at runtime, so I don't have the Editor at my disposal but only the UnityEngine.


As part of my experiments, I noticed that I get the expected result if:
- I convert the RGBA of my texture with this conversion: (255, texture.g, 255, texture.r)
- I remove the checkbox on the sRGB (Color Texture) option in the texture properties from the Editor:


So the question is: how can I deselect "sRGB (Color Texture)" via scripting at runtime?
Maybe @bgolus knows more?

Thank you everyone in advance!

 

ah yes, I see now, sorry for the confusion,

well what about creating a custom shader for your material that gets as input a normal image then converts it to normal map?

can you create a new Texture object that is of type normal map? then copy the pixel from your loaded image in this new texture?

 

I would prefer to avoid touching the shader, I'm currently using the Universal Rendering Pipeline Lit shader. I'm not very experienced with editing shaders, do you have the source of this shader so I can try edit it?

This is what I'm currently trying to accomplish. As Unity reads the original texture as sRGB, I need to copy the pixels of this texture and paste them in a new Texture2D which is set as "linear". The problem is that I cannot use the function Texture2D.GetPixels() because Unity says "the texture cannot be read (it is not readable)". That's probably because of another flag that doesn't allow the Texture to be read by scripts.

Is there a workaround for this?

It's incredible that such a simple thing is getting so complicated...

Thank you!

 

One thing you've not said is how you're creating the textures at runtime. Presumably you're using

UnityWebRequestTexture.GetTexture()

? That function has this bit of info in it's documentation:

That

TextureImporter.sRGBTexture

mentioned there is that sRGB check box in the import settings. When you uncheck it the texture is set to be a linear data texture. This does not change the data in the texture, it only changes if the GPU applies a color conversion to the values when it is sampled. However it's also a value that has to be set when the texture is created, which annoyingly the

UnityWebRequestTexture.GetTexture()

doesn't let you do! It seems like it would be a trivial thing for them to have exposed, but alas, they did not.

There are a few solutions to this. The easiest one is to use the suggested

ImageConversion.LoadImage()

, which means using

UnityWebRequest.Get()

, creating a

Texture2D

set to be linear and use

tex.LoadImage()

with the

webRequest.downloadedBytes

. Very bad pseudo code below (no guarantee I have any of the web related stuff right):
https://docs.unity3d.com/ScriptReference/Networking.UnityWebRequest.Get.html
https://docs.unity3d.com/ScriptReference/Texture2D-ctor.html
https://docs.unity3d.com/ScriptReference/ImageConversion.LoadImage.html

Code (csharp):

1. using (UnityWebRequest webRequest = UnityWebRequest.Get("myTextureURL.png"))
2. {
3.   // the rest of the code for the we request stuff up to getting a UnityWebRequest.Result.Success
4.  
5.   // resolution does not matter, it will be overridden, but the other values do!
6.   Texture2D newNormalMap = new Texture2D(1, 1, TextureFormat.DXT1, true, true);
7.  
8.   // will load a JPG or PNG and compress it to the DXT1 format
9.   newNormalMap.LoadImage(webRequest.downloadedBytes);
10.   return newNormalMap;
11. }

And yes, that'd save it as a DXT1. I'll get to that.


Now onto some other things brought up:

That "check" in the editor cannot be passed for any runtime created texture. AFAIK the check is literally "does this texture have a texture importer with

convertToNormalMap

enabled?". Runtime created textures do not have a texture importer attached to them ... and really no runtime texture does since the texture import settings aren't included with builds and can be safely ignored. The asset class for textures do not have any setting to flag them as being a normal map or not!

This is a lie. Not calling you a liar mind you, I'm saying Unity is lying when it says "DXTnm". That's not a format ... or more specifically that's not the format the texture actually is. It's a DXT5 texture. The only "special" thing is Unity moved some of the color channels around before it compressed it. The original red channel stored in the alpha, the original green channel stored in the green and blue, and the red channel set to 1.0. The reason for this is because the alpha and to a lesser extent the green channel are higher quality than the red and blue channels. Green only very minorly so, but the alpha channel uses as much data as the RGB does by itself. The memory size difference between a DXT1 texture, which only holds the RGB colors, and the DXT5 which holds RGBA, is the later is twice the size just from adding the alpha channel. But the increase in quality can be significant.

If wanted, you could use

TextureFormat.RGBA32

instead of

TextureFormat.DXT1

in the above setup, an then use

tex.GetPixels32()

on the texture to modify the color data in the way described. Then call

tex.Compress()

to convert it to a "DXTnm" DXT5 image. Or you could just make sure the PNG images you're loading from the web are already setup like that and set the texture format to DXT5 before

tex.LoadImage()

and it'll "just work".

But since Unity 5.6 or so you don't need to rearrange the channels anymore. The red channel is "white" in the DXTnm to allow that this to work. In Unity's shaders, it multiples the red and alpha channel together, and since by default any texture without an alpha channel (like DXT1) will return 1.0 for the alpha, you can store a normal map exactly as it was in the original texture, or with stuff moved around like in the DXTnm, or as the two channel BC5 or EAC formats that you can choose when importing textures. Any will work. Also, the blue / "z" is always reconstructed, so what's in the blue channel is totally ignored. It's just set to a copy of the green channel in DXTnm to slightly improve the compression quality.


I also skipped some possible steps in that I don't know if mip maps are automatically handled by

tex.LoadImage()

or not. I would hope it does, otherwise that adds a bit more pain to all this.

 

No, I don't have it, I've had the same issue with HDRP Lit shader and recreate it to have same inputs as the editor and then add my custom stuff upon was complicated.

but it depends of how many of those inputs/slots from the default lit shader you actually use. You don't need to recreate all of them.

Reading into the documentation the Textures appears to be non readable by default. I don't know how to make loaded textures to be readable

 

Hello @bgolus , thanks for your detailed reply, I basically have read all your posts regarding textures!

I honestly don't know how I load the textures as I'm using the Unity Plugin called "Trilib 2", which allows me to download models and textures at runtime.

What I can say is that the normal map is made with Substance and stores the values on RGB.
When Unity loads this texture, it says the GraphicFormat is R8G8B8A8_SRGB.

Applied as it is, it's not working at all (the model is totally wrong) so I guess I need to convert this normal map in something that Unity likes.

What I tried is:
1) copy the texture in a new one that can be read using GetRawTextureData() and LoadRawTextureData()
2) read all the pixels, apply a conversion on them (moving the red channel into alpha and setting the red and blue channels as white)
3) copy all these edited pixels in a new linear Texture2D
4) compress this new Texture2D with compress()
5) Apply it as new normal map to the existing material.

This process doesn't work as I don't get the expected result (which I can obtain "fixing" the texture manually from the editor). What am I missing?

So my conversion is wrong? I'm moving red into alpha and setting red and blue as white (1.0). You are instead saying that I should put red into alpha, green in green and blue and red set to 1.0. Is it correct?

So if I start with my RGBA32 normal map should it work just compressing it without any modification?

Thank you, you're explanations are incredibly useful!

 

So, the TLDR version is: the normal map needs to be set to be a linear (aka not sRGB) texture. Everything else after that is extra fluff. If you can get the

Texture2D

to be a linear texture, everything will work fine. Until then nothing you do really matters.

I would contact the author of that asset for support and ask how to properly import normal maps with the tool. They should hopefully already have a way that works. And if not that's a huge failing if an asset that's supposed to make importing asset at runtime easier.