Question about normal map quality and channel packing

Hi all,

I keep hearing people say if I don't import normal map as normal map in Unity, I will see some quality lost to normal data.

As far as I know, this is caused by texture compression and each channel have different number of bits, so Unity shuffle channel on normal map import to get better precisions.

Question is: If I import a PNG file where I use AG channel to store normal data to begin with, don't tell Unity it's a normal map (and import it as linear texture), but use shader to reconstruct normal data at runtime. Does it still suffer from the same quality lost?

Another question: If I want to use the Alpha channel of my Albedo texture for something else (non-transparency), is it still safe to import texture as sRGB?

Thx in advance!

 

Yes, it’ll be exactly the same.

The thing I think you’re missing is the source texture’s format has nothing to do with the format used in the game. GPUs required special compression formats. On most platforms the default normal map format is DXT5. This is also the default format for any RGBA texture. So everything else you’re doing is exactly what Unity already does. More over you don’t actually need to move the red channel to the alpha anymore as of the later versions of 2017. Apart from using an uncompressed normal the next best quality is from keeping the texture a normal map, but overriding the per platform format to be BC5 which is two DXT5 alpha channels.

The alpha channel is always linear for all textures, regardless of if sRGB is set or not. That’s why Unity stores the smoothness map in the alpha channel of the metallic or specular texture.

 

Thx, I think I do understand the part about source texture format isn't the actual in-game texture format, in particular DXT5 and BC7/BC5.

But you brought up a few interesting points:

- So sRGB setting doesn't actually affect the alpha channel, that's very good to know.

- I realize Unity 2017.2 add RG channel support for normal map while working with Amplify Shader's unpack normal node. What I don't understand is: why? Is it added specifically to allow BC5? (I also thought the RG -> AG swizzle is specifically a DXT5 normal map compression step, so other modern format isn't affected, please correct me if I am wrong.)

- Lastly, the reason I am considering BC7 instead of BC5 is because I want to limit my use of texture map to 2:

1st: Albedo in RGB + Masks in A.
2nd: Normal in RG + Metallic in B + Roughness in A.

So I can't actually set the 2nd texture to normal, and it seems like BC7 is my best bet.

 

Now that I think of it, maybe Unreal's suggestion for mobile game texture is also worth considering.

1st: Albedo in RGB + Roughness in A;
2nd: Normal in RGB.

And they put Metallic as scalar.

 

Ah, yeah, I missed read your a original post slightly.

The issue is most people who try not setting normal maps as the normal type don't know about the AG swizzle even if they know to uncheck sRGB, so the texture gets compressed as the default RGB format, which is DXT1. The swizzled DXT5 produces significantly higher quality normals than DXT1, and Unity didn't support RGB normals on desktop, and suddenly their normals look way worse. Swizzled DXT5 with the R and B channels used for other content can still produce superior results to the straight DXT1 normals simply because that alpha channel is compressed separately, but you are doing so at the cost of the normal map quality. Several AAA games do pack data like this, as well as Unity assets like Alloy and MicroSplat/MegaSplat. Valve for example packs the roughness into the R and B channels as they use an anisotropic roughness.

Yes, it's specifically added for BC5 support.

The upcoming SRP formats default to using BC5, and I wish Unity's built in forward and deferred formats would default to that as well rather than the current setup of allowing both RG and AG normal map textures. My experience is BC5 produces normal maps that are far superior to BC7. I'd rather use half dimension uncompressed RGB normals than BC7. I believe I first remember seeing this technique described by Sony in talk on making PS3 games as it allowed users to choose between DXT1 and the 2x larger DXT5 depending on quality needs. I can't remember if it originated from Insomniac, or Sucker Punch, or an internal Sony studio. It was in mainly because the PS3 didn't support the 3Dc / DXN / BC5 format though, and RAM was still at a premium. I first saw BC5 (then called DXN) used on the Xbox 360, though desktop ATI gpus had support for a few years before that (called 3Dc).

Unity does that swizzle for uncompressed and now BC7 normals too, mainly because the DXT5 format did and they didn't support RG or RGB normals on desktop with the built in shaders. They also made the "high quality" compression option use BC7 when they added that format before changing the way they added RG support to the shaders.

BC7 is a weird format. Each block can choose what kind of compression to use, be it combined RGBA compressed, or split RGB / A compressed, or straight RGB, depending on the content and the choices the compressor makes. Also as I stated above I find the quality of BC7 for normals to be highly objectionable, be it AG or RG, even with out other data packed in the extra channels. It creates a significantly more blocky look to DXT5 AG normals. Not as bad as DXT1, but still bad. I prefer DXT5 AG with extra data packed in the unused channels if you're going that route ... though I actually prefer RG -> GA rather than RG -> AG because lighting is more often coming from above or below an object and GA gives the normal map's y the higher quality alpha channel.

Note if you do any custom packing you'll have to manually reconstruct Z as the built in function supports RG and GA by multiplying the R and A channels together.

Unless you're specifically targeting mobile, I'd suggest BC5 RG normals over DXT1 / BC7 RGB normals.

 

Also, to be more specific about your original question, manually swizzling the normal map to be AG (with the R and B channels white) with sRGB off will produce identical results to an un-swizzled normal map imported as a normal map.

 

That's a lot to unpack, thx for the detailed info @bgolus.

I will give this suggestion a go, I haven't compare DXT5 GA with BC7 GA side by side too seriously, it just felt slightly cleaner when I toggle high quality compression when using RG.

(top: BC7, bottom: DXT5; both using my previous "Normal in RG + Metallic in B + Roughness in A" scheme, in shader we swizzle and unpack it using the built-in function. I will need to check the GA approach.)

(UPDATE: see no difference when comparing DXT5 GA and BC7 GA, guess it just doesn't matter much given my texel density.)

 

Hi interesting thread. please explain AG normal maps vs RGB? So you get 2 spare channels to pack some other maps into, please explain the benefits of AG?

 

... See the post directly above the one you quoted which goes into a lot of that?

Normal maps are a representation of a normalized vector (it has a length of 1), and tangent space normal maps should always be pointing away from the surface along the surface normal, so you only need two components to reconstruct the a third component of the normal using Pythagorean Theorem (z = sqrt(1 - x*x + y*y)). This is useful because...

Due to the way desktop GPU compression formats work, you can get better looking normals by using the AG channels. Specifically the alpha of DXT5 is significantly higher quality than the RGB color (it uses the same amount of memory to store the single alpha channel as the RGB). Also the alpha is "separate" in that the contents of the alpha have no effect on the appearance of the RGB values. The RGB values of a DXT1 or DXT5 are all intertwined, as they are with most image compression formats; you can't put an image in one channel of the RGB color without impacting the other color channels.

The two channel BC5 format is two DXT5 alpha channels, and was explicitly created for tangent space normals.
It was a format introduced on the ATi Radeon X800 Pro called 3Dc, but didn't see common usage until the Xbox 360 where it was called DXN, before finally being added to DX10 as BC5, but they're all the same image format.

For mobile platforms, many didn't have a compression format with alpha, or using alpha reduced the quality of all channels, so there wasn't the same benefit of moving data into the alpha. Plus a square root can be a relatively expensive operation, so removing that can be a decent performance optimization for mobile where GPU performance is lower. Thus for platforms that don't support DXT5, Unity stores the normal as the original RGB values in the available format for that platform.

 

Thanks bgolus. Really helpful information and it clears up my understanding of the benefits of using an AG approach. I am in fact focusing on a mobile game so its not worth packing the channels with an AG approach it seems.

I recently come across 2 Texture arrays, would I be right in saying you could perform the reconstruction of normal maps before you add them to the array and then you bypass the overhead of square rooting on the GPU? I use ASTC compression on my textures so I'm still unclear if that has adverse effects with 2D Texture Arrays

My focus is to optmise the size of the build more so than get better quality Normal maps, so for this reason I am interested in using the spare channels in certain textures to reduce the amount of textures in the build. If I can get better normal details that is a bonus.

Thanks again, learnt something new that is helpful.

 

If you were to store your normals packed as two channels of an ASTC with other data in the “free” channels, I would go with reconstructing the normals in the shader like on PC (you would need to write custom shaders to handle this). Extracting the two channels and reconstructing the third component means you’ll be using an uncompressed texture at that point since there’s no way to “re-compress” a texture on a mobile device. You can’t add to or otherwise modify compressed textures*, only read from them.

* On hardware that supports it, you can use CopyTexture to copy blocks from one compressed texture into another of the same format. DXTC and ETC uses 4x4 blocks, ASTC has multiple block sizes depending on which one you choose. On desktop platforms there are options to compress to DXT1 or DXT5 in real time using a low quality compressor.

Texture Arrays are just a way of letting you pass more textures to a shader than would normally be allowed, or allow objects with different textures be batched by storing their array index as another vertex attribute. Actually sampling a texture from an array is no faster than any other texture using the same size and format, and may in some cases be slower. They’re an alternative to using an atlas that adds a little complexity for the benefit for not having to worry about mip map bleeding or texture wrapping, but otherwise have a lot of the same limitations, and adds some new ones (like all textures have to be the same dimensions).

If you’re worried about build size and less about texture memory, I would suggest keeping RGB normals and instead reducing / limiting the size of your normal maps. Dropping the max size of your texture from one power of 2 to the next reduces that texture’s size by 75%. After that if you still need to reduce the size more, try channel packing and reconstructing the normal in the shader. Platforms with ASTC support likely have the performance to handle the normal reconstruction.

 

@bgolus For adreno GPUs (Oculus Quest) I currently just leave normals as-is because I'm never quite sure what Unity's actually doing for a given platform. Would you recommend importing a 4 channel texture and unpacking normals in this case? I felt it would probably be slower and consume more bandwidth than just going with 3 textures: albedo rgb+a, normal and a mask map.

I appreciate the in-depth breakdown you did above but you only lightly touched on mobile, so I thought I'd ask.

 

Exactly. I'd like to know that too. ASTC is flexible enough that it could be doing a lot of different things, but I haven't been able to find any details on exactly what. I assume it is doing RGB correlated (DXT1 analog), and RGBA with uncorrelated alpha (DXT5 analog) when an alpha channel is used. I have been using radiosity normal maps so I would have to experiment a bit to see if it does anything differently with an actual normal map texture.

ASTC does have modes for two uncorrelated channels as well as two correlated + one uncorrelated, but I don't know if there is a way to actually use them in Unity. Would be nice to have the option; especially on standalone VR where you sometimes want to pack things together in slightly unconventional ways to make the most of your limited texture bandwidth.

 

Depends on the device. Older GLES 2.0, or very early GLES 3.0 devices, it makes sense to skip the additional shader work of reconstructing the tangent space normal map's z. This is another reason why Unity's default is to not do any special packing of normal maps and just encode them as-is into ETC, or ASTC.

However something like the Quest has more than enough power to reconstruct the z from two components. And it shouldn't consume more bandwidth. Using one texture vs two, the one texture should hopefully be less bandwidth assuming the resolutions are the same between the two options. Also if you only need normals and a single mask, you can do that with an RGB texture. Though an RGB and RGBA ASTC is the same size, and thus same memory bandwidth usage, and RGBA ASTC will have more obvious compression artifacts than RGB. They both decompress into float4 values regardless too. But ASTC and mobile hardware is weird so who knows.

The ASTC format has a ton of options. For the most part it's weird to me that they expose separate RGB and RGBA options as ASTC can choose which format it uses for every individual block of the image. So one block could be RGB, one could be RGBA, one could be a single or two channel block, all in the same image. My understanding was that the ASTC texture compressor is supposed to make these decisions on the fly, but I don't know how true that is, or if Unity does / can override it.

 

Thanks! That is interesting... What would be the best packing roughly for a normal map? I guess I could have RG as the normal, and B for metal and A for something else, or do you recommend RGB?

 

RGB will give you better quality than RGBA because it is compressing less information into the same block size.

In theory, you can have one non-correlated channel in any channel and the compressor will select the best channel per-block to be non-correlated. So you can put whatever in any channel and the compressor should deal with it. Alpha is a bit special because it can be linear when the texture is sRGB, and can be LDR when the texture is HDR.

The compressor is a bit of a black box so I still would like to have more explicit control over which options is will/can/can't use, but it's possible that I am simply being irrational and the machine will always be better than I am anyway.

 

Yeah, that. If you can get away with only packing 3 channels per texture you'll get guaranteed better quality.

I mean ideally for normals you want the x and y to be uncorrelated too, so the "best" option would be if you could have all channels be uncorrelated, but that's not really an option (besides much of the compression comes from being able to treat them as correlated). When you get into the realm of channel packing data like this you just kind of have to live with some amount of compression artifacts.

 

I was wondering if anyone saw a benefit in using one of the Oct encoding schemes i.e. http://jcgt.org/published/0003/02/01/paper.pdf.
I notice that Unity (at least the SRP) has code for decoding and encoding them.
On most platforms, the cost difference should be negligible between formats, assuming that you want some sort of renormalization step (which is essential if you are using channels with as little as 5 or 6 bits precision)

 

This is incorrect, at least for DXT and ETC.

RGBA compression formats use 8 bytes per block, whereas RGB formats will use 4.

The RGB and Alpha channels have separate lookup tables. This is why DXT5 normal maps move a component into the alpha channel for higher quality. (Mobile maps however do not use an alpha-channel format for normal maps, to avoid the cost of reconstructing the Z component, as described above.)

 

I was talking specifically about ASTC for mobile which always uses 16-bytes per block regardless of how many pixels or channels you are packing together.

The important thing I had forgotten about was the error metric. The ASTC compressor has a mode for normals to minimize angle error as opposed to the difference in absolute error, and hopefully Unity is using that option when you set a texture type to "normal map" but it would be nice to use that when you are packing channels manually.

And having thought about it more, that error metric is what I actually want to have more control over. Giving different weights to different types of error (i.e. discontinuities between blocks, absolute error, angle error) and also weighting the error for some channels more than others.

 

Say Unity opened up the options for this, what kind of quality difference would we be looking at? If it's so minor that's basically min/maxing then I probably won't worry much. Still odd Unity would choose to keep these settings completely black boxed.

 

ASTC normal maps are certainly way, way better looking than the other mobile options, but that's probably more due to ASTC's overall quality being much higher than other formats. I haven't tried comparing a ASTC "normal map" vs ASTC "regular texture set to linear" to see if they compress differently at all. Wouldn't be hard to test. Part of me doubts there's a difference.

*does a quick test

Nope, no difference between a "Normal map" and a "Default" set to linear. Also no difference between using an RGB and RGBA ASTC compression format, which is good, as it means the compressor is correctly detecting the source has no alpha and compressing those blocks exactly the same. Also means Unity is probably just passing in the textures to the compressor with default settings, apart from explicitly telling it to ignore the alpha or not.

 

I am more thinking of the case where one could attach a heightmap to a normal map without messing the normal map up too much. Because errors can be pretty apparent in reflections with normal maps, but a heightmap being used for a little bit of surface parallax can be compressed quite a lot and look pretty much the same. So it might be nice to make it really prioritize the normal map.

Save thing for adding metalness to a roughness texture. The details in the roughness are what matter where there isn't usually anything interesting or important happening in the metalness channel.

Maybe mostly only relevant on really constrained platforms like mobile VR where an extra texture fetch can be particularly expensive so I would be easily convinced that it's a better and more efficient workflow to simply split up your textures if they don't pack and compress cleanly. But I am also just biased towards having more levers to experiment with.

I would be interested to see minimize for discontinuities between blocks over the absolute error because for some types of textures it might be better to look less wrong even if it actually is more wrong. It's not too surprising that minimizing for angle error versus absolute wouldn't change much because those would be highly correlated to begin with.

 

I combine textures when I can, but it takes some experimentation to see what works best.
Before you consider where to put your channels, however, I think that you should consider the resolution that you need to store the channels at.
A lot of secondary textures can be lower res without problems. Normal maps usually require the most detail.