Alternative normal packing

I'm new to texture packing and compression so I'm hoping someone can sanity check my thinking. I haven't found anyone talking about packing maps this way but I mustn't be the first to think of it.

Currently we use three maps for our objects:
Albedo (RGB) DXT1
Packed Map (R-AO, G-Spec, B-Gloss) DXT1
Normal which will be compressed with DXT5nm to (1, G, 1, A)

I've read that you want to avoid packing extra info into the R and B of the normal because it will create compression artifacts but I wonder, what if we split the X and Y of the normal map into the alpha of the Albedo and Packed maps creating:
Albedo (RGB, A-X) DXT5
Packed Map (R-AO, G-Spec, B-Gloss, A-Y) DXT5

As I understand it, this would result in higher quality normal data and the same memory profile since one DXT5 is equal to two DXT1. This also shouldn't muck up the other data in the maps any worse because in DXT5 the color is compressed the same way as DXT1 with the alpha being compressed separately.

Does this make sense?

 

Yes, that would absolutely produce much higher quality normals. The existing BC5 format is what many AAA games use because it's two DXT5 alpha channels, this is effectively produce the same results.

It should be noted that several AAA games do pack data in the R and B channels of the normals. Recent Valve titles for example as they support anisotropic specular on all surfaces, and store the xy specular smoothness in the r and b channels, using the G and A for the xy normals. Their experience was that smoothness often correlated well with the normals, so the additional artifacts weren't very obvious. However they do have a fallback to split it into two textures when the artifacts are a problem, possibly using BC5 for the normals in that case.

 

Short answer, yes, that would be better and is indeed like having a BC5 format texture.

The underlying reason is that while DXT interpolates everything to 8 bit values, the stored values are not all 8 bit. The separate alpha channels are stored as 8 bit value, but for the color part it's 5:6:5. (Which you could easily combine into one 7 bit value*, but you'd still miss half the values compared to 8 bits.)

Base system for DXT color:
- Store two 16 bit color values in 5:6:5* format
- Store sixteen 2 bit interpolation values. (Use value1, value2 or two blends of the two.)
In total 2x16 + 16x2 = 64 bits for a 16 pixel block.

Base system for DXT alpha:
- Store two 8 bit alpha values
- Store sixteen 3 bit interpolation values. (Use value1, value2 or six blends of the two.)
In total 2x8 + 16x3 = 64 bits for a 16 pixel block.

*: You can combine two n bit values into one n+1 bit value. So 5+5+6 = 6+6 = 7 here.
Encoding:

Code (csharp):

1.  
2. float single; // Our input value (between 0 and 1)
3. float3 rgb565 = floor(float3(31.75, 127.0, 63.5) * single);
4. rgb565.gb -= rgb565.r;
5. rgb565.g -= rgb565.b;
6. rgb565 /= float3(32.0, 64.0, 32.0);
7.  

Decoding:

Code (csharp):

1.  
2. float rgb565; // Our input value as 5:6:5 rgb
3. float single = dot(rgb565, float3(0.25, 0.5, 0.25));
4.  

P.S.: I know that this is not the whole story for DXT, but it's a quick breakdown.