Question Mesh/Geometry/BlendShape/Animation data improvements

Bastien in the other thread suggested to just post it, so here goes!

This is partially "asset import", partially "data build", partially "graphics" area. It's about data formats for Meshes and their friends.

Today all the "geometry data" in Unity is far from using optimal data formats. Specifically, out of the top of my head:

1. Mesh data can use either "mesh compression" import setting (quantize mesh data to lower amount of bits, uniformly over the input data range; at load time unpack into uncompressed format), or have meshes be influenced by "vertex compression" project setting (which tries to convert some vertex data components to e.g. half-precision floating point formats, but there's a lot of conditions where that may or might not work).
2. Related, if a mesh has blend shapes, I'm not sure if they are compressed at all in any way.
3. Related, any and all animation data -- Unity can do keyframe reduction at import time, but beyond that I'm not sure if anything is "compressed" at all (if it is, great!).

On one project I'm working on, all the above has led the team to basically "bypass" Unity's import pipeline for meshes and animations, and develop "custom" importers that encode and compress the data. This has led to runtime memory savings of 2x-3x compared to what Unity normally does, and on-disk savings of about 5x compared to built-in Unity functionality. If each and every "large enough" production ends up doing similar things, that sounds like a lot of wasted effort, i.e. Unity could provide that built-in!

Mesh Data, in memory

I'd suggest removing the "mesh compression" import setting and "vertex attribute compression" project setting (yeah, a breaking change), and instead rethink how mesh data should be stored at runtime. E.g. something like this could be a sane default:

• Vertex positions: float3 (12 bytes). Harder, but better: ushort3 (6 bytes), uniformly quantized over the mesh bounding box, but this one needs to pass bounding box into the shaders so it's harder to do.
• Normals and tangents: either octahedral encoding for each (4 bytes), or perhaps store the whole tangent frame quantized into 4 bytes (so you'd no longer have "normal" and "tangent" as separate attributes).
• UVs: half precision floats (4 bytes per usual 2D UV), or uniformly quantize over some "sane" range (e.g. -8..+8 for UVs).
• Skinning information: something simple could be 1-byte per weight (unorm format), and 1-byte per bone index. If input has more than 255 bones, then switch to 2-byte per bone index. Possibly better alternative: "Permutation Coding for Vertex-Blend Attribute Compression" (Peters et al 2022) that cleverly encodes things like 13 bones per vertex into just 8 bytes.

Unity almost allows doing the above already, with ability to provide your own mesh vertex attribute layouts. But nothing in the built-in importing and data processing pipeline uses that.

There could be per-asset (or per-preset, per-folder, per-project etc.) settings that allow customizing the vertex layout. But the defaults IMHO could be much better optimized for "small memory usage, typical use case" than the "everything is completely uncompressed" defaults of today.

While at it, also split the runtime mesh data memory layout so that position attribute is in a separate stream from "everything else". This primarily benefits GPU performance, especially on mobile.

Upsides: smaller runtime memory usage, less bandwidth usage for GPU, smaller initial data for data files.
Downsides: any quantizations or limitations are, well, limited compared to using just full floats for everything. Also some encodings would require interaction with shaders (a vertex shader might need to do octahedral decoding, etc.).

Mesh data, on-disk

Data quantization / encoding above could or would produce about 2x-3x smaller mesh geometry data compared to what is default now already. For data files, additional things could be done. I've had good success using meshoptimizer vertex and index compression -- these compress well and are very fast.

Blend shape data

I don't remember what is the runtime and disk data format for mesh blend shape data right now -- but the scripting API exposes all blend shapes as full 32-bit floats. Which again sounds a bit overkill for typical use case. For example, if a blend shape only contains positions and normals, then I dunno, something like "12 bit per position component, 9 bit per normal component" uniformly quantized over the range of values in the blend shape would put per-vertex blend shape data item at 8 bytes, instead of 24 bytes.

For on-disk storage, again there's a lot of data correlation between blend shape values so using something like meshoptimizer (mentioned above) to compress them might be useful.

Animation data

Again similar story -- are animation keyframes for e.g. position and rotation stored as full 32-bit floats everywhere? That also might be overkill for on-disk data at least. Animation data very likely often contains most of data for positions and rotations (it can animate anything else too, but these ones are harder to know requirements for). Having specialized data formats for positions & rotations would make a lot of sense though.

 

Was excited about this as we do our own mesh combining and already use custom mesh data layout - so an easy performance win if all I have to do is move positions to their own stream! But I found this which suggests there are no real gains to be had here?

https://forum.unity.com/threads/doe...e-streams-matter-for-gpu-performance.1335644/

+1 to all of your post though. We've found custom mesh data layouts to be a very useful to save us memory and disk space!