Search Unity

LZ4 compression chunks and memory usage

Discussion in 'Asset Bundles' started by Cramonky, Feb 8, 2020.

  1. Cramonky

    Cramonky

    Joined:
    Apr 1, 2013
    Posts:
    186
    Hello all,

    Unity's documentation about the LZ4 compression method describes it as "chunk-based compression", where only the needed "chunks" of an asset bundle are decompressed when an asset is loaded.

    I am curious about how small these chunks are. For any arbitrary object, is the chunk only made up of the assets that the object uses (meshes, materials, textures, etc)? Or is it possible that a chunk may contain assets for multiple objects that are not related to each other? I did some quick testing with the profiler and it looks like the first option is the case, at least for the objects I tested with.

    The main reason I am curious is because I am wondering if there are any benefits to breaking up assets into smaller bundles versus putting large amounts of assets into a single bundle when LZ4 is used. In my case I have built a level editor where there are a large number of assets that can be chosen basically at random.

    Thanks!
     
    Last edited: Feb 8, 2020
    Cramonky, Feb 8, 2020
    #1
    mowax74 and Upian like this.

  2. NameString

    NameString

    Joined:
    Mar 23, 2023
    Posts:
    2
    I don't know why Unity officially describes lz4 as "chunk-based compression". Because lz4 only has the concept of frame and block, but not the concept of chunk.
    When using lz4 compression, the process of Unity is like this
    1. Multiple assets (textures, meshes) will be merged into one file
    2. This file will be cut into multiple chunks, and these chunks are equally divided
    3. Compress these chunks independently with lz4.
    The above is inferred based on the 5.4 manual.In the manual there is a sentence like this:
    Chunk-based compression (LZ4) means that the original data is split to chunks (subblocks) of equal size and that chunks are compressed independently
     
    NameString, Apr 3, 2023
    #2

  3. AndrewSkow

    AndrewSkow

    Unity Technologies

    Joined:
    Nov 17, 2020
    Posts:
    91
    FYI we have revamped the documentation about AssetBundles for the 2023.1 version of the manual and it has a bit more information about compression. The same information also applies to older versions of Unity (and over time we may backport the documentation to older versions).

    https://docs.unity3d.com/2023.1/Documentation/Manual/AssetBundles-Cache.html

    The terminology of chunk is referring to how the archive content is broken into 128KB chunks, and those are compressed individually, which means we can access data without needing to fully decompress the full AssetBundle content into a memory buffer, which is the behavior with LZMA compression. For any particular chunk we have to fully decompress the entire 128KB section.

    The division of assets between bundles is definitely an exercise in trade offs, and i don't think there is a single heuristic that will work for all cases. Although LZ4 helps reduce the overhead when only parts of an AssetBundle are loaded, there are still some implications for updates/downloads if bundles are very large and only a few assets are changing between builds. And when an AssetBundle is loaded there are some datastructures that are loaded that are proportional in size to the total number of objects. Having many small assetbundles would also introduce some negatives, e.g. with some overhead when loading many files. Some people have even hit limit of file descriptors on linux or other resource limitations if the number of assetbundles being loaded is very large.
     
    AndrewSkow, Apr 4, 2023
    #3
    Ryiah likes this.