Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug Addressables - Extremely slow load time

Discussion in 'Addressables' started by Jarko89, Feb 5, 2021.

  1. Jarko89

    Jarko89

    Joined:
    Mar 8, 2017
    Posts:
    15
    Hi,

    I'm facing serious load speed issues with Addressables that are basically making them completely unusable for a project. Talking about / The project in question has about 5gb of assets (in the built player), so it's not particularly large. It's for PC (Windows).

    When I started using Addressables, I moved part of the prefabs used by the game to Addressables and kept using Resources for the rest, ending up with 10 different bundles of size ranging from 20mb or so to 500+mb. Up to that point it was all good, loading speed was very nice.

    After that, I moved everything I could to Addressables, ending up with 3.6gb of asset bundles (the largest bundle weights 900mb). All bundles contain prefabs.

    Right after loading the app, I load a few of those prefabs (located in different bundles). The issues is, the first one I load (let's call it prefab X), one of no particular weight, takes AGES to load.
    • On an SSD drive, loading prefab X takes 7-8 seconds. That's already a huge time to load such a small prefab. All the following prefabs I load, theoretically much heavier, take 10-100ms each. Those prefabs are part of different bundles.
    • On an HDD drive where load operations (e.g. scene loads) take a bit longer on average (just a bit), loading that first prefab takes... sometimes more than 100 seconds. All the following prefabs load extremely quickly, pretty much as fast as on the SSD.
    During most of the loading time of prefab X, the game is doing nothing: CPU usage is almost 0%, disk reading is around 15 MB/s, RAM usage goes up by a few MB per second. Once it's over, disk reading becomes high (as it should be), RAM usage grows quickly, and in a 3-4 seconds it has loaded the remaining 2gb of data.

    The strangest thing of all is this only happens the very first time the same game build is launched. If you close the game and reopen it, the loading time of prefab X drops to a couple seconds (acceptable), as if something got cached after the first run. Creating a new build or restarting Windows makes it happen again.

    This issue been tested and is 100% reproducible on multiple machines and multiple disks within the same machine. It affects addressables versions from 1.8.5 to 1.17.4-preview equally.

    A few things I've tried:
    • Calling Addressables.InitializeAsync() beforehand. It's not the problem, that always resolves in a few ms.
    • Switching between Uncompressed and LZ4 compression. Does nothing.
    • Setting all asset groups to "cannot change after release", disabling "send profiler events"... also does nothing.
    Thank you all for reading and for your help. I like Addressables, it's be really sad to have to throw it all away :(

    Cheers!
     
    Last edited: Feb 5, 2021
  2. Favo-Yang

    Favo-Yang

    Joined:
    Apr 4, 2011
    Posts:
    464
    Probably not going to help too much...

    LZ4 - the chunk based compression is what you need. To avoid loading the whole assets bundles into the memory https://docs.unity3d.com/550/Documentation/Manual/AssetBundleCompression.html. Though it doesn't seem to related to the issue. But you should stay with that.

    Based on the math, it seems the first load force to load the entire assets (or a large part of it) into the memory.
    • The average SSD seq read is 300~500MB/s. 7s means up to 3.5G.
    • The average HDD seq read is 25~100MB/s. It can take 100s+ to load 3.5G data.
    Then the OS cached your data until you push them out of the cache. Load another memory consumption app or a fresh restart. That's why it won't be a trouble on the second load.

    So you may want to test the hypothesis. What about reduce 1/2 part unrelated assets to your first prefab from the game, then watch if the load time reduces to 50%. If it happens then you confirmed the hypothesis.

    Then you need to figure out why it happens...
    Try to load a simple image, a single prefab see anything different?
    Anything special with your prefab?
    Anything special before loading the prefab that may touch all assets?
    Use https://docs.unity3d.com/Manual/AssetBundles-Browser.html to verify the compression format of the generate asset bundles?
    More random guess...
     
    KwahuNashoba likes this.
  3. Jarko89

    Jarko89

    Joined:
    Mar 8, 2017
    Posts:
    15
    Thank you for your reply! So it loads faster if you kill the app and reopen it thanks to the Windows memory cache? I didn't know it worked so well :)

    As for compression, yes, I used to use Uncompressed, now I'm experimenting with LZ4. My problem with LZ4 is that it might mess up the diffing by our binary patcher. We'll see. It seems that LZ4 makes that first load faster than Uncompressed (halves it), which is a bit counter-intuitive - shouldn't it be the other way around since it has the added cost of having to decompress the assets? I guess the gain from the smaller HDD read offsets completely the added CPU time from decompression, and then some...

    Anyway, it's still very slow. I can assure you that loading all the game assets (5gb) into the RAM at startup (they used to be all referenced in various lists located in Resources folders) was much faster than loading 3.5gb using Addressables, which clearly points to a big weakness of Addressables.

    I'm going to run more tests, but that prefab shouldn't be referencing much and the problem is the game seems almost idle during all of that long Addressables.LoadAssetAsync call - ram usage doesn't go up, and disk read speed sits around 15 MB/s, when it could easily go above 100 MB/s (as it does when it loads the other assets)... It seems I'm not the only one talking about this kind of behavior here on the forums, some Unity people commented on it but no update was every given...
     
    KyleOlsen and ImpossibleRobert like this.
  4. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,816
    Thank you for flagging. Could you file a bug report for us?
     
  5. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    151
    Has this been reported through the tracker yet?
    I'm facing the same issue as described in the OP, albeit on a lower scale since my project is smaller.

    I am fairly certain that this isn't related to any compression scheme, because it happens in the editor too where I don't have any asset bundles actually compiled. My project has about 1,200 addressable assets and it takes over a second to initialize at startup before loading the very first asset.

    According to the profiler, the first ever invocation of AddressableAssetSettingsLocater.Locate() is what's taking the time, with over 75% of it spent in numerous instances of ContentBuilderInterface.GetPlayerIdentifiersInAsset().
     
  6. KarimTA

    KarimTA

    Joined:
    Jan 8, 2019
    Posts:
    27
    We have an almost identical performance issue as what @Jarko89 described.
    First loading on HDD taking up to several minutes.
    Using LZ4 improving the load times significantly.

    Deep profiling indicated that all Background Jobs were reading bundles for the whole time.

    It appears that loading our first scene pulls a lot of our bundles (especially with the changes in dependency behavior in 1.1.13). We spent a lot of time trying to analyze and simplify our dependencies, but it only delayed the issue. (as a side note, a Dependency Graph visualizer would be greatly appreciated).

    As a workaround for now, we had to disable CRC checks, as the overhead seems incredibly expensive on HDD.
    This brought our Addressable Scene loading time from ~170s to ~25s.

    We are not really comfortable removing the CRC checks as they caught some Steam download failures, but that unfortunately beats the alternative.

    As for my other posts, we unfortunately are not able to submit a bug report of our project.
     
  7. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    151
    I got news for people pained by Addressables performance in the editor. I made use of Addressables.LoadResourceLocationsAsync to get resource locations for assets with a certain label and load them in a second step, because I want to store the addresses alongside the loaded assets. By default, the type parameter is set to null. Don't use this! If the type is null, the AddressableAssetSettingsLocator (which is used only in the editor) will actually open each asset's file and read its header to determine the asset type. Just by explicitly stating the type of the assets to locate, I could reduce my startup time in the editor by an order of magnitude (and that's using an SSD, the speedup should be much greater even on a HDD).

    To the Addressables team: please check if this default behaviour is really necessary. You are displaying the type of every addressable asset in the Addressables Group window, so you certainly have this information cached somewhere.
     
    Last edited: Mar 18, 2021
    Miscellaneous, apkdev and gsylvain like this.
  8. gsylvain

    gsylvain

    Joined:
    Aug 6, 2014
    Posts:
    100
    I had loading taking more than 3 minutes on console. I tried disabling CRC checks and it went to 2m30s. After messing around a bit, I got the tip to go in unity menu Windows -> Asset Management -> Addressables -> Settings and enable "Contiguous Bundles" (addressables 1.12.1 and up). The loading time went down to 37s!!!
     
  9. Arthur-LVGameDev

    Arthur-LVGameDev

    Joined:
    Mar 14, 2016
    Posts:
    228
    37 seconds, down from 2.5 minutes -- that's pretty massive.

    We also experience relatively slow-loading, definitely slower than the comparable approach was with Resources dir, though difficult to compare them because we're Async-loading much of our stuff now vs previously most everything was synchronous. That said, previously we'd see something like ~12-second loads with "blocking" via Resources dir loads; now we have 'smooth' loading/framerate during load (mostly), but it takes probably >60-seconds before all assets have been loaded.

    Going to try fiddling with these settings -- even a 25% savings would be great, but if we can shave 75% off asset loading time via turning a few knobs then that'd be pretty spectacular, hah. (And it'd probably mean the current functionality, or at least the defaults, are what I would call 'bugged' / non-sane / wrong).
     
  10. VoodooDetective

    VoodooDetective

    Joined:
    Oct 11, 2019
    Posts:
    238
    I'm having the exact same problem. I'd originally thought our problem was because we had only one asset bundle (group) containing everything.

    I tried setting the one group to "pack separately" thinking that would fix the issue, but it made no difference whatsoever. I think this might be a bug or a design flaw. Either way, it's a huge problem for us. We're scrambling to figure out our options...

    EDIT:
    We're using Addressables to load animations dynamically at runtime. We don't need our "shower" animation if we're not in the "bathroom" scene.

    Did some experimenting between addressables and loading from resources. Addressables are ~89% slower than loading from resources. So I don't know if there's been a regression in the latest version or if this is to be expected, but it is appallingly ponderous and basically unusable for our purposes.
     
    Last edited: Dec 17, 2021
    Miscellaneous, Eclectus and pdinklag like this.
  11. Arthur-LVGameDev

    Arthur-LVGameDev

    Joined:
    Mar 14, 2016
    Posts:
    228
    @VoodooDetective -- that approximately matches our experience / benchmarks from ~9mo ago as well, FWIW. =|
     
    VoodooDetective likes this.
  12. VoodooDetective

    VoodooDetective

    Joined:
    Oct 11, 2019
    Posts:
    238
    OK so I did a lot of digging late into the night yesterday. I've found a few things out that answers why I was having issues. The root cause is poor documentation.

    • With one large asset bundle, disabling CRC dramatically speeds up addressables "warm up" time
    • Addressables wasn't really meant to be used if you're not also loading scenes with addressables. If you don't load your scenes using addressables, especially if you make use of SpriteAtlases, Addressables will load duplicate data into memory.

      Our scenes have references to animations that make use of the same sprite atlases the our dynamically loaded animations use. Because of that, when both dynamically and statically loaded animations are present in the same scene, Unity will load in a copy of the sprite atlas for the "statically" loaded animations, and Addressables will load in a copy of the sprite atlas for the "dynamically" loaded animations.

      None of this was intuitive even though it makes sense once it's been explained.
    • BONUS: the addressables analysis window has an "Unfixable" category with an analyzer that will tell you exactly which assets are duplicated in both Unity's scenes and the addressable asset bundles.

    My humble request from the addressables team is to update their documentation and put together a comprehensive guide. I gathering all this information from random forum posts, videos, and scattered documentation. The learning curve is tremendous and it doesn't have to me. These links were quite helpful:

    https://forum.unity.com/threads/built-in-data-and-local-assets-duplicated.785603/
    https://forum.unity.com/threads/are-addressable-assets-loaded-twice-if-i-reference-them.697058/

    All together, our load times are at parity with Resources.Load, our memory footprint is ever-so-slightly larger with addressables, and our build sizes are dramatically smaller with addressables.


    Here's a breakdown of the cursory benchmarking I did. I'm not including pre-optimization numbers. They were basically nonsensical (30+ seconds to load on my old iPhone).

    Here's me trying to share an image through Unity forums. Notice how my eyes are bleeding:

     
    Last edited: Dec 18, 2021
  13. Eclectus

    Eclectus

    Joined:
    Oct 29, 2012
    Posts:
    20
    glenneroo likes this.
  14. aaversa

    aaversa

    Joined:
    Sep 5, 2016
    Posts:
    41
    We're having issues with load speed here. In our test case, we have about 100 TextAssets being loaded, which total less than half a megabyte. This is currently taking 13.6 seconds to load in editor mode on a 9900K w/ solid state drive. Totally insane. I have tried:

    * LZ4 vs. Uncompressed
    * Contiguous Bundles
    * Disable CRC checks
    * Addressables.LoadResourceLocationsAsync with typeof(TextAsset) specified

    None of these have lowered the load time. Is this a known issue? How could less than half a megabyte of files be this slow to load?
     
    bartofzo and Ardito92ITA like this.
  15. Tartopom29

    Tartopom29

    Joined:
    Jan 20, 2021
    Posts:
    6
    If it might help, we had a similar issue.

    Loading from Resources was extremely fast and switching to Addressables slow.
    Our problem was that for characters resources (2D images), we had 1 big asset group (1.4Gb), loading 1 prefab took about 4 seconds.
    Splitting the big file into multiple smaller groups (1 per character) drastically decreased loading time (now it's 500ms).
     
    Miscellaneous likes this.
  16. EternalSouls

    EternalSouls

    Joined:
    Apr 22, 2021
    Posts:
    3
    Lokesh2022 and abj2023 like this.
  17. Lokesh2022

    Lokesh2022

    Joined:
    May 8, 2021
    Posts:
    9
    Tartopom29 likes this.
  18. Wully

    Wully

    Joined:
    Mar 18, 2014
    Posts:
    15
    Thanks, this saved us, when we didnt use the type with LoadResourceLocationsAsync, the whole projects addressable assets were being loaded into memory as well as taking a long time to load

    For anyone in the future, you should use UnityEngine.SceneManagement.SceneInstance for getting the resource locations for scene addressables, that one caught me out at first
     
  19. bartofzo

    bartofzo

    Joined:
    Mar 16, 2017
    Posts:
    150
    Having the same issue. Very slow for small amounts of data. The solutions provided in this thread don't work for me.
     
  20. gsylvain

    gsylvain

    Joined:
    Aug 6, 2014
    Posts:
    100
    Just out of curiosity... Try this:
    - renaming the file extension from .txt to ".potato"?
    - have a prefab's component or scriptableObject expose an array of UnityEngine.Object
    - link all your .potato
    - you should be able to cast a ref as TextAsset.

    Does doing this change the speed?
     
  21. MiFrilke

    MiFrilke

    Joined:
    Dec 14, 2016
    Posts:
    41
    Has anyone in this thread ever reported this as a bug to Unity? If so, could you share the ticket ID, so we could keep up with the status of the ticket?

    We're facing similar issues on HDD and none of the suggested solutions/workarounds have helped us so far (or only made the loading times less terrible, but still too long). We're sitting at ~15-20s loading on SSD vs ~6:30min on HDD...

    Only thing left to look into is the "CRC check disable", which is currently in progress.
     
  22. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    Just moved a project over to addressables. Boot time went up by 1 minute. Switched Asset Bundle CRC to Disabled. Boot time went back down by 1 minute.
     
    Thygrrr and MiFrilke like this.
  23. icaro56

    icaro56

    Joined:
    Apr 4, 2018
    Posts:
    6
    I think the main problem is in the function GetResourceLocations used by the other functions GetDownloadSizeAsync and DownloadDependenciesAsync.

    When there are many addresses in locations the GetResourceLocations function slow down. So, this function need to be optimized.
     
  24. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    658
    Thank you. I had about 300 objects in an addressable group. I added a five more...

    And then I was seeing five second pauses in built players, but not Unity. I hit some kind of invisible boundary with quantity or hashing. I thought it was my code and I was chasing my tail until I read your reply. Thank you! It turns out the addressables CRC implementation sucks.

    Does anyone have a ticket open for their behavior? Calculating CRCs for files can and should be fast.