Search Unity

[ISSUE] Asynchronous loading takes enormous amount of time in Android (1000 assets - 17 seconds).

Discussion in 'Addressables' started by TextusGames, Oct 8, 2019.

  1. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    727
    I've found that disabling the CRC checks results in massive load time improvements (30s -> 1s on consoles)
     
    ZAUBAR likes this.
  2. netkill

    netkill

    Joined:
    Feb 12, 2020
    Posts:
    3
    Bump + 1
     
  3. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    2,517
    I am also in the same situation. LoadAssetsAsync takes way too long on consoles, android.. for me it takes like 2.5 mins ... that takes like 3-4 sec on PC
     
    TextusGames likes this.
  4. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    2,517
    I wonder, if creating many addressable groups with less counts and loading them one after another can help speed up the loading process. any one?
     
  5. nindim

    nindim

    Joined:
    Jan 22, 2013
    Posts:
    130
    Hi @unity_bill ,

    I saw you posted back in October 2019 about looking into this, was there ever a resolution or is it still on the roadmap?

    Does the new sync WaitForCompletion() functionality suffer from the same slowness (perhaps it's using the same slow async API underneath?).

    Can you advise on how to best workaround his issue till it's fixed? I've seen a few people mention disabling the CRC checks, is this wise?

    Thank you!

    Niall
     
    phobos2077 likes this.
  6. davidla_unity

    davidla_unity

    Unity Technologies

    Joined:
    Nov 17, 2016
    Posts:
    763
    @nindim Current WaitForCompletion will suffer from the same slowness. However, I would like to say engine fixes are currently in the process of being backported that should really help speed up Addressables in general. There were engine side issues that have been resolved by another part of the team. I know that's not exactly what you're asking but I wanted to throw that out.

    Current it is still on the roadmap. We have some team members that are dedicated to improving performance. I know this is something they're aware of. I'll try and have them post here if they have anything useful to add. I have also made tickets regarding people seeing APK builds being twice the size of other builds so that could have something to do with it as well.

    Disabling CRC check isn't an uncommon thing to do. If you're worried about bundles maybe being corrupted or altered in some way then I wouldn't disable it. If that isn't really a concern for you then it might be worth considering. Maybe at least as a temporary workaround. I can't really recommend one way or another, it's dependent on your specific situation.

    I will say, one option you could consider is our "Use CRC for Cached Asset Bundles". Then you can make sure that any remote bundles haven't been messed with but don't have to incur the cost of that check when loading cached bundles.
     
  7. FlyVC

    FlyVC

    Joined:
    Jan 5, 2020
    Posts:
    30
    for me, it helped to change the Bundle Mode from "pack together" to "pack separately".
    Not sure if having thousands of separate assets bundles will have another drawback later on...
     
  8. lfg_ryan

    lfg_ryan

    Joined:
    May 8, 2019
    Posts:
    7
    Does anyone have any suggested workarounds here? We were using addressable version 1.8.5 without issues. Loading of a large number of scriptable objects through LoadAsync completed on older Android phones in ~5s. Every time we have tried to update past 1.8.5 load times have exploded to over 2min. We now are forced to update do to some other random bugs involving sprite sheets between addressable 1.8.5 and the latest 2019 release. I have tried every combination of option disabling I can think off. Load only requested vs load all on request, contiguous bundles vs not, CRC checks vs not. I get small percentage improvements based on these checks but nothing close to the < 10s I used to have, we haven't been able to get it under 1m.
     
  9. Marble

    Marble

    Joined:
    Aug 29, 2005
    Posts:
    1,268
    Upgrading to 2021.2 caused async addressables to load significantly more slowly for us on Android, but not iOS or Standalone. We're seeing ~3 minutes to finish loading 100MB of assets locally (not downloaded). It takes ~3 seconds on other platforms, and took about that long on Android as well before upgrading to Unity 2021.2 from 2021.1, and to Addressables 1.19.11.
     
  10. felipe-shida-playkids

    felipe-shida-playkids

    Joined:
    Sep 16, 2020
    Posts:
    13
    Bump +1

    We are experiencing the same problem, making a stress test on our game we noticed a huge gap between prefab load times:
    Load: 419 prefabs, ~250 unique prefabs (all pretty similar in size)
    Device: Xiaomi POCO phone
    Pack mode: separately
    Compress mode: LZMA
    Remote load from Amazon S3 CDN
    Async load time: 3:45 minutes

    Resources load time: 20 seconds

    To take the possibility that it was a network connection issue aside, we made the same test loading the bundles from local storage:
    Local async load time: 3:10 minutes

    So the conclusion is that the overhead is very significant and it lies over the fetch, decompress pipeline probably? Has anyone made this test using the pure asset bundle async load, without addressables?
     
    Last edited: Dec 1, 2021
  11. Marble

    Marble

    Joined:
    Aug 29, 2005
    Posts:
    1,268
    I've made some effort to repro this in an empty project without success – the long load times seem to only occur with specific prefabs or assets. Like you, we have a bundle with a lot of prefabs in it, many of them nested. When we remove some of the root prefabs, load times are fine, but they go back up when adding them again. Our next task is to narrow it down to which asset type or prefab configuration is responsible and report a bug; this may take a couple weeks as our Android tester is busy over the holidays.
     
  12. JelteS

    JelteS

    Joined:
    Jan 30, 2018
    Posts:
    16
    You should probably check the dependencies of the asset bundles. It may be that loading your prefabs bundle, requires the game to actually load all the bundles due to the dependency graph. Check the build layout report, to see a bundle’s dependency graph
     
  13. Marble

    Marble

    Joined:
    Aug 29, 2005
    Posts:
    1,268
    That was a great suggestion, and I did find some redundant dependencies. Eliminating them decreased loading times, but it's still weirdly slow on Android (63 seconds) vs other platforms (5 seconds).
     
  14. felipe-shida-playkids

    felipe-shida-playkids

    Joined:
    Sep 16, 2020
    Posts:
    13
    Using
    Code (CSharp):
    1.  
    2. Addressables.DownloadDependenciesAsync
    3.  
    before requesting the addressables directly, then using the Synchronous API (WaitForCompletion) reduced the time to approximately 35s, that is what we expected for downloading the 21MB + instantiating from local storage.
     
    Eclectus likes this.
  15. Eclectus

    Eclectus

    Joined:
    Oct 29, 2012
    Posts:
    20
    Huge speedup on Android via Individual Addressables instead of just root folder/s

    In my use case I used 253 folders (8.92 GB on disk) each containing
    1. A 8192x4096 ASTC texture
    2. A 512x512 ASTC cubemap
    3. A 1KB json file
    I found that making all (253 * 3 = 759) of these assets into individual addressables, as opposed to just their root folder made things much faster, to the point that they behaved as fast on Android (an Oculus Quest 2) as when previously using Resources.Load()
    1. When I say 'Individual addressables' - I mean that they each have their 'Addressable tickbox' ticked and appear in the 'Addressables Group' window without being ghosted, in that window.
      1. Previously they already existed in the 'Addressables Group' window, but only because their root folder had been dragged in / marked as an addressable.
      2. So they were ghosted, indicating that they were a descendent of an addressable, not addressables themself.
      3. Note that I have 'Bundle Mode' set to 'Pack Separately'.
    2. When I say 'as fast on Android' this is only the case if you also set 'Asset Bundle CRC' to be 'Disabled' as mentioned in previous posts.

    I will update this post to include more information as I find it, but wanted to get this out there in case it helps?

    Thanks to everyone who has contributed so far!
     
    Last edited: Dec 27, 2021
  16. Marble

    Marble

    Joined:
    Aug 29, 2005
    Posts:
    1,268
    We achieved load times on Android equivalent to other platforms by choosing "Pack Separately" on our Addressables groups, as well.
     
    Eclectus likes this.
  17. Eclectus

    Eclectus

    Joined:
    Oct 29, 2012
    Posts:
    20
    Nice - I think this only works when combined with using many/all individual addressables, within the groups. So if you were to simply have a root folder as the only addressable, then 'Pack Separately' as the 'Bundle Mode' will not make a difference on it's own - you want to have an Addressable for each asset that you plan to load separately, otherwise it is my theory that it does not pack them separately.

    @davidla_unity Any thoughts on what I have suggested?
    I like the addressables system, but have reached these conclusions about individual paths / packing without documentation - feel free to link to some if I missed it. Or shoot down what I am saying if it is inaccurate. :)
     
    Last edited: Dec 27, 2021
    OmnifariousStudios and Marble like this.
  18. jmozgawa

    jmozgawa

    Joined:
    Oct 16, 2018
    Posts:
    4
    I've recently upgrade to Unity 2021.2.7 and noticed exact same issue. Terrible slow addressables / asset bundles load time.
    But I've found a workaround that seem no one mention here before. All what I had to do was to check "Use UnityWebRequest for Local Asset Bundles" in each Addressable Asset Group asset. After cleaning and rebuilding addressables, all starts work as expected. No lags, slowdowns, game loads 50x faster as it was before the 2021.2.7.

    upload_2022-1-4_15-10-38.png
     
  19. nindim

    nindim

    Joined:
    Jan 22, 2013
    Posts:
    130
    Hi @davidla_unity ,

    It seems like enabling UnityWebRequest for local bundles could be the silver bullet to this issue :)

    Can you take this to the team and find out if this should be on by default, please?

    With regards to the other improvements that were going to be backported, has this happened yet.and if so what releases should we be upgrading to?

    Thank you!

     
  20. KwahuNashoba

    KwahuNashoba

    Joined:
    Mar 30, 2015
    Posts:
    110
    It depends, I would say. As far as I understand, enabling UnityWebRequest for local bundles will repack LZMA compressed asset bundles into LZ4 and then cache it with that compression. Since, during load time, LZ4 does not need to be uncompressed as a whole but only the required blocks, it will speed up loading times. So, in case you have large bundles with a lot of assets, this is where you get performance boost. On a negative side, it will still keep LZMA compressed bundle togehter with LZ4 cache, so it will occupy more space on device storage.

    Can you confirm that you are using LZMA for those addressable groups that you noticed performance boost on by enabling this option?

    There is also another thread on the forum that looks into this abstract issue of slow loading speed of addressables.
     
    phobos2077 likes this.
  21. phobos2077

    phobos2077

    Joined:
    Feb 10, 2018
    Posts:
    350
    You described it very well. But I think UWR caches bundles in uncompressed format, not LZ4. So it can be a huge difference in storage, depending on the type and amount of content. Another reason for speed up might be that if LZMA compressed bundle is loaded directly (w/o UWR), it will be loaded completely uncompressed into memory which might have severe repercussions. Since LZ4 allows individual assets to be decompressed, loading it directly doesn't involve this up-front decompression time and memory space cost.

    The point is, what seems like an "obvious default" for one project isn't always true for another.
     
  22. KwahuNashoba

    KwahuNashoba

    Joined:
    Mar 30, 2015
    Posts:
    110
    Are you sure about this? Based of what I can found in the docs, it says:
    And also on the documentation about asset bundle compression they mention that UWR might or might not compress cache based on single parameter. I would say addressables set that parameter to true, although there is no any explicit info on this.
     
  23. phobos2077

    phobos2077

    Joined:
    Feb 10, 2018
    Posts:
    350
    It is part of the configuration, Cache Initialization Settings. I have this setting ("Compress bundles") unchecked since long time, so that's why it's uncompressed for me. I just forgot about this piece of config :)

    But my point I think still stands - you have to understand all the moving parts when using Asset Bundles and all projects are different, hence why everything is configurable.
     
    KwahuNashoba likes this.
  24. glenneroo

    glenneroo

    Joined:
    Oct 27, 2016
    Posts:
    231
    Since upgrading from 2020 to 2021, my Android (Oculus Quest 2) build Addressable loading time went from 10 seconds to 80 seconds while waiting for one of 10 Addressable groups (with 112 prefabs) to be loaded. Setting Bundle Mode to Pack Separately cut the time down to 40 seconds but this is still horrendous for a ~300MB APK and compared to 10 seconds prior to upgrading is... frustrating. I also tried setting Use UnityWebRequest but it didn't help at all. Changing CRC settings does nothing. Analyzing/Fixing duplicate dependencies made no difference at all.

    EDIT: I'm using 1.19.18 Addressables after originally downgrading to 1.17.17 because the only compatible (available) version via Package Manager was 1.19.17 which had this bug preventing me from even loading the same troublesome group without changing code - I wonder if the bug was actually exposing some other underlying issue with this group... so I tried recreating the group and moving all the prefabs to it - unfortunately no change. Good news is that 1.19.18 is available via Package Manager (but a manual upgrade is necessary it seems) and fixes the bug for me.

    EDIT #2: Ok apparently this group is somehow getting packed multiple times. In the Editor it loads 112 prefabs but in my build LoadResourceLocationsAsync(label) returns 1846 locations with the label (I use labels and groups with the same names) and each prefab is listed between 15-21 times! I have other prefabs in other groups which are loaded only once so I don't know what's wrong with these prefabs - perhaps because of all the dependencies? My solution for the time being is put in a check if it's already loaded and not load again which helps... but loading times are only down to 25 seconds.

    EDIT #3: Using the 2nd parameter of LoadResourceLocationsAsync(label, type) fixes finding 20x of each asset - perhaps 1.19.17/18 fixes a bug I was "exploiting"? Load time is now back down to almost normal... but still not like they used to be.

    Summary: Pack Separately + LoadResourceLocationsAsync with 2 parameters.
     
    Last edited: Feb 12, 2022
    gyx920820 likes this.
  25. davidla_unity

    davidla_unity

    Unity Technologies

    Joined:
    Nov 17, 2016
    Posts:
    763
    Huh, I mean that definitely doesn't seem right. If you have a repro you can file a bug against I'd be very grateful for you to do so. We have heard of some users running into problems where they request a resource location and the location is returned multiple times, but we've never been able to reproduce it locally. I think what you're running into there sounds very similar.
     
  26. glenneroo

    glenneroo

    Joined:
    Oct 27, 2016
    Posts:
    231
    I actually filed a bug due to a crashing issue which I was able to resolve myself. My project is attached to Case #1402052 and should exhibit the behavior.
     
    davidla_unity likes this.
  27. aaversa

    aaversa

    Joined:
    Sep 5, 2016
    Posts:
    41
    Nothing that we have tried improves load speed at all. Enabling local webrequests, loading by type, changing compression type, pack together/separately, contiguous bundles, disable CRC, etc. It is currently taking Unity almost 14 seconds to load 100 TextAssets of combined size <0.5 MB. How is anyone supposed to use this half-baked functionality?
     
    NyanGamedev and TextusGames like this.
  28. notunusual

    notunusual

    Joined:
    Jun 13, 2016
    Posts:
    12
    I'm told it's possible this is the issue we are facing.

    - https://issuetracker.unity3d.com/is...when-calling-waitforcompletion-freezes-editor

    We had the same issue as you reported, where individual asset loads of raw TextAssets (from 5-20k each) were taking from 0.5 to 2 seconds per load.

    Switching on the Use UnityWebRequest did resolve it, but i'm not satisfied this makes any sense, but is at least a work around for the time being.
     
  29. notunusual

    notunusual

    Joined:
    Jun 13, 2016
    Posts:
    12
    Unity support had us add a gradleTemplate.properties file in ./Assets/Plugins/Android/gradeTemplate.properties

    Code (CSharp):
    1. org.gradle.jvmargs=-Xmx**JVM_HEAP_SIZE**M
    2. org.gradle.parallel=true
    3. android.enableR8=**MINIFY_WITH_R_EIGHT**
    4. unityStreamingAssets=**STREAMING_ASSETS**, .bundle
    5. **ADDITIONAL_PROPERTIES**
    The contents of the above.

    It stops the .bundle files from being compressed again, which resolved our slow loading issue.
     
    SlawekKmiecik and glenneroo like this.
  30. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    973
    Hm, that's curious. I'm not sure why that would help, considering the bundles are already in StreamingAssets and hence part of the **STREAMING_ASSETS** macro. OTOH, that could explain why I am not seeing this issue, we screw with the aapt.noCompress list to explicitly skip .bundle already.
     
    Cupchen likes this.
  31. notunusual

    notunusual

    Joined:
    Jun 13, 2016
    Posts:
    12
    Yes it was confusing to me as well.

    However they asked us to verify if the .bundle files in the .apk were in fact compressed again, and sure enough they were.


    When using the gradle properties above they no longer were.
     
  32. NyanGamedev

    NyanGamedev

    Joined:
    Apr 11, 2022
    Posts:
    5
    After wasting 1 day of work and trying all variations of workarounds, nothing helped: replacing
    SceneManager.LoadSceneAsync
    by
    Addressables.LoadSceneAsync
    adds 5s to my startup time, on Android only (so this is not a bug in my setup, other platforms work as fast as before, this is an addressables issue on Android)

    I don't know why I believed addressables was production ready...
     
    TextusGames likes this.
  33. bellepoque

    bellepoque

    Joined:
    Dec 9, 2014
    Posts:
    3
    After explicitly specifying the type in Addressables.LoadResourceLocationsAsync(label,type)
    the problem was resolved.
     
  34. MaxGucci

    MaxGucci

    Joined:
    Mar 12, 2014
    Posts:
    4
    Only setting gradleTemplate.properties helps me.

    unityStreamingAssets=**STREAMING_ASSETS**, .bundle
     
    Merklar likes this.