Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

[ISSUE] Asynchronous loading takes enormous amount of time in Android (1000 assets - 17 seconds).

Discussion in 'Addressables' started by TextusGames, Oct 8, 2019.

  1. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Addressable asset loading takes too much time about 100x more than it shuld in virtual play mode and at android runtime build.

    I made 500 addressable scriptable objects than i load them at start of the game and waiting for them all to load and it takes about 10 seconds in virtual mode and about 5 seconds in android runtime mode.

    Tested with addressables 1.1.10 and 1.2.4
    with unity 2019.2.8 and 2019.3b4

    I have already submited bug project.

    BugStatus:
    http://fogbugz.unity3d.com/default.asp?1189850_e14nvlg7eni5d3m7

    Bug case: 1189850

    EDITED
    LInk to the advanced project:
    https://drive.google.com/file/d/1qNOH0rQOdY0JoqXeH-c7lAKW7X0pH8Tx/view
    Link to simple project
    https://drive.google.com/open?id=1VBBRoJFjBCJfFAW51cVS-GEAo3l1Kd2V:
     

    Attached Files:

    Last edited: Nov 1, 2019
  2. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    690
    Could you show us the loading code, please?
     
  3. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
  4. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Here is load code

    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEngine.AddressableAssets;
    3. using UnityEngine.UI;
    4.  
    5. public class Loader : MonoBehaviour
    6. {
    7.     public Text TextUI;
    8.     private async void Start()
    9.     {
    10.         Application.logMessageReceived += ApplicationOnLogMessageReceived;
    11.      
    12.         Debug.Log("Initializing...");
    13.         var lastTime = Time.realtimeSinceStartup;
    14.      
    15.         await Addressables.InitializeAsync().Task;
    16.  
    17.         Debug.Log("Initialized with " + (Time.realtimeSinceStartup - lastTime) + " seconds");
    18.         lastTime = Time.realtimeSinceStartup;
    19.      
    20.         Debug.Log("");
    21.         Debug.Log("Loading...");
    22.      
    23.         var operationHandle = Addressables.LoadAssetsAsync<ScriptableObject>("default", o => { });
    24.         await operationHandle.Task;
    25.      
    26.         Debug.Log("Loaded with " + (Time.realtimeSinceStartup - lastTime) + " seconds");
    27.         Debug.Log("Loaded asset count : " + operationHandle.Result.Count);
    28.  
    29.     }
    30.  
    31.     private void ApplicationOnLogMessageReceived(string condition, string stacktrace, LogType type)
    32.     {
    33.         TextUI.text += condition + "\n";
    34.     }  
    35.  
    36. }
     
  5. Flirer

    Flirer

    Joined:
    May 20, 2014
    Posts:
    5
    I have same or very similar issue.
    Packed Mode - take seconds (1-5) to download assets.
    Virtual Move - 20-30 seconds.
    On Android 20-40-60 seconds.

    I tryed to download same files with Mobile Phone, to ensure its not slow internet connection - they downloaded in few seconds.

    If im using WiFi on android device, every thing is downloaded with same speed as PC, but again 4g internet alowed to download same files manually in seconds, a bit slower then wifi, but just a bit.
     
  6. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    I am waiting for official response. For me loading 500 simple scriptable object assets from FILE takes 9 seconds))
     
  7. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    690
    Some quick calculation shows that loading 500 objects in one frame each would take 8.3s at 60FPS. Maybe something is causing it to load them sequentially for some reason, thus incurring an one frame per asset bundle penalty. @unity_bill?
     
  8. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    I actually suggested the same idea in bug report chat.
     
  9. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    194
    We're seeing something semi-related,on an iPhone 7/10 loading a folder via

    We build up a list of IResourceLocations and pass that list to the LoadAssetsAsync

    var test = await Addressables.LoadAssetsAsync<UnityEngine.Object>(resourcLoc, (result)=>
    {
    // Do stuff
    }).Task

    This takes 4seconds or so on the iPhone 7 to load about 350 files, on an iPhone 10 it takes > 30seconds

    we're not sure exactly what is going on..

    Looking into it but any ideas/comments would be useful.

    We don't believe it is anything to do with shaders or texture decompression but then i need to investigate more.

    thnx
     
    TextusGames likes this.
  10. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    To deferintiate it you can just create 500 empty scriptable objects and load just them and you should see that it also takes seconds.
    It is a bug clearly.
     
  11. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Further testing on different android devices (runtime build)
    Loading 1000 empty scriptable object addressable assets;
    Galaxy A 30 - 4 seconds;
    Huawei Honor 7C - 6 seconds;
    Xiaomy mi mix 17 seconds !!!
     
  12. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Here is link to builded Apk.
    Can someone else test on their devices how much time it is taking?
    https://drive.google.com/open?id=1DH8842JnQYk8UAtIbuv3zWep_lmEZw89

    For me loading 1000 empty scriptable object with 17 seconds is a bug (especially then on windows it takes 0.3 seconds!)

    Please, someone, test it and say your opinion.
     
    Last edited: Oct 15, 2019
    KB73 likes this.
  13. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    194
    @unity_bill - wonder if he can confirm issue or if there is further optimization to be done? what is the recommended way of loading a lot of objects in fast with addressables? We just use a live folder and load the entire thing in..but it is really really slow on device

    TextusGames - can you confirm if the time is still occurring even with the old bundle system? if so then i guess it is something more fundamental
     
  14. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Try to load on windows it should take less then a second. And if it so then I think it is a bug.

    I have not used old system. I can not say anything about it..
     
  15. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Ok.
    I have compared loading times of old Resources and new AddressableSystems.
    In all cases i load 1000 identical but different sets of empty scriptable objects ( 3 set of 1000 assets).
    The below numbers in brackets are (Resources.LoadAll / Resources.LoadAsync / Addressables.LoadAssetsAsync).

    Loading in different cases.
    1. At windows runtime, they are almost identical (0.03 / 0.28 / 0.35).
    2. In editor there is loading time increase like 20x times in asynchronous operations compared to the synchronous method of resources (and packed and fast mode for addressables) (0.25 / 10.53 / 16.98).
    3. In android only synchronous Resources.LoadAll takes (0.34 / 11.62 / 15.69).

    That make me think that there is some kind of bug or performence flow with asynchronous system both in old Resources and new Addressables system.


    That is possible to load on android via synchronious Resources.LoadAll with 0.34 seconds definetly should not take 16 seconds with asynchronous operation on same device!


    here is link to comparasing project
    https://drive.google.com/open?id=1qNOH0rQOdY0JoqXeH-c7lAKW7X0pH8Tx

    here is apk
    https://drive.google.com/open?id=1ka_p5wKpZIf4mIXjAMzJBCFGKdq9A03T

    That is enough for me. I am tired of testing and i am convinced that there is an issue with the async system. I will just wait now for official responce.
     

    Attached Files:

    Last edited: Oct 11, 2019
  16. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    194
    The only thing i can think of is the bundles have not been tagged as not to compress in the apk so you end up with the apk zip and the compressed file within the apk...which you usually don't want for assetbundles.

    In the gradle build system for android, you can specify the file extensions/folders that should not be compressed as this slows stuff down, typically things in streaming assets

    As for iOS, i don't know..
     
  17. unity_bill

    unity_bill

    Unity Technologies

    Joined:
    Apr 11, 2017
    Posts:
    965
    Right now virutal mode has some artificial slow downs that we did not intend to leave in there. We have a ticket in our backlog to remove the slowdown.
    In general virtual mode should only be used when debugging ref count type stuff using the profiler. Other than that use case, use fast or packed.
     
  18. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Virtual mode is not that problematic because you have different ones to use. But thanks for Reply.

    What about loading times on android?
    In my tests loading 1000 lightweight scriptable assets take 0.35 seconds on windows runtime build. and up to 17 seconds on Android runtime build (may take more or less on different devices but it takes seconds) ( though loading 1000 separate assets synchronously via Resources.LoadAll() takes 0.3 on android).
    I think loading times of 1000 lightweight objects should be less than 1 second ( and tests prove that) and not take 17 seconds.
    Why Asynchronous loading takes so much time on android? (both in Resources.LoadAsync and addressables). What do you think? Are there any plans to fix this?
     
  19. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Changing assets to pack separately decreased loading time in virtual play mode from 13 seconds to 0.4 seconds and made it almost equal to fast play mode time = 0.35 seconds.
    But it also increased android loading time from 17 to 34 seconds(
     
  20. shiqiang_vts

    shiqiang_vts

    Joined:
    Jun 12, 2019
    Posts:
    8
    I think this is a issue about AssetBundleProvider, each asset load request will use
    LoadFromFileAsync create AssetBundle object. If you want to load many small assets, it will create lot of IO request. on PC SSD is OK,but on HDD or some slow storage medium, like phone,console.. will take too much time. On my project,we request load one thousand lua files on switch, will take 20 minutes.
    I have two solution:
    1. Modify addressable, use LoadFromMemoryAsync create assetbundle handle. First step is move all small files to one asset group, and extend addressable support LoadFromMemoryAsync. this need read whole assetbundle file to memory, then unity can direct load from memory.
    2. Make all small file to one file, then we only have one IO request.

    I hope official can start to optimizing it.
     
    KB73 and TextusGames like this.
  21. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Minutes? Really?
     
  22. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    I hope they at least approve that problem exists an say that they will try to fix that some day.
    Loading 1000 files at start of the game is quite a realistic scenario. And if it takes 20 seconds then no one will wait an play your game. And this is a problem.
     
  23. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    I have copied same builded project from windows SSD to HDD and there is no difference it takes 0.3 seconds to load from both discs.
     
  24. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    194
    Ya our issue is not the editor, it is just Android/iOS
     
    TextusGames likes this.
  25. LilMako17

    LilMako17

    Joined:
    Apr 10, 2019
    Posts:
    2
    My project also uses roughly the same pattern and produces similar results (loading 1 asset bundle from streaming assets containing a couple thousand scriptable objects). My scriptable objects reference some sprites that get packed into the asset bundle, so this load time also includes loading those:

    Editor fast mode: 2,507 objects loaded in 1.37 seconds
    Android: 2,507 objects loaded in 16.8 seconds

    Code (CSharp):
    1. /// <summary>
    2.     /// Asynchronously Loads all static data.
    3.     /// </summary>
    4.     public void LoadAssetsFromAddressables(Action userCallback = null)
    5.     {
    6.         _startDataLoad = false;
    7.         _isDone = false;
    8.         _userCallback = userCallback;
    9.         _stopWatch = new Stopwatch();
    10.         _stopWatch.Start();
    11.  
    12.         var task = Addressables.LoadResourceLocationsAsync(STATIC_DATA_LABEL);
    13.         task.Completed += (result =>
    14.         {
    15.             if (result.Status == AsyncOperationStatus.Failed)
    16.             {
    17.                 Log.Error("failed to find location of " + STATIC_DATA_LABEL);
    18.                 _isDone = true;
    19.                 return;
    20.             }
    21.  
    22.             Log.Debug("found " + result.Result.Count + " files. starting load");
    23.             _startDataLoad = true;
    24.             var keys = result.Result;
    25.             _loadOperation = Addressables.LoadAssetsAsync<StaticData>(keys, (a) => { });
    26.             _loadOperation.Completed += OnAllDataLoaded;
    27.         });
    28.     }
    29.     /// <summary>
    30.     /// Async callback to integrate data all StaticData assets have finished loading.
    31.     /// </summary>
    32.     private void OnAllDataLoaded(AsyncOperationHandle<IList<StaticData>> op)
    33.     {
    34.         _stopWatch.Stop();
    35.         Log.Debug("static data load complete: " + _stopWatch.Elapsed);
    36.  
    37.         if (op.Status == AsyncOperationStatus.Failed)
    38.         {
    39.             Log.Error($"Failed to load StaticData: {op.OperationException}");
    40.             return;
    41.         }
    42.  
    43.         Log.Debug("loaded " + op.Result.Count + " data");
    44.  
    45.         _isDone = true;
    46.         if (_userCallback != null)
    47.         {
    48.             _userCallback.Invoke();
    49.         }
    50.     }
    51.  
     
  26. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Thanks all. Now I know am not the only one.

    I quess on some phones it can take more than a minute to load (on mine mi mix 1 probably).
    In that state, no one will wait for the game to load that amount of time.
    That makes this approach unusable on mobile devices...

    Anyway, we need an answer from the Unity team about enormously slow asynchronous loading times on android (and possibly IOS, Switch ) (@Ryanc_unity, @unity_bill)?
     
    Last edited: Oct 16, 2019
  27. shiqiang_vts

    shiqiang_vts

    Joined:
    Jun 12, 2019
    Posts:
    8
    I want to share some we found clues.
    I found unity load assets from the asset bundle to use a very small buffer to read files. it's 64kb. I think it's should be 2 MB or more to reduce IO request times. Luckily Unity gives us a function that can adjust read buffer size: LoadFromStreamAsync, it has a parameter set read buffer size.
     
    KB73 likes this.
  28. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Could you try to use that function and compare loading times?
     
  29. shiqiang_vts

    shiqiang_vts

    Joined:
    Jun 12, 2019
    Posts:
    8
    I found why our game needs 20 minutes to load the asset bundle, unity will calculate asset bundle CRC32, need to read the whole file to calculate. our game's asset bundle very large( 2-3 GB), so this is a very slow process.
    I will try to close the CRC32 check. Maybe Unity team can remind the developer to check it? @unity_bill
     
    KB73 likes this.
  30. LilMako17

    LilMako17

    Joined:
    Apr 10, 2019
    Posts:
    2
    following up on my last post. I have solved the problem on my project, I reduced the load times on android down to ~2.5s loading 2500 scriptable objects and around 50Mb of uncompressed sprites (turns into ~1.5s if you remove all the sprites), which is about the same time it takes to load in the editor. It looks like the problem for me was the inter-dependencies in my scriptable objects. For example, I have a lot of classes that look like this:
    Code (CSharp):
    1. public class AClass : ScriptableObject
    2. {
    3.     [SerializeField]
    4.     public BClass BReference;
    5. }
    6.  
    7. public class BClass : ScriptableObject
    8. {
    9.  
    10. }
    Both AClass and BClass were being requested to load as part of my blob of 2,500 objects (loading by tag where all 2500 entries were tagged). However, I noticed if I only requested to load AClass without BClass, the fields in AClass would still populate correctly, and it reduced the load time by a little bit. For my project, I still need a flat list of all AClasses and BClasses, so I went ahead and made a new scriptable object:
    Code (CSharp):
    1.  
    2. public class StaticDataCatalog : ScriptableObject
    3. {
    4.     [SerializeField]
    5.     public List<ScriptableObject> DataCatalog = new List<ScriptableObject>();
    6. }
    and wrote an editor tool to populate the DataCatalog with the list of 2,500 entries i need to load. I then bundled this one StaticDataCatalog with the rest of the data in same bundle, and changed the loading code to just load this one StaticDataCatalog asset instead of requesting 2,500 keys. Reading the DataCatalog field gives me the same information as requesting to load every file in the bundle, but reduces the load times down to expected levels as mentioned at the top of this post. I'm not sure exactly whats going on under the hood, but I'm happy I found a solution that works for me. Good luck to everyone else in this thread, I hope you find solutions too!
     
  31. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    The problem is not related to interdependencies because my repro projects contain just lightweight scriptable objects ( with just one integer field) and loading of 1000 of them took 17 seconds in android.
    I think the problem is related to multiple asynchronous operations that occur at the same time (maybe it kind of happens in sequence instead of parallel).
    Or possibly the problem is how the asynchronous operation decides how much time it could spend on each frame.
    Anyway, it is a deep asynchronous problem that happens both in new addressables and old resource systems.

    LilMako17, Your solution works because you now load just one scriptable object asynchronously instead of 2500(maybe they now load at the start of the game synchronously). Also, check asset duplication rules, there is a probability those 2500 assets are now copied somewhere else.

    I am glad you found the solution that works for you,
    But this is not a proper solution it is just a workaround for the asynchronous problem that still exists and needs to be fixed!
     
  32. unity_bill

    unity_bill

    Unity Technologies

    Joined:
    Apr 11, 2017
    Posts:
    965
    we'll definitely look into loading time of large numbers of assets. thanks for the feedback.
     
    TextusGames, awallick-sd and KB73 like this.
  33. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Thank you.
     
  34. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    161
    Nothing changed in 1.3.3.

    Sync Resources.LoadAll() vs Async Addressables.LoadAllAsync()

    Windows (NoteBook Acer Aspire 5 A-517-511G from HDD):
    Loading 1000 assets from the Resources folder synchronously takes 0.006 seconds.
    Loading of another 1000 assets asynchronously via Addressables takes 0.367 seconds.

    That is 60x slower.....
    It is still ok because it takes only 0.3 seconds.


    Android (Xiaomi Mi Mix):
    Loading 1000 assets from the Resources folder synchronously takes 0.33 seconds.
    Loading of 1000 another assets asynchronously via Addressables takes 18 seconds,

    That is 55x slower.....
    And this is definitely not OK because it takes almost 20 seconds.....

    Android vs Windows:
    Loading on Android is 50x slower than on Windows (18 /0.367).

    ((50 - 60)x seems like magic consistent pattern's number)
     
    Last edited: Nov 1, 2019
    unity_bill likes this.