[ISSUE] Asynchronous loading takes enormous amount of time in Android (1000 assets - 17 seconds).

Addressable asset loading takes too much time about 100x more than it shuld in virtual play mode and at android runtime build.

I made 500 addressable scriptable objects than i load them at start of the game and waiting for them all to load and it takes about 10 seconds in virtual mode and about 5 seconds in android runtime mode.

Tested with addressables 1.1.10 and 1.2.4
with unity 2019.2.8 and 2019.3b4

I have already submited bug project.

BugStatus:
http://fogbugz.unity3d.com/default.asp?1189850_e14nvlg7eni5d3m7

Bug case: 1189850

EDITED
LInk to the advanced project:
https://drive.google.com/file/d/1qNOH0rQOdY0JoqXeH-c7lAKW7X0pH8Tx/view
Link to simple project
https://drive.google.com/open?id=1VBBRoJFjBCJfFAW51cVS-GEAo3l1Kd2V:

EDITED.
Unity decided to not address this adressable issue
(for a time being) (at 04.14 2020)
https://forum.unity.com/threads/iss...d-1000-assets-17-seconds.757151/#post-5784628

 

Could you show us the loading code, please?

 

Here is project.
https://drive.google.com/open?id=1VBBRoJFjBCJfFAW51cVS-GEAo3l1Kd2V

 

Here is load code

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.AddressableAssets;
3. using UnityEngine.UI;
4.  
5. public class Loader : MonoBehaviour
6. {
7.     public Text TextUI;
8.     private async void Start()
9.     {
10.         Application.logMessageReceived += ApplicationOnLogMessageReceived;
11.      
12.         Debug.Log("Initializing...");
13.         var lastTime = Time.realtimeSinceStartup;
14.      
15.         await Addressables.InitializeAsync().Task;
16.  
17.         Debug.Log("Initialized with " + (Time.realtimeSinceStartup - lastTime) + " seconds");
18.         lastTime = Time.realtimeSinceStartup;
19.      
20.         Debug.Log("");
21.         Debug.Log("Loading...");
22.      
23.         var operationHandle = Addressables.LoadAssetsAsync<ScriptableObject>("default", o => { });
24.         await operationHandle.Task;
25.      
26.         Debug.Log("Loaded with " + (Time.realtimeSinceStartup - lastTime) + " seconds");
27.         Debug.Log("Loaded asset count : " + operationHandle.Result.Count);
28.  
29.     }
30.  
31.     private void ApplicationOnLogMessageReceived(string condition, string stacktrace, LogType type)
32.     {
33.         TextUI.text += condition + "\n";
34.     }  
35.  
36. }

 

I have same or very similar issue.
Packed Mode - take seconds (1-5) to download assets.
Virtual Move - 20-30 seconds.
On Android 20-40-60 seconds.

I tryed to download same files with Mobile Phone, to ensure its not slow internet connection - they downloaded in few seconds.

If im using WiFi on android device, every thing is downloaded with same speed as PC, but again 4g internet alowed to download same files manually in seconds, a bit slower then wifi, but just a bit.

 

I am waiting for official response. For me loading 500 simple scriptable object assets from FILE takes 9 seconds))

 

Some quick calculation shows that loading 500 objects in one frame each would take 8.3s at 60FPS. Maybe something is causing it to load them sequentially for some reason, thus incurring an one frame per asset bundle penalty. @unity_bill?

 

I actually suggested the same idea in bug report chat.

 

We're seeing something semi-related,on an iPhone 7/10 loading a folder via

We build up a list of IResourceLocations and pass that list to the LoadAssetsAsync

var test = await Addressables.LoadAssetsAsync<UnityEngine.Object>(resourcLoc, (result)=>
{
// Do stuff
}).Task

This takes 4seconds or so on the iPhone 7 to load about 350 files, on an iPhone 10 it takes > 30seconds

we're not sure exactly what is going on..

Looking into it but any ideas/comments would be useful.

We don't believe it is anything to do with shaders or texture decompression but then i need to investigate more.

thnx

 

To deferintiate it you can just create 500 empty scriptable objects and load just them and you should see that it also takes seconds.
It is a bug clearly.

 

Further testing on different android devices (runtime build)
Loading 1000 empty scriptable object addressable assets;
Galaxy A 30 - 4 seconds;
Huawei Honor 7C - 6 seconds;
Xiaomy mi mix 17 seconds !!!

 

Here is link to builded Apk.
Can someone else test on their devices how much time it is taking?
https://drive.google.com/open?id=1DH8842JnQYk8UAtIbuv3zWep_lmEZw89

For me loading 1000 empty scriptable object with 17 seconds is a bug (especially then on windows it takes 0.3 seconds!)

Please, someone, test it and say your opinion.

 

@unity_bill - wonder if he can confirm issue or if there is further optimization to be done? what is the recommended way of loading a lot of objects in fast with addressables? We just use a live folder and load the entire thing in..but it is really really slow on device

TextusGames - can you confirm if the time is still occurring even with the old bundle system? if so then i guess it is something more fundamental

 

Try to load on windows it should take less then a second. And if it so then I think it is a bug.

I have not used old system. I can not say anything about it..

 

Ok.
I have compared loading times of old Resources and new AddressableSystems.
In all cases i load 1000 identical but different sets of empty scriptable objects ( 3 set of 1000 assets).
The below numbers in brackets are (Resources.LoadAll / Resources.LoadAsync / Addressables.LoadAssetsAsync).

Loading in different cases.
1. At windows runtime, they are almost identical (0.03 / 0.28 / 0.35).
2. In editor there is loading time increase like 20x times in asynchronous operations compared to the synchronous method of resources (and packed and fast mode for addressables) (0.25 / 10.53 / 16.98).
3. In android only synchronous Resources.LoadAll takes (0.34 / 11.62 / 15.69).

That make me think that there is some kind of bug or performence flow with asynchronous system both in old Resources and new Addressables system.


That is possible to load on android via synchronious Resources.LoadAll with 0.34 seconds definetly should not take 16 seconds with asynchronous operation on same device!


here is link to comparasing project
https://drive.google.com/open?id=1qNOH0rQOdY0JoqXeH-c7lAKW7X0pH8Tx

here is apk
https://drive.google.com/open?id=1ka_p5wKpZIf4mIXjAMzJBCFGKdq9A03T

That is enough for me. I am tired of testing and i am convinced that there is an issue with the async system. I will just wait now for official responce.

 

The only thing i can think of is the bundles have not been tagged as not to compress in the apk so you end up with the apk zip and the compressed file within the apk...which you usually don't want for assetbundles.

In the gradle build system for android, you can specify the file extensions/folders that should not be compressed as this slows stuff down, typically things in streaming assets

As for iOS, i don't know..

 

Right now virutal mode has some artificial slow downs that we did not intend to leave in there. We have a ticket in our backlog to remove the slowdown.
In general virtual mode should only be used when debugging ref count type stuff using the profiler. Other than that use case, use fast or packed.

 

Virtual mode is not that problematic because you have different ones to use. But thanks for Reply.

What about loading times on android?
In my tests loading 1000 lightweight scriptable assets take 0.35 seconds on windows runtime build. and up to 17 seconds on Android runtime build (may take more or less on different devices but it takes seconds) ( though loading 1000 separate assets synchronously via Resources.LoadAll() takes 0.3 on android).
I think loading times of 1000 lightweight objects should be less than 1 second ( and tests prove that) and not take 17 seconds.
Why Asynchronous loading takes so much time on android? (both in Resources.LoadAsync and addressables). What do you think? Are there any plans to fix this?

 

Changing assets to pack separately decreased loading time in virtual play mode from 13 seconds to 0.4 seconds and made it almost equal to fast play mode time = 0.35 seconds.
But it also increased android loading time from 17 to 34 seconds(

 

I think this is a issue about AssetBundleProvider, each asset load request will use
LoadFromFileAsync create AssetBundle object. If you want to load many small assets, it will create lot of IO request. on PC SSD is OK,but on HDD or some slow storage medium, like phone,console.. will take too much time. On my project,we request load one thousand lua files on switch, will take 20 minutes.
I have two solution:
1. Modify addressable, use LoadFromMemoryAsync create assetbundle handle. First step is move all small files to one asset group, and extend addressable support LoadFromMemoryAsync. this need read whole assetbundle file to memory, then unity can direct load from memory.
2. Make all small file to one file, then we only have one IO request.

I hope official can start to optimizing it.

 

Minutes? Really?

 

I hope they at least approve that problem exists an say that they will try to fix that some day.
Loading 1000 files at start of the game is quite a realistic scenario. And if it takes 20 seconds then no one will wait an play your game. And this is a problem.

 

I have copied same builded project from windows SSD to HDD and there is no difference it takes 0.3 seconds to load from both discs.

 

Ya our issue is not the editor, it is just Android/iOS

 

My project also uses roughly the same pattern and produces similar results (loading 1 asset bundle from streaming assets containing a couple thousand scriptable objects). My scriptable objects reference some sprites that get packed into the asset bundle, so this load time also includes loading those:

Editor fast mode: 2,507 objects loaded in 1.37 seconds
Android: 2,507 objects loaded in 16.8 seconds

Code (CSharp):

1. /// <summary>
2.     /// Asynchronously Loads all static data.
3.     /// </summary>
4.     public void LoadAssetsFromAddressables(Action userCallback = null)
5.     {
6.         _startDataLoad = false;
7.         _isDone = false;
8.         _userCallback = userCallback;
9.         _stopWatch = new Stopwatch();
10.         _stopWatch.Start();
11.  
12.         var task = Addressables.LoadResourceLocationsAsync(STATIC_DATA_LABEL);
13.         task.Completed += (result =>
14.         {
15.             if (result.Status == AsyncOperationStatus.Failed)
16.             {
17.                 Log.Error("failed to find location of " + STATIC_DATA_LABEL);
18.                 _isDone = true;
19.                 return;
20.             }
21.  
22.             Log.Debug("found " + result.Result.Count + " files. starting load");
23.             _startDataLoad = true;
24.             var keys = result.Result;
25.             _loadOperation = Addressables.LoadAssetsAsync<StaticData>(keys, (a) => { });
26.             _loadOperation.Completed += OnAllDataLoaded;
27.         });
28.     }
29.     /// <summary>
30.     /// Async callback to integrate data all StaticData assets have finished loading.
31.     /// </summary>
32.     private void OnAllDataLoaded(AsyncOperationHandle<IList<StaticData>> op)
33.     {
34.         _stopWatch.Stop();
35.         Log.Debug("static data load complete: " + _stopWatch.Elapsed);
36.  
37.         if (op.Status == AsyncOperationStatus.Failed)
38.         {
39.             Log.Error($"Failed to load StaticData: {op.OperationException}");
40.             return;
41.         }
42.  
43.         Log.Debug("loaded " + op.Result.Count + " data");
44.  
45.         _isDone = true;
46.         if (_userCallback != null)
47.         {
48.             _userCallback.Invoke();
49.         }
50.     }
51.  

 

Thanks all. Now I know am not the only one.

I quess on some phones it can take more than a minute to load (on mine mi mix 1 probably).
In that state, no one will wait for the game to load that amount of time.
That makes this approach unusable on mobile devices...

Anyway, we need an answer from the Unity team about enormously slow asynchronous loading times on android (and possibly IOS, Switch ) (@Ryanc_unity, @unity_bill)?

 

I want to share some we found clues.
I found unity load assets from the asset bundle to use a very small buffer to read files. it's 64kb. I think it's should be 2 MB or more to reduce IO request times. Luckily Unity gives us a function that can adjust read buffer size: LoadFromStreamAsync, it has a parameter set read buffer size.

 

Could you try to use that function and compare loading times?

 

I found why our game needs 20 minutes to load the asset bundle, unity will calculate asset bundle CRC32, need to read the whole file to calculate. our game's asset bundle very large( 2-3 GB), so this is a very slow process.
I will try to close the CRC32 check. Maybe Unity team can remind the developer to check it? @unity_bill

 

following up on my last post. I have solved the problem on my project, I reduced the load times on android down to ~2.5s loading 2500 scriptable objects and around 50Mb of uncompressed sprites (turns into ~1.5s if you remove all the sprites), which is about the same time it takes to load in the editor. It looks like the problem for me was the inter-dependencies in my scriptable objects. For example, I have a lot of classes that look like this:

Code (CSharp):

1. public class AClass : ScriptableObject
2. {
3.     [SerializeField]
4.     public BClass BReference;
5. }
6.  
7. public class BClass : ScriptableObject
8. {
9.  
10. }

Both AClass and BClass were being requested to load as part of my blob of 2,500 objects (loading by tag where all 2500 entries were tagged). However, I noticed if I only requested to load AClass without BClass, the fields in AClass would still populate correctly, and it reduced the load time by a little bit. For my project, I still need a flat list of all AClasses and BClasses, so I went ahead and made a new scriptable object:

Code (CSharp):

1.  
2. public class StaticDataCatalog : ScriptableObject
3. {
4.     [SerializeField]
5.     public List<ScriptableObject> DataCatalog = new List<ScriptableObject>();
6. }

and wrote an editor tool to populate the DataCatalog with the list of 2,500 entries i need to load. I then bundled this one StaticDataCatalog with the rest of the data in same bundle, and changed the loading code to just load this one StaticDataCatalog asset instead of requesting 2,500 keys. Reading the DataCatalog field gives me the same information as requesting to load every file in the bundle, but reduces the load times down to expected levels as mentioned at the top of this post. I'm not sure exactly whats going on under the hood, but I'm happy I found a solution that works for me. Good luck to everyone else in this thread, I hope you find solutions too!

 

The problem is not related to interdependencies because my repro projects contain just lightweight scriptable objects ( with just one integer field) and loading of 1000 of them took 17 seconds in android.
I think the problem is related to multiple asynchronous operations that occur at the same time (maybe it kind of happens in sequence instead of parallel).
Or possibly the problem is how the asynchronous operation decides how much time it could spend on each frame.
Anyway, it is a deep asynchronous problem that happens both in new addressables and old resource systems.

LilMako17, Your solution works because you now load just one scriptable object asynchronously instead of 2500(maybe they now load at the start of the game synchronously). Also, check asset duplication rules, there is a probability those 2500 assets are now copied somewhere else.

I am glad you found the solution that works for you,
But this is not a proper solution it is just a workaround for the asynchronous problem that still exists and needs to be fixed!

 

we'll definitely look into loading time of large numbers of assets. thanks for the feedback.

 

Thank you.

 

Nothing changed in 1.3.3.

Sync Resources.LoadAll() vs Async Addressables.LoadAllAsync()

Windows (NoteBook Acer Aspire 5 A-517-511G from HDD):
Loading 1000 assets from the Resources folder synchronously takes 0.006 seconds.
Loading of another 1000 assets asynchronously via Addressables takes 0.367 seconds.

That is 60x slower.....
It is still ok because it takes only 0.3 seconds.


Android (Xiaomi Mi Mix):
Loading 1000 assets from the Resources folder synchronously takes 0.33 seconds.
Loading of 1000 another assets asynchronously via Addressables takes 18 seconds,

That is 55x slower.....
And this is definitely not OK because it takes almost 20 seconds.....

Android vs Windows:
Loading on Android is 50x slower than on Windows (18 /0.367).

((50 - 60)x seems like magic consistent pattern's number)

 

Is there any news about this?

 

We have same problem with AssetBundle.LoadFromFile.
500 bundles loaded from streaming assets takes
on PC 0.6 seconds
on Android ranging from 12 to 24 seconds depending on device(powerful - slower).
Since Addressables work by loading bundles from streaming assets i think its the same problem.

 

Addressables uses asset bundles under the hood, so you should compare it against the different ways of loading bundles to get a better idea of what is happening.

 

Any news on this? I am seeing crazy delays with InitializeAsync and LoadAsync on Xaomi phones. It is killing our users in certain countries where that phone is ubiquitous.

 

Only the bad news. Below is response:


Re: (Case 1189850) [Bug] Addressable assets loading takes 50x more time than it should in virtual play and Android


Jan 14, 2020, 7:01 PM


to me


Hi,

We really appreciate your contribution to the Unity product.
Thank you for reporting this issue. We take every case submitted to us seriously by investigating the impact on you, our customer, as well as the impact it may have on the engineering and experience of our product.

This particular case has been investigated thoroughly and we have decided, in the interest of protecting the stability and features of Unity for users that rely upon the affected versions, to not address this fix for the time being. We understand that this will cause problems for some users, and so may address in a future version.

Thanks,

 

1400 gameObject Prefabs with texture and mesh dependencies

PC standalone 8 seconds
Switch installed package 215 seconds

using 1400 calls to AssetReference.LoadAsssetAsync<GameObject>

then 1400 calls to isDone in a coroutine spinning away

right now I regret my time wasted on addressables.

 

That is not only Xiaomi. I tested it on 5 different phones and results are really bad.

I personally can never understand how something that makes things load 50x slower than it is possible can be backlogged...

 

You are not alone ))

 

Actually, I would not accept that answer and at the very least ask for a hell of a lot more information. Also, ping Bill and Ryan for any comments. This is unfortunately a typical response from the QA team not understanding that the issue makes an entire system entirely unusable for your use case, as opposed to "causing problems". Digging deeper at to what is happening here is definitely something to do. Where is the delay coming from?

At the very least they should come up with any possible workarounds. Different compression perhaps?

 

That's scary. Please list the devices so I can maybe test my game on them.

Edit: I see you listed:
Galaxy A 30 - 4 seconds;
Huawei Honor 7C - 6 seconds;
Xiaomy mi mix 17 seconds

Any others?

 

Bill has actually replied in this thread before and said :

I personally think this issue is coming from asynchronous load system not from addressable system itself (because Resources.LoadAsynchronously also produces significant delay comparing to sync Resources.Load)( And if so, I thing it exists for a very long already). I do not know who is responsible for asynchronous load system.

If unity ignores replies like this and decides to backlog such scary issues what else can I do. I already spent some time on the addressables and this particular issue it would be no good to waste more.

 

I do not remember. Maybe Pocketbook surf pad 4. There where about 6 of them. But all have big slowdowns.

You can download the project and test it on devices you have yourself.

I actually did not test it for quite some time. Maybe something has changed.

 

We are noticing long load times as well, any more updates on this?

 

UT created a survey to understand what community needs. THIS is what we need, for the critical issues like this to be fixed in a reasonable time frame! And for QA team to be more cooperative.

 

bump

 

upvote +1