Addressable Load Performance

I have been experiencing some longer than expected load times on ios.
I have found several other threads on the same issues.

My addressables use case is the simplest. The addressable bundles are packed with the build locally.
No downloads.

I have setup some test to look at different ways of loading.
Initially on my PC. This is a high end PC, M2 SSD, i9, 3080. 64gb ram.
Unity 2021.1.25f1, addressable 1.18.19

It’s a simple test.
1. Load scene (additive) with model prefab in it.
2. Load scene (additive) async with model prefab in it.
3. Load addressable of the same model prefab as in the scenes.

Here are the results.
~250 ms Sync Scene
~259.05 ms Async Scene
~977 ms Addressable
I ran this a few times in a windows Il2PP build.

My earlier tests were showing the a huge difference in the sync and non sync.
But when I added a frame delay they are now coming about about the same.
I think when the sync loadscene call finishes unity has not actually loaded the assets to render them yet.
And the prefab has some other scripts on it that could be doing work on the following frame.
This was really noticeable on my iphone test. Sync showed a few ms to load but then a bit later model appeared. Adding the frame to all tests makes the time reported and model appearance match up.

The extra frame increased the sync loadscene test but had no real affect on the others.

When I deployed this same setup to iphone12 and a iphone 7+.
The 12 sync loadscene is about 2x as fast about 250ms vs 500ms.
But the async scene and the addressable are in the same range.
About 5s and about 8s. +- .5s

I was a little surprised by this. But maybe while the CPUs are faster the flash read times maybe similar.
But in either case the async is longer and the addressable is considerably longer.

I was thinking about possibly using Resources.LoadAsync and testing or possibly asset bundles.
But I thought Resources.Load was deprecated even though I think the LoadAsync maybe newer.
And I thought addressables loaded the asset bundle under the hood anyways so not sure I will get an speed up there.

Unless the slowdown comes from the async aspect and i could load the asset bundle synchronously.
I may try it see what happens.
Otherwise for now i think i am stuck just loading everything and hoping to not run out of memory.
Initially that will be ok. But as our game grows this may become a problem to let you select from lots of characters and having to have them all in memory.

I can probably submit a bug with a paired down project if that would be useful.

The code for the test is as follows if anyone wants to try it.

Code (CSharp):

1. public class TestController : MonoBehaviour
2. {
3.     [SerializeField]
4.     private Button _LoadSceneSync;
5.     [SerializeField]
6.     private Button _LoadSceneAsync;
7.  
8.     [SerializeField]
9.     private Button _LoadAddressable;
10.  
11.     [SerializeField]
12.     private AssetReferenceGameObject _Fighter;
13.  
14.     [SerializeField]
15.     private TMPro.TextMeshProUGUI _Text;
16.  
17.     // Start is called before the first frame update
18.     IEnumerator Start()
19.     {
20.         _LoadAddressable.onClick.AddListener(LoadAddressable);
21.         _LoadSceneSync.onClick.AddListener(LoadSceneSync);
22.         _LoadSceneAsync.onClick.AddListener(LoadSceneASync);
23.  
24.         AsyncOperationHandle<IResourceLocator> handle = Addressables.InitializeAsync();
25.         yield return handle;
26.     }
27.  
28.     private void LoadSceneSync()
29.     {
30.         IEnumerator LoadSceneCo()
31.         {
32.             System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
33.             sw.Start();
34.  
35.             UnityEngine.SceneManagement.SceneManager.LoadScene("FighterScene", UnityEngine.SceneManagement.LoadSceneMode.Additive);
36.             yield return 0;
37.  
38.             sw.Stop();
39.  
40.             _Text.text = string.Format("Load Scene Sync: {0}", sw.Elapsed.TotalMilliseconds);
41.             Debug.Log(_Text.text);
42.         }
43.  
44.         StartCoroutine(LoadSceneCo());
45.     }
46.  
47.     private void LoadSceneASync()
48.     {
49.         IEnumerator LoadSceneCo()
50.         {
51.             System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
52.             sw.Start();
53.  
54.             yield return UnityEngine.SceneManagement.SceneManager.LoadSceneAsync("FighterScene", UnityEngine.SceneManagement.LoadSceneMode.Additive);
55.             yield return 0;
56.  
57.             sw.Stop();
58.  
59.             _Text.text = string.Format("Load Scene Async: {0}", sw.Elapsed.TotalMilliseconds);
60.             Debug.Log(_Text.text);
61.         }
62.  
63.         StartCoroutine(LoadSceneCo());
64.     }
65.  
66.     private void LoadAddressable()
67.     {
68.         IEnumerator LoadSceneCo()
69.         {
70.             System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
71.             sw.Start();
72.  
73.             yield return _Fighter.InstantiateAsync();
74.             yield return 0;
75.  
76.             sw.Stop();
77.  
78.             _Text.text = string.Format("Load Addressable: {0}", sw.Elapsed.TotalMilliseconds);
79.             Debug.Log(_Text.text);
80.         }
81.  
82.         StartCoroutine(LoadSceneCo());
83.     }

 

I have a similar issue, where loading a scene with addressable take a huge amount of time, and in some situations, the same operation will not take the same amount of time (a scene can take 2 times much time to load for no reason). None of the threads in the forum I found had a solution from an Unity tech. So the following is just my assumptions.

I think that all async operation are managed behind the scenes by the PreloadManager. This Preload manager seems to decide from itself the amount of time it will allocate, per frame, to load a resource. It seems to also decide or not if it will resume the loading started to the next frame or not.
Unity give us Application.backgroundLoadingPriority to modify the priority of the background thread, which should be the parameter we need to use to force async operation to operate, but none of that had effect during my tests. Also, during my tests, I was able to see in the profiler clear situations were the PreloadManager was doing nothing during a certain amount of frame, which mean they are other parameters that dictate its behaviour.



You can see in the profiler, that after starting the preload "Application. Preload Assets" phase, the PreloadManager do nothing during for the rest of the frame. This is also visible when you get the PercentComplete value of an AsyncOperationHandle. The value stays blocked for a long amount of time.

My personal concern about that is that we do not have any way/ideas of what is going on internally. As I said, none of the Unity dev answers any of the thread related to that. At this point, not relying on async operation seems to be better than using it. I am working on a project 1.16.16, so I do not have any choice right now. Could it be possible to have a dev talking to us about that ? Are they aware of that ? Is there anyway to modify the Addressables package to control in a better way the async behaviour ? @TreyK-47 @davidla_unity

 

That's interesting I have seen other posts mentioning the backgroundLoadingPriority and it not having any affect. My guess that is for other systems since its not in the addressable namespace.

I don't have time to dig in and try and profile and debug unity. Right now for my next few builds my character count is low enough i can just load them all

Initial startup time is not noticeable loading all 6 characters. At the beginning by having them in my initial loading scene.
Now switching between characters in the menu is pretty much instantaneous.

My only issue with preloading everything and my main reason for addressables was memory management. This wont work moving forward when adding more characters.Since this is cross platform on mobile.

I believe the addressable system is ultimately producing asset bundles.
And poking around it uses the AssetBundle apis to load them. Maybe at some point I will try loading them directly async or snyc and see if performance is better. I would rather have a 1 sec pause than a 8s spinner.

My other attempt was to try and keep a cache of most accessed characters for the user and preload those on startup. This still will cause long load times when choosing one of the non cached characters. Adds complexity and still not a solution to speed up loading. At this point I really think it would be faster to load a new scene every time a character is chosen.

My test about loading the addressable character is probably a little skewed because this loads a bunch of common stuff that may not all be used because of the deduplication analysis. But even so switching characters take way to long and should not have to be loading all that common stuff.

 

Obviously unity and the hardware is capable of loading the data I need quite fast. It would be great to have a way to control that. One point I could say 100% another place use 10%. I think even a few FPS would be enough for a user to not feel the game has frozen, spinner move, progress bar change.

If it would save loading time and give the much need control over memory usage.

 

Yes, that would be cool. In the meantime, we are thinking of not using the an addressables scene asset for our specific problem, until Unity give us more info on the nature of the issue.

 

I was talking to a coworker today. Unless the addressable system is doing something really inefficient. I believe ultimately it just calls AssetBundle.LoadAsync. So that is probably where the slowness is.

I feel like this was designed or tested with the idea to use this to page in levels or other data while the game is running and tuned for that. Not to load a bunch of stuff behind a load screen where you want it to use be fast as possible. Which is why there does not seem to be any controls for the amount of CPU time being given. (A guess)

I have moved away from using addressables for the moment. And may try loading the asset bundles directly using AssetBundle API load functions and profile them. But for my current beta build I can fit what I need into memory just barely. My worry is I will see the same slow performance because my understanding it Addressables just wraps and makes easier the AssetBundle system.

 

Have you played with

AssetBundle.memoryBudgetKB

and

Application.backgroundLoadingPriority

to see if tweaking these settings improve performance with your project?

https://docs.unity3d.com/ScriptReference/AssetBundle-memoryBudgetKB.html

This setting basically provides a trade-off between smooth-loading and faster-loading by accepting some choppiness. The higher the priority, the faster the load should perform.
https://docs.unity3d.com/ScriptReference/Application-backgroundLoadingPriority.html

Addressables also has the

Contiguous Bundles

setting, which according to the documentation "Produces a more efficient bundle layout" which sounds like it would improve loading performance.
https://docs.unity3d.com/Packages/com.unity.addressables@1.19/manual/AddressableAssetSettings.html

According to the documentation, the AssetBundle compression affects loading performance as well. Make sure to use the appropriate compression for your content for optimal performance:
https://docs.unity3d.com/Packages/com.unity.addressables@1.19/manual/AddressablesFAQ.html

 

Thanks @Peter77

No I have not tried the

AssetBundle.memoryBudgetKB

First time I have seen this mentioned relating to the addressable system. I will try to find some time to see if it may help.
Is there anywhere to check if this is an issue? Profiler or something indicating its out of memory?

I have not tried changing

Application.backgroundLoadingPriority

. This has been mentioned many times on the forums and this thread and it has never seemed to have any effect related to addressable loading speed. I will try it when I look at the memoryBudgetKB.

This is a bigger issue than just me as there are many posts about performance of the system.

It would be great if these actually help and if they do I will go and update every forum post.
But I have a feeling they will not help.

But good to have things to try at least.

 

So went ahead and decided to test these new things.

Continuous Bundles was already enabled.
AssetBundle.memoryBudgetKB no noticeable changes
Application.backgroundLoadingPriority no noticeable changes

I went ahead and decided to try loading the bundles directly.

I am loading the bundles built by the addressable system using the following settings.
Continous Bundles, LZ4, CRC=Enabled, Excluding Cached, Pack Togther

I know the bundles needed to be loaded. Its about 330mb of data on disk.
Spread across 7 bundles.

Load Scene = 322 ms
Load Scene Async = 287ms
AssetBundle.LoadFromFile / AssetBundle.LoadAsset / Instantiate = 500 ms
AssetBundle.LoadFromFileAsync / AssetBundle.LoadAssetAsync / Instantiate = 415 ms
Addressables = 1600 ms !!

So luckily for me there seems to be a decent amount of overhead in the addressable system on top of asset bundles.
Loading the asset bundles directly is 3-4x faster (PC).
And happily the Async versions seem to be a little bit faster.

They are a little bit slower than loading the scenes directly but I can live with ~100ms difference.
Looks like I will be only using addressable to build asset bundles and loading them directly.

I will try on mobile as well to see if these results stay the same.

 

Well dang,
I thought I had tried this but looking at my last post I remembered the mention of about CRC checks.
I thought I had turned it off before and tested it. Or maybe I thought since I was loading from streaming assets it wouldn't be used when set to Enabled, Excluding Cached.

Turning it to off, rebuilding the bundles and build.
Now the loading of the addressable is the same as the AssetBundle Async.
Will confirm it as well on mobile.

@BeeNvizzio May want to try that depending on if you are downloading from online sources or not.

 

Almost success.

On iOS iphone7+ the Assetbundle Async and Addressables are now the same.
Unfortunately they still take 6s compared to 8 before.

While the AssetBundle Load Sync function takes 500-700ms.
Scene loading takes around 300-500ms.

So look like still have to abandon Async on mobile and switch to Sync Assetbundles.

 

Well some more success. I decided to take a look at the code for the addressable package to see if I could replace the LoadAsync with synchronous versions. Luckily there are only a few places where it makes calls to AssetBundle.Load....

And actually there is basically a synchronous loading path already there that is used when !UNITY_2021_1_OR_NEWER.
I added some extra defines to enable this path and now I have addressable using sync loading.

I am now seeing the same speeds as my synchronous AssetBundle.Load test
iPhone 7+ = 500-800ms
iPhone 8 = 500-800ms
iPhone 12 Pro Max = 400ms

And on all 3 platforms my AssetBundle.LoadAsync test returns about 5000ms.
Which was the same as the Addressables Async original code without CRC enabled.

I think I can live with these numbers now and not have to go about writing my own loading system.
I hope the unloading and memory usage should work as well.

There were lots of posts about addressable synchronous but I think they were all API issues.
This is still the Async API, just the bundle and catalog loading is Synchronous.
Not looking to change the API, its easy enough to work around if your really need to spin wait.
But the performance I am seeing is huge. And having to modify the package to obtain it is a down side.

@unity_bill If feel like this should be an official option in the addressable package enabled by a #define.

 

I will try that next week! I hope it will also work for me. Thank you @daxiongmao !!!

 

I was finally able to test to disable CRC checks.
Before that, I made a build using SceneManager.LoadScene to compare the loading times difference.
So far, disabling CRC give me similar result than using SceneManager.LoadScene, which improve loading times from a 3s to 1.5s at best in my game.
But I still got an issue where the addressables are stuck for a long time without reason, sometimes. (Problem that I described here:
https://forum.unity.com/threads/addressable-load-performance.1186984/#post-7604068
And also visible here: https://forum.unity.com/threads/addressables-loadsceneasync-seems-to-pause-on-ios.1186000/
and here: https://forum.unity.com/threads/addressables-loadsceneasync-stucked-for-a-lot-of-time.1173983/

Because of those issues, I will stick for now with the SceneManager.LoadScene to load the scene I need. It's a bit sad, but until the package give us better tool to control the way the PreloadManager works, I will probably not be able to fix this problem.

Also, I think Unity should be clearer with the way CRC impact the loading time of addressables.
I found this forum post few days ago (https://forum.unity.com/threads/are-crc-checks-worth-the-insane-overhead.1192090/) and it is not normal that Unity do not have any lines about that in the documentation.

@andymilsom your own comment (https://forum.unity.com/threads/are-crc-checks-worth-the-insane-overhead.1192090/#post-7638295) should be part of the doc.

 

@BeeNvizzio I was only able to get async loading to be the same as scene loading on PC. On iOS it was still many times slower using async vs scene load. It was definitely faster without CRC checks. 5s vs 8s

Depending how you are timing your Scene Load you may actually need to wait one extra frame. The scene load completes before all the assets are loaded. So it originally gave me false results being faster.

While sync loading is much much faster 1s. not as fast as scene but much much closer.

I modified the addressable code to be able to use AssetBundle.Load instead of AssetBundle.LoadAsync. This gives me acceptable load times. Like maybe 1s vs 5s vs 8s. And I still get the benefits of the addressable system to unload and track references and build bundles etc.
But this makes the load freeze the game. But still worth it. Rather freeze everything for 1s then show a 5s spinner.

With a little more work could probably have it be able to do either way.
Async if you know you have time to wait for the load or Sync to load faster.

I can probably put the code modifications up somewhere if you are interested. It turned out to only be a few lines. They actually have It in their code only enabled for earlier versions of unity.

Now I have loading working fast enough I am focusing on optimizing my bundles contents to minimize data not needed for each character. Which should speed up loading more too.

 

It would be wonderful to see your implementation. Could you share them with us?

 

Has Addressable improved? I'm using Addressable and also see the very long loading time when the first asset is loaded. The asset is a 50 KB database file but it takes 5 seconds to finish the Addressable.LoadAssetAssync.

 

I have not tried anything newer at the moment as this code is working currently for us.
Probably will not try updating to the latest until we start another project.

I realized I posted this in a conversation not a thread to someone.
You can check out the code if you like and see if it helps with performance.
The issues were mainly on mobile if i recall. But would have to read back through the posts.


I put it up in a github repo.
https://github.com/ncs-jeremy/addressables_sync
It is from a bit earlier version 1.18.19.

And most of the code was already there in unity. I just added the extra defines to turn it on regardless of version of unity.
I did not commit everything because i think most was just code format changes.
Let me know if it doesn't build.
The changes of interest are in commit "Changed to allow for synchronous loading"

I have not looked into the latest versions if that code is still there.
I had thought I had it done by platform but it could easily be done by changing where ASSETBUNDLE_SYNC is defined.

I think when I benchmarked, the sync loading only helped on mobile. On PC async was as fast or faster.
I think I put the results in my forum posts.