The problem with Addressables and the new AssetBundles system

Hi, guys
First of all I would like to appreciate the whole job with asset bundles during these years, many things have been improved a lot. However I think many useful things are getting lost from the asset bundles legacy system.

When I talk about this legacy system I am referring to the very old legacy functions that even though are marked as obsolete are still working today. We have been working with this legacy option for 5 years so far because they were added into one of our projects (still being updated) long ago when it was the only option available. The system had a lot of issues but we were able to create a workflow where dealing with all this was almost transparent

Recently we have been taking a look into the new system, and more specifically Addressables and even though all the improvements are really welcome we are missing a lot of features... I'd like to show you what we have been doing first so you can understand our problem easier.

1. Asset bundle Creation
We created all our builds using one script called "CustomBuilds". This script manages a lot of things for properly delivering the different builds we needed. It is also the script responsible of the creation of our Asset bundles using BuildPipeline.PushAssetDependencies, BuildPipeline.PopAssetDependencies, BuildPipeline.BuildAssetBundle and BuildPipeline.BuildPlayer. So at the end of our build process we have a code similar to this

BuildPipeline.PushAssetDependencies();

foreach (string assetBundle in BundlesList) {
BuildPipeline.BuildAssetBundle(...);
}
BuildPipeline.BuildPlayer(...);

BuildPipeline.PopAssetDependencies();

Basically, every call to BuildAssetBundle was putting some assets into the specific asset bundle but also was removing them from the list of assets to include in the next asset bundles and the final build. You could create dependencies between assetbundles this way. There were already plugins to manage these dependencies and also to explore and create the asset bundles (we were using one of these available plugins on the store to manage the assetBundles content but in the end we were creating them ourselves).

AssetBundles dependencies are not possible now with the new system. If you add an asset to a bundle and also to the build then this asset is duplicated. There is no way to create a "reference" to an asset included in an asset bundle, imagine a material not included in any bundle referencing a texture inside a bundle for instance. You might think this is a good thing since you remove this annoying dependence but this also breaks what we consider the most useful part of the bundle system (see below)

2. Asset bundle loading
There were two ways to load assets from a bundle: direct access and indirect access.
- In direct access we called AssetBundle.LoadAsset similar to when you call Resources.Load. This function is not marked as obsolete (yet) so we can still use it. There is also AssetBundle.LoadAssetAssynch that we never used basically because we never needed it and also because it complicated things a lot. Interestingly this is the approach the new Addresables system is taking: assynch loading. You may have already guessed by the posts of other people in the forums that this is not a good idea, we need non-assynch functions
- Indirect access is more interesting. This way of using asset bundles was poorly documented which I think is the main reason why most people were not using it (and probably didn't even know about its existance). At some point of our application we just opened the bundles calling WWW.LoadFromCacheOrDownload (note this is marked as obsolete) from a local uri usually pointing to StreamingAssets folder or Application.persistentDataPath where we had previously downloaded them. After this call all asset dependencies were magically resolved and we didn't have to change a single line of code. In the previous example where there is a material on the build referencing a texture on an AssetBundle if we used this material after opening the bundle it was working. No extra lines of code loading the texture from the bundle was needed. Also, we never closed any of the opened bundles because the memory impact was minimum (opening the bundle was not loading all its content into memory) and asset references were managed the same way any other asset not included in bundles was, no extra ref count were needed (another big problem of the new system)

Indirect access required to work with some limitations on the asset bundles. The kind of stuff we put on a bundle were basically textures, sounds and FBXs. We never included any scripts or shaders to avoid annoying issues that were very known back in the day. For this same reason we were never including prefabs or materials because they could potentially lead to the inclusion of scripts or shaders.
But the main benefit of indirect access was that we didn't need to change any line of code when moving stuff to bundles. This was important for us since we included Assetbundles after being working in the project for a couple of years (imagine changing all our loading stuff to assynch calls). We needed to open the bundles before using any resource from them and we had to download everything manually but that was ok. The approach now seems to make this transparent but this was never a problem for us. We clearly prefer to manage the download of assets (because we know the app and we know when we are going to need to do it) than having to move all our loading stuff to an assynch approach, besides it is very common to move to a loading screen where all needed assets bundles are being loaded in the background. It is impossible to make this transparent

Not sure if I have properly explained it but, our biggest concern now that we are starting a new project is that we are losing a lot of benefits we had and we are getting a lot of issues that we never had to deal with.

So now that I have explained our workflow, here are the things I wanted to ask:
- What is the main reason to deprecate WWW.LoadFromCacheOrDownload? Why not fix all the issues it had instead.
- Why are you guys deprecating assetbundles dependencies? I understand they create some problems but let us deal with them, the benefits are higher

 

- UnityWebRequest completely replaces WWW (see https://docs.unity3d.com/ScriptReference/Networking.UnityWebRequestAssetBundle.GetAssetBundle.html)
- dependencies within asset bundles are now handled automatically (e.g. if an asset in A references an asset in B, then asset bundle A has a dependency on asset bundle B). this enables to give a proper error when you load things wrongly, instead of missing references. Addressables will manage correctly loading asset bundles with dependencies.

I don't want to make a build, have half of my assets missing, and I don't know why, if I loaded a scene before loading the bundles...

also with your workflow (having direct references to stuff in bundles) you are still forced to have everything downloaded and loaded in memory when loading the scene, that does not provide any benefit vs having them in the build.

 

Yes, you are right. But I was talking about references with the build itself, not between asset bundles. When you use an asset in a build (contained in a scene) and also in any asset bundle, the asset is duplicated. That's very different to how it was, and it forces you to change the code when moving stuff to bundles

Neither do I, but that is going to happen to you anyway both with the current system and the legacy one. Now with addressables you have a worst scenario where you load an asset from a bundle and that can take minutes to get downladed forcing you to pause your game execution meanwhile

No, as I said assets are not loaded into memory when you open a bundle, we checked this with the profiler and have been using this on several games. Also the main benefit of not including assets in the build is reducing the build size

 

but you still need to download them, and also you lose the automatic stripping that normal builds have.

example:
- you have a scene in the build with objects that reference a material that references a texture. you have 3 other unused material in the project.
- you build the game. Unity pulls the scene from build setting, and the material and texture that you used.
- you change stuff and rebuild. Unity naturally tracks the change and your build has the new mat/tex and not the old ones.
now you use AB your way:
- your scene has a hard reference to material, but material and texture are in AB
- you either have an AB with all your assets (so you have a game effectively 3 times bigger) or you need to manually sync the AB build with the game build.
- if you load the scene with the wrong AB, you simply don't see the object (or you get a missing/null reference somewhere without explanations)

also, you mentioned in your post that you only put meshes and textures in the bundles, because of limitations of old system.
we have put basically everything in AB with the new system: prefabs, materials, sprites, scriptable objects (with references to other assets) everything works, and we manage duplication using the AssetBundle Graph Tool (and we have our own AssetBundleManager based on the unity one, customized for our apps) I plan to use Addressables when it will become stable and feature-parity.

I think your use case of having working hard references to stuff in bundles was undocumented / undefined behaviour, so you were basically hacking with Unity internals.

 

Of course I need to download the asset bundles, how is it going to work otherwise?

This is what happens with our system in the scenario you are suggesting:
- We create the AB with the Texture (we never included materials), this is nothing different from the regular workflow. You have an AB with a Texture wether it is used or not in the final build
- If you build without using the material and the Texture (which doesn't make sense really) you will never use it and nothing happens
- If you create a new build using the scene with the material, the AB content hasn't changed so you can still use the previous one (there is an option that has to be enabled to ensure internal ids doesn't change between builds that has to be enabled)
- Now you need to open the AB before loading the Scene (this doesn't load the texture into memory) where the material is used and because it is treated as a dependancy it is loaded when the material is loaded (this is exactly what happens now with dependencies between ABs, first you need to open the bundle where the dependency is)

The problem with prefabs and shaders could have been fixed instead of creating a whole new system (I am pretty sure in this new system scripts are not included in the ABs because that doesn't make sense)

And this way of working was perfectly documented. As I have said already the trick was to include dependecies between the build and the AssetBundles. It is as risky and undefined as doing it with the AssetBundles with the current system. I have used the functions that were available back then, never have to hack anything

And again, the main benefit was that we never had to change a single line of code to move stuff to Asset Bundles.

 

you can do this with the new system:
- move EVERYTHING into asset bundles
- put only a single script in the build that loads the first scene from Ab, and nothing else
- direct references between bundles are correctly treated as dependencies, just as before

scripts are never included in anything, they are compiled into a DLL and that is included in the build (you can include DLL in asset bundle, then manually load it in the mono runtime from its

byte[]

contents. it's not allowed on iOS (and probably other platforms) though)

 

We thought of that approach, but all calls to Resources.Load need to be changed to AssenBundle.Load and... well, putting everything into an AssetBundle sounds a bit scary. I am actually wondering now how the .obb file in Android is working which is supposed to do something similar to what you are suggesting (so probably the functionallity is never going to be deprecated although marked as obsolete...)

Thanks for your answers, still I would like someone from Unity staff to take a look into all this since I think a lot of changes that are being implemented could benefit a lot from the old system

PS: scripts were added into legacy ABs, it was something we had to be very careful with . That's what I was referring to

 

this is (almost) exactly what Addressables would provide (except it's async): when you first move from direct references (i.e. no resource management at all) to something else, instead of Resources.Load, you call Addressables.Load, then configure Addressables to point at Resources.
then you can switch from "resources" to asset bundles (local or remote) you don't change any code because everything is taken care of.

btw you can still keep resources into Resources/, then your built game will contain: code, Resources and the bootstrap scene. your bundles will contain everything else.

that's nasty. do you mean the literal script file was included as-is and not compiled? or it was still compiled but a copy of the source was put in the AB? and it was breaking references to said script?

ps one last question about your usage (I am curious): let's simplify saying you have only one scene, and one bundle with textures/meshes.
is the behaviour like this:
- you open the game, scene is loaded, you don't see anything because no textures/meshes are there
- you called WWW.LoadFromCacheOrDownload("bundle")
- when the download completed and without reloading the scene, now all meshes and textures became loaded and visible
or you do still need to (re)load the scene after loading the bundle?
and, that was the intended and documented behaviour?

in the first case, then I agree that the legacy system was useful (free streaming of content, albeit magic), and you should post a feature request in the Addressables forum to restore it (at least partially).
if not, then it is equivalent to my setup of everything into AB

 

References didn't work with scripts, so they were still included in the build (thank god). But as you see here there was a way to compile them in real time... Our main concern was that since the code changes constantly with new updates we weren't sure how that was going to affect a full prefab included in an assetbundle being loaded in later builds.

For what I see shaders can still be included... we didn't like this either. Imagine you create an assetbundle including shaders, then a new version of unity appears where that shader doesn't compile anymore. If you open that assetbundle the shader will not work so you need to update the bundle with the shader fixed, but then the older version of your app will see a new version of this bundle and will update it too... and it will not work for them. Now you need a system where depending on the app version you need to get a different version of the bundle too making things more complicated

No, that didn't work. You needed to be careful and never use any references to stuff contained into assetbundles before loading the bundle or all these references never worked again, even if later you loaded the bundle and reloaded the scene. That was one of the most annoying parts... that we think could have been fixed

 

ok. that's what I thought (the last part). if so, then I think the new flow is better, as it forces you to load the dependencies (and you can still have references by putting everything into bundles)

regarding the shaders:
1) not only shaders can be incompatible between version, but all asset types (including your own scripts if you change the serialized fields)
2) in any case, when you upgrade Unity (or otherwise make the assets incompatible) you should put the new bundles in a new location, so that old clients still see the old bundles and updated clients see the updated ones. we handle this (hosted on s3) by creating folders V1, V2, etc... and changing the base url hardcoded in the client. Addressables also handle this through the content catalog (but I still need to study how it works in the details. mostly waiting for 1.0 / out of preview / docs available though)

regarding the scripts: they are compatible as long as their serialized data (i.e. public and

SerializeField

fields) is not changed in conflicting ways (e.g. changing a field type).
you can:
- add new fields (they get the default value if it's not present in the serialized data)
- remove fields (they will get ignored)
- rename fields, by attaching the

FormerlySerializedAs

attribute)
- rename the class/script file, as long as the GUID and FileID of the script does not change (i.e you do it inside Unity)
-- (FileID of classes in assemblies are derived from their full name, so you can't rename those)
be sure to include the TypeTree in the bundle, as that is what contains the information needed to deserialize the scripts. (there are some blog posts "Serialization in Unity" that explain that in details)

 

Thanks for the well thought out feedback. It looks like some of your concerns have been discussed between yourself and @M_R. I’ll try to speak to each of your outstanding concerns, and in addition, I have some notes on what you’re doing today that could be helpful

I’d like to make two comments on this workflow. For one, I see you are managing dependencies yourself. I’m not sure how you’re doing it, but I want to make sure you are aware that using the AssetDatabase interfaces to do that can provide false positives. Obviously this isn’t the purpose of your post, but in case this tip is helpful, I thought I’d pass it along. For example, if you set the Metallic texture on a material, then changed the shader to one that didn’t take a Metallic texture, the editor keeps up with that now-invalid dependency. AssetDatabase calls will still give you that texture, but the actual build will strip it. There are a handful of ways to manage dependencies, so if you aren't using this method, disregard.


Second, you are using this methodology to allow for direct dependencies between the player and bundles. Moving forward, that is on our list to support through the Scriptable Build Pipeline (SBP). Currently that package can only build bundles, but we are working towards adding player-build support. At that point, cross dependencies should be doable. That being said, I would recommend not actually doing that. As M_R states in this thread, you can “move EVERYTHING into asset bundles” to get the same behavior. Your response was that sounds “scary”. That solution scares me *far* less than one where you have to load things in a particular order or it just fails. Keep in mind you can put bundles in StreamingAssets if you don’t want them to be downloaded.


In discussing M_R’s suggestion, you also lament having to move things from Resources to Asset Bundles. I’d give this a read: https://blogs.unity3d.com/2017/04/12/asset-bundles-vs-resources-a-memory-showdown/ as it discusses some of the other advantages of moving out of Resources.

Indirect Loading.

I already hit on the player->assetBundle dependency aspect of this above, so I just want to mention the memory implications. You said you never unload these bundles because the memory impact is small. This is true for the bundle itself, but once something in a bundle is loaded, it can not be removed from memory except by unloading the entire bundle, or by calling Resources.UnloadUnusedAssets. I’m guessing that means you are either calling UnloadUnusedAssets explicitly, or you are opening scenes in Single mode, which calls it under the hood. That is a very expensive call performance-wise, and we generally recommend users not do that, or only do it during loading screens where a hitch won’t be evident. The better option for performance is to keep up with what you’ve needed in each bundle, and unload the bundle when you can. Addressables does this for the user.


A philosophical digression.

One thing I wanted to discuss was that one of the key purposes of Addressables is exactly what you mention for your system… “no need to change a line of code”. The intent is that Addressables a simple user experience such that users would use it in their project from day one. Even if everything is local, or even in Resources. Then, as they need to move things around as their game grows, they change no code. It’s worth noting, you do change some of your code. If you had all things local, directly referenced by your scene, then changed to having some content in remote bundles, you now need to add a bunch of build/download/load/prepare code. With addressalbes, you don’t do any of that. You literally change nothing if you move an asset from local to remote.


Asynchronous interfaces…

Which brings me to your concern over everything being asynchronous. The main reason is that it enables the game code to not care where the thing is it’s asking for. Loading or instantiating can happen regardless of the need for a download. We also feel it is a better design choice, but would point out, that you don’t have to do all things asynchronously even if using our system. The main example being that you could asynchronously load your assets (perhaps during a loading screen, which you mention you already have), then keep a hold of all the loaded assets. Those can be directly instantiated using Unity’s current APIs.


I hope this sheds some light on the addressables system for you. I think in many ways it aligns with your desires, and in some it does not. For those it does not, our intent is to continue to provide you with options and opportunities to inject your own code to manage things the way you wish.


Thanks for the feedback,

-Bill

 

Hi Bill, thanks for the answers, really appreciated.

To be honest we are not sure either, because Unity takes care of everything under the hood for us. We can guess what is going on and why is working: the assetdatabase dictionary that is created on launch can be expanded later. When an asset bundle is loaded new entries are added referencing the resources included in them. Then if any asset has a dependency it is atomatically resolved because the dictionary contains this new entries... something like this?

We are not loading things on a specific order. We just need to open the bundles before loading any content from them. And that's the only change needed. Even with the current Assetbundles system you need to do that, and the same thing happens with Addresables (I know it is thought to make things transparent but in a real scenario you can't start a game without having your main character loaded, you need to know before hand that you need it, then load it and then load the scene)

In the meantime we are always taking a look into https://docs.unity3d.com/2019.1/Documentation/ScriptReference/BuildPipeline.BuildAssetBundle.html to ensure it hasn't been removed from the API. It's been marked as obsolete for a while se we are a little bit scared

Resources.UnloadUnusedAssets is a function that we unity users tend to use a lot. We are aware of its expensiveness but there are certain situations were you just have to use it. Sometimes you can wait until the loading screen appears and then call it, but there are some other times where you need to do it manually or your app will crash (imagine a photo album where you are loading big images from a bundle). Uploading the bundle and loading it again also has a hiccup so not sure why that is a better option (besides there is the risk of having duplicated resources on memory)

Ok, I understand your point of view, but from our side, the people using it I can ensure this is not true. It is true that once you have everything working with Addresables you don't need to change anything if you decide to move stuff to bundles or remove it from them. But the problem is that first step: moving everything to Addresables. And it is not a small problem. In big projects, those that are still being maintained after a couple of years doing this is a nightmare... You are creating a system that from the start of the project makes you think everything is in Addresables, but that's not how unity works. Resources.Load is used a lot and changing it is not trivial
When I say "No need to change a line of code", I literally mean it. With our old system we didn't need to know if your stuff was in a bundle or not. It was completelly transparent and this new system is not

Again this makes you change a lot of things because now you need to know before loading your scene which stuff is going to be used. In the common scenario it is the scene itself during its Awake or Start methods the one loading all this, you don't need to plan ahead.

With this post I just wanted to show you guys how we managed to introduce AssetBundles in less than a week to some of our projects that had been in development for a long time. We were very happy with our solution (it has been working without any issues for 4 years so far) and now we are going to lose it. We are aware of the problems it had but this new system is not problem free either and the changes in the code for us are way riskier than before

 

Since the intent for addressables is to be able to have code that works the same whether the content is remote or local (which is great for those that want this), when are you going to introduce a system whose intent is to actually replace Resources.Load?

 

Hello, I wonder how can I get all contents from a specific folder, I mean I have 1 folder marked as addressable and I can access all data by path but I'd like to load i.e all sprites from a sub folder like folder1/sprites/ <- get all list of sprites instead of creating a doc where I put all addressables list.

Thanks in advance.

 

Thanks for the quick response, I understand, but what if I have 3 bundles and each one has this label, isn't going to download 3 of them because label is included ? could it be separated by merge?
Another thing is that by adding a label to the content folder all subfolders are marked by this label, but we have a different label for each kind of content i.e. models3d or stickers we get GameObjets in one and Sprites in the other. if this is the case then we have to split our bundles much more than required.

Our main goal is:
1- download bundles individually with the folder key and only the content inside of it (DownloadDependencies solves that I think but I'm not sure if the parent folder/container ID marked as addressable would work to download it)
2- Getting the different kind of assets of that bundle/sub folders (In the case of label all sub folders are marked with the same label, we want to reuse the same label i.e sticker but without downloading many bundles with the same label)

Thanks in advance and thanks for your time.
I really appreciate your help.

Best regards.

 

Resources.Load is synchronous local load.

Addressables is (as I understand it) an asynchronous API that is remote/local agnostic.

As such, migrating to addressables means you have to write more code and restructure things to account for it being async. And you are doing that not because it is something you want, but because addressables was specifically designed to fill a different need.

Or in meme-speak

Thanks, I hate it.

 

Thank you, we added that option, those features would be very helpful.
Best regards.

 

Hello again,
Do you guys know if the property is working properly on the DowloadDependeciesAsync() method, it always returns 0 on IEnumerator but only shows 1 when completed event is executed or if I yield the async method.

I'm using the 0.8.4 Addressables version, packed play mode, hosting working properly, download size also working. and teting on the editor, Unity version 2018.3.14

Code (CSharp):

1.  IEnumerator AcceptDownload(string BundleTag)
2.         {
3.             var async = Addressables.DownloadDependenciesAsync(BundleTag);
4.             while (!async.IsDone)
5.             {
6.  
7.                 Debug.Log($"donwloading {async.PercentComplete}");
8.                 yield return null;
9.  
10.             }
11.             //DownloadBundleAddressable(BundleTag);
12.  
13.  
14.         }

Hope you can help me with this, i`d like to show the progress to the user.
Best regards.

 

Another Question, do I have to Include all dependencies for a certain prefab or does the addressable system includes them automatically into the bundle?
i.e. I created a prefab, that prefab is from a 3d model with texture, material, other settings, etc.. and I just put that prefab inside an addressables group

 

We have seen this bug as well. (It seems that the internal class ChainOperation should override PercentComplete, but it does not.) We created this workaround:

Code (CSharp):

1.         private static float calculatePercentComplete(AsyncOperationHandle asyncOperation)
2.         {
3.             var deps = new List<AsyncOperationHandle>();
4.             asyncOperation.GetDependencies(deps); // deps is added to! (weird API...)
5.             float percentCompleteSum = 0;
6.             foreach (var asyncOperationHandle in deps)
7.             {
8.                 percentCompleteSum += asyncOperationHandle.PercentComplete;
9.             }
10.  
11.             return percentCompleteSum / deps.Count;
12.         }
13.  

PS: You should probably start a new thread for new questions.

 

I have been limited by Unity to an under 4GB file size for building my game. So, I have been forced to now use asset bundles. I was wondering sort of what the original poster was doing, how can I package all of my "Resources" from my resources folder (which I was told to use in legacy versions, but apparently am now told not to), and then load all of the resources I need at the beginning of my game (say before or during my "starting screen" scene). And then have ALL of the resources I would have normally had available in a regular (non-assetbundle) build?

In other words, I want to load all of my stuff at the beginning of my game once, then have it available and working for every scene in my game. But now I have to do it using the asset bundle system. How do I do this?

 

No that is not right!

Here I described why.
https://forum.unity.com/threads/instant-synchronous-load-methods.724928/

In short.
Your asynchronous asset system is not a straightforward replacement to the previous system it requires to make significant changes to previously synchronical code, there you can use things like Resource.Load("MyObject").DoSomethingImmediatly().

 

I commented on that thread directly.

 

We have Addressables somewhat working but need a good use cases for reducing the load time of instantiating assets from asset bundles to the minimum time necessary. Right now to instantiate a single asset from a loaded asset bundle is taking 4-7 seconds. Not OK.
Compression is turned off in all cases. These asset bundles reside on the Hard drive not on the Web. Our asset bundles consist of FBX's which contain meshes and textures. Eash FBX is roughly 20MB in size.
Also we tried Application.backgroundLoadingPriority = ThreadPriority.High; to speed up this process and this appears to do absolutely nothing! Is Unity really setting thread priority?

 

I know this post is a little old, but I think the foremost question is:
I have a project in 5.6 that a client need to be refactored to new releases. This project have a LOT of AssetBundles created in the original version.
It´s possible read those AssetBundles using Addressables or not?
PS: Those assets are in a directory within the executable, like "/AssetBundles".