New AssetBundle System Issues and Quirks

We have been using the Unity AssetBundle system since the earliest 4.x versions. Our application runs on iOS, Android, and WebGL. While the new updates found in Unity 5.x provide some incremental advantages and very nice tutorials, there still seem to be a number of issues in real world use.

We'd like to share some of our findings to compare notes with others to see if there are helpful workarounds or whether these issues can be scheduled into future releases of Unity 5.x. These are not meant to be a criticism of all of the hard work by the Unity team but rather an open discussion to help improve the product.

Everyone's feedback is appreciated!

1) Asset Dependencies On Resource Assets
------------------------------------------------------------
Asset dependencies are a very useful feature and the new manifests make things much clearer as to which bundles are dependent on other bundles. But what about dependencies to assets that are in the "Resources" folder.

Since Assets in the Resources folder are already included in the binary, and since Unity 5.x forces the use of BuildAssetBundleOptions.DeterministicAssetBundle, why would multiple copies of these "dependent" Assets be included in each AssetBundle that uses them? Shouldn't Unity be able to use the unique GUIDs to infer that an asset is located in the "Resources" folder and make it available to Assets loaded from an AssetBundle at load time without needing a copy of it inside the AssetBundle?

According to the latest AssetBundle tutorial for Unity 5.2:

Asset dependencies are never lost. Dependent Assets will be added to the AssetBundle automatically along with the selected Asset if that dependent Asset has not been assigned to any AssetBundle when the AssetBundles are built. This is very convenient and prevents the loss of dependent assets. However, this can also cause the duplication of Assets.
We have some resources that we consider "system" resources such as fonts, graphics used for system dialogs, etc. We have found that trying to put these inside an AssetBundle located in StreamAssets and loading them at runtime is exceedingly slow resulting in very slow load times for our app. We need these Assets to be nearly instantaneously available to the binary but would also like to share them with AssetBundles. Instead of having to put these "system" resources into an AssetBundle, we would like for downloaded AssetBundles to be able to reference them in place from the Resources folder without duplicating them into each AssetBundle that references them.

Interestingly, the AssetManifest generated by Unity 5.x does not show these dependent Resource Assets in it's file list or in it's dependency list. Are they actually being included in the AssetBundle without being explicitly listed? They are listed in the Editor log build output as "Used Assets".

2) AssetBundle Caching and Hash Identifiers
------------------------------------------------------------
The new system provides the ability to append an AssetBundle's hash identifier to the generated AssetBundle file name using BuildAssetBundleOptions.AppendHashToAssetBundleName. While this seems like a useful feature, it doesn't seem to work as expected in practice when using WWW.
LoadFromCacheOrDownload.

The problem is that by including the hash in the file name, each change to an AssetBundle will result in a new file name and will be considered unique in regards to its presence in the cache. For example, assume changes are made to assets in an AssetBundle called 'abc' resulting in the creation of 3 different hash values and hence 3 different files (for simplicity the "_xxx" suffixes represent simplified versions of the actual Hash128 values generated by Unity):

abc_123.unity3d
abc_345.unity3d
abc_456.unity3d

Using WWW.LoadFromCacheOrDownload with these 3 different file names will cause all 3 AssetBundles to be loaded into the cache and remain in the cache.

According to the latest LoadFromCacheOrDownload documentation for Unity 5.x:
Cached AssetBundles are uniquely identified solely by the filename and version number; all domain and path information in url is ignored by Caching. Since cached AssetBundles are identified by filename instead of the full URL, you can change the directory from where the asset bundle is downloaded at any time.

To circumvent this issue, we have had to use our own naming convention where the hash value is included as part of the path instead of part of the file name. Because LoadFromCacheOrDownload ignores path information, each of the following AssetBundle files are considered the same (assuming we always pass the same version number into LoadFromCacheOrDownload as well).

/somepath/abc/123/abc.unity3d
/somepath/abc/345/abc.unity3d
/somepath/abc/456/abc.unity3d

Instead of having to remap our AssetBundle files, we would like the LoadFromCacheOrDownload to ignore the hash portion of AssetBundle file names. Perhaps the addition of some specific restricted naming convention could be included such as abc_hash123.unity3d. If Unity made it clear that the inclusion of "_hashxyz" was restricted to AssetBundle hashes, LoadFromCacheOrDownload could be modified to work without file renaming and made the BuildAssetBundleOptions.AppendHashToAssetBundleName more useful.

3) AssetBundle Cache Flushing (Not Busting)
-------------------------------------------------------------
There still exists no way to flush an AssetBundle out of the cache. It's understood that existing AssetBundles can be overwritten with new versions but there should be a way to explicitly remove AssetBundles from the cache altogether and free up precious space on user's cell phones and PCs.

For example, we make use of lots of short-lived DLC like promotions, timed quests, timed store discounts, etc. Once these short-lived DLC items expire, there's no reason for them to persist in the AssetBundle cache but we have no way of explicitly removing them.

This has been a glaring issue across multiple major versions of Unity. Is there a technical reason this can't be easily implemented or just a major oversight?

To put this in perspective, our long time users often have several hundred megabytes of "dead" AssetBundles in their cache. Only a complete re-install of the app (not something we want them to have to hassle with) will flush the these "dead" AssetBundles for good.

Feedback Welcome!

 

To start with, I don't think this is the right forum for this.

1) Unity has been subtly dropping hints that the concept of the Resources directory is going to go away. There are already things that you can do with asset bundles that you can't do with resources (such as variants). Asset bundles going forward is going to be the better way to go. My recommendation would be to put these "system" resources into an uncompressed asset bundle and then keep the asset bundle open. This should be just about as fast as loading it form the resources directory, and you get async loading possible as well!

2) I never really understand the concept of the versioning system that is built into the asset bundles. In order to increment the asset bundle you need to check a manifest of the files that you should download. This is either embedded into the .exe or it's queried from a web location. This manifest might as well contain the version and the URL that you're downloading from. For example: MyAppAssets/1.0/Textures.unity3d

Then when you release an upgrade you point the url to: MyAppAssets/1.0.1/Textures.unit3d using a version 101 to download. This will then replace the file in the cache correctly (asset name is the same) and download the appropriate download from the URL.

3) I agree, having more granular control over the cache would be nice.

 

The main issue with actually making asset bundles depend on things in the resource directory is that you can't guarantee the assets in the resource directory are synced with the current version of the asset bundles. In other words, once you leave the application's build process, you pretty much have to have everything you are dependent on outside of the app as well, otherwise it's possible for your data to get into an invalid state.

On 3 - there's another issue with the cache; the current cache design free's the largest bundle from the cache first when the cache is full. In our current game, the bundle holding the UI is the largest bundle, which would be immediately re-loaded after being flushed out of the cache, only to be immediately flushed again. Meanwhile, all those tiny asset bundles you downloaded last year will stick around forever.

Note, I believe you can clear the entire cache without an app reinstall from the Cache class; but obviously that's not ideal either.

---
One of my main issues with the Asset Bundle system is that it's a nightmare to work with at scale. We have thousands of asset bundles in our game, and growing constantly. Some of my issues include:

- The system requires you to organize your assets twice; once in the project folders, and another for bundles. This gets confusing fast, and people easily make mistakes.
- Far to easy to waste memory/bandwidth. Forget to bundle a texture into it's own bundle? Good chance you'll see it in memory several times. Make a package of textures and stop using one? Now everyone is going to have to download that extra data which is never used. Asset bundles are like a fully manual build process, since you basically have to specify everything by hand. Sure, it will collect the dependancies for you, but if you don't manually manage everything perfectly you'll have issues.
- Variants are not well thought out in terms of the Unity workflow. We have 4 variant levels representing different target performance/memory targets. This means most assets, especially textures and sounds, are in our tree 3 or 4 times so we can set the compression/size options on them for each variant. Duplicating assets like this is terrible for so many reasons, as it's easy for them to get out of sync, makes our texture compression time 4x as long, and is 4x as much crap to manage, and they show up as individual options in every reference menu (and unity throws build warnings if you don't have every reference using the same variant level!). Rather, we should be able to generate these assets as part of the compile/import process, not have multiple variants on disk to manage/etc, and not suffer a 4x import time. For instance, if you watch the cache server when doing texture compression you'll see it do this:

- Compress Highest mip of 2048x2048 hd texture
- Compress first mip (1024) of hd texture
- Compress next mip (512) of hd texture, etc, down to the lowest mip
- Compress Highest mip of 1024x1024 sd texture
- Compress....

Really, I just want to control the import process at the level where I can say "Compress this texture and generate all the mips; put them all in the hd version. Now put the everything but the highest level mip into the sd bundle". Then I wouldn't need multiple copies of the assets, and wouldn't suffer massive compression times which take down peoples machines for hours some days.

 

Thank you for your responses Ferazel:

You may be right that this isn't the proper forum although I do think the discussion covers more "cutting edge" topics around AssetBundles. I find that there ends up being little to no discussion about real-world AssetBundle issues in any other forums.

To your points:

1) Unity has been subtly dropping hints that the concept of the Resources directory is going to go away. There are already things that you can do with asset bundles that you can't do with resources (such as variants). Asset bundles going forward is going to be the better way to go. My recommendation would be to put these "system" resources into an uncompressed asset bundle and then keep the asset bundle open. This should be just about as fast as loading it form the resources directory, and you get async loading possible as well!​

We will try the "Resources-as-uncompressed-local-AssetBundles" approach and see how it performs. Do you have any experience with this approach?

2) I never really understand the concept of the versioning system that is built into the asset bundles. In order to increment the asset bundle you need to check a manifest of the files that you should download. This is either embedded into the .exe or it's queried from a web location. This manifest might as well contain the version and the URL that you're downloading from. For example: MyAppAssets/1.0/Textures.unity3d

Then when you release an upgrade you point the url to: MyAppAssets/1.0.1/Textures.unit3d using a version 101 to download. This will then replace the file in the cache correctly (asset name is the same) and download the appropriate download from the URL.​

I think we're in agreement here. We have been creating our own custom web server loaded AssetManifest that we construct from the new .manifest file generated by BuildPipeline.BuildAssetBundles. Our AssetManifest file contains URL paths to the proper AssetBundles for the client to use. The client sends up it's version number (eg 1.1.3) and receives back something like AssetManifest-1.1.3.json.

Inside the AssetManifest, we have a mapping of AssetBundle name to AssetBundle URL like this:

"myassetbundle" : {

"route" : "myassetbundle/hashgeneratedbyassetbundlebuild/",
"file": "myassetbundle.unity3d" ​

}
​

Given a root cdn URL, we construct the full URL to the uniquely identified AssetBundle. It would be nice for Unity to generate a suitable web server loadable AssetManifest file like this. The new .manifest files are helpful but not what's really needed for real-world CDN deployment.

3) I agree, having more granular control over the cache would be nice.​

Does anyone know the proper person at Unity to contact regarding this? This has been a glaring omission for 2 years+.

Our ideal solution would be for our custom AssetManifest described above to contain an extra field indicating the desire to clear an AssetBundle from the internal cache like this:

"myassetbundle" : {

"route" : "myassetbundle/hashgeneratedbyassetbundlebuild/",
"file": "myassetbundle.unity3d",
"clearfromcache" : true​

}
​

If "clearfromcache" is true, we would like to call a Cache API to clear myassetbundle from the cache.

 

Thank you for your responses jbooth. See my comments below:

The main issue with actually making asset bundles depend on things in the resource directory is that you can't guarantee the assets in the resource directory are synced with the current version of the asset bundles. In other words, once you leave the application's build process, you pretty much have to have everything you are dependent on outside of the app as well, otherwise it's possible for your data to get into an invalid state.​

If you read my reply to Ferazel above, you will see that we maintain our own AssetManifest per client version. This AssetManifest ensures a set of AssetBundles will work with a given client using the hash values returned from BuildPipeline.BuildAssetBundles. We never ship a new client version without ensuring there is a compatible set of AssetBundles. Given this model, it should be possible for Assets in AssetBundles to access resources with their GUID without fear of things being out of sync and without the need to duplicate the resources inside the AssetBundle.

On 3 - there's another issue with the cache; the current cache design free's the largest bundle from the cache first when the cache is full. In our current game, the bundle holding the UI is the largest bundle, which would be immediately re-loaded after being flushed out of the cache, only to be immediately flushed again. Meanwhile, all those tiny asset bundles you downloaded last year will stick around forever.

Note, I believe you can clear the entire cache without an app reinstall from the Cache class; but obviously that's not ideal either.​

Agreed that neither of these are very helpful. Is there a technical reason why providing a Cache.RemoveBundle would be difficult?

One of my main issues with the Asset Bundle system is that it's a nightmare to work with at scale. We have thousands of asset bundles in our game, and growing constantly. Some of my issues include:​

Sounds like they need to expand the Asset import settings to support settings per AssetBundle variant so you maintain only a single asset in your folder structure and the import per variant is handled automatically.

 

[q] You may be right that this isn't the proper forum although I do think the discussion covers more "cutting edge" topics around AssetBundles. I find that there ends up being little to no discussion about real-world AssetBundle issues in any other forums. [/q]

The forums are generally great for starting a feature, but anything which only happens in the scale of shipping a large game you'll basically get crickets. I kind of wish there was a "Unity-Advanced" forum or something where this stuff could be discussed with a higher signal:noise ratio.

[q] If you read my reply to Ferazel above, you will see that we maintain our own AssetManifest per client version. This AssetManifest ensures a set of AssetBundles will work with a given client using the hash values returned from BuildPipeline.BuildAssetBundles. We never ship a new client version without ensuring there is a compatible set of AssetBundles. Given this model, it should be possible for Assets in AssetBundles to access resources with their GUID without fear of things being out of sync and without the need to duplicate the resources inside the AssetBundle. [/q]

Yes, but my point is that they can't make any guarantee about these being in sync, because the system doesn't have any of these concepts in it. Loading from a cached asset bundle should be just as fast as loading from resources, as the resource directory is actually stored as an asset bundle in the client (or at least will be soon according to Unity, but I think it is already?). The only difference is that it's included with the app. If they allowed dependencies into the resource directories, they really need to solve for the CDN/Versioning specific issues as well, which IMO is outside of the scope of the feature as currently defined.

[q] Agreed that neither of these are very helpful. Is there a technical reason why providing a Cache.RemoveBundle would be difficult? [/q]

If they just switched to an LRU cache and let you (or the client) set the cache size, then you could basically forget about it for the most part. If there was a clear function, I personally would prefer one where I can pass an array of manifest files, and any cached data not found in those manifest files or not matching the hash could be cleared automatically. This could happen on startup and keep the client as lean as it could be.

[q] The client sends up it's version number (eg 1.1.3) and receives back something like AssetManifest-1.1.3.json. [/q]

We just create a path to a folder on the cdn instead, using the built in manifests. That prevents us from having to parse our own manifest on the client. So, for instance, the base path for all downloads gets munged to something like public/bundles/[client_version]/[bundle_version]/[platform]/[optional-android-compression]/