Deterministic Asset Bundle Building

We have been doing some further testing of Addressables and have run into some issues.

In one simple example, we have addressables referencing some scenes and some prefabs. When I build the bundles on my Mac, they are all built and play correctly. When they are built on a Windows PC - exactly the same setup, they are all built again but the asset bundles containing any scenes are named with a different hash and a few bytes different. On another Windows PC they were built again successfully, but different again to the other two machines.

What it looks like is scenes in asset bundles are built slightly differently between each machine, causing the bundles to be named and sized slightly differently. This will cause issues with patching later on down the track - if we build the bundles on PC1 for the initial release, commit everything to git, then attempt to build some update on PC2, all bundles that include scenes will be rebuilt and modified, even if they have not changed. I also couldn't guarantee that if we rebuild the bundles on the same PC a few months down the track that the bundles would be built the same.

Is there any way to work around this, or some setting perhaps that might need to be enabled?

The issue I see is that I have a setup such as this:

LocalStaticBundleA - Scene1, Scene2, Scene3
LocalStaticBundleB - Scene4, Scene5, Scene6

After building them, I then make a change to scene 2 and scene 3, and preparing for content update would look like this:

LocalStaticBundleA - Scene1
LocalStaticBundleB - Scene4, Scene5, Scene6
RemoteContentUpdateA - Scene2, Scene3

Now building the bundles would produce a modified StaticBundleA (because Scene2 and Scene3 have been removed), and also a modified StaticBundleB (because Unity won't rebuild bundles identically as mentioned above), and a new Content Update bundle (as intended).

What we would ideally like is to re-use the Local bundles built initially so that those files are not marked as changed, but the client would look for the Remote folder and grab the newer bundles from there.

I believe as a work-around I can save the asset bundles built initially as a backup, and then when its time to build the update I can copy the files into the Library/com.unity.addressables/StreamingAssetsCopy folder so that the build pre-processor will pick up the initial bundles and use them in the build. But this feels incorrect to have to manually replace the built folder location.

Additionally, this is not possible for a Remote update, e.g. if I make a change to Scene2 again for another update, I would have:

LocalStaticBundleA - Scene1
LocalStaticBundleB - Scene4, Scene5, Scene6
RemoteContentUpdateA - Scene3
RemoteContentUpdateB - Scene2

In this situation, if I had marked RemoteContentUpdateA as static, it would have the same issue as the local bundles - the bundle has changed and would have to be re-downloaded. But if the RemoteContentUpdateA stayed exactly the same, and only RemoteContentUpdateB was new, the download size is smaller. The only way I can think to resolve this is to manually modify the content update .json and .hash file (which might be fragile as I am not certain of the format of those files or how they work exactly), or to manually replace the RemoteContentUpdateA bundle with the previous bundle and rename it to match the exact file name (which may or may not work).

 

Following on from this, likely related but not sure if its worth its own separate post so adding it here.

From the documentation at https://docs.unity3d.com/Packages/c...manual/AddressableAssetsDevelopmentCycle.html it explains a nice example of how content gets update when preparing for content update regarding static and non-static bundles. This sounds right and how we would need it to work, but doesn't play out that way in practice.

The example listed there said that after changing an asset in Local_Static, Remote_Static and Remote_NonStatic, the following happens:

1. Local_Static remains on the device, with a reference to an asset no longer used (moved to new content update).
2. Remote_Static is unchanged and will remain cached on device or downloaded when assets M/N are requested.
3. Remote_NonStatic will be rebuilt with a new version.

The problem I have is with the static bundles. Particularly the reference that Remote_Static would be unchanged - however in my testing, when building the content update, the Remote_Static bundle is changed - it gets rebuilt with one less asset in it. So when the game client requests asset M or N, it would no longer retrieve it from the cache but instead think it needs to get it from this re-generated asset bundle.

I have done some tests trying to trick the system into loading the previously unchanged asset bundle (which includes assets L, M and N) but the CRC check fails (as it should). I would rather not have to disable CRC checks and do some dodgy copy-paste file replacing to get the bundles to remain unchanged.

------------------------

For some context on exactly why we need the static bundles to be exactly unchanged - we are building patches for platforms, particularly Switch. In our setup, we load the Local bundles from StreamingAssets/... (default location), and the Remote bundles from StreamingAssets/Remote/... - this way the game will request updates from the 'remote location' which is actually still bundled with the game.

Why do we do it this way? So that we can keep the base Local bundles in the game, and only add Update bundles into the remote directory. The total build file size may increase, but the patch will compare file binaries, and it should find that all the bundles in the Local Bundles directory (StreamingAssets/...) have been unchanged and so the patch size for those files would be 0 bytes. The patch would find changes in the Remote Bundles directory (StreamingAssets/Remote/...) and only include those in the patch.

If the local bundles are changed, even by one byte, the entire file is flagged as dirty and included in the patch, pushing its size up when no change is actually needed. This is why we need to have bundles that can be built identically across any machine, and also ensure that after patching the Local Static (and Remote Static) bundles that have had assets pulled out of them into content update groups also remain unchanged.

 

Update: I have been able to make it partially work by disabling the CRC check, and manually replacing the updated bundle with the initial build's bundle. This has new issues though when working with Scenes.

If you assume from the example linked above that asset L, M, and N are all scenes, then it changes the loading process. My understanding of asset bundles is that when loading an asset (e.g. texture, mesh, prefab) we can load it by name within a specific bundle. So if there were two bundles with the same named texture, we would say "Load cats.png from bundle A" and it would work fine. However, when it comes to Scenes in asset bundles, they work a bit more magically - once an Asset Bundle containing scenes has been loaded by Unity, you can automatically load into that scene using SceneManager.LoadScene. There is no way to say "Load Scene1 from bundle A" - so if the same scene was included in multiple bundles, there is no way of specifying which bundle to load the scene from.

This causes issues because the game would load the Local_Static bundle, containing 3 scenes (L, M, N). It would also load the Content Update group, containing Scene L. Now when the game says "Load Scene L" it is not clear which bundle it should load from. In my test, it ended up being unable to load the scene, and some scenes became corrupt / missing elements. It might be fixable by unloading the bundle you dont want explicitly (i.e. Unload Local_Static bundle, then load Content Update bundle, then load Scene L) but that is not happening automatically with addressables currently.

 

A big +1 to all of this - in particular the need to specify asset bundles from older builds to replace the built ones (e.g. a with a Bundle Replacement Schema & build script). In addition it would also be beneficial to temporarily and automatically generate patch/remote groups (the ones created from the "Prepare for Content Update" from static groups) at the start of the build process (i.e. as an early build script task step). Otherwise you keep needing to roll back changes you don't want to commit to source control.

Also I wasn't aware of the scene issue BFS-Kyle. That could be solved if each individual scene was in a group that's not marked as content static AND the scenes are deterministic upon being rebuilt - even months or years later? It would be safest if you could just alias to the latest patch bundle though.

As always thanks for all the work on Addressables, I'm sure you don't hear that enough Bill and team!

 

Can I ask why asset bundles are not determenistic upon the asset GUIDs? If I'm not mistaken, asset GUIDs never change once imported into the project besides the instances where you delete the meta file or heavily mess around in the structure. There is an option for that in the asset bundle build APIs, but why doesn't it work?

It is exceptionally crucial for many games on consoles and this issue needs to be clarified.

 

Two separate issues being discussed here.

1. Determinism.

and

If you build bundles, with the same data, on the same machine you will get the same result. If this ever doesn't happen, it is a very important bug to us, and we get it fixed. If you build bundles on different machines, especially if those machines are running a different OS, we cannot guarantee they will be identical. We shoot for that, but with all the third party plugins we use, and the way different hardware/OS will handle things like floating point arithmetic, it's not realistic. We don't like it, but in some instances, we just cannot fix it.

This is and has been the case with Unity for a long time, and is unrelated to Addressables. For your situation, we highly recommend having a build server, or a designated person on your project to create your builds. Having different people on different machines build the game is not recommended.

2. Updating bundles.
Here I need to clarify the intended workflow, and point out we currently have a bug in this flow.
Lets take your example...

In this case, the intended workflow is that LocalStaticBundleA and B both get rebuilt, but then discarded. We should probably find a way to not write them (or at least delete them) ourselves, but don't yet. Here's where things diverge a little from the "common" path to the "switch patch update" path. For the common path (a game with an already distributed local player, and a remote server with updates), what we build as "local" doesn't matter, because when doing an update, you don't rebuild the player (if you were rebuilding the player, you'd want fresh bundles). If the S1/2/3 bundle was remote and static RemoteStaticBundleA(s1, s2, s3), this also shouldn't really matter. You could either not upload the new RemoteStaticBundleA(scene1), or if you upload it, that's harmless as it has a unique file name (appended hash) and won't ever be referenced.

But, your setup isn't common. Honestly I had expected that people in your situation would just rebuild (no update flow at all). Using your example,the entire file of LocalBundleA would be rebuilt, causing the patch to see that LocalBundleA was different, and LocalBundleB was not. This means the update download would include scene1 unnecessarily unless Switch patching could do partial-file updates (I do not know if it can).
The fix for you I think is that whenever you ship, I think you need to put all your built bundles into StreamingAssets (copying them from our Library cache), and commit them to version control.

Also... here's our bug. What should happen, is you end up with a catalog like this:
Scene1 -> LocalStaticBundleA(original... s1, s2, s3)
Scene2 -> RemoteContentUpdateA (new.. s2, s3)
Scene4 -> LocalStaticBundleB(original.. s4, s5, s6)
...

unfortunately, the catalog has Scene1 pointing to the new local bundle, which the user won't have on their device. We are working on fixing this bug now, and it should be out within a couple weeks. We have it on our to-do to clean up the unused bundles ourselves, and I'm now also adding a to-do to clarify our docs a bit here.

This last bit is something I need to look into. The temporary fix is to mark your static groups with scenes as "BundleMode=PackSeparately". This will put each scene into it's own bundle. That can have some overhead if we're talking about 100's of scenes, but in general may be worth the tradeoff depending on your situation.



Hopefully this gets you pointed down the right path. Thanks so much for the feedback.

 

ok, yeah, that is a problem. Not only due to the loading you mention, but if several scenes are built into the same bundle, there's actually some efficiency in the build that causes the scenes to depend on each other. Pulling one out significantly alters the makeup. So realistically what this means is we need to change Prepare to say "if a scene has changed, all scenes in that bundle are considered changed". And obviously update the docs.

So the best bet if you expect the possibility of a scene update is to pack scenes separately

 

Thanks for the replies @unity_bill - much appreciated.

This is understandable, and I have been aware of it in the root asset bundles since way back when Obviously would be ideal to have it identical across all machines, but sounds like a tricky problem. As for using a build server, we do currently have that, but due to the volume of builds we run, we have a build farm of numerous machines running builds, and it could pick any of those to perform the build depending on the load. We could always configure it and dedicate one physical machine to a specific game, but ideally having identical builds across machines would be perfect.

After more testing since initially posting, this is much closer to what I think we need to do. If we need to patch a single scene from a group of scenes, we would essentially mark ALL scenes in that group to be changed, and rebuild them all.

As for why we would push this into the remote instead of local - this is because we want to keep all the other local untouched bundles the same - if we could guarantee that they would all be rebuilt identically to the v1 built bundles, then that wouldn't be a problem (see above) but instead what we would have to do is save / commit the v1 build local bundles, and manually insert them (along with the newly produced 'remoted' bundles) into the build when preparing an update. But what we would be doing is marking ALL of "LocalBundleA" to be changed (including scene1, even though it was not explicitly changed) so that the entire bundle will simply be fully re-built. At this point I think it will work, however we may run into issues when we get to the point of creating a second update with new remote-bundles (but i believe we can resolve this by copying over our first patch bundles in-place, renaming as required, and ensuring the 'verify CRC' is turned off - this will ensure identical files are copied over). Note: Switch patch detection seems to be based on full files - it doesn't handle partial file similarities (unlike Steam which is far easier to work with!)

That sounds like the right idea. Also regarding packing separately, I think that's good in theory, but it loses these efficiencies that you mentioned, and the total file size can go up significantly. (This can be mitigated if the project also makes good use of not only scene bundles but also resource bundles and loading of resources separately to share assets between scenes that way).

 

In theory, if it's the same hardware with the same software, it should work. Our CloudBuild service does not just have one machine running, but still manages to support determinism (as far as I'm aware). They way they pull it off (I think) is to very carefully ensure that each server is identical. Perhaps that could work for you. Related note, we are working on getting full addressables support within CloudBuild, in case that's appealing to you.

yeah. Over the course of developing addressables we argued a lot on how much "automatic optimization" we should do. We landed on "almost none" because of situations exactly like this. Building together causes optimized, smaller bundles. Building separately is better for updating (and maybe downloading, depending on the situation). There is no clear "best practice" here. I wish there was.

 

Sorry for the off-topic, but what would be the issues right now with Addressables + Cloud Build?

 

There's no issue with the building, per se, there's just not a way to get the built bundles after the build is over. I think it'll work if all your addressables are local, but not if you want to build some remote bundles that you need to the copy onto a server. So really it's a front-end issue. Cloud build team has a lot on their plate, but I believe this is coming.

 

For the record, I'm currently working with Addressables and Cloud Build. As of I'm not sure exactly when Cloud Build settings now has a switch for building Addressables and an option to specify what profile to use. When on, successful builds provide a zip file with the catalog and bundle files, so I guess the Cloud Build team has made progress since @unity_bill's last post, but I still can't figure out how to do a Content Update, as per this post: https://forum.unity.com/threads/add...pdating-remote-resources.741884/#post-5033060

If anyone sees this and has any suggestion please let me know.

 

@lloydv Have you been able to do content updates with Unity Cloud Build? We are considering dropping Unity Cloud Build entirely, since we can't make it work for updates, nor find documentation about it.

 

@unity_bill, Hi, thanks for your replies!
Currently we considering Unity Cloud Build as our main solution to build Android and iOS versions and Addressables for them. We also plan after release dynamically update Addressables without releasing new version of the app.

As @DiegoWw mentioned above, Unity Cloud Build is not suitable to reach content updates after release.
Do we need to use local build machine for that?
What is current state of the determinism in Unity Cloud Build?

 

@unity_bill We have since relied on our own local builds, but I would like to know where to request this feature formally.

Also, I have a very specific question. Since it is not possible to generate updates on cloud build, can we just do them locally and upload them to cloud build, and then generate a build from that? We tried that, and it didn't work, but we might be missing a step. Do you know of any tutorials, or resources to help us do this workaround?

 

I honestly don't understand how this could be. I have 2 Macs running the same OS, using they same build of Unity 2019.4.2f1, each downloaded the latest version of a project from Unity Collaborate. Whatever plugins being used in the project by one machine are obviously the exact same ones as the other machine, but we're getting different size bundles.

It's odd to me that processing the same dataset with the same software on different computers doesn't produce the same output.

 

@unity_bill we have an issue with even building on same Mac. Our case is about texture compression is set to ETC2 for iOS. I just made quick test by creating bundle with only one image with ETC2 compression. After removing Library folder, reimport project and build bundle again the result is not same hash for bundle. On PC everything works as expected - hash remains always same.

 

Some info regarding bundle builds was recently posted in another thread: https://forum.unity.com/threads/content-update-unnecessary-bundles.768710/#post-6169220

 

In Unity 2020.2.1, on Windows, if I delete the Library folder/open project/build bundle, I always get a different hash value appeneded to filename.

Anyways does anyone here any have updates to this issue in general?

 

@delexaet I am also facing this issue and I think this breaks the whole system, since I want to deploy via somepipeline and not always have the library folder and others cached.

My assumption is that with the very same codebase and assets the bundles should look exactly the same. No matter on which machine...

 

when I <Clean Build> and then <New Build> with the same machine in one project I get server bundles with the same hash and in another project I get bundles with different hash every time so it seems inconsistent. Then when I use the build server, I always get different hashes, which forces me to upload bundles again to the distribution server. Both running OSX
I'm not sure if I'm doing something wrong
The only solution I found for "deterministic" bundles is kind of crazy:
I build on my desktop, then copy the content of /Library/com.unity.addressables/aa to a version-controlled folder, e.g.: "
rootProjectPath + "/StreamingAsetsAAData/"

Then on the build server, I never run the script to buid addressables. In a pre-process script I just copy the version-controlled build from /StreamingAsetsAAData/ to the real StreamingAssets (Application.streamingAssetsPath + "/aa"), matching what the addressables build script would do at build time.

I'm not even using Content Updates, just basic embedded addressables for localization plus a few remote bundles

 

It seems some versions of addressalbe has the problem. I test 1.18.1, build the assets twice, and some of the assets bundles changed. And upgrade to 1.19.19, build twice, there is no change.

 

Did you change unity version as well by any chance @Vanimalsdev ?

 

No, uniy version is 2020.3.8f1

 

@unity_bill Is this still the case with addressables v1.19.19? Also, if determinism in hashes was achieved, does that mean we don't need the content update flow anymore? We just build the bundles and upload to S3 and point clients to the new catalog and they will download only the changed ones since unchanged ones will have the same hash?