Content Update Unnecessary Bundles

Hello, we are trying to figure out why this is happening. In the screenshot you can see what packed asset groups/bundles the system say will update with a single, simple texture change. The question is that we have no idea WHY all these bundles are in this list. As far as we can tell there should only be one in the list (the actual changed texture).



How we got here (using 2018.4.11f1 and Addressables v1.2.4):
1. We set all our Groups to be "Static Content" and they are also set to pack separately and all have encryption turned OFF for now.
2. The modified file involved is set up as an addressable directly (it's not implicit).
3. We built all our addressables as a baseline and created a build (for Switch if it matters).
4. We then, in Photoshop, modified a single texture file that is used by only one material (BackRider.mat, that is itself only used on one prefab - BackRider.prefab. Both are their own addressables and packed separately already)
5. We then prepared a content update using the .bin file created from step 3 above.
6. The results are what is shown in the screenshot.

What we expected to happen was that ONLY the texture's direct bundle would change as that is the only change made in the game.Instead we also see the bundles for the material, prefab and SOME levels that creature is in was impacted (but not all the levels).

It is critical for our target platform that we can make as small of a patch size as possible and are trying to limit the number of changes Unity is making to files/bundles unnecessarily.

Is this a bug? Are we doing something wrong? Any insight and advice would be much appreciated. Thank you!

 

Also of note - we figured for our use-case (only using Addressables for patching purposes where all files will always be included locally in the build and on device) that we should not even use the "Static Content" and remote server approach at all. We just did this as our latest test because just rebuilding the addressables caused many random bundles to be rebuilt using this same simple texture change test.

Below is a diff of the baseline build (left) and the single texture change build (right) bundles of a Windows build. It shows how many bundles are "touched"/changed for no explainable reason to us - and because the bundles are binary we can not even see what changed in all these files, though it seems to be a very tiny bit of data.

 

Could someone from Unity please advise us regarding minimizing patch sizes to meet Nintendo's strict size criteria? We are really stuck on if we are doing something wrong on our end with Addressables and how we should proceed. Thank you!

 

Thanks for the reply David! I'm not sure if this will help better clarify what is happening, but I'll give it a shot

1. We created a baseline build and built all our bundles (any platform is the same - this one was Windows).
2. We then changed a single texture file called "Backrider-d.psd". This texture is only used in two materials and both those materials are only used in a single prefab in the game. That prefab is then used in various game level unity scene files. This texture is in its own bundle (as are the materials, prefab and each unity scenes)
3. We then rebuilt the bundles and made a new build. All the bundles shown in the screenshots in the original posts above changed (really, we think the second screenshot with the larger set of bundle changes show in Beyond Compare is more relevant as we don't think we should be using "Static Content" bundles in our use case).

What we would expect to happen is that just the "Backrider-d.psd" bundle would change as that was the only change between builds.

Here is an image to visualize this as well as showing all direct and indirect dependencies of the texture file:

 

If you think what we are seeing is a bug, we will be happy to create one. We are also fine with Unity using our project as a test case if you like. It is a completed game (already out on Steam with console ports in the works) and the Unity project is about 15GB in size (after deleting the Library) so we might need to figure out how to transfer it over.

 

One last bit of info - we just tried making a fresh baseline build of the bundles (cleared cache and such) and then immediately just built the bundles again - and several bundles showed changes between the two though nothing in the game was changed at all.

We then just built the bundles again for a third time with no changes and this time many more bundles were marked as different from the baseline build with no explainable reason.

When doing compares on the bundles that were marked different, it seems to just be small bits of data that is exactly the same, but moved randomly to a new location within the file. How can we avoid Unity from doing this? Thanks!

 

Thanks again. I tried to repro the "rebuild bundles without any changes will change some bundles" in a clean project, but no luck there. I will look into if we can whittle down our actual project where the issue is reproducible. I'm also curious if potentially a middleware or custom editor script could be the cause (though no actual proof of that yet).

I was however able to easily recreate the "many bundles being rebuilt when only one thing in one bundle changed" issue in a clean project and submitted that bug. The case number for that is: 1197035

 

@DavidUnity3d and @unity_bill
We have finally had the time to pair down our game's project and submit a bug for the "Addressables Rebuilding Bundles Unnecessarily" issue. The supplied project gave a 100% repro case for us when modifying the specified texture in the report. The case number is: 1200150

This is the "big one" for us as it greatly adds risk to being able to create any Switch patches due to their patch size restriction. For us in our full project, a simple file change is causing a rippling effect of touching many, many unrelated bundles and bloating our patch sizes. We are happy to help in any way we can to get this resolved. Thanks!

 

Any update on case number 1200150? A while back it was marked as closed, but is clearly a 100% reproducible issue with the provided test project. Can someone please look at the additional replies with more repro cases and screenshots we provided.

We'll be contacting Nintendo about this as well so they are aware since it can cause a big issue with their own patching size limit policy with Unity engine games. If we are doing something wrong, please just let us know but it sure does look like an actually issue of bundles being touched on the binary level that should not be. We will be submitting our release build to them very soon and would really like to have this figured out before we do so that any future patches will be as small as we are designing them to be. Thanks!

 

Hello again. We wanted to just post an update to what is going on regarding this issue.

While some of the problems have been resolved, we are still seeing an issue regarding bundles being modified unexpectedly when they shouldn't. We spent a lot of time testing this on multiple versions of Unity and see the issue in all tested versions.

We really hope this can be identified and fixed, as well as all these fixes back-ported to 2018.4 (LTS) so that we will not have issues making small patches for the Nintendo Switch (and other platforms for that matter).

Because we were told the issues regarding PSD's and restarting the editor were fixed (at least in 2019 and 2020 released builds), we have created a new bug for this issue:
Case: 1210984
Bundles changing when they shouldn’t after modifying an asset
https://fogbugz.unity3d.com/default.asp?1210984_qkts2f0cabti9t2l

Also, here is a video of the issue (see video index in description to quick jump around - bad bundles at 3:37):

Thanks again for looking into this @DavidUnity3d , @unity_bill , and others!

 

I'm not familiar with the Addressables source, but it sounds like it's not using the deterministic asset bundle flag when building. Without that, the asset IDs inside the bundles are regenerated on every build, requiring all bundles that depend on the modified bundle to be regenerated (I can't imagine why would anyone ever want determinism to be optional).

This is one of the reasons I didn't even bother with addressables and went straight into coding our own asset bundle manager using a modified asset graph. That's the only way to be sure something as crucial as this works reliably. The Addressables team posted not long ago they were focusing on mobile, I'm not even sure they actually tested it mucb on consoles, and if they did, I seriously doubt they got as far as testing patch generation determinism.

I hope I'm reading wrong, but are you saying that this is actually by design? Because if the SBP goes through the dependency chain flagging as modified every asset that references a modified asset, this is absolutely not what anyone would want. If a texture is modified, only the bundle containing that texture should be updated, it shouldn't go all the way up and rebuild the material, prefab and scene bundles. The system should strive to keep changes to the bare minimum.

 

We did look into BuildAssetBundleOptions.DeterministicAssetBundle, but it didn't seem like something we use with Addressables (no setting for it in Addressables or documentation on its use with that system) or the cause of what we are seeing, so we didn't look into that further.

Happy to hear from Unity on this as we would try anything to get this working before our impending launch on consoles.

 

BuildAssetBundleOptions.DeterministicAssetBundle became obsolete around unity 5.something. All bundle builds are <shooting to be> deterministic now.

@Greyborn - the third ticket you filled (with the youtube video) I think has finally given us the repro scenario needed to track this down. It will likely be a few weeks before we can get the actual fix out, but we'll keep you posted. This repro case has been hugely helpful as we've had a nagging & intermittent determinism issue for a long time.

 

That's great! Any idea if this is limited to the SBP bundle generation process, or should it also affect the built-in build pipeline? I can try running those steps in our project that is using built-in and a similar granular bundle setup generated by AssetGraph to make sure.

 

Great to hear! If there is anything else we can do to help, please just let us know. Looking forward the bug's squashing

 

it shouldn't affect built in pipeline, but we won't know for sure until we work out exactly how to fix it. Which will likely be next week or so.

 

BTW - when it is fixed we would plead that this is back ported to 2018 LTS so we can utilize it for the game we are about to ship.

 

Thanks. I assumed both used the same underlying mechanism when it comes to writing the actual serialized data to the bundle files. It would be nice to have more technical details on how the SBP serializes bundles at a low level compared to the built-in.

And since this thread is about bundles, determinism, patching and Nintendo Switch, it would be amazing if there was a way to setup an alignment value to be used when serializing asset data into bundles. This way we wouldn't need to resort to creating several thousands of bundles with very few assets each: if bundles have padding added between each asset data blob to ensure start offsets are always aligned to the chunk size used by the platform's patching mechanism, changes to one asset would not affect any other asset inside the bundle.

 

Hello, any update on this issue? We're close to certification submission and we're having the same problem.

 

Thanks for the update @DavidUnity3d

 

Great news, thanks!

 

Hello there. We're wrestling with this as well as @Greyborn. @DavidUnity3d- any news about a fix for this issue? Or any suggestions for alternative solutions / workarounds?

 

Yes, and update would be nice as it has almost been 5 months since that "few weeks" quote above

We have actually had other Nintendo Switch developers reach out directly to us in hopes we found some kind of workaround (we haven't). This is actually a big deal for Switch developers as the bug can directly violate a Lot Check for patch submission.

Thanks in advance for any updates to this issue!

 

Hi David! We'd also be super interested in a fix for this issue. In our experience, building bundles is a bit of gamble- even if you build the same exact bundles twice in a row, without touching anything in the editor in between, sometimes they will come out with significant binary differences.

 

@unity_bill & @DavidUnity3d - popping in again to see if we could all get an official response to this for us Switch devs? Thank you!

 

Have you tried Addressable 1.10, I think they have fixed this issue based on the release note. The only thing will trigger unnecessary build is changing serilization layout, basicly means you can't add or remove new class or struct.

 

We're in the same boat. Would love to get info about this.
This tread dragged on long enough that we have now already submitted on Switch and we'll surely have to request a waiver to do any kind of patching.

 

Could you please point us to anything documented about this fix?

Unity has still not marked the actual bug as fixed, nor have made any further comment to us or any other developer beyond it would in "the next week or so" - six months ago. :/

Here is a link to the bug again for reference: https://fogbugz.unity3d.com/default.asp?1210984_qkts2f0cabti9t2l

 

@Greyborn I believe @Thermos was referring to this: https://docs.unity3d.com/Packages/c...11/changelog/CHANGELOG.html#1100---2020-05-28, but they can correct me if I'm wrong.

 

Checking in again a month after my last request for an official response to this issue. Any update or acknowledgment at all , or even just an official word from Unity to tell us Switch devs to go pound sand, would be appreciated @unity_bill or @DavidUnity3d

 

Hi Ryanc_unity, thanks for the update!

This would be awesome- I'd imagine that many of us would be really grateful for official advice on the implications of using (particular versions of) 2018, 2019 or even 2020, on bundling content.

 

I'm using Unity 2019.2.9f1 with Addressables 1.11.2 and am having this same issue. I'd hate to upgrade this project again (it started on 2018 with Addressables 0.7.something) but if it is fixed it would be huge for us.

Our project has tens of thousands of assets using Addressables that are all contained in lists of Asset References on Scriptable Objects. A change to one list, such as adding an item to the list or changing the order of the list elements, somehow causes every single item to need to be redownloaded up and down the chain. This is a live B2B service and content is added monthly if not multiple times per month, so it is pretty annoying to deal with (though this is still the best way we've found to handle loading/unloading content efficiently and handle remote content easily).

 

We're not using Sprite Atlas or reimporting, but I will try a newer Unity version and see if it was fixed. The issue is the same as the original post, just in our case every single asset is tied together like this:
Main Database (Scriptable Object, asset reference)
--Group (scriptable Object, asset reference)
----Item (scriptable object, asset reference)
------Texture2D, asset reference
------Sprite, asset reference
------some variables

With dozens of groups and over a thousand items, each with their own texture and sprite addressable objects. So in our case if we change one thing, we experience the OP's problem across the entire project. It doesn't happen 100% of the time, but probably around 75% of the time updates are made.

Optimally, we'd like the Groups to be in one Addressables Group that is non-static, and the Items + their references to be in a static for each new update (once or twice a month) to keep the updates small, but that doesn't work, because prepare for content update marks everything as needing to be updated even things in static groups that were not touched.

 

We appreciate getting an update, thank you. We'll just have to do the best we can with 2018 LTS as trying to port our game up to a newer version just before submission is too risky (and didn't go well when we tried it).

 

@Ryanc_unity We found that asset bundle aren't deterministic in 2020.2. We didn't cache the Library folder but use Unity accelerator as cache server. As we build in CI, the machine is newly created every time and we checked out the same source from git but the hash of the asset bundle may change a bit. Is this a bug? We found that there is no problem in 2019.2.

 

Hi, sorry for distracting for another issue we met under this thread.
We get a assetbundle including shader updated but have no resource changed. The text diff of deserialization .ab shows only the builtin keywords (_Normal, _AlphaTest, etc) order of subprogram is changed. We are on 2017 LTS, does this related to bugs you mentioned here?

 

We're currently experiencing minor changes on asset bundles from build to build - in our case byte #153 in the bundle has a different value each build, nothing else changed. Unpacking & converting the bundle to text (via the editor tools WebExtract and binary2text) doesn't show any differences in the text representation of the bundle data.
Unity 2020.2.4f1 + Addressables 1.17.6-preview + SBP 1.16.1

 

The same problem. Some asset bundles are stable, but others are randomly changing each build.
I don't remove the Library directory.
2020.2.1f1