Automate build w/ custom Addressables build script (ie Variation in example repo)

Hi,

We currently use a custom "build maker" script to automatically make builds for us on different platforms, etc.

This is the code we use to make builds & to automate the creation of the Addressables bundles:

Code (CSharp):

1.  
2. // Set Build Target
3. EditorUserBuildSettings.SwitchActiveBuildTarget(BuildTargetGroup.Standalone, target);
4.  
5. // Handle Addressables, which are actually Asset Bundles.
6. AddressableAssetSettings.CleanPlayerContent();
7. AddressableAssetSettings.BuildPlayerContent();
8.  
9. // Build it
10. BuildPipeline.BuildPlayer(Scenes, FullPathForSingleBuild(buildDate, branch, target), target, GetBuildOptions(optionsMask));
11.  

Few questions:

1. We have recently implemented "variations" by following the example shown in the sample repo, pretty much an exact copy of it ('Advanced/Addressables Variants' example), for shipping assets in different resolutions.
   It appears that the example shown requires you to build Addressables bundle via the Addressables window -> Build button -> New Build -> "Packed Variations" button.
   Question: How do we automate this / what's the equivalent way to achieve this via code?
   
   
2. We've been trying to speed up our builds (and load times), and I see that this package is supposed to already support a semi-intelligent level of "incremental/delta" package builds/updates.
   Question:
   A) Is there something specific needed to get incremental building to work, for faster builds?
   B) Does our code above (specifically the 'CleanPlayerContent()' call) defeat the incremental functionality -- meaning, are we shooting ourselves in the foot?
   
   
3. If we know that none of our Addressables have changed, or if we know that our "video frames" did not change but our models did, is there a way [via code] for us to trigger a build for only those groups -- or is it all or nothing? Basically we'd love to add to our 'buildmaker UI' a way to check/uncheck which groups are re-built & which aren't; often we've just made a small code hotfix & we'd love to get that shipped rapidly, without waiting on a full Addressables 'packaging' process to occur [for each target platform heh].
   Question: Is there a way to build only certain Addressable groups -- or is it an all-or-nothing operation?

Thank you!

 

Bump -- this is holding us up and we've scoured the docs. @unity_bill perhaps?

 

Thank you very much.

 

Thank you very much, extremely helpful.

One follow-up (which may admittedly be a dumb/newbie Q): Our "build maker" code runs N builds, one for each platform we're building for, so usually 2-3 platforms/build targets/actual builds. Does BuildPlayerContent() need called for each build target, or is the built bundle cross-platform usable / do we only need to call it once?

Ideally some of this -- the first few answers, especially -- could make it into the docs (and/or into the sample repo). Especially the "build automation" aspect, which there isn't really a great example of anywhere (at least that I've seen), and it's super important & likely very commonly used stuff.

Knowing that the "CleanPlayerContent()" call removes any 'incremental' gains -- that's super great to know, we've been using the package for over a year now & didn't really know that, would have been great to see as an example in the repo (a "how to automate builds" example would be lovely TBH).

 

It is automated code but we don't run them via CLI right now, it's an EditorWindow with checkboxes and a "run" button. It is setup to be able to run from CLI as it just takes an integer/bitmask for "options" and then runs the appropriate builds, but that's most useful for running a "remote build" on another machine so that it doesn't tie up a dev's working directory.

We totally understand that Addressables doesn't include code, is just assets. What is less clear is how to correctly/safely "preserve" the built Addressable bundles between builds -- especially when build targets are changing. The issue is that we (like everyone, I'd think) need an extremely repeatable and universal build process -- it needs to 'just work' for everyone on the team, every time, without question/doubt/concern.

Here's a more "complete" example of our build automation code:
https://gist.github.com/ArthurD/954a7b4dbc1eece53ec75e6024d1b8aa

What happens here is that we will be building for 2-3 platforms (ie macOS, Windows, occasionally Linux); our build code 'enqueues' the proper build targets and then runs the builds needed sequentially until all have been made.

Since Addressables bundles are platform-specific, how are we supposed to do this without calling CleanPlayerContent()? Is the answer to write code that looks at the filesystem (at the build output path), and use System.IO to manually mutate the outputted contents to remove extraneous (ie non-platform relevant) stuff from the resulting StreamingAssets folders in each of the builds? That seems kind of hacky/messy & error-prone.

Our ideal world is we hit "build" and the result that Unity spits out is guaranteed to be "golden" -- at the end, our build code spits out some commands to upload the build to Steam. We hit build, walk away for 30 minutes, come back & copy the "deploy commands" that our build-maker outputted to the console, run them in Steamcmd, and we're done.

It works well & is extremely repeatable [though admittedly I think switching build targets via code was error-prone on macOS editor last I tried it] -- it's just awfully slow, we don't get any 'incremental' benefit. Sure, we could skip the addressables build calls (both 'clean' and 'build') -- but that means we're expected to have the bundles already built in StreamingAssets, and my understanding is that it means we'd need to manually remove the "mismatched" bundle for each platform (ie ensure the Windows build doesn't contain the macOS bundle, and vice versa) -- correct? Or am I missing something here?

 

We checkout the project for each platform to a separate folder. We never have to change build targets inside Unity between builds.

All folders contain the same project, we pull the latest changes from git before we build, but the "secret" to make the build fast is to keep the projects "Library" folder. It's where the build system caches all its data and it seems, at least with Asset Pipeline v1, changing the build target invalidates the cache. Using a different project folder for each platform seems to by-pass the cache issue.

 

Heh, I see. I assume you then run the builds via CLI so you can run 2 commands & have them both being built in parallel + have the full speed of the cache too, eh?

This should be documented somewhere, I swear. I knew switching build targets felt like it was doing quite a bit of work, and we've occasionally [relatively rare] needed to nuke WORKINGDIR/Library/* to fix some kind of "stateful" bug -- but I never thought switching build targets was 'nuking cache' anywhere even remotely similar to that degree.

Even just anecdotally, I'd be curious to hear how quick your builds are (and I assume you're always building Addressables eh?) -- our project weighs in around 2-3GB (post-build) and we are seeing >= 30min total build times for two platforms (win/mac, both on Mono / not IL2CPP).

Edit to add: Those build times are on new & very high-end machines; latest-gen CPUs, copious amounts of RAM, extreme-performance disk/RAID/IO config, etc...

 

Yes it's via CLI and yes they do run in parallel.

However, the Unity build process is quite resources hungry and building more than a single project at a time on the same machine doesn't lead to faster builds. I don't recommend to build in parallel on the same physical machine, I would recommend to distribute builds across different physical machines instead.

Yes, building Addressables always. It totally depends on:

1. How well is your asset bundle content structured, see here.
2. How much content changed since the last build, because Unity supports incremental builds, which happens when you don't clear the Library.

Incremental builds take most of the time less than 10 minutes here, but if there are a lot of content changes, it takes longer.

 

Wowzers. Massive gains via the guidance provided above, and especially via changing our builds to be CLI-based with 1 repo per build target as @Peter77 suggested. Previously our build times for 2 platforms were >30 minutes total.

After changing our build automation around (CLI driven w/ 1 repo each), well, the results really say it all: we're now seeing builds for 2 platforms consistently taking under 10 minutes. Back-to-back builds (ie no changes at all) are lightning speed, relative to before, and routinely are <=5 minutes total for two platforms (!!). Pretty remarkable difference.

This stuff should be documented -- ideally a "automated project build w/ addressables" example. Thinking about just how much time we wasted over the year or so, just waiting on builds to run, it's a little bit frustrating since the 'solution' (ie massive speed gains) were so close at hand. Unless these kinds of things are documented though, quirks and all, the only way to discover this would have been via trial-and-error (or community / word-of-mouth, as was the case here).

Massive thank you @Peter77 -- if you're ever in the Vegas area LMK, I'd love to buy you a few drinks. Ty!!

 

Alright, so we've got this setup now -- using exactly the code supplied above for the packing -- but we're getting an error in builds (on both macOS and Windows targets).

Code (CSharp):

1. Exception encountered in operation Resource<ResourceManagerRuntimeData>(settings.json), status=Failed, result= : Invalid path in TextDataProvider : 'D:/SteamLibrary 2/steamapps/common/SimCasino/SimCasino_Data/StreamingAssets/aa/settings.json'.
2. (Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 39)
3. RuntimeData is null.  Please ensure you have built the correct Player Content.
4. (Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 39)

It works fine in editor play mode. Editor logs don't show any issues during the builds. We tried upgrading to the absolute latest Addressables release (Preview release; then re-vendoring it + patching it) -- same issue.

We're running editor version 2019.4.23f1 and we tried both our original Addressables version and the latest preview version -- results are the same, the content is missing in the builds.

LMK what we can provide to help debug this. It seems similar to the issue in this post -- but so far nothing we've tried has resolved the issue. Any guidance much appreciated.