Addressables + Cloud Build: Updating remote resources

I'm new to Addressables and working on a project that builds with Cloud Build.

I was happy to find Addressables already integrated into the dashboard, so I turned the switch on and on my next build there was a package with the remote ServerData to upload. Awesome!

However, in the downloaded zip file there is also a Library/com.unity.addressables/StreamingAssetsCopy/aa/iOS folder and I don't know what the contents are intended for. Is there some documentation on this somewhere?

Also I am very vague on how remote assets can be updated when using Cloud Build, because I don't see anywhere to create or select the magical '.bin' file like in the demo videos.

I hope someone can share their expertise or experience. Thank you.

 

Here's a tutorial may help in general https://forum.unity.com/threads/demo-tutorial-for-remote-hosting-using-amazon-s3.724022/

It's not well documented. Just known they're settings file (where to load catalog) and local catalog (to load local content) is good enough.

You may possible to upload the bin file and other remote assets of RemoteBuildPath, to your content server in PostExport method of Cloud Build (using Sync IO). There's no example code so far, so please share if you reach there.

You may also fire a ticket to Unity Cloud Build team to ask them adding the feature to download content from RemoteBuildPath, like another zip file from the build panel, or the a link in webhook.

 

@Favo-Yang Thank you for the info, now I have a little more direction to work in.

I understand they're settings files, but the fact that they're included with the bundle build makes me think I need them for something. Am I supposed to manually put these files back in the library folder on my machine, maybe?

So far I've got my addressables built on Cloud Build, hosted remotely, and loading correctly in an iOS app. Next step is to try and update remotely, and for that I need the bin file. I've already got a post-build shell script set up that uploads the ipa to Deploygate, so hopefully this won't be too headache-inducing...

Will post back here when I've tried a few things.

 

Only if you intend to use those cloud build bundles (local / remote) in your local development (or share with other devs to save their build time). In such case, you will need sync the setting files to locate remote catalog. Depends on whether you have local contents, and other details, those setting files may have no change during build time and can be safely committed into repo. See this thread #7, #8 for details.

The bin file will be located at the RemoteBuiltPath. However it only required by preparing a content update, and building for content update. In such case you will need download the saved bin file from previous build, the default addressable build panel won't be useful for that. You need write a custom script as PreExport hook.

However I wonder if you really need the *content update* thing. It's a process designed for preparing incremental content build, for public releases. As you described, the major use case is testing purpose through Deploygate, you won't get much benefits I guess.

 

Got the bin file using a post-build .sh script. Here's how:

Code (CSharp):

1.  
2. # Unity Cloud Build manifest file location
3. UCB_MANIFEST=$WORKSPACE/Unity/Assets/__UnityCloud__/Resources/UnityCloudBuildManifest.json.txt
4.  
5. # log its contents
6. cat $UCB_MANIFEST
7.  
8. # extract the build number using regex
9. CLOUD_BUILD_NUMBER=$(grep -o '"buildNumber":"[^"]*' $UCB_MANIFEST | grep -o '[^":]*$')
10.  
11. echo "Cloud build number: $CLOUD_BUILD_NUMBER"
12.  
13. # Addressables bin file location
14. ADDRESSABLES_BIN=$WORKSPACE/Unity/Assets/AddressableAssetsData/iOS/addressables_content_state.bin
15.  
16. # Print byte count (for proof that the file exists in the log)
17. wc -c $ADDRESSABLES_BIN
18.  
19. # Send addressables bin file to firebase function
20. curl \
21.   -X POST \
22.   -H "Content-Type: multipart/form-data" \
23.   -F "buildNumber=$CLOUD_BUILD_NUMBER" \
24.   -F "file=@$ADDRESSABLES_BIN" \
25.   https:/XXXX.cloudfunctions.net/my_upload_function
26.  

The bin file and build number are submitted to a custom firebase function that simply saves the file to firebase storage so I can download it when I need it (when I want to update remote addressables for a production build).

Two things to note:

• I've hardcoded the iOS path, will make it dynamic later
  
• There's surely a better way to write that json-parsing regex

It would be really nice if there were ANY documentation at all on shell scripting for Unity Cloud Build. Even something as simple as getting the cloud build number requires a lot of effort and snooping around in the dark. I pretty much had to hack this together over about 20 build iterations (partly just because I suck at shell scripting), the last few of which took almost 2 hours each because the build manifest json doesn't even exist when the pre-build script is run, so I had to run the build to its completion to test the result.

Alternatively, yes, I could have written a c# script that does the same thing, but I like the idea of isolating my build logic from the rest of the codebase. It's also a heck of a lot cleaner to curl a file from the shell than to use a UnityWebRequest.

 

Thanks and you may want to cross post on cloud build sub-forum.

 

I'm still having some trouble.

I want to do a Content Update for a Cloud Build build. Since there is no option to do a Content Update on the dashboard or in any build settings that I can find, I'm trying to do the Build For Content Update locally, uploading the updated catalog and bundles to my server afterward.

To align my local project as closely as possible with the relevant cloud build I have:
1. For all Library files in the Addressables zip file downloaded from Cloud Build, I've replaced my local library folder files with them
2. I've placed the downloaded catalog/bundle files in my server data generation folder, removing all other files in that folder

When I perform Build For Content Update locally, selecting the content state bin generated during Cloud Build, an update is performed, but the built client gives errors when it tries to update with the new catalog.

My biggest issue seems to be that the updated catalog which I built locally results in a different hash for the defaultlocalgroup_assets_all bundle than the cloud build catalog.

Can anyone suggest a workaround?

Has anyone else successfully done a Content Update for a Cloud Build build?

 

Here are my steps

1. Build my project with Cloud Build
2. Confirm my local branch and Cloud Build are on the same git commit, and no files are modified locally
3. Confirm my local project is using the same Addressables Build Profile as Cloud Build (Default)
4. Confirm my local platform is set the same as Cloud Build target (iOS)
5. Confirm my local Unity version is the same as the Cloud Build target Unity version (2018.4.8)
6. Confirm the Addressables package is up to date (1.2.4)
7. Addressables Window > Build > Clean > All
8. Download Cloud Build Addressable Assets generated zip file and move ServerData to my local ServerData build folder
   • downloaded catalog: catalog_2019.09.28.22.58.33
   • downloaded catalog hash: f11ad94c95216661d89518b2f2ad22f9
   • downloaded remote bundle: remotepackedassets_assets_all_ad5d3e1ea9645181a87581baa60420b6
9. Upload ServerData from previous step to my development end point
10. Uninstall previous version of app from my test device
11. Clean install (uninstall + re-download) the Cloud Build built app from Step 1 on my test device (iPhone 8+)
12. Open app and confirm it loads correctly, reflecting assets from Step 9
13. Check downloaded Library assets
    • Library/com.unity.addressables/StreamingAssetsCopy/aa/iOS/catalog.json
    • Library/com.unity.addressables/StreamingAssetsCopy/aa/iOS/settings.json
    • Library/com.unity.addressables/StreamingAssetsCopy/aa/iOS/iOS/defaultlocalgroup_assets_all_4362a3c42058e6161d9dedb96aa7c444.bundle
14. Confirm all files are new (not replacing existing files), copy into my local project Library folder
15. Copy Cloud Build-generated addressables_content_state.bin to Assets/AddressableAssetsData/iOS/addressables_content_state.bin
    • obtained via post-build shell script that sends the file to Firebase, as described in my earlier post
16. Addressables Window > Build > Prepare for Content Update
    • Result: "No Addressable groups with a BundledAssetGroupSchema and ContentUpdateGroupScheme (with StaticContent enabled) appear to have been modified."
17. Addressables Window > Build > Build For Content Update
    • Result: Success with no relevant console warnings or errors
    • resulting catalog hash: dc25f477c8e77ab0def998a70324ccbb
    • resulting asset bundle: remotepackedassets_assets_all_ad5d3e1ea9645181a87581baa60420b6
18. Check local project Library/com.unity.addressables/StreamingAssetsCopy/aa/iOS/iOS
    • new defaultlocalgroup_assets_all_b319345d231e06fb3b7d15d01f881900.bundle created
19. Upload generated ServerData to my development end point
20. Restart the app on my test device. It fails with the following error:

Code (CSharp):

1. Exception encountered in operation Resource<IAssetBundleResource>(defaultlocalgroup_assets_all_b319345d231e06fb3b7d15d01f881900.bundle): Invalid path in AssetBundleProvider: '/var/containers/Bundle/Application/.../Data/Raw/aa/iOS/iOS/defaultlocalgroup_assets_all_b319345d231e06fb3b7d15d01f881900.bundle'.
2.  

So it seems that no matter how closely I try to align with the Cloud Build state, running Build For Content Update will ignore the defaultlocalgroup_assets_all bundle downloaded from the Cloud Build dashboard and create a new one.

Could this be a bug? How can I get my project to accept the defaultlocalgroup provided by Cloud Build?

 

The problem is about deterministic build in Unity, which only guarantees on the same machine. See more discussions on,
https://forum.unity.com/threads/deterministic-asset-bundle-building.703826/
https://forum.unity.com/threads/randomized-non-reproducible-hashes-in-asset-bundle-names.731771/

When run a content update build, the system calculates the hash for the default bundle on the local machine, and turns out different from the cloud build. So the bundle file is missing, then it builds the bundle for you. I'm not an export for content update workflow, but you may want to try no hash for the default group, to see how the system reacts to it.

 

Wow, thanks for the solid advice! I'll definitely try that and post back here with how it goes.

 

Finally got around to testing this all the way through and unfortunately it still does not work. Here is the error in case anyone is interested

Code (CSharp):

1. Exception encountered in operation Resource<IAssetBundleResource>(defaultlocalgroup_assets_all.bundle): Unknown error in AsyncOperation
2. Exception encountered in operation Dependencies: Unknown error in AsyncOperation
3. Exception encountered in operation Resource<GameObject>(BackButtonLayer.prefab): Dependency Exception

So until something on the Addressables or Cloud Build side of things changes, I have to conclude that it is impossible to perform a Content Update for Addressables content for a Cloud Build installation of the app.

So I suppose there are four possible workarounds for me.

1. Accept that I will never be able to perform a content update. Addressables loses some of its meaning in this scenario, but still allows me to get the build size down and to avoid asset bundle dependency/loading chaos.
2. Accept that I must perform production builds locally. I loathe building with XCode. I despise it. And it despises me. Every time I open it to do something I find myself crawling under a rock, shaking for about a week for reasons that I'm never totally clear about.
3. I can whine to the Addressables team and the Cloud Build team and hope that they resolve this issue (after all, it does seem like a pretty significant flaw).
4. I might win the lottery, in which case I could stop coding/building myself and just hire people to do everything for me.

I'll probably go with a combination of 1, 3 and 4. (Yes, I hate XCode that much)

 

Thank you for this thread, I was in a similar position today and this helped a lot with understanding the problem.

I might have found a solution for the problem but since we don't urgently need to do a content update for addressables for an app built by cloud build I won't invest the time into actually testing / implementing it.

The idea is to:

1. Use a Pre-Export Method that loads the addressables_content_state.bin file from your server (for example using FTP).
2. If the file is available you use it to build addressables for content update (this is the part that would need some work but according to this threat it should be possible: https://forum.unity.com/threads/prepare-for-content-update-by-script.694873/).
3. Otherwise you do a clean build as described in this script:
   :https://gist.github.com/favoyang/cd2cf2ed9df7e2538f3630610c604c51
4. After building your addressables you would have to upload both the addressables_content_state.bin and the catalog hash / json files as well as the bundle files to your remote server.

Limitations: You would still have to trigger a full build for the content update to happen because you can't simply call the script on the build server. Maybe you could somehow make the build stop after building the addressables though.

Did you find anything new yourself or did you abandon the ideas after your last post?

Cheers!

 

Bump! Any news from Unity guys here?

 

Is there any update regarding this issue?