Question Building multiple catalogs

Let's say I want optional DLC to be packed up separately or set up a Unity project that players can use to create mods / extension packages. The way to do this using Addressables would be to pack them into a catalog. I could then have users install them to a predetermined directory and my game detects them and loads them at will using LoadContentCatalogAsync, is that right?

The default Addressable build pipeline (BuildScriptPacked) is currently hardcoded to build a single catalog (catalog.json) that contains all asset groups. I realize I can write my own build script - but that reads way easier than it is, because nearly all useful functionality in the Addressables package is declared as internal, and so using it would mean going through reflection hell. I can also just embed the package into my project and edit it directly, but that means that updating Addressables will cause extra work in the future.

So before I start with any of that, in case I missed something, is there already any way to build multiple catalogs, each containing a selected set of asset groups? In my case, in fact, I'd actually want one catalog per asset group.

 

For anyone interested, I managed to implement a post processing of the built catalog to split it into multiple catalogs. I had to modify the Addressables source code to get this done, because reflection alone didn't help here (many build steps are embedded in one massive function DoBuild).

In the end, it's now working exactly as I inteded, but it'd be nice if in the future, Addressables were to support building a separate catalog from an asset group.

 

Hi @pdinklag

I'm pretty interested in how you accomplished this as I'm a bit in the same boat. We have an optional DLC package which we also put in a separate directory. We currently do this by making 2 addressable builds. One for the main content and one for the DLC content. While it currently works for us, our DLC builds this way grew very large in size and are pain in the ass so I'm curious how you managed to do this, if you're willing to share, of course.

Thanks in advance!

 

Alright, I'll do some commenting and make a git patch for the Addressables package in the coming days. This is a bit tricky because unlike other packages, afaik, the Addressables package doesn't have an official public code repository on Github, and I'm not familiar enough with the license (Unity Companion License) to go ahead and upload the entire modified package. Maybe if a Unity representative stumbles on this, can you answer whether that would be OK?

In any event, I didn't make too many modifications - it's all in that big ol' function BuildScriptPacked.DoBuild that I mentioned. Please give me up to a week for this!

 

Thanks!

In case no github repo is available, I'm also open to receive your modifications over email if the license allows it. I'm not afraid to dive into the code as I did quite some modifications myself and I'm somewhat familiar with it.
If you want to get it touch, please send me at jan@lugus-studios.be.

Much appreciated!

 

The companion license seems to be fine, it's just that Unity pretty much owns whatever is released under it. But that's OK. I forked a mirror of Addressables on Github and uploaded my changes there, including an example and instructions: (deprecated, use LuGus-Jan's version)

The multiple_catalogs branch is based off of 1.16.15 as 1.16.16 was causing trouble (see other threads around here) and I don't want to use the 1.17 previews yet.

I'd like to keep this public because I feel it's a feature that Addressables definitely need. After all, this is also ultimately the answer to Modding with Addressables. So if there's any questions, please ask them here and I'll try to clarify.

 

Thanks a ton!
I'll have a look at this over the course of this week! If this pans out to work in our case as well, this might be a nice feature (request) for the Addressables team!

 

I wanted to thank you for the work since it was pretty easy to transform to our own build system.
I did run into some trouble though, and I was wondering whether you encountered the same.

Did you add any scenes to the external/additional catalogs? It seems like 'base' assets load fine, but scenes have a lot of missing script references. Loading the same scenes from the main catalog when I put them in that one seem to work fine.

It seems like this is our final blocker. Perhaps this is what you refer to with issues in 1.17? I used 1.17.17. Maybe I should give 1.16.15 a try then.

 

Glad it's of any help!
I simply didn't try 1.17 yet, now that it's out of preview I'll update soon.

However, I also didn't test my modifications against loading scenes. Did you get this warning by any chance when building a catalog containing scenes? It may be as simple as getting the load order of scripts right if scripts are contained in another catalog. If the scripts are contained in your project maybe I missed something.

 

I managed to figure it out. It indeed had to do with that warning. It seems like in your modifications you don't look up for any further dependencies if they can't be found inside the groups of the new catalog.

Since we have a lot of scenes depending on assets in the main catalog, it's vital to us that they are included. So I added the dependency asset entries to the new/additional catalog as well, but kept the entries tied to the main catalog.

My other issue was with how I loaded in the additional catalog, for which I added a provider suffix. This causes the dependencies in the additional catalog to not resolve with the assets in the main catalog. So removing this provider suffix made everything work for us.

One other modification I did to your work was to define which asset groups should be separated in an additional catalog, rather than working with name prefixes.

If you want and are interested in these changes I made, I can try and make a PR to your repository.

 

Is there an upper limit on the number of catalogs you would want to load? How about 500 or 1000? Has anyone ran stress tests at that scale?

 

Could you provide some more details on what you mean by "dependency asset entries?" Thanks!

 

Cool!

I have the same question as @chorst_genies really. To be frank, I was too lazy to dig into how exactly dependencies are stored and postponed that for later, so a PR would be a good insight!

No stress tests yet. For Unity, it shouldn't make a difference whether you have one huge catalog or many small catalogs. However, if assets are split across many catalogs, loading routines gathering them from different files / disk locations may cause a considerable impact.

 

@pdinklag @chorst_genies Of course. I created a pull request on your multiple catalogs, @pdinklag

My case on this is that we have a main catalog and a 'DLC' catalog, which contains assets that may depend on assets found in the main catalog. Previously, our DLC catalog and bundles had duplicate assets copied with them, almost as having these DLC packs function as 'standalone' packages. This made our build times extremely long, and users had to download a lot of data every time we performed an update to this DLC. So wanted something so that the DLC catalog could depend on certain assets being present in the game, reducing size, build time and update size.

@pdinklag gave a huge leg up in the starting point for this. The only issue remaining seemed to be that dependency assets couldn't be found. And looking through the code, they're also discarded... So that made sense... (And also integrating it in our build process/tools, which is the reason from stepping from the name prefixes)

To resolve the dependencies I actually just brute-forced dependency IDs of every asset to be included in the external catalog to match with one of the keys found in the default catalog. So, let's say that I have a scene, which in turn uses a lot of addressable assets, e.g. pause menu prefab, music player prefab, etc., then the scene depends on these assets being available (and linked in the external catalog). So in processing the asset entries that belong to the external catalog, I check their dependencies, and look these up in the default catalog. If I find a match, I literally copy the entry and include it in the external catalog as well (just the asset entry, I don't want to duplicate the actual bundle data as well, so that no conflicts arise in loading the same bundle twice (which Unity does complain about).

Link to the PR: https://github.com/pdinklag/com.unity.addressables/pull/1

Code (CSharp):

1. // Process dependencies
2. foreach (CatalogSetup additionalCatalog in catalogSetups)
3. {
4.     // Process dependencies recursively, and only once.
5.     var dataEntries = new Queue<ContentCatalogDataEntry>(additionalCatalog.BuildInfo.Locations);
6.     var processedEntries = new HashSet<ContentCatalogDataEntry>();
7.     while (dataEntries.Count > 0)
8.     {
9.         var dataEntry = dataEntries.Dequeue();
10.  
11.         // If already processed or no dependencies, then skip.
12.         if (!processedEntries.Add(dataEntry) || (dataEntry.Dependencies == null) || (dataEntry.Dependencies.Count == 0))
13.         {
14.             continue;
15.         }
16.  
17.         foreach (var entryDependency in dataEntry.Dependencies)
18.         {
19.             // Search for the dependencies in the default catalog only.
20.             var depLocation = defaultCatalog.Locations.Find(loc => loc.Keys[0] == entryDependency);
21.             if (depLocation != null)
22.             {
23.                 dataEntries.Enqueue(depLocation);
24.  
25.                 // If the dependency wasn't part of the catalog yet, add it.
26.                 if (!additionalCatalog.BuildInfo.Locations.Contains(depLocation))
27.                 {
28.                     additionalCatalog.BuildInfo.Locations.Add(depLocation);
29.                 }
30.             }
31.             else if (!additionalCatalog.BuildInfo.Locations.Exists(loc => loc.Keys[0] == entryDependency))
32.             {
33.                 Debug.LogErrorFormat("Could not find location for dependency ID {0} in the default catalog.", entryDependency);
34.             }
35.         }
36.     }
37. }

 

Thank you so much for sharing this! I was wondering if there is a way to generate hash files for the extra catalogs? It doesn't seem to do it by default and I think this will prevent the loading app from caching any bundles on device.

 

I think it's possible to generate the hash files, but they weren't needed in our usecase, so I didn't spend time looking into that. Having a quick look at where they're generated, I'd say you'd have to fiddle a bit with the for-each looping over the catalogs to create their file contents, perhaps setting something additional in the

CatalogSetup

class that denotes it's a remote one that needs a hash file to be generated and then adjusting the

CreateCatalogFiles

method accordingly.

 

Thanks for your quick reply! I'll have a look at that. Also, perhaps I can just copy the hash generated for the default catalog and use it for the additional ones too. Doesn't seem like it needs to be unique between catalogs just unique between versions to denote an update is needed?

I do have one other question though... It looks like the default catalog that is generated isn't including the correct assets? I have the following setup:

DefaultLocalGroup (Should be in default catalog)
bracelet-0001-bangle (Assigned to additional CatalogContentGroup 1)
mask-0014-panda (Assigned to additional CatalogContentGroup 2)

I would expect the main default catalog generated to (only) have the DefaultMaterial from the DefaultLocalGroup but for some reason it seems like a copy of mask-0014-panda's catalog. It only has the mask prefabs and some dependencies in there.

I can't find "DefaultMaterial" in any of the generated catalogs.

 

In case anyone interested, I was able to build/load multiple catalogs in slightly different way, but still using 1 project. I does require modifying Addressables package but instead of relying on just separate catalog override it extends Unity's implementation by allowing multiple settings instead of 1 static. That come with the whole list of benefits and you can cycle through those different settings in the Settings window, have different profiles, addressables assets and categories all referenced and used safely.

 

I'm having great success with pdinklag / LuGus-Jan solution, but I'm interested to hear more about your solution.

 

I have to second @jeppe79 here, always good to see more solutions, especially seeing that mine is probably due a rebase to the latest Addressables version and I also didn't really look at @LuGus-Jan 's modifications yet.

 

Just coming back here to applaud @LuGus-Jan for creating an intuitive workflow around this, keeping it up to date with newer Addressables versions and providing a documentation. I'm going to archive my repository now and will migrate to his. ( https://github.com/juniordiscart/com.unity.addressables )

 

The fork for multiple addressable settings is located here:
https://github.com/theokcompany/com.unity.addressables

There are a few parts to achieve building of completely separate catalogs including remote support:
1) There is no easy way around having 1 addressable settings object always loaded. AddressableAssetSettingsDefaultObject is the main reason and it's a static reference. However, in this PR: https://github.com/theokcompany/com.unity.addressables/pull/1/files I extended it to instead allow multiple settings by only 1 loaded as a default. This is basically facilitated by a new serializable class which the above class a list of

 public class AddressableAssetSettingsDefaultPair 

 {

        public AddressableAssetSettings addressableAssetSettings;

        public bool isDefault;

 }

2) To support Editor fast initialization mode you need to build all settings, not just one. This PR achieves it by naively iterating over the above list: https://github.com/theokcompany/com.unity.addressables/pull/2/files

3) Optional step, but adding an additional output folder based on active profile id: https://github.com/theokcompany/com.unity.addressables/pull/3/files

4) Optional step, but highly recommended. Unity hardcodes default catalog id in whatever active settings is build. There are ways to override it, but the simplest solution is to go in the above AddressableAssetSettingsDefaultPair and add an optional catalogAddress string and override

var contentCatalog = new ContentCatalogData(ResourceManagerRuntimeData.kCatalogAddress);

with it.
https://github.com/theokcompany/com.unity.addressables/pull/4/files

After these changes, you now can add multiple settings each with it's own data builders and content.

 

For reference the above AddressableAssetSettingsDefaultObject looks like this with 2 catalogs which then can by cycled through using isDefault check.

 

Hi, @AndreyD ! Thanks for providing your solution. I tried on an empty project but when I hit the create addressables settings I run into an error. The AddressableAssetSettingsDefaultObject.Instance is null when the create is getting called

 

Hi, @AndreyD！Thanks for providing your solution. I am testing it. I found that you change "addressables_content_state" from binary format to JSON. But the convert had bugs, many keys' values are null. It will make data update check not work. I tried use Newton.JSON to solve it. It does not work either.

 

Hey all, I'm having trouble to use @LuGus-Jan's Multiple catalog version of addressables.
rep link: https://github.com/juniordiscart/com.unity.addressables

1.I build one additional addressables group beside default and built-in groups and add some prefabs in this groups (default and additional).


2.Create an ExternalCatalogSetup change its properties


3.Add this "ExternalCatalogSetup" to "BuildScriptPackedMultiCatalogMode" scriptable object and included it in build and play mode scripts

after building with multi-catalog build script i get this errors:

Code (CSharp):

1. D:\Projects\Multi-Catalog\Library\com.unity.addressables\aa\Windows\StandaloneWindows64\additionalgroup_assets_all_9d95d8c2ebcd569d278ee174f833d47c.bundle does not exist
2. UnityEditor.GenericMenu:CatchMenu(Object, String[], Int32)

Code (CSharp):

1. Addressable content build failure (duration : 0:00:00)
2. UnityEditor.GenericMenu:CatchMenu(Object, String[], Int32)

Does anybody knows anything about this errors?

 

@m1mostwanted

Everything seems to be set up correctly from what I can see in the shots. The only thing I'm not sure about is that you're building your external catalog to the same location as where the default content is placed.
Does this still happen if you move the build path to a different location that is outside the Library folder, e.g. to a folder on your Desktop?

 

I changed the path but still i get the errors, Can you give me a glimpse of what you doing in code so maybe I know where should I start to investigate my problem?

 

All code is publicly accessible and you can check out which parts have been changed on my end in this repo. But perhaps you can send the project as an issue to the repo with some concise description of how to reproduce it?

 

@LuGus-Jan Hey - I'll abuse this thread to pick your brain about your repo; thanks a lot for making it public and this really should be built-in functionality.

I've managed to build a DLC test package and catalog, but the multi-catalog builder doesn't seem to handle addressable folders with subassets. The first problem seems to be in ExternalCatalogSetup.cs:IsPartOfCatalog() and is fixed by:

Code (CSharp):

1. //                                      return assetGroups.Exists(ag => ag.entries.Any(e => loc.Keys.Contains(e.guid)));
2.                                         return assetGroups.Exists(ag => ag.entries.Any(e => loc.Keys.Contains(e.guid) || (e.IsFolder && e.SubAssets.Any(a => loc.Keys.Contains(a.guid)))));
3.  

Another problem is that if the group only contains folders with subassets, the bundle is not moved to the build path together with the catalog at the end of the build. This seems to happen because CatalogSetup.Files is empty, which results in BundleFileId not getting set for the AddressableAssetEntry, but that's as far as I got before I ran out of brain with the default build script.

I can work around these issues, but I thought I'd mention them here in case you have some ideas.

 

I don't have experience using sub assets in folders but I'll adjust the code to include your fix. Thanks for spotting this!

Regarding the moving of bundles, do you have a small repro example I could have a look at?

 

A simple repro case is:

1. Make a folder with some asset in it, and mark the *folder* as addressable.
2. Move the folder into an asset group that has been added to the external catalog definition, so that the asset group only contains the addressable folder and no direct asset references:

Now when you build addressables, the catalog is moved to the build path defined in the external catalog setup, but the bundle stays in the default build location with other bundles.
This is easily worked around by including a dummy asset in the group.

 

@GroundCombo Apologies for taking a long time to get this fixed. In the 1.20.3 of the multi-catalog variant of Addressables, both of your reported issues should've been fixed.

 

No worries, thanks again! I'll definitely give this a spin once I get to implementing our DLC.

 

@LuGus-Jan I follow the setting up of the multi catalog to Default building script - Multi-Catalog, but only get the catalog as normal. is there a example project like the picture show on github?

 

There's no demo project. I haven't found the time to set that up.

Have you assigned the external catalog objects to the data builder as well?

 

Hi~ i found if set Remote.LoadPath the same with Remote.BuildPath, it work.




if set different , i got error :

Could not find file '/Users/......./AddressableTest/http:/192.168.2.156:8080/AiProject/StandaloneOSX/dlc_01_assets_all_848583625fc259ac9c88fd2b07e735de.bundle'.

But dlc_01_assets_all_848583625fc259ac9c88fd2b07e735de.bundle is in the Remote.BuildPath



it seem the error happen when try to copy file from build path to extenal catalog path

 

Ah, you're building remote catalogs. Check your console, there should be a warning that's logged that remote catalogs are not supported (BuildScriptPackedMode.cs on line 560).

I have never used remote loading of catalogs, and the person who laid the foundation for this work of splitting up catalogs also didn't need this functionality, so this didn't get implemented. We both needed the functionality for 'traditional DLC' stuff where the content is installed locally (e.g. by Steam or another launcher) and a license check is done before it gets loaded by the game.

I'm also not keen on implementing this functionality myself since I have personally no experience with all of the possible edge cases and workarounds that currently exist or are needed, since it's heavily tied in with the Addressable's feature of updating an existing build.

Perhaps I should add a warning to the read me file, that this is currently not supported. Feel free to fork from my repo though if you want to have a go at adding support for this feature.