Search Unity

  1. Get the latest news, tutorials and offers directly to your inbox with our newsletters. Sign up now.
    Dismiss Notice

Help Wanted Building multiple catalogs

Discussion in 'Addressables' started by pdinklag, Feb 6, 2021.

  1. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    99
    Let's say I want optional DLC to be packed up separately or set up a Unity project that players can use to create mods / extension packages. The way to do this using Addressables would be to pack them into a catalog. I could then have users install them to a predetermined directory and my game detects them and loads them at will using LoadContentCatalogAsync, is that right?

    The default Addressable build pipeline (BuildScriptPacked) is currently hardcoded to build a single catalog (catalog.json) that contains all asset groups. I realize I can write my own build script - but that reads way easier than it is, because nearly all useful functionality in the Addressables package is declared as internal, and so using it would mean going through reflection hell. I can also just embed the package into my project and edit it directly, but that means that updating Addressables will cause extra work in the future.

    So before I start with any of that, in case I missed something, is there already any way to build multiple catalogs, each containing a selected set of asset groups? In my case, in fact, I'd actually want one catalog per asset group.
     
  2. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    99
    For anyone interested, I managed to implement a post processing of the built catalog to split it into multiple catalogs. I had to modify the Addressables source code to get this done, because reflection alone didn't help here (many build steps are embedded in one massive function DoBuild).

    In the end, it's now working exactly as I inteded, but it'd be nice if in the future, Addressables were to support building a separate catalog from an asset group.
     
  3. LuGus-Jan

    LuGus-Jan

    Joined:
    Oct 3, 2016
    Posts:
    62
    Hi @pdinklag

    I'm pretty interested in how you accomplished this as I'm a bit in the same boat. We have an optional DLC package which we also put in a separate directory. We currently do this by making 2 addressable builds. One for the main content and one for the DLC content. While it currently works for us, our DLC builds this way grew very large in size and are pain in the ass so I'm curious how you managed to do this, if you're willing to share, of course. :)

    Thanks in advance!
     
  4. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    99
    Alright, I'll do some commenting and make a git patch for the Addressables package in the coming days. This is a bit tricky because unlike other packages, afaik, the Addressables package doesn't have an official public code repository on Github, and I'm not familiar enough with the license (Unity Companion License) to go ahead and upload the entire modified package. Maybe if a Unity representative stumbles on this, can you answer whether that would be OK?

    In any event, I didn't make too many modifications - it's all in that big ol' function BuildScriptPacked.DoBuild that I mentioned. Please give me up to a week for this! :)
     
  5. LuGus-Jan

    LuGus-Jan

    Joined:
    Oct 3, 2016
    Posts:
    62
    Thanks!

    In case no github repo is available, I'm also open to receive your modifications over email if the license allows it. I'm not afraid to dive into the code as I did quite some modifications myself and I'm somewhat familiar with it. :)
    If you want to get it touch, please send me at jan@lugus-studios.be.

    Much appreciated! :)
     
  6. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    99
    The companion license seems to be fine, it's just that Unity pretty much owns whatever is released under it. But that's OK. I forked a mirror of Addressables on Github and uploaded my changes there, including an example and instructions: https://github.com/pdinklag/com.unity.addressables/tree/multiple_catalogs

    The multiple_catalogs branch is based off of 1.16.15 as 1.16.16 was causing trouble (see other threads around here) and I don't want to use the 1.17 previews yet.

    I'd like to keep this public because I feel it's a feature that Addressables definitely need. After all, this is also ultimately the answer to Modding with Addressables. So if there's any questions, please ask them here and I'll try to clarify. :)
     
  7. LuGus-Jan

    LuGus-Jan

    Joined:
    Oct 3, 2016
    Posts:
    62
    Thanks a ton! :D
    I'll have a look at this over the course of this week! If this pans out to work in our case as well, this might be a nice feature (request) for the Addressables team!
     
  8. LuGus-Jan

    LuGus-Jan

    Joined:
    Oct 3, 2016
    Posts:
    62
    I wanted to thank you for the work since it was pretty easy to transform to our own build system. :)
    I did run into some trouble though, and I was wondering whether you encountered the same.

    Did you add any scenes to the external/additional catalogs? It seems like 'base' assets load fine, but scenes have a lot of missing script references. Loading the same scenes from the main catalog when I put them in that one seem to work fine.

    It seems like this is our final blocker. Perhaps this is what you refer to with issues in 1.17? I used 1.17.17. Maybe I should give 1.16.15 a try then.
     
  9. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    99
    Glad it's of any help! :)
    I simply didn't try 1.17 yet, now that it's out of preview I'll update soon.

    However, I also didn't test my modifications against loading scenes. Did you get this warning by any chance when building a catalog containing scenes? It may be as simple as getting the load order of scripts right if scripts are contained in another catalog. If the scripts are contained in your project maybe I missed something.
     
  10. LuGus-Jan

    LuGus-Jan

    Joined:
    Oct 3, 2016
    Posts:
    62
    I managed to figure it out. It indeed had to do with that warning. It seems like in your modifications you don't look up for any further dependencies if they can't be found inside the groups of the new catalog.

    Since we have a lot of scenes depending on assets in the main catalog, it's vital to us that they are included. So I added the dependency asset entries to the new/additional catalog as well, but kept the entries tied to the main catalog.

    My other issue was with how I loaded in the additional catalog, for which I added a provider suffix. This causes the dependencies in the additional catalog to not resolve with the assets in the main catalog. So removing this provider suffix made everything work for us. :)

    One other modification I did to your work was to define which asset groups should be separated in an additional catalog, rather than working with name prefixes. :)

    If you want and are interested in these changes I made, I can try and make a PR to your repository.
     
    chorst_genies likes this.
  11. chorst_genies

    chorst_genies

    Joined:
    Jun 3, 2020
    Posts:
    7
    Is there an upper limit on the number of catalogs you would want to load? How about 500 or 1000? Has anyone ran stress tests at that scale?
     
  12. chorst_genies

    chorst_genies

    Joined:
    Jun 3, 2020
    Posts:
    7

    Could you provide some more details on what you mean by "dependency asset entries?" Thanks!
     
  13. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    99
    Cool! :)

    I have the same question as @chorst_genies really. To be frank, I was too lazy to dig into how exactly dependencies are stored and postponed that for later, so a PR would be a good insight!

    No stress tests yet. For Unity, it shouldn't make a difference whether you have one huge catalog or many small catalogs. However, if assets are split across many catalogs, loading routines gathering them from different files / disk locations may cause a considerable impact.
     
  14. LuGus-Jan

    LuGus-Jan

    Joined:
    Oct 3, 2016
    Posts:
    62
    @pdinklag @chorst_genies Of course. :) I created a pull request on your multiple catalogs, @pdinklag

    My case on this is that we have a main catalog and a 'DLC' catalog, which contains assets that may depend on assets found in the main catalog. Previously, our DLC catalog and bundles had duplicate assets copied with them, almost as having these DLC packs function as 'standalone' packages. This made our build times extremely long, and users had to download a lot of data every time we performed an update to this DLC. So wanted something so that the DLC catalog could depend on certain assets being present in the game, reducing size, build time and update size.

    @pdinklag gave a huge leg up in the starting point for this. The only issue remaining seemed to be that dependency assets couldn't be found. And looking through the code, they're also discarded... So that made sense... :p (And also integrating it in our build process/tools, which is the reason from stepping from the name prefixes)

    To resolve the dependencies I actually just brute-forced dependency IDs of every asset to be included in the external catalog to match with one of the keys found in the default catalog. So, let's say that I have a scene, which in turn uses a lot of addressable assets, e.g. pause menu prefab, music player prefab, etc., then the scene depends on these assets being available (and linked in the external catalog). So in processing the asset entries that belong to the external catalog, I check their dependencies, and look these up in the default catalog. If I find a match, I literally copy the entry and include it in the external catalog as well (just the asset entry, I don't want to duplicate the actual bundle data as well, so that no conflicts arise in loading the same bundle twice (which Unity does complain about).

    Link to the PR: https://github.com/pdinklag/com.unity.addressables/pull/1

    Code (CSharp):
    1. // Process dependencies
    2. foreach (CatalogSetup additionalCatalog in catalogSetups)
    3. {
    4.     // Process dependencies recursively, and only once.
    5.     var dataEntries = new Queue<ContentCatalogDataEntry>(additionalCatalog.BuildInfo.Locations);
    6.     var processedEntries = new HashSet<ContentCatalogDataEntry>();
    7.     while (dataEntries.Count > 0)
    8.     {
    9.         var dataEntry = dataEntries.Dequeue();
    10.  
    11.         // If already processed or no dependencies, then skip.
    12.         if (!processedEntries.Add(dataEntry) || (dataEntry.Dependencies == null) || (dataEntry.Dependencies.Count == 0))
    13.         {
    14.             continue;
    15.         }
    16.  
    17.         foreach (var entryDependency in dataEntry.Dependencies)
    18.         {
    19.             // Search for the dependencies in the default catalog only.
    20.             var depLocation = defaultCatalog.Locations.Find(loc => loc.Keys[0] == entryDependency);
    21.             if (depLocation != null)
    22.             {
    23.                 dataEntries.Enqueue(depLocation);
    24.  
    25.                 // If the dependency wasn't part of the catalog yet, add it.
    26.                 if (!additionalCatalog.BuildInfo.Locations.Contains(depLocation))
    27.                 {
    28.                     additionalCatalog.BuildInfo.Locations.Add(depLocation);
    29.                 }
    30.             }
    31.             else if (!additionalCatalog.BuildInfo.Locations.Exists(loc => loc.Keys[0] == entryDependency))
    32.             {
    33.                 Debug.LogErrorFormat("Could not find location for dependency ID {0} in the default catalog.", entryDependency);
    34.             }
    35.         }
    36.     }
    37. }
     
    Last edited: Apr 30, 2021
    pdinklag and chorst_genies like this.
  15. chorst_genies

    chorst_genies

    Joined:
    Jun 3, 2020
    Posts:
    7
    Thank you so much for sharing this! I was wondering if there is a way to generate hash files for the extra catalogs? It doesn't seem to do it by default and I think this will prevent the loading app from caching any bundles on device.
     
  16. LuGus-Jan

    LuGus-Jan

    Joined:
    Oct 3, 2016
    Posts:
    62
    I think it's possible to generate the hash files, but they weren't needed in our usecase, so I didn't spend time looking into that. Having a quick look at where they're generated, I'd say you'd have to fiddle a bit with the for-each looping over the catalogs to create their file contents, perhaps setting something additional in the
    CatalogSetup
    class that denotes it's a remote one that needs a hash file to be generated and then adjusting the
    CreateCatalogFiles
    method accordingly.
     
    chorst_genies likes this.
  17. chorst_genies

    chorst_genies

    Joined:
    Jun 3, 2020
    Posts:
    7
    Thanks for your quick reply! I'll have a look at that. Also, perhaps I can just copy the hash generated for the default catalog and use it for the additional ones too. Doesn't seem like it needs to be unique between catalogs just unique between versions to denote an update is needed?

    I do have one other question though... It looks like the default catalog that is generated isn't including the correct assets? I have the following setup:

    DefaultLocalGroup (Should be in default catalog)
    bracelet-0001-bangle (Assigned to additional CatalogContentGroup 1)
    mask-0014-panda (Assigned to additional CatalogContentGroup 2)

    I would expect the main default catalog generated to (only) have the DefaultMaterial from the DefaultLocalGroup but for some reason it seems like a copy of mask-0014-panda's catalog. It only has the mask prefabs and some dependencies in there.

    I can't find "DefaultMaterial" in any of the generated catalogs.

    upload_2021-5-12_17-24-50.png
     
    Last edited: May 13, 2021 at 1:34 AM
unityunity