Feedback Automatic duplicate asset resolving

So we have a project with a lot of art assets (textures, animations, etc.), not all of them are being used in the final build, so naturally we want Unity to resolve the dependency tree from the scenes and root assets we load explicitly down to the individual shaders, animations, and sprites being used.

Another requirement is we need to split our content into bundles based on when it's needed (when loading the menu scene, when going into action, etc.). But here comes the problem - duplicate assets.

Let's say there are Asset A which references Asset B which references Asset C (so A->B->C). Now we have a bundle 1 which has asset A explicitly referenced via Addressables location, and also we have bundle 2 which has asset B explicitly. Now as you may know, this will result in asset C being duplicated in both bundles 1 and 2.

The solution currently being offered means going to Addressables Analyze and pressing Analyze then Fix on the "Check Duplicate Bundle Dependencies". Which will put asset C explicitly into its own group. This has issues:

1. This is manual action that needs to be done before every build. This is not an option for CI pipeline, so we would have to dig into source code and try & run this operation via script, then copy the result assets from the "Duplicate Asset Isolation" group that it creates into our own pre-configured group.
2. It places all duplicate assets into one group, which is also not ideal for certain configurations. We might have thousands of duplicates due to how asset groups are used.

Another solution might be to explicitly place entire folders of art assets into separate groups so that we don't have to use Analyze before every build, but it's also not a good option because it will not check if asset is referenced anywhere at all and ends up bloating the final build with unused assets.

So this begs the question.. why doesn't Addressables automatically resolves asset duplicates during build? It might have used the obvious solution of placing the duplicated asset into the bundle with the shortest reference path. So that means if I added a Scene with an FBX into one bundle, and FBX itself into another bundle, it's obvious where I would want to store the art that FBX uses...

Is there any such solution planned? Seems like another feature that's needed for practically 99% of projects.

 

Great suggestion.

I would love to have this as well, it would greatly improve the addressables build workflow.

I think the solution presented would work really well for my project and be a standout benefit of addressables over a normal assetbundle workflow.

 

How about adding automatic dependency resolution as an option that defaults to the current behavior? In our case we don't have two groups with the same explicit entries, duplicates only happen because of implicit dependencies, so just following the shortest path rule will be perfect. I'm sure it will be beneficial for many projects.

Having to add duplicate dependencies explicitly into serialized group data seems like a hack, to be honest. Ideally you'd want to setup your project with a given structure (based on folders, etc) and then never touch Asset Groups again until you need significant structural changes in the project. I can't imagine asking artists or designers to go and add something manually to Asset Groups... this is just not going to work on bigger project.

 

Was there any progress made on this issue?
The main problem we are facing with the current de-duplication system is the following:

You have group: LocalLevel1Assets, RemoteLevel2Assets and RemoteLevel3Assets

Now there is a certain duplication between RemoteLevel2Assets+RemoteLevel3Assets, however those assets are not referenced in LocalLevel1Assets - the current algorithm still puts everything in the local "Duplicate Asset Isolation" group which will be included in the main build. However in reality it would be much better to create a "Remote Duplicate Asset Isolation" group for Level2+Level3 shared assets.

The minimum viable solution would be to have a rule create 2 duplicate asset isolations (1 for local, 1 for remote)

What we would love would be "automatic asset groups" where we can define which content can be de-duplicated by using those (e.g. a Level2Level3DynamicDeduplicationGroup). The rest of the content would still be part of the "Duplicate Asset Isolation" - Those automatic asset groups could be manually configured to de-duplicate certain other asset groups, and would be automatically updated with every build.

What could also help would be a grouping view in the Duplicate Asset Isolation Group:
> Prefab1 [LocalLevel1Assets, RemoteLevel2Assets]
-- Mesh1
-- Mesh2
-- TextureA
> Scene1 [RemoteLevel2Assets, RemoteLevel3Assets]
-- ..
and even allow us to ideally move a whole asset group to a different duplicate asset isolation group

 

I'm not sure if this is useful to you or not but I've used this to help cleanup after fixing the duplicates (i.e. if you already have a Duplicate Asset Isolation group and then you do it again, they end up in Duplicate Asset Isolation1:

Code (CSharp):

1. #if UNITY_EDITOR
2. using System.Collections.Generic;
3. using System.Linq;
4. using ArtificialArtists.Tools;
5. using UnityEditor;
6. using UnityEditor.AddressableAssets.Settings;
7. using UnityEngine;
8.  
9. namespace ArtificialArtists
10. {
11.     public class DuplicateAssetIsolationCleanup
12.     {
13.         [MenuItem("AA_Tools/Builds/Clean Up Dependencies")]
14.         private static void CleanUpDepenedcies()
15.         {
16.             AddressableAssetGroup actualDependencyGroup = null;
17.             AddressableAssetGroup temporaryDependencyGroup = null;
18.             AddressableAssetSettings addressableSettings = AA_Tools.GetAddressablesSettings();
19.            
20.             foreach (var addressableAssetGroup in addressableSettings.groups)
21.             {
22.                 switch (addressableAssetGroup.name)
23.                 {
24.                     case "Duplicate Asset Isolation":
25.                         actualDependencyGroup = addressableAssetGroup;
26.                         break;
27.                     case "Duplicate Asset Isolation1":
28.                         temporaryDependencyGroup = addressableAssetGroup;
29.                         break;
30.                 }
31.             }
32.  
33.             if (temporaryDependencyGroup == null)
34.             {
35.                 Debug.LogWarning("No Extra Duplicates Group found");
36.                 return;
37.             }
38.  
39.             addressableSettings.MoveEntries(temporaryDependencyGroup.entries.ToList(), actualDependencyGroup);
40.             addressableSettings.RemoveGroup(temporaryDependencyGroup);
41.         }
42.     }
43. }
44. #endif

 

I was wondering if there is a way to analyse and then fix the duplicates by script without using reflect. I'm building out our dev-ops pipeline and this process currently involves manually clicking buttons ‍♀️