Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

Resolved Finding duplicates missed by standard addressables analyze rule

Discussion in 'Content Pipeline Dev Blitz Day 2023 - Q&A' started by StephanieRowlinson, Jun 8, 2023.

  1. StephanieRowlinson

    StephanieRowlinson

    Joined:
    Jul 23, 2014
    Posts:
    136
    We have a lot of textures and materials in our project that were copied around outside Unity. Afterwards each copy was separately imported and as a result Unity sees them as unique objects. Of course this massively bloats our bundle size, but when I run the standard find duplicates rule it misses them.

    I want to write my own custom rule to fix this, but I'm having some trouble figuring out how to get to the relevant data for comparisons. All the samples work based on path name or the GUID of the object. Both of these are unique in our case, but the contents behind them isn't.

    How can I best go about finding these duplicates?
     
  2. davidla_unity

    davidla_unity

    Unity Technologies

    Joined:
    Nov 17, 2016
    Posts:
    736
    Hey,

    Given that those assets have been duplicated, I'm not sure what could be possible as far as detecting those. I'd probably recommend some kind of naming convention for your textures/materials that could be used to detect duplicates, or potential duplicates.

    Otherwise you could compare a content hash or something similar to see if a matching hash was already detected in your assets. I'm not sure how feasible that is and would probably be quite slow, but it might be an option.

    Maybe someone from the AssetDatabase team will be able to chime in and give some advice since this sounds like a problem space they'd be more familiar with.

    I hope that helps some
     
    StephanieRowlinson likes this.
  3. StephanieRowlinson

    StephanieRowlinson

    Joined:
    Jul 23, 2014
    Posts:
    136
    Thanks that does help! I'll go look over the AssetDatabase forum to see if anyone has tried something similar. :)
     
  4. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    175
    Hey, just in case the question didn't come up on the forums, you could do something like this, where you get all hashes in a project and then any assets with duplicated content will trigger the Debug message:

    Code (CSharp):
    1. var allAssets = AssetDatabase.GetAllAssetPaths();
    2.         var allHashes = new HashSet<Hash128>();
    3.         for (int i = 0; i < allAssets.Length; ++i)
    4.         {
    5.             var hash = AssetDatabase.GetSourceAssetFileHash(AssetDatabase.AssetPathToGUID(allAssets[i]));
    6.             if (!allHashes.Contains(hash))
    7.                 allHashes.Add(hash);
    8.             else
    9.             {
    10.                 Debug.Log($"Found duplicated item at {allAssets[i]}");
    11.             }
    12.         }
    However, AssetDatabase.GetSourceAssetFileHash is an internal API, and the only way to access it is to do the following:
    1. Make a folder called "Editor" under the "Assets" folder (so it becomes "Assets/Editor")
    2. In there, add a Testable Assembly Definition, with the name "Assembly-CSharp-Editor-testable.asmdef"
    3. The contents of that file should look like this:
    Code (CSharp):
    1. "name": "Assembly-CSharp-Editor-testable",
    2.     "references": [],
    3.     "optionalUnityReferences": [
    4.         "TestAssemblies"
    5.     ],
    6.     "includePlatforms": [
    7.         "Editor"
    8.     ],
    9.     "excludePlatforms": [],
    10.     "allowUnsafeCode": false,
    11.     "overrideReferences": false,
    12.     "precompiledReferences": [],
    13.     "autoReferenced": true,
    14.     "defineConstraints": []
    15. }
    This means that all code under this folder is to be used for "Testing" and as such, internal functions become available to your project.
     
  5. StephanieRowlinson

    StephanieRowlinson

    Joined:
    Jul 23, 2014
    Posts:
    136
    Thanks for the reply! I added this to an existing editor folder and got a bunch of errors from my editor window. I guess this means I'd need to run this code as a test too?
     
  6. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    175
    What errors are you getting?
    If there was an ASMDEF in there before then this might prove to be a little bit more tricky.
     
  7. StephanieRowlinson

    StephanieRowlinson

    Joined:
    Jul 23, 2014
    Posts:
    136
    It was an Editor Window that used Addressables and now I'm getting this

    Code (CSharp):
    1.  
    2. Error    CS0234    The type or namespace name 'AddressableAssets' does not exist in the namespace 'UnityEditor' (are you missing an assembly reference?)    Assembly-CSharp-Editor-testable    C:\Stephanie\wkspaces gluon\XVR.one_content\ContentPipeline\Assets\Scripts\Editor\Addressables\AssetBundleWindow.cs    5    Active
    3.  
    I managed to get your code working, by moving it to a test class. Figured that would be the easiest way to ensure I didn't run into more namespace issues.
     
  8. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    175
    Great :D

    What I usually do is decorate any of this code with the MenuItem attribute, so it looks like:

    Code (CSharp):
    1. public class MyClassWithCodeToRun
    2. {
    3.   [MenuItem("AssetDatabase/MyFunction")]
    4.   public static void MyFunction()
    5.   {
    6.     Debug.Log("Success!");
    7.   }
    8. }
    And then the item becomes available on the top bar menu, so I can just run the code by clicking it.

    So, now that the code is running, did you manage to find some duplicate assets?
     
    StephanieRowlinson likes this.
  9. StephanieRowlinson

    StephanieRowlinson

    Joined:
    Jul 23, 2014
    Posts:
    136
    Ah good trick! I'll use that next time I do something like this.

    I did indeed find exactly the duplicates I was expecting to find in my test so it looks like we can use this to aid our analysis of the current situation. Thanks for all your help. :D
     
    Unity_Javier likes this.