Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Feedback [IN-36148] Huge performance drop in BuildLayoutGenerationTask (1.20.3 -> 1.21.9)

Discussion in 'Addressables' started by Alan-Liu, Mar 20, 2023.

  1. Alan-Liu

    Alan-Liu

    Joined:
    Jan 23, 2014
    Posts:
    390
    After I upgraded our project to use Addressables 1.21.9 from 1.20.3, BuildLayoutGenerationTask became very slow. Before, it took only less than 6 seconds to finish, but after upgrading, the Editor stuck in BuildLayoutGenerationTask for over 10 minutes and I had to kill the Editor process.

    After some investigation, I found it's due to the following code:
    Code (CSharp):
    1. public class BuildLayoutGenerationTask : IBuildTask
    2. {
    3.     // ...
    4.  
    5.     private LayoutLookupTables GenerateLookupTables(AddressableAssetsBuildContext aaContext)
    6.     {
    7.         // ...
    8.  
    9.         foreach (BuildLayout.File file in lookup.Files.Values)
    10.         {
    11.             // ...
    12.  
    13.             // Cache all object types from results to find type for when implicit asset
    14.             Dictionary<ObjectIdentifier, Type[]> objectTypes = new Dictionary<ObjectIdentifier, Type[]>(1024);
    15.             foreach (KeyValuePair<GUID, AssetResultData> assetResult in m_Results.AssetResults)
    16.             {
    17.                 foreach (var resultEntry in assetResult.Value.ObjectTypes)
    18.                 {
    19.                     if(!objectTypes.ContainsKey(resultEntry.Key))
    20.                         objectTypes.Add(resultEntry.Key, resultEntry.Value);
    21.                 }
    22.             }
    23.    
    24.             // ...
    25.         }
    26.  
    27.         // ...
    28.     }
    29. }
    In our project, lookup.Files.Values has 26336 items, m_Results.AssetResults has 36242 items and objectTypes has 695450 items.

    It seems the code of constructing objectTypes can be moved outside the loop:
    Code (CSharp):
    1. public class BuildLayoutGenerationTask : IBuildTask
    2. {
    3.     // ...
    4.  
    5.     private LayoutLookupTables GenerateLookupTables(AddressableAssetsBuildContext aaContext)
    6.     {
    7.         // ...
    8.  
    9.         // Cache all object types from results to find type for when implicit asset
    10.         Dictionary<ObjectIdentifier, Type[]> objectTypes = new Dictionary<ObjectIdentifier, Type[]>(1024);
    11.         foreach (KeyValuePair<GUID, AssetResultData> assetResult in m_Results.AssetResults)
    12.         {
    13.             foreach (var resultEntry in assetResult.Value.ObjectTypes)
    14.             {
    15.                 if(!objectTypes.ContainsKey(resultEntry.Key))
    16.                     objectTypes.Add(resultEntry.Key, resultEntry.Value);
    17.             }
    18.         }
    19.  
    20.         foreach (BuildLayout.File file in lookup.Files.Values)
    21.         {
    22.             // ...
    23.         }
    24.  
    25.         // ...
    26.     }
    27. }
    With this modification, BuildLayoutGenerationTask took about 80 seconds to finish, which is still worse than the old version, but is usable in our project.

    Hope Unity developers can take a look at this issue.
     
    Last edited: Mar 23, 2023
    einWikinger likes this.
  2. Alan-Liu

    Alan-Liu

    Joined:
    Jan 23, 2014
    Posts:
    390
    Here is the comparision between 1.20.3 and 1.21.9 (with my modification above):
    1.20.3:
    upload_2023-3-23_12-30-32.png
    1.21.9:
    upload_2023-3-23_12-33-38.png
     
  3. unity_shane

    unity_shane

    Unity Technologies

    Joined:
    Jun 3, 2021
    Posts:
    106
    @Alan-Liu Hey there - nice catch as always. I'll keep an eye on this and see if we can get this in soon. I'll also take another look at the build report code to see if there's any other straightforward optimizations we missed. Thanks for pointing this out! BuildLayoutGenerationTask is pretty much always going to be slower on the newer versions given that the build reporting we do now is much more robust, but 10 minutes is definitely unacceptable.
     
    Alan-Liu likes this.
  4. einWikinger

    einWikinger

    Joined:
    Jul 30, 2013
    Posts:
    97
    @Alan-Liu, did you also notice that the BuildLayoutGenerationTask is requiring more RAM than before? In addition to it becoming slow, it also started to require RAM in the realm of nearly a hundred of gigabytes (for our project).
     
  5. Alan-Liu

    Alan-Liu

    Joined:
    Jan 23, 2014
    Posts:
    390
    I didn't notice that before, but I did some tests today, and confirmed there is an unexpected large memory usage in BuildLayoutGenerationTask of Addressables 1.21.9.

    Here is the data from my test in Windows Editor (Unity 2020.3.30 + Addressables 1.21.9):
    upload_2023-4-7_16-34-27.png

    1,2,3 refer to the locations in the code:
    Code (CSharp):
    1.  
    2. public class BuildLayoutGenerationTask : IBuildTask
    3. {
    4.     // ...
    5.  
    6.     private BuildLayout CreateBuildLayout()
    7.     {
    8.         AddressableAssetsBuildContext aaContext = (AddressableAssetsBuildContext)m_AaBuildContext;
    9.         LayoutLookupTables lookup = null;
    10.         BuildLayout result = null;
    11.         using (m_Log.ScopedStep(LogLevel.Info, "Generate Lookup tables"))
    12.         {
    13.             // ----------- 1 -----------
    14.  
    15.             lookup = GenerateLookupTables(aaContext);
    16.  
    17.             // ----------- 2 -----------
    18.         }
    19.         using (m_Log.ScopedStep(LogLevel.Info, "Generate Build Layout"))
    20.             result = GenerateBuildLayout(aaContext, lookup);
    21.  
    22.         // ----------- 3 -----------
    23.         return result;
    24.     }
    25.  
    26.     // ...
    27. }
    28.  
    Apparently, something in GenerateLookupTables caused the memory increased about 8GB.

    Looking through the code, I think it's due to the following code:
    Code (CSharp):
    1. private static Dictionary<long, string> GetObjectsIdForAsset(string assetPath)
    2. {
    3.     UnityEngine.Object[] assetSubObjects = AssetDatabase.LoadAllAssetsAtPath(assetPath);
    4.     Dictionary<long, string> localIdentifierToObjectName = new Dictionary<long, string>(assetSubObjects.Length);
    5.     if (assetSubObjects != null)
    6.     {
    7.         foreach (var o in assetSubObjects)
    8.         {
    9.             if (o != null && AssetDatabase.TryGetGUIDAndLocalFileIdentifier(o, out _, out long localId))
    10.                 localIdentifierToObjectName[localId] = o.name;
    11.         }
    12.     }
    13.  
    14.     return localIdentifierToObjectName;
    15. }
    As you can see, it calls AssetDatabase.LoadAllAssetsAtPath to load assets and during the execution of BuildLayoutGenerationTask, Unity dosen't have any chance to unload them, so the memory keeps increasing until all assets that should be packed in asset bundles are loaded.

    A workaround is that unloading unused assets after BuildLayoutGenerationTask.GenerateLookupTables processes some asset bundles. Here is an example (Diff for BuildLayoutGenerationTask.cs):
    Code (CSharp):
    1. ===================================================================
    2. @@ -241,6 +241,8 @@
    3.              Dictionary<string, List<BuildLayout.DataFromOtherAsset>> guidToPulledInBuckets =
    4.                  new Dictionary<string, List<BuildLayout.DataFromOtherAsset>>();
    5.  
    6. +           long memory = UnityEngine.Profiling.Profiler.usedHeapSizeLong;
    7. +
    8.              foreach (BuildLayout.File file in lookup.Files.Values)
    9.              {
    10.                  Dictionary<string, AssetBucket> buckets = new Dictionary<string, AssetBucket>();
    11. @@ -464,6 +466,9 @@
    12.                          }
    13.                      }
    14.                  }
    15. +
    16. +                if (UnityEngine.Profiling.Profiler.usedHeapSizeLong - memory >= 1024 * 1024 * 1024)
    17. +                    EditorUtility.UnloadUnusedAssetsImmediate();
    18.              }
    19.  
    20.              foreach (BuildLayout.File file in lookup.Files.Values)
    21.  
    The following data is from the same test using the workaround above:
    upload_2023-4-7_16-51-9.png
     
    Last edited: Jul 28, 2023
    einWikinger likes this.
  6. Alan-Liu

    Alan-Liu

    Joined:
    Jan 23, 2014
    Posts:
    390
    Hi, @unity_shane, can you take a look at the memory usage issue above?

    Update:
    I submitted a new bug report for this issue: IN-37708.
     
    Last edited: Apr 7, 2023
    einWikinger likes this.
  7. unity_shane

    unity_shane

    Unity Technologies

    Joined:
    Jun 3, 2021
    Posts:
    106
    @Alan-Liu @einWikinger, Thanks for bringing this issue up - we'll do what we can to take a look at the memory issue ticket soon. Thanks for making a ticket! Also, I was able to get the fix you described before into the next release, so it'll be shipping with 1.21.10. There hasn't been time to do a deeper dive into optimizing BuildLayoutGenerationTask yet, but it's definitely on the list. Thanks again!
     
    einWikinger and Alan-Liu like this.
  8. einWikinger

    einWikinger

    Joined:
    Jul 30, 2013
    Posts:
    97
    @Alan-Liu thanks for looking into it that deeply!
    @unity_shane We've hotfixed it on our branch now as well, but waiting for the official 1.21.10 release now :)
     
    Alan-Liu likes this.
  9. Alan-Liu

    Alan-Liu

    Joined:
    Jan 23, 2014
    Posts:
    390