Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Out of Memory Building Content

Discussion in 'Addressables' started by chris_hellsten, Aug 19, 2019.

  1. chris_hellsten

    chris_hellsten

    Joined:
    May 5, 2014
    Posts:
    4
    Every time I try to build content for my project I get out-of-memory and crash, Unity 2019.2.0f1 & Addressables@1.1.7.

    I have approximately 5GB of models (FBX) in my project, not all can be loaded into memory at once. I am streaming based on player proximity at run-time and this was working on a previous slightly smaller batch of models. To facilitate the streaming the models are packed separately, so I have a very large number of asset bundles being generated (~3000).

    My developer machine has 16GB of RAM but I would have hoped it would be enough to do a successful build. The memory management in the build asset bundles stage is actually very efficient, but when we reach GenerateLocationListsTask.Run, memory spikes up to maximum (virtual memory allocation can't cope either) and the process crashes. The crash always occurs in the call to CreateResourceLocationData, possibly related to the large amount of data generated in the dictionaries and lists being created in the surrounding loops.

    The build succeeds if I only build 1 asset group at a time, but each time I perform a build, the catalogue replaces the entries from the previous build. Is there a way I can work-around these memory limitations by additively building my assets into the catalogue instead of replacing it when I do a new build?

    The log crashes with:
    DynamicHeapAllocator out of memory - Could not get memory for large allocation 4804516!
    Could not allocate memory: System out of memory!
    Trying to allocate: 4804516B with 32 alignment. MemoryLabel: VertexData
    Allocation happened at: Line:171 in C:\buildslave\unity\build\Runtime/Graphics/Mesh/VertexData.cpp

    The reference to VertexData.cpp leads me to believe the assets are being loaded into memory in this step? Is this necessary?
     
  2. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    839
    We had exactly that in Addressables 0.5.3 when building for Windows. Building for iOS worked fine. We have now rebuilt out catalogue from scratch for the later versions of addressables and it's stopped happening, so I am not entirely certain what was causing it.
     
  3. unity_bill

    unity_bill

    Unity Technologies

    Joined:
    Apr 11, 2017
    Posts:
    1,013
    this may be impossible due to the size of your project, but if you could file a bug against unity with your repro case, that would be a huge help. We'd love to track this down, but can't seem to create a synthetic test that triggers it.
     
  4. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    839
    When I have a little bit of time, I can try to instrument the build step to find what asset triggers it and see if I can isolate it into a repro. From what I remember, the memory allocation it attempts is ridiculous, many GiB.
     
  5. chris_hellsten

    chris_hellsten

    Joined:
    May 5, 2014
    Posts:
    4
    I did put together a simple repro project with nothing but Addressables and a (big) bunch of FBX files, but I tried to submit the bug-report 4 times, and after uploading I get:
    upload_2019-8-20_21-55-40.png
    Time to report a bug on the bug reporter >.<
     
  6. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    839
    Sounds like a transient server issue to me, I'd try again in a couple of hours and see what happens. If that fails, I'm sure you can ask the Unity guys for a Google Drive account to upload the repro project to instead.
     
    unity_bill likes this.
  7. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    839
    Yikes. I’m currently staring at a build that has allocated 65GB of RAM... so far!
     
  8. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    Same problem compiling a Windows build (our Assets folder size is 34.6 GB, using Unity 2019.2.9).
    We need now a PC with 32GB to build our project, is it normal?
    Unity use 29GB at the beginning, until it start to build scenes:

    upload_2019-10-21_12-29-32.png
     
    Last edited: Oct 21, 2019
  9. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    839
    Out of curiosity, do you have any really big monolithic asset bundles that reference native assets (textures etc.) from ScriptableObjects and/or MonoBehaviours? As this very much looks like the behaviour I'm seeing here. To see if it is the same issue, you could run your build through a native profiler (like Visual Studio's Performance Profiler) and see if the Editor is spending a ton of time loading objects it should not be loading in CalculateSortIndex().

    If it is that issue, we have worked around it by having smaller bundles but to do that we have had to write our own packing mode.

    PS: Amiga forevah. ;-)
     
    pyroakm likes this.
  10. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    Hi AlkisFortuneFish.
    We set groups of big scenes to be in AssetBundles. It seem that Unity load all resources in memory:
    upload_2019-10-21_15-15-30.png
     
    Last edited: Oct 21, 2019
  11. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    839
    At that stage in looks intentional, as that's an explicit call to LoadAssetAtPath from managed code, rather than a PPtr dereference of the already exported assets in native code, which in our case causes textures to be decompressed from the native GPU format to ARGB. At the same time, it looks as if most time is spent processing textures, which is precisely what happens in my issue too.

    Since it's triggered by managed code, the Unity profiler might be able to shed some extra light. I do suspect however that it's just part of the build process.

    Sorry I couldn't be of more help.
     
    pyroakm likes this.
  12. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    I will try to use smaller AssetBundle, thank you anyway.
    Amiga rule :)
     
  13. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    With smaller AssetBundles, nothing change, Unity continue to load all bitmaps in memory when building:

    upload_2019-10-22_16-59-51.png
     
  14. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    839
    Ugh. @Ryanc_unity any ideas?
     
  15. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    283
    Hmm, the building OOM crash fix should be in that version as it was backported to 2019.2.0f1.

    Looking at the call stack though, it looks like there is something wrong in script land, as it's currently building the Player at the start of that callstack, and in the middle of it during a C# callback something is calling LoadAssetAtPath. I'd really need access to a repro project with similar behavior / callstack to track down where the problem is occurring at this point as it really could be a ton of things, like MonoBehaviors flagged with [ExecuteAlways] or [ExecuteInEditMode] doing something on load that should have a BuildPipeline.isBuildingPlayercheck added if running in UNITY_EDITOR.
     
    pyroakm likes this.
  16. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    Thank you AlkisFortuneFish and Ryanc.
    Then I searched who's calling LoadAssetAtPath, look like this is HDRenderPipeline (we use HDRP):
    upload_2019-10-22_17-51-2.png
     
    Last edited: Oct 22, 2019
  17. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    Here, HDRP seem load all assets in memory, that will be a problem because our project is far from finish and will continue to grow:
    upload_2019-10-22_18-9-59.png
     
    Last edited: Oct 22, 2019
  18. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    283
    ...ouch
     
    AlkisFortuneFish likes this.
  19. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    839
    What is it even doing with that data?
     
  20. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    It's used to strip HDRP shaders and varients, that should be more optimized by avoiding textures and meshes loading, or discharging them progressively, but maybe only materials is useful.
     
    Last edited: Oct 25, 2019
  21. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
  22. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    283
    Thanks @pyroakm
    I poked the Render Pipeline team the other day about this code line, so hopefully depending on their current tasks this issue gets picked up fairly quickly.
     
  23. SebLagarde

    SebLagarde

    Unity Technologies

    Joined:
    Dec 30, 2015
    Posts:
    734
  24. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    That work fine, thank you.
     
  25. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    Hi.

    We've got similar memory problem when starting Unity, when it import new or changed assets (using 2019.3.0f6).
    It may be in https://github.com/Unity-Technologi...tprocessors/ModelImporterPostProcessor.cs#L43
    At line 43, there is: var embeddedMaterials = AssetDatabase.LoadAllAssetsAtPath(assetPath).OfType<Material>();
    Same as previous problem on HDRP, that load all assets in a path in memory to finaly get only materials.
    Some paths in our project contain lots of textures or meshes, and it seem to take lot of memory when importing (28GB).
     
    Last edited: Feb 14, 2020
  26. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    Last edited: Feb 26, 2020
  27. pyroakm

    pyroakm

    Joined:
    May 18, 2016
    Posts:
    28
    Fixed in 2019.3.5.
     
unityunity