Out of Memory Building Content

Every time I try to build content for my project I get out-of-memory and crash, Unity 2019.2.0f1 & Addressables@1.1.7.

I have approximately 5GB of models (FBX) in my project, not all can be loaded into memory at once. I am streaming based on player proximity at run-time and this was working on a previous slightly smaller batch of models. To facilitate the streaming the models are packed separately, so I have a very large number of asset bundles being generated (~3000).

My developer machine has 16GB of RAM but I would have hoped it would be enough to do a successful build. The memory management in the build asset bundles stage is actually very efficient, but when we reach GenerateLocationListsTask.Run, memory spikes up to maximum (virtual memory allocation can't cope either) and the process crashes. The crash always occurs in the call to CreateResourceLocationData, possibly related to the large amount of data generated in the dictionaries and lists being created in the surrounding loops.

The build succeeds if I only build 1 asset group at a time, but each time I perform a build, the catalogue replaces the entries from the previous build. Is there a way I can work-around these memory limitations by additively building my assets into the catalogue instead of replacing it when I do a new build?

The log crashes with:
DynamicHeapAllocator out of memory - Could not get memory for large allocation 4804516!
Could not allocate memory: System out of memory!
Trying to allocate: 4804516B with 32 alignment. MemoryLabel: VertexData
Allocation happened at: Line:171 in C:\buildslave\unity\build\Runtime/Graphics/Mesh/VertexData.cpp

The reference to VertexData.cpp leads me to believe the assets are being loaded into memory in this step? Is this necessary?

 

We had exactly that in Addressables 0.5.3 when building for Windows. Building for iOS worked fine. We have now rebuilt out catalogue from scratch for the later versions of addressables and it's stopped happening, so I am not entirely certain what was causing it.

 

this may be impossible due to the size of your project, but if you could file a bug against unity with your repro case, that would be a huge help. We'd love to track this down, but can't seem to create a synthetic test that triggers it.

 

When I have a little bit of time, I can try to instrument the build step to find what asset triggers it and see if I can isolate it into a repro. From what I remember, the memory allocation it attempts is ridiculous, many GiB.

 

I did put together a simple repro project with nothing but Addressables and a (big) bunch of FBX files, but I tried to submit the bug-report 4 times, and after uploading I get:

Time to report a bug on the bug reporter >.<

 

Sounds like a transient server issue to me, I'd try again in a couple of hours and see what happens. If that fails, I'm sure you can ask the Unity guys for a Google Drive account to upload the repro project to instead.

 

Yikes. I’m currently staring at a build that has allocated 65GB of RAM... so far!

 

Same problem compiling a Windows build (our Assets folder size is 34.6 GB, using Unity 2019.2.9).
We need now a PC with 32GB to build our project, is it normal?
Unity use 29GB at the beginning, until it start to build scenes:

 

Out of curiosity, do you have any really big monolithic asset bundles that reference native assets (textures etc.) from ScriptableObjects and/or MonoBehaviours? As this very much looks like the behaviour I'm seeing here. To see if it is the same issue, you could run your build through a native profiler (like Visual Studio's Performance Profiler) and see if the Editor is spending a ton of time loading objects it should not be loading in CalculateSortIndex().

If it is that issue, we have worked around it by having smaller bundles but to do that we have had to write our own packing mode.

PS: Amiga forevah. ;-)

 

Hi AlkisFortuneFish.
We set groups of big scenes to be in AssetBundles. It seem that Unity load all resources in memory:

 

At that stage in looks intentional, as that's an explicit call to LoadAssetAtPath from managed code, rather than a PPtr dereference of the already exported assets in native code, which in our case causes textures to be decompressed from the native GPU format to ARGB. At the same time, it looks as if most time is spent processing textures, which is precisely what happens in my issue too.

Since it's triggered by managed code, the Unity profiler might be able to shed some extra light. I do suspect however that it's just part of the build process.

Sorry I couldn't be of more help.

 

I will try to use smaller AssetBundle, thank you anyway.
Amiga rule

 

With smaller AssetBundles, nothing change, Unity continue to load all bitmaps in memory when building:

 

Ugh. @Ryanc_unity any ideas?

 

Thank you AlkisFortuneFish and Ryanc.
Then I searched who's calling LoadAssetAtPath, look like this is HDRenderPipeline (we use HDRP):

 

Here, HDRP seem load all assets in memory, that will be a problem because our project is far from finish and will continue to grow:

 

What is it even doing with that data?

 

It's used to strip HDRP shaders and varients, that should be more optimized by avoiding textures and meshes loading, or discharging them progressively, but maybe only materials is useful.

 

I've opened a bug report here:
https://fogbugz.unity3d.com/default.asp?1193585_hlsaf5ehcjun62no

 

That work fine, thank you.

 

Hi.

We've got similar memory problem when starting Unity, when it import new or changed assets (using 2019.3.0f6).
It may be in https://github.com/Unity-Technologi...tprocessors/ModelImporterPostProcessor.cs#L43
At line 43, there is: var embeddedMaterials = AssetDatabase.LoadAllAssetsAtPath(assetPath).OfType<Material>();
Same as previous problem on HDRP, that load all assets in a path in memory to finaly get only materials.
Some paths in our project contain lots of textures or meshes, and it seem to take lot of memory when importing (28GB).

 

@Ryanc_unity, @SebLagarde, can you help us on this problem again?

 

Fixed in 2019.3.5.

 

This issue continues in unity 2020.1.5 and latest addressables. On my windows machine I run out of memory and unity crashes while building addressables. I transferred the project file to my 2011 mac. I switched the target platform to mac. Addressables built with no problem. I haven’t tried building for windows on mac. That is the next thing I will do. But, on my windows machine I can’t build addressables. Both machines have 16gb ram. Help is appreciated.

P.S.: Project is HDRP 8.2.0.

 

Yes, my early 2011 model macbook pro built addressables for windows and built the hdrp project with success. I tried the built game on windows machine and confirmed everything worked fine (those macs do not support hdrp with their 256 mb gpu so, this was the way to be sure that addressable library was properly built).

2 possible outcomes of this: either my windows machine bios settings are messed up, or windows version of unity has problems with build algorithm(s). In a few days, when I am ready for an alpha, I will reset bios and try to build addressables on windows machine to find out which. Also, before that, I will try to build with latest unity 2020.1. If nothing works, I will zip the 200gb project folder, copy that to old mac, unzip there, and do the addressables build job and the project build job there. A huge time loss, but, well, at least a solution if I have nothing on hand.

 

I also got out of memory error when building addressable on Unity 2020.2.2f1. Stuck at Writing serialized file

 

An update on this issue. There was an error in my bios settings. Fixed that. Then, I still had crashes when building addressables. Even computer crashes... On my windows machine... My macbook pro 2011 was always successful in building addressables. When I monitored memory usage, I was observing that the whole 16 GB of memory was used and when unity could not get any more mem for the next operation when building addressables, it or computer was crashing.

Then, I ordered new ram and upgraded windows machine to 32 GB. Every now and then, unity crashes when building addressables. But, in general, it succeeds to build.

Mac still successfully does the job.

I have lots of 8k images uncompressed used for imposters in the projects. I guess those are the most memory consuming assets.

Still using unity 2020.1.4 on windows machine since upgrading causes an issue with imposters and breaks the project. Tried with 2020.2.0 on an M1 mac and got no problems. 2020.2.1 on mac caused an issue with imposter asset so could not try to build addressables with that.

To summarize, I guess, mac version runs completely ok, while windows version has something to do with memory management.

Edit: my project is like 80gb assets most of which is assigned as addressables (not all assets are assigned as addressables). I get 17-18 gb build size with addressables archive. That might give an impression of how much of that asset size is going through the addressable system. Also, I am not sure if this is important or not, but, my library folder has more than 120 gb of build cache. Could not dare to delete them, for now, since I have left little tasks to complete the project.

 

Same issue here. We have a small 2D game 2021.3.1f1, built-in render pipeline. It's building fine on all platforms except Windows where it eats up all the memory and then dies with this stacktrace:

Code (CSharp):

1. Opening scene 'Assets/Scenes/Yacht/Top Deck (Exterior).unity'
2. Unloading 195 Unused Serialized files (Serialized files now loaded: 0)
3. Loaded scene 'Assets/Scenes/Yacht/Top Deck (Exterior).unity'
4.     Deserialize:            10.272 ms
5.     Integration:            57.520 ms
6.     Integration of assets:  0.063 ms
7.     Thread Wait Time:       0.007 ms
8.     Total Operation Time:   67.862 ms
9. Unloading 51 unused Assets / (2.3 KB). Loaded Objects now: 31669.
10. Memory consumption went from 53.90 GB to 53.90 GB.
11. Total: 57.788200 ms (FindLiveObjects: 2.726900 ms CreateObjectMapping: 3.223800 ms MarkObjects: 51.680200 ms  DeleteObjects: 0.156300 ms)
12.  
13.  
14. =================================================================
15.     Native Crash Reporting
16. =================================================================
17. Got a UNKNOWN while executing native code. This usually indicates
18. a fatal error in the mono runtime or one of the native libraries
19. used by your application.
20. =================================================================
21.  
22. =================================================================
23.     Managed Stacktrace:
24. =================================================================
25.       at <unknown> <0xffffffff>
26.       at UnityEditor.Build.Content.ContentBuildInterface:WriteSerializedFile_Internal_Injected <0x000da>
27.       at UnityEditor.Build.Content.ContentBuildInterface:WriteSerializedFile_Internal <0x0007a>
28.       at UnityEditor.Build.Content.ContentBuildInterface:WriteSerializedFile <0x001e2>
29.       at UnityEditor.Build.Pipeline.WriteTypes.AssetBundleWriteOperation:Write <0x00262>
30.       at UnityEditor.Build.Pipeline.Tasks.WriteSerializedFiles:UnityEditor.Build.Pipeline.Utilities.TaskCachingUtility.IRunCachedCallbacks<UnityEditor.Build.Pipeline.Tasks.WriteSerializedFiles.Item>.ProcessUncached <0x00332>
31.       at UnityEditor.Build.Pipeline.Utilities.TaskCachingUtility:RunCachedOperation <0x00c84>
32.       at UnityEditor.Build.Pipeline.Tasks.WriteSerializedFiles:Run <0x001b2>
33.       at UnityEditor.Build.Pipeline.BuildTasksRunner:Run <0x00346>
34.       at UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles <0x00a52>
35.       at UnityEditor.AddressableAssets.Build.DataBuilders.BuildScriptPackedMode:DoBuild <0x008aa>
36.       at UnityEditor.AddressableAssets.Build.DataBuilders.BuildScriptPackedMode:BuildDataImplementation <0x001d0>
37.       at UnityEditor.AddressableAssets.Build.DataBuilders.BuildScriptBase:BuildData <0x00284>
38.       at UnityEditor.AddressableAssets.Settings.AddressableAssetSettings:BuildPlayerContentImpl <0x00149>
39.       at UnityEditor.AddressableAssets.Settings.AddressableAssetSettings:BuildPlayerContent <0x0012a>
40.       at UnityEditor.AddressableAssets.Settings.AddressableAssetSettings:BuildPlayerContent <0x0002a>
41.       at UnityEditor.AddressableAssets.GUI.AddressableAssetsSettingsGroupEditor:OnBuildPlayerData <0x0001a>
42.       at UnityEditor.AddressableAssets.GUI.AddressableAssetsSettingsGroupEditor:OnBuildScript <0x00032>
43.       at UnityEditor.GenericMenu:CatchMenu <0x00062>
44.       at <Module>:runtime_invoke_void__this___object_object_int <0x0009d>
45. =================================================================
46. Received signal SIGSEGV
47. Could not allocate 5544 (1 * 5544) bytes
48. Crash!!!

@Ryanc_unity @unity_bill

 

I'll give that shot! Also, just learned this is a known issue:
https://issuetracker.unity3d.com/issues/high-memeory-usage-when-building-addressables

 

so... the issue was marked as "Won't fix", so what now?
I can't use "Disable Visible Sub Asset Representations" since I need to address my assets at runtime, but I have over 30GB of assets and building takes up all of my 32GB of memory. Is the only solution really to get a Mac?

 

Hi,
There's an issue in recent Addressables versions, which causes all assets are loaded into the memory during the buildlayout.json/txt is being generated. Here is a workaround. Or you can disable the generation of buildlayourt.json/txt to avoid the issue.