Increased memory usage with IL2CPP on HoloLens

Hello,

We have noticed that our application uses much more memory on the HoloLens with IL2CPP than with .Net Backend. Our application consist of a main scene for where user selects a scenario from the server and starts it, this switches a scene and load some meshes from the server. The meshes are manually parsed from a custom format (asset bundles are not used) and Unity’s Mesh class is created.

.Net backed (when scenario is opened is highlighted in red):


Il2cpp backend (when scenario is opened is highlighted in red):


Did somebody also notice different memory usage on HoloLens between IL2CPP and .Net? Seems the memory usage is slightly higher on il2cpp but the biggest issue seems to be memory not freeing when exiting scenario back to selection screen. In Editor and when targeting .Net framework the memory is correctly freed, only with il2cpp on device is the memory an issue. The scenario tested here has a big mesh where a single mesh is bigger than 64k vertices, so we use 32bit index format. The array to create the mesh are short lived and should be garbage collected after Unity mesh is created.

When profiling memory using Unity profiler the memory seems to be stable, no weird spikes, or undestroyed meshes/texture/materials.

We had investigated differences in GC between UWP il2cpp (uses Unity’s not compacting Boehm GC) and UWP .net (uses .net CLR compacting GC). And there are differences due to compacting / not compacting memory. Although the memory issue is not reproducible in the editor which also uses Boehm GC.

We are using Unity 2018.3.1, build is Release, targeting 10.0.17763.0 sdk, scripting runtime version 4.x Equivalent, API level 4.x.

Upon profiling the memory using VS memory profiler. It seems like the heap size is stable but the memory is increasing? Could it be this severe due to memory fragmentation?
Both snapshots are from selection screen with main content unloaded.


So to sum up, is there some information about what could cause the memory to increase and not being collected with il2cpp? Or is this a Unity IL2CPP bug?
Thank you.

 

Great, thanks. I am out of office today, will test it out on Monday.

 

Hello,
sorry for the later response.

I tried gathering some data as you mentioned. Seems the VirtualAlloc Commit provider reports the memory to rise but then remains stable. But the application still crashed with out of memory.

 

Also how is the memory allocated on HoloLens?
According to this paragraph from https://files.unity3d.com/zilys/ETWPerfGuide/data/VirtualAllocCommitProvider.html:

I was not sure if HoloLens counts as desktop or mobile device. Since it is x86 UWP platform.

 

Thanks for the tip, I can see the memory much better now.



I can debug where we use memory, but this does not explain why the memory consumption graph from Unity GC looks stable, the memory is not growing. But the memory chart from system as mentioned in the first comment seems to be growing. Tracing with the Memory Profile on HoloLens shows the same chart as the initial post.

We are using System.Threading.Tasks for most of our background async loading. Could this influence the memory consumption? Since as mentioned in the page the memory is allocated per thread and the Threads are in this case managed by the ThreadPool and use the SynchronizationContext.

I will try to run the tracing on my local machine this week. Maybe it will give us more data? Or try to remove all usages on Tasks and run all on main thread to see if the memory issues persist? What would your recommendation be?

 

Hi, I sent you the data. I will continue profiling next week.

 

As in the first post and then in the traces themselves it ends with the application crashing. And in system memory usage you can see the memory rising, consistently with the graphs in the first post.

 

Sadly I cannot give you access to our repo. Is there nothing you could think of? Or somebody from the IL2CPP/UWP team? No reports of Memory fragmentation? Or corruption of the GC to not release the memory?

 

Last year I reported a GC related memory problem (1054493). It's still not fixed but it could be related to the problem you have.
We were upgrading from the old .Net 3.5 runtime to .Net 4.x. Since then our 32 bit Windows Standalone builds use more memory or crash after some time because of "Out of memory" exceptions. And as far as I know Hololens runs a 32 bit OS.

I once got a reply from an Unity developer for it which included the following:

For this I modified the original code in a way so that it always produces potential pointers:

Spoiler

Code (CSharp):

1. using System;
2. using System.Collections.Generic;
3. using System.Threading;
4. using UnityEngine;
5. using UnityEngine.Assertions;
6.  
7. namespace Assets
8. {
9.     public class MemoryTest : MonoBehaviour
10.     {
11.         private Thread _thread;
12.         private uint _offset;
13.  
14.         private void Update()
15.         {
16.             if (_thread == null || !_thread.IsAlive)
17.             {
18.                 IList<TestObject> meshes = null;
19.                 _thread = new Thread(() =>
20.                 {
21.                     meshes = GenerateData();
22.                 });
23.                 Assert.IsNull(meshes); //Dummy code to prevent warnings
24.                 _thread.Start();
25.             }
26.         }
27.  
28.         private IList<TestObject> GenerateData()
29.         {
30.             var buffer = new byte[1024 * 1024 * 16];
31.             Assert.IsTrue(buffer.Length > 0);
32.  
33.             var objectCount = 1024 * 16; //Enough objects to completely point to every possible area in the 32 bit address space
34.             var meshObjects = new TestObject[objectCount];
35.             for (int i = 0; i < objectCount; i++)
36.             {
37.                 var meshObject = new TestObject();
38.                 meshObject.MeshIdentifier = _offset;
39.                 _offset += 1024 * 256; //point to every 256KB
40.                 meshObjects[i] = meshObject;
41.             }
42.  
43.             return meshObjects;
44.         }
45.     }
46.  
47.     public struct TestObject
48.     {
49.         public uint MeshIdentifier;
50.         public object DummyObject;
51.     }
52. }
53.  

The code doesn't cause any problems with the old .Net runtime or with a current (5.x) Mono release. But any IL2CPP or .Net 4.x Mono build has problems.
It also only causes problems with multiple threads. You were mentioning that you used Tasks in your code. So maybe this is related.

 

Hmm, so the problem is that arrays containing structs created on another thread with reference types, then the memory is leaked?
Not sure what that message from Unity dev means. Since they need to scan the whole array for references is not a reason for it to leak.
What did you do in the end to solve this? Remove usage of reference types from big arrays?
Initialize the array on the main thread? Did the issue also occur with il2cpp and x64 build?

 

Unfortunately I don't have many answers to your questions. We're still having these issues but hope that most of our customers use the 64 bit versions. I don't know why it needs a thread first to cause it. Maybe it has something to do with scanning the stack for any references.
I also don't know why they can't scan arrays precisely. I thought the Boehm GC has a precise mode. But I guess there is some reason Unity can't use it.
Our codebase is too large to find and change every possible location where it might cause problems. Especially as we have some external .Net libraries which we can't control.
64 bit builds aren't really affected as far as I know. The probability for a false pointer to hit a memory address within the GC heap is relative low. In my IL2CPP tests (Windows Standalone, UWP, Android) only the 32 bit versions had the problem. 64 bit builds didn't cause any issues.

 

Reproduction repo for memory leaks in UWP IL2CPP x86 but not in x64 builds.

https://bitbucket.org/reflekt/unity_il2cpp_memory_bug/src/master/

 

We've been trying to track down a similar issue on HoloLens error but it manifested in HttpClient downloads. Prior to moving to IL2CPP we saw no issues, post IL2CPP we get random jumps in RSS as we allocated/de-allocated byte arrays from image downloads. I assumed it was large object heap fragmentation. It is worth noting that we didn't see the issue (as much) when using UnityWebRequest which is using a native buffer as its backing store.

Native memory profiling was kind of useless to track it down as IL2CPP stack traces appear to be junk if you are using coroutines. We saw regular large heap allocations (I think it was 4MB + 4 bytes), but couldn't trace them back to what was allocating.

Any progress or feedback as to how to avoid the issue on HoloLens would be great.
Thanks.

 

Hi,

I seem to be having a similar situation and I'm also doing huge downloads (gltf files and its resources) using HttpClient.
Is there any news on this?

Running on the editor doesn't leak memory but running on Hololens it quickly runs out of memory on the 3rd or 4th download gltf...

 

https://issuetracker.unity3d.com/issues/windows-memory-leak-on-net-4-dot-x-and-x86-builds

I found the issue here. It says it's fixed in 2020.1. 2019.3 just released, is this bug not getting backported? This is a major blocker for hololens development... and who knows when 2020.1 will be released...

 

Thanks, we really need to know if it will be backported to Unity 2019 LTS

 

It would be great to get the fix backported to at least Unity 2019 LTS for x86 builds so that HL1 can be supported with IL2CPP. I've confirmed the following:

• https://bitbucket.org/reflekt/unity_il2cpp_memory_bug/src/master/ broken in HL1 x86 release build in Unity 2019 LTS
  
• https://bitbucket.org/reflekt/unity_il2cpp_memory_bug/src/master/ broken in HL2 ARM release build in Unity 2019 LTS
  
• https://bitbucket.org/reflekt/unity_il2cpp_memory_bug/src/master/ works with no memory leak in HL2 ARM64 release build in Unity 2019 LTS
  
• My application in HL1 x86 .NET scripting backend in Unity 2018 LTS works with no memory leak
  
• My application in HL2 ARM .NET scripting backend in Unity 2018 LTS works with no memory leak
• My application in HL1 x86 IL2CPP scripting backend in Unity 2018 LTS broken
• My application in HL2 ARM IL2CPP scripting backend in Unity 2018 LTS broken
  
• My application in HL1 x86 IL2CPP scripting backend in Unity 2019 LTS broken
• My application in HL2 ARM IL2CPP scripting backend in Unity 2019 LTS broken
  
• My application in HL2 ARM64 IL2CPP scripting backend in Unity 2019 LTS works with no memory leak

In summary,

• If you have HL1, you are currently forced to stay on Unity 2018 LTS with the .NET scripting backend if this memory leak is affecting your application; however, I have yet to build on Unity 2020 to confirm if 2020 actually fixes this issue.
• If you have HL2, you can use ARM64 in Unity 2019 LTS to get around this issue.

 

We just did some testing with our app on Hololens 2 that matches your results: ARM has massive memory leaks while ARM64 does not.

Using ARM64 was not an option when using MRTK2 until recently. Do we know if this is getting backported? Updating to 2020 is currently not an option for us.

 

We have the same problem here on 2018 LTS on armv7 andorid , but no problem on armv8 anroid.
If we change scripting runtime version to Net3.5， there is no memory leak。
We also confirm 2020 has fixed the issue.
Is there any plan to backport the fix to 2018 and 2019?

 

I faced the same issue. My app memory consumption is getting higher and after some point, the app crashes. I tested it with Task.Run and without Tasks but the result is the same. I also tested by changing the configurations like Master - ARM - Device or Release - ARM64 - Device but still the same. Everything works fine in the editor but when I deploy my app to HoloLens 2, I can't catch what causes the memory leak. I also analyzed my memory usage with the above file. It seems net.dll and kernel consume all memory.

 

I used the HoloLens performance analyser and I got the same graphics as above. Virtual allocation is getting higher. I tried to call GC after each UDP packet but nothing changed. I also tested the 2019 LTS with legacy and 2020 LTS with open XR it seems the same. I also created my own post if you like feel free to take a look please. https://forum.unity.com/threads/memory-leak-on-hololens-2.1261091/

 

Hi...I'm using Unity 2021.3.18 LTS, memory leak is happening when scanner is launched in screen and once the scanner is closed the memory is not coming down in HoloLens 2 device, Is their any solution ?