Search Unity

Increased memory usage with IL2CPP on HoloLens

Discussion in 'Windows' started by MartinGrman, Feb 21, 2019.

  1. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    Hello,

    We have noticed that our application uses much more memory on the HoloLens with IL2CPP than with .Net Backend. Our application consist of a main scene for where user selects a scenario from the server and starts it, this switches a scene and load some meshes from the server. The meshes are manually parsed from a custom format (asset bundles are not used) and Unity’s Mesh class is created.

    .Net backed (when scenario is opened is highlighted in red):


    Il2cpp backend (when scenario is opened is highlighted in red):


    Did somebody also notice different memory usage on HoloLens between IL2CPP and .Net? Seems the memory usage is slightly higher on il2cpp but the biggest issue seems to be memory not freeing when exiting scenario back to selection screen. In Editor and when targeting .Net framework the memory is correctly freed, only with il2cpp on device is the memory an issue. The scenario tested here has a big mesh where a single mesh is bigger than 64k vertices, so we use 32bit index format. The array to create the mesh are short lived and should be garbage collected after Unity mesh is created.

    When profiling memory using Unity profiler the memory seems to be stable, no weird spikes, or undestroyed meshes/texture/materials.

    We had investigated differences in GC between UWP il2cpp (uses Unity’s not compacting Boehm GC) and UWP .net (uses .net CLR compacting GC). And there are differences due to compacting / not compacting memory. Although the memory issue is not reproducible in the editor which also uses Boehm GC.

    We are using Unity 2018.3.1, build is Release, targeting 10.0.17763.0 sdk, scripting runtime version 4.x Equivalent, API level 4.x.

    Upon profiling the memory using VS memory profiler. It seems like the heap size is stable but the memory is increasing? Could it be this severe due to memory fragmentation?
    Both snapshots are from selection screen with main content unloaded.


    So to sum up, is there some information about what could cause the memory to increase and not being collected with il2cpp? Or is this a Unity IL2CPP bug?
    Thank you.
     
  2. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    6,436
    Could you try profiling memory allocations with Windows Performance Analyzer to make sure that they're coming from the garbage collector? Here's a guide on it: https://files.unity3d.com/zilys/ETWPerfGuide/data/VirtualAllocCommitProvider.html

    You can record a trace on HoloLens by going into Device Portal, then into Performance Tracing tab and finally using a custom profile that I attached to this post.
     

    Attached Files:

  3. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    Great, thanks. I am out of office today, will test it out on Monday.
     
  4. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    Hello,
    sorry for the later response.

    I tried gathering some data as you mentioned. Seems the VirtualAlloc Commit provider reports the memory to rise but then remains stable. But the application still crashed with out of memory.
    upload_2019-3-4_15-2-38.png
     
  5. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    Also how is the memory allocated on HoloLens?
    According to this paragraph from https://files.unity3d.com/zilys/ETWPerfGuide/data/VirtualAllocCommitProvider.html:
    I was not sure if HoloLens counts as desktop or mobile device. Since it is x86 UWP platform.
     
  6. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    6,436
    Drag the commit stack column to the left of the yellow marker. That will actually make it readable.

    Regarding your second question: it treats HoloLens as desktop.
     
  7. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    Thanks for the tip, I can see the memory much better now.
    upload_2019-3-5_15-47-48.png
    upload_2019-3-5_15-48-3.png

    I can debug where we use memory, but this does not explain why the memory consumption graph from Unity GC looks stable, the memory is not growing. But the memory chart from system as mentioned in the first comment seems to be growing. Tracing with the Memory Profile on HoloLens shows the same chart as the initial post.

    We are using System.Threading.Tasks for most of our background async loading. Could this influence the memory consumption? Since as mentioned in the page the memory is allocated per thread and the Threads are in this case managed by the ThreadPool and use the SynchronizationContext.

    I will try to run the tracing on my local machine this week. Maybe it will give us more data? Or try to remove all usages on Tasks and run all on main thread to see if the memory issues persist? What would your recommendation be?
     
  8. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    6,436
    I don't this it's related to tasks per se. However, it's really weird that your memory consumption looks different in Visual Studio and Windows Performance Analyzer. Are you saying that in Windows Performance Analyzer the memory consumption looks stable while it's not in VS? You could try calling this API from scripts to see which one is more accurate: https://docs.microsoft.com/en-us/uwp/api/windows.system.memorymanager.appmemoryusage

    Could you also send me the recorded trace in a PM?
     
  9. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    Hi, I sent you the data. I will continue profiling next week.
     
  10. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    6,436
    The trace you sent me is indeed very weird - it doesn't display any leaks or memory usage increasing over time. Did the game actually crash at the end of it? Or does it take much longer than that for it to crash?
     
  11. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    As in the first post and then in the traces themselves it ends with the application crashing. And in system memory usage you can see the memory rising, consistently with the graphs in the first post.
     
  12. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    6,436
    Alright, I'm out of ideas. Can you report a bug with a repro project so we could dig in?
     
  13. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    Sadly I cannot give you access to our repo. Is there nothing you could think of? Or somebody from the IL2CPP/UWP team? No reports of Memory fragmentation? Or corruption of the GC to not release the memory?
     
  14. Hagn

    Hagn

    Joined:
    Apr 1, 2013
    Posts:
    8
    Last year I reported a GC related memory problem (1054493). It's still not fixed but it could be related to the problem you have.
    We were upgrading from the old .Net 3.5 runtime to .Net 4.x. Since then our 32 bit Windows Standalone builds use more memory or crash after some time because of "Out of memory" exceptions. And as far as I know Hololens runs a 32 bit OS.

    I once got a reply from an Unity developer for it which included the following:
    For this I modified the original code in a way so that it always produces potential pointers:
    Code (CSharp):
    1. using System;
    2. using System.Collections.Generic;
    3. using System.Threading;
    4. using UnityEngine;
    5. using UnityEngine.Assertions;
    6.  
    7. namespace Assets
    8. {
    9.     public class MemoryTest : MonoBehaviour
    10.     {
    11.         private Thread _thread;
    12.         private uint _offset;
    13.  
    14.         private void Update()
    15.         {
    16.             if (_thread == null || !_thread.IsAlive)
    17.             {
    18.                 IList<TestObject> meshes = null;
    19.                 _thread = new Thread(() =>
    20.                 {
    21.                     meshes = GenerateData();
    22.                 });
    23.                 Assert.IsNull(meshes); //Dummy code to prevent warnings
    24.                 _thread.Start();
    25.             }
    26.         }
    27.  
    28.         private IList<TestObject> GenerateData()
    29.         {
    30.             var buffer = new byte[1024 * 1024 * 16];
    31.             Assert.IsTrue(buffer.Length > 0);
    32.  
    33.             var objectCount = 1024 * 16; //Enough objects to completely point to every possible area in the 32 bit address space
    34.             var meshObjects = new TestObject[objectCount];
    35.             for (int i = 0; i < objectCount; i++)
    36.             {
    37.                 var meshObject = new TestObject();
    38.                 meshObject.MeshIdentifier = _offset;
    39.                 _offset += 1024 * 256; //point to every 256KB
    40.                 meshObjects[i] = meshObject;
    41.             }
    42.  
    43.             return meshObjects;
    44.         }
    45.     }
    46.  
    47.     public struct TestObject
    48.     {
    49.         public uint MeshIdentifier;
    50.         public object DummyObject;
    51.     }
    52. }
    53.  

    The code doesn't cause any problems with the old .Net runtime or with a current (5.x) Mono release. But any IL2CPP or .Net 4.x Mono build has problems.
    It also only causes problems with multiple threads. You were mentioning that you used Tasks in your code. So maybe this is related.
     
  15. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11
    Hmm, so the problem is that arrays containing structs created on another thread with reference types, then the memory is leaked?
    Not sure what that message from Unity dev means. Since they need to scan the whole array for references is not a reason for it to leak.
    What did you do in the end to solve this? Remove usage of reference types from big arrays?
    Initialize the array on the main thread? Did the issue also occur with il2cpp and x64 build?
     
  16. Hagn

    Hagn

    Joined:
    Apr 1, 2013
    Posts:
    8
    Unfortunately I don't have many answers to your questions. We're still having these issues but hope that most of our customers use the 64 bit versions. I don't know why it needs a thread first to cause it. Maybe it has something to do with scanning the stack for any references.
    I also don't know why they can't scan arrays precisely. I thought the Boehm GC has a precise mode. But I guess there is some reason Unity can't use it.
    Our codebase is too large to find and change every possible location where it might cause problems. Especially as we have some external .Net libraries which we can't control.
    64 bit builds aren't really affected as far as I know. The probability for a false pointer to hit a memory address within the GC heap is relative low. In my IL2CPP tests (Windows Standalone, UWP, Android) only the 32 bit versions had the problem. 64 bit builds didn't cause any issues.
     
  17. MartinGrman

    MartinGrman

    Joined:
    Apr 13, 2017
    Posts:
    11