Search Unity

  1. If you have experience with import & exporting custom (.unitypackage) packages, please help complete a survey (open until May 15, 2024).
    Dismiss Notice
  2. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
Feedback request: Understanding import & export package workflows Unity 6 Preview now available

Bug Unity Core Android ARMv7 Memory Leak

Discussion in 'Android' started by Vlad_Grk, May 22, 2023.

  1. Vlad_Grk

    Vlad_Grk

    Joined:
    May 22, 2023
    Posts:
    6
    Hello, I've found a memory leak on Android ARMv7 platform in Unity Core. This script illustrates a problem the best. Once a memory reaches ~800 MB, it never cleans up. The best scenario for that issue is the following:
    • Make a build on Android ARMv7 platform with IL2CPP selected,
    • Connect a memory profiler,
    • Press a test button until memory reaches ~800 MB,
    • Clear a memory using GC Button,
    • Continue doing the above steps until you get a crash.
    I fully understand, that it's not an appropriate way of using memory, but exactly this case illustrates, that something wrong with GC on Android platform. What is more, even if I'm allocating 4 MB of RAM and clearing it, the GC doesn't clear it fully. It releases like 3.5 MB of RAM out of 4 MB. The problem exists on both Mono and IL2CPP scripting backends. Is there a workaround for that kind of problem? Thank you in advance.

    Code (CSharp):
    1. public class MemTest : MonoBehaviour
    2. {
    3.   [SerializeField] private Button _testButton;
    4.   [SerializeField] private Button _gcButton;
    5.  
    6.   private string _testString;
    7.  
    8.   private void Start()
    9.   {
    10.     _testButton.onClick.AddListener(RunTest);
    11.     _gcButton.onClick.AddListener(GcCollect);
    12.   }
    13.  
    14.   private void RunTest()
    15.   {
    16.     _testString += _testString + "x";
    17.   }
    18.  
    19.   private void GcCollect()
    20.   {
    21.     _testString = string.Empty;
    22.     GC.Collect();
    23.     GC.WaitForPendingFinalizers();
    24.     GC.Collect();
    25.     Resources.UnloadUnusedAssets();
    26.   }
    27. }
     
    Vlad_Grk, May 22, 2023
    #1

  2. devopsoxe55

    devopsoxe55

    Joined:
    Feb 14, 2023
    Posts:
    3
    Hi @MartinTilo Hopefully you've seen this issue, we would very much need your thoughts here. thank you
     
    devopsoxe55, May 23, 2023
    #2

  3. MelvMay

    MelvMay

    Unity Technologies

    Joined:
    May 24, 2013
    Posts:
    11,544
    Android stuff is best posted on the Android forum.

    I'll move the thread.
     
    MelvMay, May 23, 2023
    #3
    devopsoxe55 likes this.

  4. devopsoxe55

    devopsoxe55

    Joined:
    Feb 14, 2023
    Posts:
    3
    devopsoxe55, May 23, 2023
    #4

  5. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,469
    If you take 2 Memory Snapshots, each after pressing GC collect button, and compare them, what does it say is occupying extra memory, both on the summary page and on the All Off Memory page (using version 1.0 or 1.1 of the package, it you're not on 2022+ then by using an empty project with the memory Profiler package installed for analysis, snapshots can be taken from there or imported)

    And if it looks like mostly the empty reserved space grew, what does the Memory Map in version 0.7 of the package look like? Are there huge stretches of reserved managed memory that are entirely empty or are there tiny allocations in each and if so, what are these?
     
    MartinTilo, May 24, 2023
    #5

  6. Vlad_Grk

    Vlad_Grk

    Joined:
    May 22, 2023
    Posts:
    6
    After a memory profiler I got the following results: upload_2023-5-24_16-2-29.png

    upload_2023-5-24_16-3-11.png

    On the first screenshot "Empty Fragmented Heap Space" looks the most suspicious for me. But I don't even know what I can do from C# side to clear it.
     
    Vlad_Grk, May 24, 2023
    #6

  7. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,469
    What if you:
    1. Start that player
    2. Allocate a little bit just to initializ everything
    3. Hit the GC Collect button
    4. Take a snapshot, let's name it "A"
    5. Allocate a good deal more
    6. Hit GC Collect button
    7. Take another snapshot, let's name that B
    8. Switch the memory Profiler from single to Compare mode
    9. Open both A and B
    What does it say in terms of Managed Memory usage that increased? It it only Empty Fragmented Heap Space?

    If not, if you go to the All If Memory page and expand the Scripting Memory section: what does it say changed?

    It could be that small allocations sprinkled into the fragmented heap space keep some predominantly empty pages from getting unloaded.

    Technically every 6th GC.Collect (implementation detail and subject to change) should unload empty pages, which should happen automatically as you allocate more and more memory because every heap exapnsion happens after GC.Collect failed to make enough space (or was happening incrementally in the background and would therefore not have finished in time for the currently needed allocation). That is unless you've set the GCMode to manual or disabled, which would also explain the eventual crashing OutOfMemory...

    Building on the above, aside from avoiding managed memory fragmentation, not a lot. To get a clearer picture though you may get some value (for these test purposes only, there is usually no real reason to do so) from calling GC.Collect 6 times (or early out when Profiler.GetMonoHeapSizeLong() returns a lower value after a GC.Collect, indicating that the last GC.Collect was n%6==0.

    If the memory usage stays similarly high in empty fragmentated heap memory, then either:
    • GCMode is set to disabled
    • Or there's something on a thread's (including main thread) stack that looks to the GC like a pointer to memory in this otherwise empty appearing heap space (the memory Profiler doesn't capture the stack)
    • Or the memory is actually that fragmented.
    • Or there is a bug, e.g. in the memory Profiler or the GC
    • Or I might have forgotten some alternative options
    Either way, looking at the Memory Map view of the com.unity.memoryprofiler@0.7.1-preview.1 could then maybe shed some further light on this.
     
    MartinTilo, May 24, 2023
    #7

  8. Vlad_Grk

    Vlad_Grk

    Joined:
    May 22, 2023
    Posts:
    6
    Looks like the problem is in empty fragmented heap, but I didn't set GCMode to disabled, no other threads are running, no bug in memory profiler (because Android Studio Profiler shows the same) - I just have an empty project and above script attached. I've been trying calling GC.Collect more than 6 times, but the results are the same.

    upload_2023-5-25_16-22-6.png

    upload_2023-5-25_16-23-53.png
     
    Vlad_Grk, May 25, 2023
    #8

  9. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,469
    Hmm, what Unity version are you on?
    Also does this also independently of whether or not the GC is set to Incremental?

    And how does the managed memory (blue) look in the 0.7 version of the Memory Profiler package on the Fragmentation page?
     
    MartinTilo, May 26, 2023
    #9

  10. Vlad_Grk

    Vlad_Grk

    Joined:
    May 22, 2023
    Posts:
    6
    The issue persists on all possible versions of Unity 2018 LTS, 2019 LTS, 2022, 2023. Unfortunately, it doesn't matter what mode for GC I select incremental or not.

    I've spent plenty of time on that issue, I was trying: to load the test script via Asset Bundle, Addressable, DLL, but once that amount of memory is allocated in the current domain, it'll be released only after domain unloading. I've been thinking to make a hacky solution via another App Domain or another Unity Activity on Android platform, but from what I know, Unity doesn't support more than 1 App Domain and 2 instances of Unity can't be run in the same Android App.

    Here is fragmentation page from snapshot B:

    upload_2023-5-26_17-5-7.png

    upload_2023-5-26_17-5-31.png

    upload_2023-5-26_17-5-46.png
     
    Vlad_Grk, May 26, 2023
    #10

  11. Vlad_Grk

    Vlad_Grk

    Joined:
    May 22, 2023
    Posts:
    6
    Vlad_Grk, May 30, 2023
    #11

  12. devopsoxe55

    devopsoxe55

    Joined:
    Feb 14, 2023
    Posts:
    3
    devopsoxe55, May 31, 2023
    #12

  13. RaventurnPatrick

    RaventurnPatrick

    Joined:
    Aug 9, 2011
    Posts:
    250
    I can confirm that the memory fragmentation in Unity is generally way higher than expected - especially on Android (but for us it's also true on Arm64 Builds)
     
    RaventurnPatrick, Jun 1, 2023
    #13

  14. Vlad_Grk

    Vlad_Grk

    Joined:
    May 22, 2023
    Posts:
    6
    @JoshPeterson, @MartinTilo, is there a way to update a bdwgc to some recent version?
    is there a way to force use SG garbage collector instead of bdwgc?
    is there anyway I can influence on GC under hood logic?
     
    Vlad_Grk, Jun 1, 2023
    #14

  15. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,469
    There is not. If you follow Josh's Blog Post series on updating the .Net runtime, you can see how there are several deep lying reasons within the Unity codebase for why anything but the currently used, non-moving, non-generational, Boehm GC, is not yet compatible with Unity at this point (and how work towards a future in which it will be possible is already on the way).

    As for your issue here, I've just last week realized that the Memory Profiler package code does not treat any pointer sized value as a potential pointer to the heap in the same way as the conservative Boehm GC does. So it currently only finds managed Objects that are referenced from a root via actual reference type fields. This means it also ignores IntPtr fields for reference tracking and finding of all held objects.

    I can't be sure but particularly since you mentioned that this happens with ARMv7, where pointer size == sizeof(int), this would increase the possibility that the GC thinks that some int is referencing these strings and keeps them in memory, and that the current versions of the Memory Profiler wouldn't find these.

    Additionally there is the chance that some thread local variables or internal .Net type objects that do not have their type info exposed to the Memory Profiler capture process are holding on to this memory.

    I'm suspecting your actual use case behind this artificial repro script is not in exponentially increasing the size of a string through concatenation with itself? Is it in string memory? Are you using StringBuilder? Have you tried keeping the data as a list of smaller chunks of data (strings or otherwise)?

    Have you tried moving your logic off of the main thread and onto a scripting thread that you terminate once you no longer need the memory? That could take care of stack local variables holding on to the memory.
     
    MartinTilo, Jun 19, 2023
    #15