Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. The 2023.1 beta is now available for testing. To find out what's new, have a look at our 2023.1 feature highlights.
    Dismiss Notice

How to investigate and improve memory fragmentation?

Discussion in 'Profiler Previews' started by bagelbaker, Feb 19, 2021.

  1. bagelbaker

    bagelbaker

    Joined:
    Jun 5, 2017
    Posts:
    63
    Hi,

    We have released a game on most platforms (PC and consoles) and the game is often crashing on Switch due to being out of memory. I've posted on the Nintendo dev portal but haven't got any response there. It seems it might be a general problem but crashing on this platform because it has the least amount of memory.

    A test we're doing right now is reloading a level many times and comparing the results. If we check the Tree map, there is not a big difference between the level being loaded once and the level being loaded 30 times. But after 30 times, the allocatable memory is almost reaching zero where it will eventually crash. We lose about 12MB of allocatable memory each loading. Our loading system has this flow: Addressables.LoadSceneAsync() -> Addressables.UnloadSceneAsync() -> Resources.UnloadUnusedAssets() -> Addressables.LoadSceneAsync() -> etc

    Another test that we are doing is to load a level and then loading an empty scene to see if there's any leaking. There's minimal stuff remaining.

    So we think that we might have a memory fragmentation problem. The problem is that we don't really know how to go about investigating and fixing this. When looking at the memory map, what should we look for?

    Although we are already pooling some stuff, there's some other stuff that we can pool if needed. However, it be nice to able to identify what to pool that would have the most impact.

    We've been on this issue for 2 weeks now. We've been able to reduce the general memory footprint and fix some leaks but have not seem to improve memory fragmentation so any help would be appreciated.

    Attached is screenshot of the memory map after 30 reloading of the same level.

    Cheers
     

    Attached Files:

  2. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,906
    Hi there,

    This looks like you are running on IL2CPP and the Memory Map resolution set to 6.4 MB per line?

    This means that we can't see the Virtual Machine Memory Allocated by IL2CPP because it is not going through Unity's Native Memory Manager Allocators, so there might be an unaccounted for amount of memory here that is tied to Type Metadata, Generics and Reflection.

    Beyond that, you look to be loosing some 10.24 MB to Managed Memory Fragmentation. Analyzing the Native Memory like this is a bit trickier from a screenshot.

    As for how I did that analysis: Managed Memory is allocated in Sections that are either VM Memory (not yet visible in IL2CPP, in Mono it's dark blue and currently (Update: as of version 0.4.0 of the package it is visible for snapshots taken with from builds made with Unity Editor versions 2021.2.0a12, 2021.1.9f1, 2020.3.12f1, and 2019.4.29f1 or newer) undistinguishable from empty Heap Sections, but those are rare as they would get returned to the OS after some GC.Collect cycles) or Heap Sections used for Managed Object allocations. The Memory in these sections is either used for Managed Object memory (light blue) or unused (dark blue). Generally speaking, the Heap Section which has the highest Virtual Address Values of all Heap sections is the Active Heap Section.

    New Object Allocation will be first placed in the Active Heap section. If that is not possible, the allocator will scan the free block list ( think of these as reclaimed chunks of memory of specific sizes in old sections) and if a block that can fit the allocation is found we'll allocate in that. Otherwise a new heap section will be created (gc heap expansion) at a higher virtual address value than the highest one that was ever used for an allocation in this run, i.e. at the very bottom of the map.

    All Inactive Heap sections are abandoned for new block allocations and only monitored by the GC, which will check if they still have uncollected Objects in them during each GC. When objects are collected and the Block they were allocated in is thereby emptied, that Block gets added to the free block list. If a section only consists of free blocks, the section will be returned to the OS, freeing up the Virtual Address space (i.e. that will then show the black background color in its place).
    Therefore, all dark blue space in heap sections with lower virtual address values is (potentially, but likely) wasted to fragmentation. If all the Objects within these sections would get collected, that memory could be returned to the OS.

    Now obviously, the next run might result in entirely different memory distribution so it might not be as simple as: "if this object(s) would be more short lived, I'll get back x amount of RAM". But they can provide evidence to the moment in which this heap section was created and what other short lived allocations might have shared that space and been collected already. If these objects are purposefully long lived, they can also show which kind of objects might be good to allocated at the same time as some other long lived allocations in order to have them packed together more tightly and make it less likely that they share a heap section with a short lived allocation.

    Now that is a bit much to take in I guess but, try and see if that helps you making any sense of what is going on in your memory there, because there's only so much I can tell you from screenshots. :)

    P.s. you can tell the bottom area of the Memory Map to show you the objects in a selected Virtual address range.

    P.P.S: Oh and, Native Memory is often allocated as pools or buckets and then used and reused for objects of a certain size, in an attempt to avoid fragmenting them, by mixing allocations of varying sizes. Still, you might have too many large size pools which are only sporadically filled to their full size, or too many small ones that can not be used for bigger ones then. Native Memory usage is a bit trickier to control though.

    [May 6th 2021 Update]: I had previously glossed over the free block list a bit, that should technically help reuse some of that wasted space. The memory profiler doesn't yet have any info over managed Blocks or the free Block list (we're looking into adding that).
     

    Attached Files:

    Last edited: Oct 18, 2021
    Dmitry-Fofanov likes this.
  3. bagelbaker

    bagelbaker

    Joined:
    Jun 5, 2017
    Posts:
    63
    Hi,

    Thanks for the detailed answer. I think I understand your explanation of heap memory management. I'm aware that the objects are shown in the bottom area but there's so much different types of objects and so many sections that I needed to better understanding how it works to be able to investigate efficiently and find the problem.

    The memory map resolution is set to 8MB per line.

    Where do you see the 1.6 line lengths of estimated memory lost due to fragmentation? I'm not sure what you mean by "Managed Memory Lost to Fragmentation, stitched together. Collected up to the above line" on the picture you attached. 10MB lost due to fragmentation doesn't seem to be a lot.

    Here are some stats:
    • loading level 1x: Mem.Allocated:996 M Reserved:1124 M GC.Heap:300920 K Used:292620 K Allocatable:764 M
    • reloading 10x: Mem.Allocated:1019 M Reserved:1158 M GC.Heap:300920 K Used:283468 K Allocatable:602 M
    • if we continue, none of the values changes that much except for Allocatable who decreases by about 15MB per reload. Once this values reaches zero, the game will crash during an allocation.

    I've put a few snapshots here https://www.dropbox.com/sh/fm3qoel0qead5hn/AAA1BKuXxw_z9fqRSckNGW0ta?dl=0 , I'm not sure which is the best to check but if you could take a look, that would be very much appreciated.

    Some takeaways from your post:
    • When implementing a pool management system, it's better to just allocated the maximum space needed once and not progressively. Keep that memory pool in memory between scene changes if possible.
    • To investigate heap memory fragmentation, we should look at blue section that are not the last one. Allocated memory there that is not released is potential reason the space is not garbage collected and therefore creating fragmentation.

    Thanks
     
  4. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,906
    Right, so that's 12.8MB lost to fragmentation then.

    Right, so, I stitched your screenshots together, then copied together pieces of managed memory with bigger chunks of unused memory in them together and lined them up. While lining them up, I tired to compensate for actual used memory in these sections by eyeballing them and aligning them somewhat parallel. Stitched together like that, I continued until I had one full line, i.e. 8 MB of wasted memory. Between the last line I had taken a snippet from and it's next line, I split the screenshots, added a black background and left that cobbled together line there, starting with a new one until all pieces where lined up, totaling 1.6 lines x 8 MB per line: ~12.8MB eyeball measurement. I'm planning to adjust the memory Profiler to give a fragmentation analysis like that in a future version, so that this manual process shouldn't be needed but thought I'd prototype it like that, in a way, and use it to explain managed fragmentation while at it.
    Nope it doesn't, so the issue is likely elsewhere, or at least bigger gains are to be had when looking at other areas of optimization.

    Which tool are you getting these numbers from, specifically the Allocatable?


    That sounds like the issue is in memory not tracked by Unity, e.g. native plugin memory or IL2CPP? What platform are you running on and have you tried using a native platform profiler to see what that memory could be?

    Yep
    Yes but just to clarify: the GC does monitor it and collect unused objects in there, but that's little more than deleting the GC.Handle pointing at them. I.e. it's just deleting the thing used to find the object in the haystack. Once no GC.Handles point at that section anymore, this section of memory is just freed and returned to the OS.

    Not sure I'll have some time to look at the snapshots but if it's, as suspected, in memory untracked by unity, that wouldn't lead far...
     
    Last edited: Apr 19, 2021
  5. bagelbaker

    bagelbaker

    Joined:
    Jun 5, 2017
    Posts:
    63
    Ah, now I understand the "Managed memory lost to fragmentation" part!

    The numbers are outputed automatically to the debug output of the Switch. Descriptions from Unity:
    • Mem.Allocated - Calculated by walking the list of Unity internal allocators and summing the bytes allocated
    • Reserved - Similar to above but includes both allocated and unallocated memory held by internal allocators
    • GC.Heap - Size of the Garbage Collected/Managed Heap. Not included in Mem.Allocated or Reserved
    • Used - Currently allocated memory from the GC Heap
    • Allocatable - The largest block of memory available for allocation from the system allocator
    • There are a few one time allocations at start up that may not be tracked like the graphics firmware memory. Also, any allocation directly to malloc, realloc, etc from a native plugin will not be tracked.
    This issue happens on Switch. Like I said in the initial post, I posted on the Nintendo dev portal and didn't get any response. A lot of other posts had similar issues and Nintendo support often suggested it was a memory fragmentation issue and to try the memory map view of the memory profiler. I thought it might be a general issue (which is the reason I posted here) but only crashing on Switch because it is the platform with the most memory constraint. There's a native platform tool to inspect the heap and I'll explore that and get back to you with my findings.

    Thanks!
     
  6. bagelbaker

    bagelbaker

    Joined:
    Jun 5, 2017
    Posts:
    63
    With the native heap inspector, seems most of the memory lost per scene load is memory leaking due to wwise! We are unloading the soundbanks but somehow there's some memory not deallocated. Not sure what the cause is yet but we're going to further investigate this!
     
    MartinTilo likes this.
  7. bagelbaker

    bagelbaker

    Joined:
    Jun 5, 2017
    Posts:
    63
    Turns out it was a memory leak with wwise 2019.2.5 and updating to the latest version of 2019.2.9 has resolved the issue. Yay!
     
    MartinTilo likes this.