Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Bug Memory Profiler v1.1 is a regression

Discussion in 'Profiler Previews' started by strich, Jan 4, 2024.

  1. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    Unity 2022.3.15f1.

    Tried using the Memory Profiler recently on our development and editor builds with very poor results - Only a couple times of 10+ attempts did I successfully open a snapshot. All others would sit on Crawling GC Handles for 30+ mins before I quit.

    I downgraded to v1.0 and it opens the snapshots within a couple of minutes.
     
  2. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    Hi Scott,

    Could you do me a favor please and check if you modify version 1.1 by adding
    Code (CSharp):
    1. if(i >= 1000) break;
    here (i.e. ManagDataCrawler.cs line 642)?

    If that solves the issue, then it's this bug that we're aware of and for which I'm currently working on a fix (spoiler alert, the above isn't really one). The issue there is that we previously did not correctly scan the fields of structs/value types for potential pointers to managed objects, possibly missing out on references to some objects or even missing entire managed objects on the heap. 1.1. fixed that but we somehow didn't expect some worst case scenarios here. I.e. I've seen how having arrays of structs with over a billion entries can congest in the code linked to above. There is a question of whether or not such code should use NativeArrays or other Native Collections instead but the Memory Profiler should obviously still be able to deal with that.

    If this hotfix does not help, I'd like to ask you to file a bug report, attach a snapshot that won't open, and ping me just the issue ID as soon as you get it, so that I can look at this asap :)
     
  3. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    Yep that "fixes" it.

    Note that we don't use any Native Arrays or Collections if you mean ones of the Unity kind. Though we do use a lot of big lists for things as we've written our own ECS system, but its all C# stuff. Certainly we'd have a few list that are too big (one of the reasons we're trying to use the tool!), but surely not billions big.

    Can you provide an ETA on a new preview build with your fix in it at this point?
     
    MartinTilo likes this.
  4. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    Thanks for confirming the "fix", that could be a workaround until we push a fix version.

    I'd recommend Native Collections over managed ones. The Boehm GC treats every pointer-sized field as potential pointer to parse for garbage collection, so when the Memory Profiler starts getting perf troubles like this, the GC is likely suffering as well, without providing any benefits. Larger continuous managed memory allocations used for these will also contribute to Managed Memory Fragmentation.

    No ETA but I'll hurry the fix along. We need to have a bit if a huddle on the release and version strategy for the next versions but I'm hopeful we'll have something up soon.
     
    strich likes this.
  5. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    Our ECS has specifically been built to allocate contiguous large arrays for itself, duplicated per CPU thread for multi-threading. Its been in production for years now and we're in the last few months prior to release, so unlikely we'll be able to look into whether Unity Native Arrays are worth a genuine look at this stage unfortunately.
    That said there could be memory issues we're experiencing - That's why I'm diving into the Memory Profiler right now.

    Case in point - We're currently experiencing either a memory leak or a massive GC thrashing that we're trying to resolve. We're experiencing massive memory bloat of 200-300MB added per reload into the same level from main menu multiple times.
    upload_2024-1-5_21-14-1.png
    upload_2024-1-5_21-16-45.png

    • Can you suggest any resources for controlling the Incremental GC? We could definitely mark out time to give it during load but the API is pretty opaque.
    • I'm currently using Memory Profiler v1.1 with your workaround fix - Could that be causing a lot of red herrings in the numbers above?
    • Any additional advice you can give at this stage?
    • Running `GC.GetTotalMemory(false)` reports for example 2.5GB when the memory used as reported by Task Manager is 5GB. Even after the below GC Collect. What could be a cause of this? Fragmented free heap memory? Though a compaction (below code) doesn't seem to improve it.
    Update: Running the following GC Collect does not really affect the total memory usage as reported by Task Manager:
    Code (CSharp):
    1. Debug.Log($"Performing forced GC Collect. Current Total Memory: {GC.GetTotalMemory(false) / 1000000f}MB.");
    2. GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
    3. GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced, true, true);
    4. Debug.Log($"Finished forced GC Collect. New Total Memory: {GC.GetTotalMemory(false) / 1000000f}MB.");
     
    Last edited: Jan 5, 2024
  6. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    Your assuming the Boehm GC version shipping with Unity cares about any of those settings. It doesn't. It's way to old for that. It's non-generational, non-compacting/non-moving/non-defragmenting.
    The one extra call that could make a difference is GC.WaitForPendingFinalizers, though the pure usage of Finalizers is, in my eyes, an anti-pattern in Unity that will bloat and fragment your memory even harder, especially if you revive objects in a Finalizer.

    You're hitting pretty much what I'd expect here: Memory fragmentation because of way too large allocations. I'd seriously recommend switching to Native collections. There's a reason DOTS and Unity's ECS use them over managed arrays.

    Also, the Memory Profiler can currently not help you see what in the empty fragmented heap space would be objects that are ready to collect, vs those already collected. That is because it doesn't have data on the free lists and buckets, and through that crawler code you had to modify, just finds objects that are rooted and those referenced by rooted objects (or x levels down the line). Non-referenced objects are just not found, and neither are those only referenced by thread stacks.

    Further, unused pages of heap memory are freed during every 6th GC Cycle (warning, implementation detail and therefore could be subject to change) on a page by page basis. Calling GC.Collect 6 times could tell you if those pages are actually empty for debugging purposes, but that, as well as any manual call to GC.Collect is nothing I'd encourage as it's just as well liable to fragment the heap harder. Because it would be triggered automatically when there's not enough empty heap space for an allocation. And once the pages are unloaded, you loose that empty contiguous heap space that could've been used for the next big allocation.

    The incremental GC is neat for small, non-trivially avoidable allocations in moments where a synchronous GC spike would be bad. But it also means, when it triggers (because there's not enough space) a new heap section needs to be allocated for the new object right now even if the triggered incremental GC might free enough space a few frames down the line. This further fragments your memory.

    Version 0.7 of the package still has the fragmentation page that can be somewhat useful, if armed with enough knowledge of how the Boehm GC works, to analyze this all a bit, but it's also lacking some improvements to the crawler. Particularly: if any of those arrays of structs contained reference to managed objects, or even just an 8 byte (on 64 bit platforms) value that happens to match the address of an object, that memory isn't technically empty (at least to the eyes of the Boehm GC), but with that stopgap solution, just not found by the package's crawler.
     
    Per-Morten and strich like this.
  7. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    I do have some good news though, I've come up with an approach to speed up that array of structs processing that hopefully speeds it up enough (and allows for async multithreading it) that we might not have to cut our losses after x entries in an array and left to guess if Boehm might've misinterpreted the second to last element in it as a pointer.
     
    strich likes this.
  8. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    Firstly, that's a F***ing great amount of information mate. Thanks for taking the time.

    Unfortunately `GC.Collect()` 6 times doesn't do much.

    The other thing that worries me is that snapshot B has +400,000 managed objects. The 2 snapshots should really be near-identical. Just to be sure I'm reading it right - That's certainly our code leaking objects that are somehow still referenced somewhere right?

    I'll take a look at the v0.7 fragmentation page to help validate the theory. How does one actually download previous packages these days?
     
  9. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    Apologies for the delay:

    You can try turning it off entirely, or turn it of temporarily from code while you churn over (allocate or reallocate) big arrays.

    That's 2.5 GB of MANAGED memory, I assume your app also uses some GameObjects and other Unity subsystems (Native Memory) Textures (Native and Graphics Memory), DLLs (Native though likely non-resident memory) and any kind of C# Type data (Managed Virtual Machine Memory, which is Native in IL2CPP and "Managed" in Mono), the latter doesn't need reflection, just using code, though Reflection might balloon this usage up unnecessarily.

    As I explained over in this thread, you could also use ProfilerRecorders to monitor
    GC Used Memory
    and
    GC Reserved Memory
    or check the Memory Profiler Module's manual page for more counters to check that would include the Native and Graphics memory, though the latter only in Development builds. GC Used Memory would likely be the same as what the GC.GetGetTotalMemory() API returns and as that documentation page also says, this excludes fragmentation. Fragmentation would be included in the Reserved counter.



    Yes, the References panel should tell you what is keeping them around, but yeah, I'd assume it's most likely something in your code.

    Page size is likely 4 KB on your platform, so any blue strips that survived 6 GCs and are entirely empty, those have something pointing to them that the memory Profiler missed. Either something stack local, or in those arrays that weren't fully processed.

    All 4KB blocks that are mostly empty are fragmented and will stick around because of the few objects still in them.
     
  10. stefan_furcht

    stefan_furcht

    Joined:
    Feb 10, 2017
    Posts:
    10
    Hi there,
    I'm one of Scott's work mates and the main author of our ECS, which is based by its nature on large struct arrays.
    So we look forward to a fix for this case.

    I was trying to use the memory profile quite a bit, but I still struggle to really understand the output it creates and concrete documentation about comparing memory snapshots is quite scarce.
    So it wasn't very helpful yet to find our memory leak(s), since there are many unknowns to make sense of the output of the snapshot comparison of "All Of Memory".

    The leaks occur due to loading and quitting the same level again and again, so I did snapshots after each level load for some loads.
    To avoid any effects of lazy loading first time operations, I tried to compare the snapshot after second load with a later one.

    What we see is a huge increase in the sub category "Untracked*", a smaller increase of "Reserved", and a small increase in "Managed" and the latter is even negative at times. As of yet we are unsure what this means.

    image.png

    Description shown when clicking on "Untracked*":
    image (1).png

    Can you tell us more about this "Untracked*" type and what this what we see here actually means?
    How can our code create so much untracked memory?


    Obviously we tried to make use of the details viewed in the "Managed" category, but it doesn't make much sense or maybe is even incorrect.
    For example it found a "diff" on an array used in our terrain map between second and fourth level load.

    image (3).png

    If I understand the output correctly, then the tool tells us there wasn't a terrain map in snapshot A(second load), but there is now one in the snapshot B (fourth load).
    But this can not be true, since there will be always a terrain map or there wouldn't be a game.

    Almost all the diffs are based on such arrays popping out of nothing not recognized in the first snapshot but present in the second, while other disappear but should be there (or the game wouldn't work).

    So what is this "Diff" actually based on and how can it help in any way to find a leak?

    We have an hard time to leverage the tool to help with finding leaks.

    By just looking at the basic Profiler's memory section instead, we could figure the leak could have to do with save game serialization.

    The Memory Profiler - if it worked correctly, or it would be better understood by us - respectively, it seems it could be potentially a way better tool to find the leak, but so far we fail to interpret what it is actually telling us.

    Any pointers on these questions would be much appreciated.
     

    Attached Files:

    Last edited: Jan 18, 2024
  11. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    That's not what it is trying to tell you. It's only telling you something about the SplarseIntegetSet object. The Map object is just a direct referee to this object. SplarseIntegetSet is a managed object and has (at least) two different instances at different addresses in memory since it's getting created new for the reentry. The Memory Profiler has no chance to know your mental concept of "this is the same object", because it isn't and there are no hints it could use to make that determination.
    You could see of an object of the same type has been delete between the snapshots and make the mapping in your head. If there is none, maybe it's kept and you might want to turn on the option to "Show Unchanged" in order to see that you've leaked it, and then check the references to it to see why it was leaked.

    You could also find that Map Object (search by type name was added in 1.1) and check the value of the field holding the SplarseIntegetSet and see how that changed from the version in A to the one in B.

    I have plans to make it a bit more intuitive that could help in this instance as it'd be using the strongest binding root to structure the data, sort of providing you with that "understanding" that your level contains a map. Though that's a bit further off than the perf regression fix. I got sidetracked on something else for a bit but am back on this and hoping to be able to speed up processing so much that this super stopgap workaround won't be necessary, but have yet to get it to run and see if that hypothesis holds water.
     
  12. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    I've noted your comment about the documentation, but really, it'd make more sense to get to that improved workflow on the double rather than document it better, as it'd have to be rewritten and currently would need quite some more complex explanations that'd no longer be necessary then.

    So I'd rather revisit the documentation after that work is done.
     
  13. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    For Unity versions ore 2022.2 we've put that list into the tool there and in the first page of the package documentation for the versions up until 0.7:
    As of 2022.2+ the 2nd and 3rd element have been taken off that list.

    The first one can be avoided if the native Plugin uses the Native Plugin uses our Memory Manager API to allocate it's memory.
     
    stefan_furcht likes this.
  14. stefan_furcht

    stefan_furcht

    Joined:
    Feb 10, 2017
    Posts:
    10
    Thanks for this list.
    In our case it shows by far the largest positive diff between loads.
    As far as I'm aware, the stack wouldn't grow each new fresh load, there would be no reason for it. We do not use currently any Marshal.Alloc... calls in our project by ourselves.
    With 2nd and 3rd off the List "Native Plugin allocations" would be the only one left.
    So does that mean we are using a native plugin that leaks? And eventually not destroyed GOs dangling around, which are "DontDestroyOnLoad" (we switch scene between levels)?
     
  15. stefan_furcht

    stefan_furcht

    Joined:
    Feb 10, 2017
    Posts:
    10
    I will see tomorrow if I can now make better sense of it.
    For now big thanks for taking the time to leave these important pointers.
     
  16. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    That or something related to the math of the approximation mentioned in your screenshot. But I don't get the DontDestroyOnLoad comment, as that would be tracked.
     
  17. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    @MartinTilo we discovered a leak during our scene reloads and plugged it, however we still have a colossal amount (1.6GB) of memory sitting in "untracked memory" and no strong ideas on how to determine what it is or how to reduce it. We don't have any native plugins, this is just a few minutes into gameplay on a development PC build (not IL2CPP).

    The only lead I have is that we currently spawn some 150,000 gameobjects, with a lot of that for cached UI RectTransforms and components. I do wonder if that would outlay a lot of native Unity buffers. But I don't know.

    If you have any further investigation paths we could take that would be amazing! Or could this be a red herring?
     
  18. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    My colleague went a bit into the details of 1.1.0 + 2023.1+ and how that lists details for Untracked in his United talk, maybe that helps?



    As for the opening time bug / ManagedDataCrawler speedup work, that is still ongoing. It's gotten a bit more complex than I'd have hoped but the basic structure looks solid now, so I'll hopefully "just" have to debug out the bugs in that solution and make sure the data is correct. That said, it should be a nice speedup for the opening times.
     
  19. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    Thanks. Yeah we'd already watching the talk - Unfortunately not much new info vs whats in this thread already.

    Looking forward to the next version of the Memory Profiler! But yeah the "Untracked Memory" is currently our biggest problem and AFAIK despite all our efforts so far there is no way to further isolate what that is.
     
  20. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    Have you tried updating to 2023 to take a capture from that to see what it says untracked is made up off?

    I.e. just upgrading a branch for profiling, making a build with 2023 and profiling that
     
  21. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    Oh does 2023 have memory and/or profiling improvements? Upgrading the project to 2023.2 would be several days of hard work unfortunately, since it appears that TextMeshPro has been outright removed and added into the UGUI package - Its quite the little surprise to find and a massive spaghetti to unwind.

    By the way, another interesting point: We have come to believe our UI (UGUI) is a potential major issue as it is a poor implementation with I'm not kidding 50,000+ RectTransforms all up, and more in terms of total components. We hacked it out and ran a profiling session to compare:
    image (8).png image (7).png
    Note that Untracked Memory has a 440MB diff! There must surely be something the Memory Profiler is missing here as this is all pure Unityland stuff that is being created.
     
    Last edited: Feb 9, 2024
  22. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    Yes. While I seem to have misremembered and 2022.3 already has a bit of a breakdown the details for Untracked, seeing what part of that is Resident vs not is only possible with captures from 2023.1+. I also think it adds a bit more details in that category in general, but the level of detail might depend on the platform you're profiling. It's what Anton shows here in his talk. Don't skip past the Graphics section though as, as he explains there, graphics resources might need to be expunged by the OS and Untracked might therefore encompass the CPU copy of a texture as retained by the Graphics Driver, though that might not be resident memory.


    :eek: no kidding. The RectTransforms themselves and serializing their hierarchy likely contributes to a good chunk of that native memory difference there. The Textures might, as outlined above, also contribute to Untracked via their driver side backup copy.
     
    strich likes this.
  23. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    I tried 2023.2 on a fresh empty project against a PC debug build of the game and unfortunately it doesn't show much if anything new. Untracked still only shows 2 sub-items, unlike in the video where it shows a large list of interesting items - I guess this is only available on mobile?

    So far we've been whittling away on the Unity objects list and our UI implementation, and its improving so that's good. But its a shame we cannot capture any further info in Untracked.
     
  24. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    So you didn't build with 2023? The version of the capturing Editor is unimportant for what data is available. The entire capture logic is in the player.
     
    strich likes this.
  25. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    I didn't no. Its quite a large task to upgrade to 2023.1, and 2023.2 is a massive task due to TextMeshPro changes.

    No chance of a backport?
     
  26. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,490
    Sadly no. You could make a build with an empty or example project just to see what kinda extra info you you might get though.
     
  27. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    We have a new data point - We've worked with a few other gamedevs to test their own projects:
    • One of those projects is on 2023.2 and the Untracked is only slightly expanded in detail from 2 to 4 sub-categories. Not much to glean for that information unfortunately.
    • We've been able to have a few other gamedevs test their own large projects in Memory Profiler and they've all reported large 1GB+ Untracked memory regions. So its starting to tell us its not something specifically in our project, which is good. On the other hard its a bit sad we're loosing 1GB+ to "unknown" - On Xbox 1 that is 20% of the total RAM allocation (including shared GPU memory!).
      • Are you guys absolutely sure this Untracked category is truly externalities? We don't have the evidence yet coz there is a lot of white noise, but a general feeling is that the more Unity objects that are used the bigger that region gets. It can't be just GPU driver asset copies - We don't even have that much texture usage at this point.
     
  28. HanHanunited

    HanHanunited

    Joined:
    Mar 9, 2019
    Posts:
    1
    We also run into the problem that the "reserved" spaced took up to 1.3GB of our memory. Since we are porting our game to Switch we need to tackle down this problem .

    We are currently using Unity 2022.3.8f1 so i upgraded to 2023.2.11f1 to try the new memory profiler 1.1. After doing some tests i saw that the overhead moved from reserved to untracked into the "private" section. So what is this "Private" Section about and why has it about 1.01GB allocated size and only 282mb resident size. Does it mean that most of it could be empty?
    upload_2024-2-29_2-4-55.png
    Also if i count the resident size of all reserved sections plus untracked: 270mb (Native) + 490mb (Managed) + 282mb (private Untracked) => ~ 1.1GB we have over 1GB memory usage which actually isn't in use. We are using Addressables + Sprite Atlases to improve our memory usage but currently think that with this overhead we have no chance to get under 2GB to port to the Switch. We can improve our addressables and textures sprite atlases more but in the screenshot above Textures only takes up to 450mb.

    I also have a snapshot from our switch (But with Unity 2022.3.8f1) right before it crashes because we got over the memory limit. There we have an unknown section in Native -> Unity Subsystems -> Unknown with about 0.51GB.
    Also we have an Untracked -> Untracked with about 220mb upload_2024-2-29_2-10-26.png upload_2024-2-29_2-11-31.png

    Am I missing something how we couldt investigate this problem further down?
     

    Attached Files:

    strich likes this.