Search Unity

  1. Unity 2019.1 is now released.
    Dismiss Notice

Assets [WIP] Heap Explorer - Memory Profiler, Debugger and Analyzer for Unity

Discussion in 'Works In Progress' started by Peter77, Apr 22, 2018.

  1. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    header.png

    The documentation lists all the things Heap Explorer currently can and cannot do.

    cs_objects_view.png compare_snapshot.png
    Click thumbnail to enlarge. The documentation contains more screenshots.

    Introduction

    Heap Explorer is a memory profiler, debugger and analyzer for Unity. I spent quite some time identifying and fixing memory leaks, as well as looking for memory optimization opportunities in Unity applications in the past. During this time, I often used Unity’s MemoryProfiler and while it’s an useful tool, I never was entirely happy with it.

    I then decided to write my own memory profiler, that helps me in future projects to achieve my work more easily and faster.




    Which Unity version is supported?

    I tested Heap Explorer with Unity 2017.4.6f1 LTS. Earlier versions are not supported on purpose. Capturing a memory snapshot with Unity 2017.4 and Scripting Runtime .NET 4.x does not work, see here for a workaround.


    What's new?
    Don't miss to watch this forum thread, in order to get notified about future updates!


    Download

    I decided to provide Heap Explorer beta for free for now. I don't know if I change my mind in the future, but here you have a Memory Profiler, Debugger and Analyzer for Unity, that is able to work with various larger projects where Unity's Memory Profiler fails. You're welcome.

    Download the latest version from:
    http://www.console-dev.de/bin/heapexplorer.zip


    Source code

    As of February 24th, the entire C# source code to build Heap Explorer is available at:
    https://bitbucket.org/pschraut/unityheapexplorer/src


    Known Issues

    Heap Explorer uses Unity’s experimental MemoryProfiling API, which contains various bugs, from cosmetics to major issues that make memory snapshots not trustworthy.

    These bugs occur in every application that use Unity’s MemoryProfiling API, such as Unity’s own MemoryProfiler tool. I hope Unity Technologies is going to fix them.

    I added a "Known Issues" section to the Heap Explorer documentation, that lists all MemoryProfiling API bugs that I reported.
     

    Attached Files:

    Last edited: Apr 14, 2019
    _met44, El-Tun, angrypenguin and 9 others like this.
  2. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    [reserved for future content]
     
  3. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    [reserved for future content]
     
  4. JJJohan

    JJJohan

    Joined:
    Mar 18, 2016
    Posts:
    185
    Very interesting, I'd love to get my hands on this tool when it comes out. I work on an Enterprise-orientated application that involves constant data streaming and our resource handling systems are quite complex. Especially now that we're targeting WebGL it's been critical for us to keep memory low.

    I've used Unity's MemoryProfiler (a somewhat modified version so it doesn't struggle as badly with a large memory heap in use) but I've found it limiting. I particularly like the sound of the delegates view as I wouldn't be surprised if we have a couple of those leaking in places.

    Do you have a rough timeline of further work on the tool before you consider publishing it on the Asset Store - perhaps even with a price in mind? Personally I could already see it being valuable enough to a lot of people in its current state.

    Out of curiosity, what .NET "target" have you been testing with? The only reason I ask is because I was quite surprised to see strings not sharing memory, and am wondering if this was an issue with the older Mono runtime or if it's actually something Unity is doing internally.
     
  5. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    If you're interested, I can send you an alpha build. The tool works with Unity 2017.3 and newer only though. The current alpha build works with the limited amount of projects I have and tested it with, except for the various known Unity issues I documented in the manual.

    I'm particularly interested to see whether Heap Explorer works with complex projects, because this was one of my goals to make possible. Heap Explorer does have a lower memory footprint than Unity's MemoryProfiler and saves as well as loads memory snapshots faster according to my tests.

    The "Delegates" and "Object Duplicates" views caused many "Oh really" moments on my end, I found them very insightful.

    It pretty much depends on Unity Technologies and whether they're able to repair their Memory Profiling API, which, to be fair, is marked as experimental.

    However, I would feel very uncomfortable to provide a tool through the Unity Asset Store, where I know that it's built on top of an API that does not work in various cases. I mean, their Memory Profiling API seems to report incorrect data in some cases, which makes memory snapshots not very trustworthy. The "good" thing is that it affects every tool, including Unity's MemoryProfiler as well.

    If Unity Technologies repairs their software and provides an API that does work, that would be moment when I would consider to provide it through the Asset Store.

    Until then, I would like to test the tool in a smaller circle to collect feedback.

    I tested with "Stable (.NET 3.5 Equivalent)".

    Btw, thanks for your post!
     
  6. JJJohan

    JJJohan

    Joined:
    Mar 18, 2016
    Posts:
    185
    I'd definitely be interested in the alpha build - even if it's not a bit experimental I'm sure it'd still give us some interesting insights nonetheless, even if some of the memory sizes and connections are occasionally out of whack. We upgraded to 2017.3 not too long ago so we should be good to go. I'm surprised there hasn't been more interest yet.
     
  7. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Please check your private messages, I just sent you a download link to the tool.

    I developed Heap Explorer with/for Unity 2017.3.0f3. I recently upgraded Unity to 2017.3.1p2 and Unity seem to have introduced a new crash bug in a version between 0f2 and 1p2 :confused: The crash occurs when capturing a memory snapshot of a build. It works in the editor though.

    If someone from Unity Technologies is reading this, please fix this crash:
    (Case 993250) PackedMemorySnapshot: RequestNewSnapshot causes Player to crash
     
  8. RG_Keith

    RG_Keith

    Joined:
    Mar 14, 2017
    Posts:
    26
    I'd be very interested too - I've tried Unity's profiler on Bitbucket, and found it practically unusable for a moderately-sized project (we have a ~2gb footprint, and will get a lot bigger before we ship).

    I just saw a new talk announced for Unite Berlin, entitled "Memory Profiler: The Tool for Troubleshooting Memory-Related Issues" - I wonder if they're working on a new profiler, or if they're just going to talk about the existing one?
     
  9. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    I just sent you a private message with a download link.

    Interesting! I hope it's going to be available on youtube as well.
     
    RG_Keith likes this.
  10. RG_Keith

    RG_Keith

    Joined:
    Mar 14, 2017
    Posts:
    26
    I'm sure it will, they're pretty good at uploading every Unite talk to YouTube pretty quickly after the conference.
     
  11. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Hello,

    I tried it on few of my personal small projects, and there is immediately one feedback I can give - being able to do a capture and save it right away without analyzing it automatically after capture would be cool - especially on a large projects, when I guess the analyzing would take a long time, and I would rather do a few snapshots quickly during gameplay, and analyze them later separately, e.g. by using the diffing feature. :)

    Also, when I tested it with 2018.1, I immediately stumbled upon an exception when analyzing the snapshot.

    Code (CSharp):
    1. IndexOutOfRangeException: Array index is out of range.
    2. HeapExplorer.PackedManagedObjectCrawler.InitializeCachedPtr () (at c:/Users/crash/Documents/HeapExplorer/Code/Code/PackedTypes/PackedManagedObjectCrawler.cs:65)
    3. HeapExplorer.PackedManagedObjectCrawler.Crawl (HeapExplorer.PackedMemorySnapshot snapshot)
    If it's not reproducible for you with 2018.x (since you said 2017.3 and newer), I'm willing to send you the project where I'm hitting the exception privately, since it is just a personal student project I did like a year ago or so. :)
     
    Peter77 likes this.
  12. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Were you able to successfully capture and analyze a snapshot with Heap Explorer of any of your projects?

    That's actually a very useful feature. It's implemented in alpha 1.2, which you can download using the original download link I sent you in the very first message.

    I was able to reproduce this issue with 2018.1.0f2 and 2018.2.0b3.

    It seems they introduced a new bug in 2018.1 where the types description array in the memory snapshot is empty. I just reported this issue:
    https://forum.unity.com/threads/cas...pshot-typedescriptions-array-is-empty.530047/

    That's a bummer, because the MemoryProfiling API did work in various 2018.1 beta builds (or this specific issue did not exist). Sad to see how it falls further apart with pretty much every Unity update. It seems there is no recent Unity build anymore, where the API actually works.

    I really hope UT is going to fix these issues soon, but it's more likely that it takes a few months before those fixes land.

    Thank you for giving Heap Explorer a try and providing feedback. I really appreciate that!
     
    pavelkouril likes this.
  13. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Yes, but since I have the 2017.3.1p2 installed, I just tried the in-editor capture. I just wanted to look at the UI and get familiar with it a little bit, before I try it on the bigger project I wrote you about in PM. :)

    I hope I'll be able to capture the bigger project from a build and not editor, but I'm afraid I'll hit the stupid "Receiver can not keep up with the amount of data sent" error we also hit when I tried the Unity's profiler (which is something you can't do nothing about, since it is an issue with profiler directly).

    Thanks!

    Let's hope for a fast fix then. Being able to profile just a few specific versions of 2017.3 would really really suck. :/

    No, thank YOU for making it!
     
    radiantboy and Peter77 like this.
  14. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Good to hear that it works at least in the editor on a different computer than mine.

    I'm looking forward to what you'll find out. I hope this affects the Profiler only, but not the MemoryProfiling API, but I don't know. If you run into this issue, perhaps it makes a difference if you turn off 'Record' in the Profiler window, if it's enabled at all.

    I've just uploaded an update again. Here is the changelog...
    • In Heap Explorer toolbar, added “Capture > Open Profiler”. This opens the Unity profiler window, which allows you to connect to a different target. It’s simply a convenience feature, if you figure the editor is not connected to the correct player. This saves a few mouse clicks to get to the Unity Profiler to connect to a different player.
    • The “Load” button in the “Compare Snapshot” view now features a “most recently used” snapshot list. This allows you to switch between previously saved snapshots with fewer mouse clicks.
    • In Heap Explorer toolbar, added “File > Recent”, which allows to re-open the “most recently used” snapshots.
     
  15. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Here's some more feedback I stumbled upon:
    • Be able to swap (A) and (B) in compare easily. It is "intuitive" to load "before leak" capture first and "after leak" as second, but to see the diffs more clearly and not inversed, you should load it in the other direction to get B-A, not A-B. :)
    • Do not reset current View when opening a new snapshot.
    • Do not reset the chosen second snapshot in diff view. Wouldn't mind unloading it when exiting the view, but analyze + load it back again when returning to the view? But I understand this might make things slow with really really big captures. Maybe a checkbox to "remember selection" could be an ok solution?
    • Click on Address in Memory Sections view does not open its detail, but acts as a rename.
    • And a totally cool feature would be the option to filter the results by selecting some new "root" object, and show objects that are referenced by the selected object (recursively), to e.g., see how much some terrain engine/audio engine/etc. takes memory (approximately) - with the option to optionally remove some objects from the hierarchy, so you don't reference a global manager and include all the other subsystems anyways.
    • Sums of the filtered size/count rows.
    Other than that, the tool is really really superb!
     
    Last edited: May 7, 2018
  16. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Great, thanks a heap! :)

    I uploaded a new version that contains some of your feedback. I didn't implement all feedback yet, I went with the low hanging fruit first...

    Here is the 1.6 change log:
    • "Compare Snapshots", changed diff from "A - B" to "B - A" comparison. I'll see how to best implement your suggestion to swap snapshots and avoid unloading the other one.
    • Fixed being unable to select a memory section by clicking on its address in the "C# Memory Sections" view. I used a selectable label first, to be able to select and copy the address. I've implement a context menu when right-clicking on any address now that provides a "Copy" option.
    • Loading a snapshot restores the previously active view when done. I still throw out the old snapshot before loading, to release its memory. So while the snapshot is loading, no view is visible except for the "Loading..." label. But it restores the view afterwards.
    Can you elaborate on this please? I don't get it :)

    Is this meant for the existing views or for the "option to filter the results by selecting some new root object" suggestion? What do you mean with "filtered", if you enter a search text? Or is this perhaps meant for the "Compare Snapshot" view only?

    Thanks again!

    PS: Unity was able to reproduce the "missing typeDescriptions" problem, why heap explorer doesn't work with 2018.1. Turns out this affects a lot more Unity releases, but not 2017.3 (yet). It's even broken in their long term support stream 2017.4, so nobody is able to take a memory snapshot with any tool using the most stable Unity release yet. :eek: Crossing fingers that it's being fixed soon.
     
  17. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    I'll try to do a more textual description of the thing, with a made up example. :)

    Ok, let's say you have some "GameManager", that has fields that contains "WorldManager" and "MusicManager" (among others), but also both managers have a "backref" to the GameManager. The "WorldManager" has some data about spawned objects (units/vegetation/...), and also data about terrain, and the terrain data has arrays.

    Being able to select "WorldManager" as a root object, and calculate how much memory does it + all the objects it references contain (recursively), would make it easy to determine for example that your world engine part of the game takes approximately ~100 MB. Being able to exclude some objects from this calculation is however really important - for example for the references to other systems (either directly or through back references), so you can "isolate" the memory footprint of a subsystem.

    Basically, in a pseudocode, something like this.

    Code (CSharp):
    1.  
    2. Queue toVisit = { worldManagerInstance };
    3. while (toVisit.Count > 0) {
    4.     var obj = toVisit.Pop();
    5.     if (typesToSkip.Contains(obj.Type)) continue;
    6.     foreach (var field in obj) {
    7.         if (field.IsReferenceType) toVisit.Push(field.Value); // adds the pointer/whatever to the queue to visit it later
    8.     }
    9. }
    10.  
    Basically for any view in the datagrid, having a sum size/count rows at the bottom would be beneficial. If you do not filter using the search, you get the value next to the search bar, e.g. 110243 managed objects, 11.7 MB memory; so these numbers reacting on any "filtered" view (e.g. by the root selection mentioned previously, or text search), would make it again much better in usability IMHO.

    Also, there's a bug when I sometimes need to type twice (when deleting the text in between searches) to actually make it trigger the search, but I can't find a repro. :(

    Cool!

    I tried capture today with our bigger project (in a huge level, it downright crashed the editor after a few seconds - submitted a bug report to unity, we'll see what I'll get as a reply). For a smaller level it felt like some data were missing in the snapshot? Like, by doing a calculation about the number of objects/arrays I definitely know of, it showed like ~1/5th of them? Is it possible there are some objects in the snapshots missing? Also, there was a HUGE disparity in the profiler vs heap explorer. The HE showed 35.8MB managed memory in the managed objects, the managed sections said 0.5GB over 3.5GB fragmented, and profiler showed these numbers:

    upload_2018-5-8_20-0-2.png
     
    Peter77 likes this.
  18. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Thank a lot for the explanation. I get it now! :)

    Did you get a reply, what did they say?

    Maybe it's related to the custom type in your project where Heap Explorer has problems with? As far as I remember it's an array, perhaps it pulls in these managed objects which are missing, because the array is ignored?

    Other than that, if you're able to capture a memory snapshot using Unity's Memory Profiler from bitbucket and send me that screenshot, I can check if the managed object crawler in HE is reconstructing different objects than Unity's tool.

    That was basically my approach when I implemented the object crawler. I captured a memory snapshot using Unity's MemoryProfiler. Then I extended their profiler to output all managed objects (addresses, type, etc) to a file which I imported in Heap Explorer. Then I was able to read the original snapshot of Unity's Memory Profiler, run the managed object crawler in Heap Explorer over it and afterwards compare if HE reconstructed the same objects as Unity's MemoryProfiler.

    The object crawler is a complicated beast, perhaps it still has some issues, even though all the snapshots I tested were ok. If you can provide the Unity MemoryProfiler snapshot, I can use it to check whether Heap Explorer is failing.

    I came across this one as well and reported it a while ago as:
    https://issuetracker.unity3d.com/is...edheapsections-do-not-match-stats-in-profiler

    It's not possible to "reconstruct" the numbers of Unity's Profiler using their MemoryProfling API.

    Which further leads to this issue:
    https://issuetracker.unity3d.com/is...-when-checking-through-a-packedmemorysnapshot

    ...and eventually to this issue, if you use 2018.1 and native buffers:
    https://issuetracker.unity3d.com/is...shot-nativearray-memory-is-not-captured-in-it
     
  19. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Haven't replied yet. I'll keep you posted when they do. :)

    Actually, in this case, this is an array of struct that has two byte values. By a simple calculation, there should be around ~150MB of this byte data arrays, but it shows only ~45MB. I'll try to do more experiments and stuff next week though, and will probably iterate over the all known arrays of this type manually and debug.log how many are there and how many the heap explorer shows (maybe my calculation was just wrong).

    The Unity memory profiler wasn't even able to capture a main menu for us, yet alone the level.
     
    Peter77 likes this.
  20. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    So, yeah, the snapshot really doesn't contain everything. :(


    I counted it using for loop and traversing, and it displayed about ~15% only of the byte data there really is (~2500 instances instead of ~20k).

    The good thing is that it's really easily reproducible. Just create a new project and put this script on some object, enter play mode and then do a capture. But I guess this will be an issue with Unity's API and not your Heap Explorer anyways? :(

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4.  
    5. public struct ByteData
    6. {
    7.     byte v1;
    8.     byte v2;
    9. }
    10.  
    11. public struct Block
    12. {
    13.     public ByteData[][] Datas;
    14. }
    15.  
    16. public class Test : MonoBehaviour
    17. {
    18.  
    19.     public Block[] Blocks;
    20.  
    21.     void Start()
    22.     {
    23.         Blocks = new Block[75];
    24.         for (int i = 0; i < 35; i++)
    25.         {
    26.             Blocks[i].Datas = new ByteData[512][];
    27.             for (int j = 0; j < 500; j++)
    28.             {
    29.                 Blocks[i].Datas[j] = new ByteData[4096];
    30.             }
    31.         }
    32.     }
    33. }
    34.  
     
    Peter77 likes this.
  21. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Excellent, many thanks for the reproduce, this is definitely going to help me to understand what's going on. These are managed objects, so it's very likely that the issue is in Heap Explorer.

    Unity does not provide "ready managed objects" in the snapshot, but just the "raw managed memory" and some GCHandles. Heap Explorer is then reconstruction those objects with that data. That's the responsibility of the PackedManagedObjectCrawler, which caused a few issues in your projects so far.

    I'll report back as soon as I found something. It probably takes me a few days, because debugging snapshots that contain a lot of objects was always quite difficult so far.
     
    pavelkouril likes this.
  22. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Yeah; I tried to capture the repro case using the "official" memory profiler, and it shows 17500 items with 137 MB of memory, which means it shows all of the objects. So it's definitely an issue with HE.

    But the official profiler is uttery inefficient, it took unbearable amounts of time (tens of minutes) to unpack a snapshot of the empty scene in-editor with just this script attached. No wonder it can't capture and unpack a snapshot of a real game. :/
     
    Peter77 likes this.
  23. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Good news, I fixed it and uploaded a new version! :)

    Interestingly, it was actually a feature why it didn't show all objects. My assumption was "If HE crawled more than 10.000.000 objects, then something is probably wrong and I just stop crawling further objects". However, 35*500*4096 = more than 10 million objects.

    Soooo.... I removed the safe guard code and now Heap Explorer is able to crawl and find all 17500 arrays, which contains 4096 elements each. However, the problem then was: Crawling is utterly slow.

    I then came up with a pretty neat optimization, that helps in this specific case. I need to crawl only those objects, that are able to hold references to other objects. I now implemented that Heap Explorer checks if the type to crawl contains any non-ValueType instance field. If it's just a simple struct with no references to other objects (like the ByteData struct of your example), then the entire array does not need to be crawled and not doing any work is always the fastest thing :p

    Heap Explorer can now skip millions of objects that never would pull in any other objects, analyzing the test-code you provided is now done in a few seconds. \o/

    PS: I implemented a few unit tests during development and those still pass, so I'M pretty confident the optimization didn't break crawling ;)
     
    pavelkouril likes this.
  24. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Cool!

    I'll retest it tomorrow with the big project then. It's kinda crazy we have so many managed objects in the memory tho. :O

    Just one thing; do the old .heap files work, or do I need to recapture?
     
    Peter77 likes this.
  25. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    The old .heap files still work, the data layout didn't change. Just the way I interpret the data changed.

    I'm totally looking forward your feedback. I'm crossing fingers that it now works with the big project!

    BTW, if you're able to create a reproduce for the issue with those few types that generate errors, I would still be very interested to take a look at that :)

    Thanks again for your help!
     
  26. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Interesting. Because with the old snapshots, I'm seeing completly empty memory sections (of the 16MB etc. sizes, not the small ones).

    Or is that expected that some of the big sections do not contain a single object at all?
     
  27. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    With new snapshots, I see more that with the old. So are you completly sure the old snapshots still should work? Also, with new snapshots, I'm not getting the wierd errors anymore. So that's solved too.

    Also, the loading etc. is definitely a lot slower than in previous versions, but I guess this is a fair price to pay for the "full" info.
     
    Peter77 likes this.
  28. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    That's odd. I can't explain why this information would miss in old snapshots, because I don't save the crawled data as far as I remember, but only the raw memory snapshot Unity provides. But if that's the case, then there must be something different, I just can't think of what. :confused:

    I'm not really certain. I came across this as well and here are the options I believe are possible...
    • If the entire memory section is all zero, then the section has no objects. You can check this if you click the C# icon next to the memory section address. It opens a hex memory view. A function to verify that is probably helpful, but does not exist yet.
    • The memory section might contain untracked objeccts. This would be at least GCHandles. Unity provides a list of GCHandles, but not at which memory address the GCHandle is located, only its target object. Means there is no way for me to figure out at which address a GCHandle is located and if a memory section contains a GCHandle, I can't resolve it.
    • If you click the C# icon next to the memory section address, it is actually contains data, perhaps you can see what data it is.
    • If there is a bug in HE which reconstructs the managed objects.

    If you have a snapshot, that I can use to reproduce, I would be happy to look at it.:)
     
  29. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    I uploaded an update, same download link as before.

    Here is the changelog:
    • Compare Snapshots: Added "Swap" button to swap snapshot A <> B as suggested here.
    • Compare Snapshots: Snapshot B doesn’t get unloaded anymore, when loading or capturing snapshot A, as suggested multiple times.
    • Compare Snapshots: Loading Snapshot B now displays the same loading status information as when loading Snapshot A.
    • C# Static Fields: Added toolbar menu with functionality to save the memory of a selected static field as a file.
    • C# Memory Sections: Added toolbar menu with functionality to save the all C# memory sections as a file.
    • Moved statics to bottom status bar.
    • Debugged why some memory sections do not contain references to managed objects, but contain non-zero bytes.

    Finally understood why some managed memory sections appear empty!

    Furthermore, I finally understand why some managed memory section appear to have no managed object references, but do contain non-zero bytes.

    This is caused by static field memory being stored in managed heap sections as well. Unfortunately, the information Unity provides regarding staticFieldBytes is very limited in their Memory Profiling API.

    A PackedMemorySnapshot contains for every managed type an array, that represents the memory that is used to store its static fields memory, named staticFieldBytes. Unfortunately, it does not provide at which address in the managed heap this would be located.

    Additionally, the staticFieldBytes are not only stored in their corresponding types, but also stored across managed memory sections. So the PackedMemorySnapshot contains this information at least twice.

    Due to missing functionality in the Memory Profling API, I’m unable to resolve what staticFieldBytes are stored in which MemorySection.

    That means, if a MemorySection contains staticFieldBytes, I’m unable to detect this and can’t visualize it in Heap Explorer.

    Here is how I figured it out: I searched for the bytes of largest staticFieldBytes in all managed memory sections. It was found in a memory section, which appears to be empty. I was able to reproduce this with all staticFieldBytes arrays I tested.
     
    Last edited: May 20, 2018
    pavelkouril likes this.
  30. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Great! Love the additions! Will make using the tool a little bit more enjoyable again. :)

    So - are you going to report this to Unity so they fix it?

    It's a shame Unity's memory profiling API seems to be really broken and incomplete, and therefore just limits this otherwise awesome tool. Especially that it doesn't work at all on 2017.4 and higher is really really dumb. :/
     
  31. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    I'm going to report it and add the case number to the Known Issues section in the documentation.

    I hope a text description is sufficient for them, because coming up with a simple project to reproduce this would be very difficult and time consuming. Also not sure whether they see it as bug or a missing feature. We'll find out.

    Yep, not cool at all. Crossing fingers they're going to fix that :( Too bad I don't have access to Premium Support anymore, this could have been useful to perhaps speed this up a little.
     
  32. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    I uploaded an update, same download link as before.

    Changelog
    • Overview: Added percentage % next to the size
    • Overview: Added visualization of managed heap fragmentation in the operating system memory
    • Overview: Fixed some layout issues
    • Overview: Removed GCHandles and VirtualMachine Information from the overview, as this wasn’t very interesting nor something that is normally using much memory.

    Here is how the Overview looks now:
    overview_view.png
     
    Last edited: May 21, 2018
    pavelkouril likes this.
  33. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    125
    Wow, the update looks cool.

    Just - I'm personally against the removal of the VM info (or is it still somewhere else?); knowing the basic info about the VM, including the architecture/pointer size might be good, especially when capturing stuff for different archs/targets and then looking at them retrospectively to find similar issues in the snapshots etc.

    Are you sure that memory fragmented over 450 MB of operating system memory means the app will eat as much in the process explorer? I'm honestly not sure, but if there are completly empty "pages" that the system can give to someone else, then this result might be kinda skewed?
     
    Peter77 likes this.
  34. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    I can add it back.

    I don't know. All I know is that the Unity memory snapshot contains a bunch of managed memory sections spread across that address space. Which, according to the "Understanding the managed heap" documentation, is allocated from the operating system.
     
  35. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Uploaded an update, same link as before. I brought back the GCHandles and Virtual Machine Information.

    Changelog
    • Overview: Brought back "GCHandles" and "Virtual Machine Information"
    • When loading a snapshot from the "Start Page", it opens the "Brief Overview" afterwards, rather than the "C# Objects" view.

    overview_view.png
     
  36. QFSW

    QFSW

    Joined:
    Mar 24, 2015
    Posts:
    2,333
    Looks super cool! Haven't had time to read the whole thread, but what's the current stance of this since op? Going on asset store?
     
  37. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    The tool is based on Unity's experimental Memory Profiling API. This API is so experimental, that a lot of stuff is not working and missing. Using the API causes many different Unity versions to crash. The only version that kind of works is Unity 2017.3.0f3, it has the fewest issues in that API. However, Unity's Memory Profiler on Bitbucket suffers from these issues too, so I hope UT is going to fix that.

    According to the feedback I received so far, it seems Heap Explorer is a helpful tool. It's able to analyze snapshots that Unity's Memory Profiler can't. This was one of my goals, to provide a tool that can be used in bigger projects and this seem to be working according to feedback.

    Before I can make this decision, Unity needs to fix their Memory Profiling API. I'm not interested to provide it via the UAS, as I know the API where it's based on, is not functional.
     
  38. QFSW

    QFSW

    Joined:
    Mar 24, 2015
    Posts:
    2,333
    Thanks! Looking forwards to trying
     
  39. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Uploaded an update, same link as before.

    Changelog
    • C# Memory Sections: Visualization of managed heap and individual memory section usage/fragmentation
    I recorded a video this time, where I show off the recent changes. I hope the video isn't too bad and proves useful :eek:




    @Lucas-Meijer You suggested to add a fragmentation view to Unity's memory profiler, what do you think of this implementation in Heap Explorer? Anything you would change or think is missing?
     
    Last edited: May 25, 2018
  40. QFSW

    QFSW

    Joined:
    Mar 24, 2015
    Posts:
    2,333
    Awesome! Hyped to try it
     
    Peter77 likes this.
  41. skycc

    skycc

    Joined:
    Oct 1, 2010
    Posts:
    11
    does it support mobile devices connected to editor?
     
  42. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    I tested Windows Standalone and Xbox One (Mono & IL2Cpp) only. Both work(ed).

    Heap Explorer uses the same underlying API as Unity's Memory Profiler, which is able to capture snapshots from mobile devices, therefore it should work with my tool as well.

    Unity just broke the Memory Profiling API in almost every recent Unity release, means currently neither my nor Unity's Memory Profiler work on any platform but the editor itself, otherwise I would have tested it with an Android Tablet now.
     
  43. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    I just noticed the following fix in the 2018.2.0b6 release notes:
    • Editor: Fixed "Can not send network message. Receiver can not keep up with the amount of data sent" when capturing memory snapshot using bitbucket profiler.
     
  44. SugoiDev

    SugoiDev

    Joined:
    Mar 27, 2013
    Posts:
    220
    Oh, I've been having issues with that for a long time. Sadly, it's still present in recent versions. Maybe not as frequent, but I'm not sure.

    That changelog is for the entire 2018.2 beta, probably one of the earlier versions.
    I'm hoping they'll give it another pass before 2018.2 is final.
     
  45. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Uploaded an update, same link as before.

    Changelog
    • Added functionality to guess asset duplicates
    • Added functionality to search for a native object as asset in Unity's project view


    Let me know whether you think the new feature comes in handy.
     
    SugoiDev and AcidArrow like this.
  46. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    It seems Unity Technologies fixed the Memory Profiling crashes in Unity 2017.4.6f1. I was able to sucessfully capture a memory snapshot of the editor and of a windows standalone build as well.

    From the Unity 2017.4.6f1 release notes:
    • (1011626) - Scripting: Fixed memory profiler.
    https://unity3d.com/unity/qa/lts-releases

    Thank you people at Unity!
     
    MartinTilo, QFSW and SugoiDev like this.
  47. QFSW

    QFSW

    Joined:
    Mar 24, 2015
    Posts:
    2,333
    We gonna see a release now? :O
     
  48. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Nope, not yet. Unity Technologies announced at Unite Berlin 2018 they're working on a new Memory Profiling API and tools, such as an overhauled Memory Profiler :(


    Sooo... I'll wait for the new stuff, to check out what they came up with, which might be around summer this year, I believe. From there, I decide whether it's worth for me to provide Heap Explorer through the asset store.

    However, I also have good news :) I tested Unity 2017.4.6f1 today and they fixed various memory profiling issues. No crash bugs found so far. I updated the "Known Issues" section in Heap Explorer documentation with my latest findings.

    Unfortunately, while revisiting some of my bug-reports, I also ran in other issues:

    (Case 1052308) PackedMemorySnapshot: nativeObject.isPersistent or documentation incorrect
    https://forum.unity.com/threads/cas...persistent-or-documentation-incorrect.537484/

    (Case 1052281) IMGUI.Controls.TreeView keyboard navigation is slow
    https://forum.unity.com/threads/cas...-treeview-keyboard-navigation-is-slow.537468/
     
    Last edited: Jun 27, 2018
  49. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,581
    Uploaded an update, same link as before.

    Many thanks to the people at Unity Technologies for fixing several Memory Profiling API related bugs!

    Changelog
    • C++ Objects View: Replaced generic "C++" icon with icons that indicate whether it’s an asset, scene-object or run-time-object.
    • C++ Objects View: Added popup menu to toolbar to exclude native objects from the list, depending on whether the native object is an asset, a scene-object or a run-time-object.
    • C++ Objects View: Added popup menu to toolbar to exclude native objects from the list, depending on whether the native object is marked as "Destroy on load" or "Do NOT destroy on load".
    • C++ Objects View: Display icon in object inspector in the top-right corner.
    • C++ Objects View: Fixed count and size display in bottom status bar.



    Let me know what you think!
     
    Arkade likes this.
  50. Pavelluden

    Pavelluden

    Joined:
    Jun 9, 2013
    Posts:
    2
    You are doing a great job. This tool looks very promising and useful for me. Looking forward for release!
     
    Peter77 likes this.