Utilities ░ ▒ ▓ █ Heap Explorer - Memory Profiler, Debugger and Analyzer for Unity

Woohoo \o/

Most of the stuff just worked fine with 2019.3. Very surprising to me, since Unity replaced the underlying memory profiling back-end as far as I know. Thumbs up towards @alexrvn , @MartinTilo and the rest of the gang who worked on that.

The only things I noticed that did break in Heap Explorer were related to the overhauled UI in 2019.3, but also only in specific views.

In the profiler forum I found a post where they discuss "empty managed shell objects". It's basically a native object that was destroyed, where its managed object representation is still hanging around, which seems to indicate some sort of memory leak. According Martins descriptions, I believe it would be fairly simple to implement detection for that in Heap Explorer, so I might implement that sooner or later

 

I've tested the Heap Explorer package with Unity 2020.1.0b6 and it seems to work.

I wasn't sure if it does, because of this comment:

So you didn't change the existing API, I guess? Otherwise I'm wondering why Heap Explorer still works

BTW, documentation seems to be missing for the "Connection" type:
https://docs.unity3d.com/2020.1/Doc...ence/MemoryProfiler.PackedMemorySnapshot.html

 

Are you doing it??

 

Do you know how to produce such "empty shell object"? I would need that to have something to test against when I implement detection for it. Or do you have a Heap Explorer memory snapshot that contains such object, that I could use?

 

I would imagine if you stored a reference to a GO then deleted said GO, you would end up with an empty shell that can't be GCd as it's still referenced

 

@Peter77 yeah it's like @QFSW said, but I'm happy to provide a project and/or snapshot if you need it.

...but it's something like this:

Code (CSharp):

1. public class SomeClass : MonoBehaviour {}
2.  
3. public class AnotherClass : MonoBehaviour {
4. private SomeClass _someClass;
5. private void Awake()
6. {
7.     GameObject go = new GameObject();
8.     _someClass = go.AddComponent<SomeClass>();
9. }
10.  
11. private void Update()
12. {
13.    // take a snapshot before pressing any key and one after
14.     if (Input.anyKeyDown)
15.     {
16.         Destroy(_someClass);
17.     }
18. }

 

EDIT: In the profiler forum I found a post where they discuss "empty managed shell objects". It's basically a native object that was destroyed, where its managed object representation is still hanging around, which seems to indicate some sort of memory leak.

Thanks for the snippet.

I've created a branch with the empty shell object detection. If you use Unity 2019.3 or newer, you can just add this git URL via the Unity Package Manager:
https://github.com/pschraut/UnityHeapExplorer.git#empty-managed-shell-detection

Otherwise please download and add to your project manually:
https://github.com/pschraut/UnityHeapExplorer/tree/empty-managed-shell-detection

Can you please let me know if this works for you? Because if it works, then I'm turning it into a proper update and merge it into master.

You can find it under the "View" popup:



Opening this view will analyze the snapshot for empty shell objects and displays them like:


As shown in the image above, the C# MeshRenderer object has no native object anymore (m_CachedPtr = null) and is referenced by "AnotherClass".

This was the code I used for testing:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. public class AnotherClass : MonoBehaviour
6. {
7.     private UnityEngine.Object _someClass;
8.  
9.     private void Awake()
10.     {
11.         GameObject go = new GameObject();
12.         _someClass = go.AddComponent<MeshRenderer>();
13.     }
14.  
15.     private void Update()
16.     {
17.         // take a snapshot before pressing any key and one after
18.         if (Input.anyKeyDown)
19.         {
20.             Debug.Log("Destroy");
21.             Destroy(_someClass);
22.         }
23.     }
24. }
25.  

 

Heyo! This is potentially a dumb question, but I've had issues using this and Unity's official memory profiler.
Trying to capture a standalone player in either of them results in

I've tried letting it run for over half an hour, I have a very high end PC so I don't really foresee that being an issue. The editor log file mentions no errors, it just stays on that progress bar forever. However, if I capture an editor player, it works completely fine and finishes quickly.

My project isn't massively sized or anything either.

Tried using 2019.3 and 2020.2.

 

Does the issue occur with both, "Capture and Save" and "Capture and Analyze"? I'm wondering if it's related to capturing or analyzing. From what platform do you try to capture a memory snapshot? Just from PC or some mobile device or console?

If "Capture and Save" works, I could take a look at the snapshot why it's not loading. If "Capture and Save" and Unity's Memory Profiler don't work, it could be a problem on Unity's side.

 

Sorry about the delayed response :/

Not sure if it was something I changed with my project, or if it was updating, but I went to Unity 2020.2a12 and everything seems to be working now, both in Heap Explorer and the Unity Memory Profiler. ^_^ Thanks guys!

 

Hi @Peter77
I'm sorry I took so long to answer, I was busy with other things in the project.
I tried to use the tool but I get an exception. On the other hand, my colleague was able to use it and said it is really helpful!

I get this when I change the view to "Empty C# Shell Objects":

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. HeapExplorer.AbstractManagedObjectsView.OnGUI () (at Assets/Plugins/UnityHeapExplorer/Editor/Scripts/ManagedObjectsView/ManagedObjectsView.cs:303)
3. HeapExplorer.ManagedEmptyShellObjectsView.OnGUI () (at Assets/Plugins/UnityHeapExplorer/Editor/Scripts/ManagedEmptyShellObjectsView/ManagedEmptyShellObjectsView.cs:54)
4. HeapExplorer.HeapExplorerWindow.DrawView () (at Assets/Plugins/UnityHeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:486)
5. HeapExplorer.HeapExplorerWindow.OnGUI () (at Assets/Plugins/UnityHeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:398)
6. System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at <567df3e0919241ba98db88bec4c6696f>:0)
7. Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
8. System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at <567df3e0919241ba98db88bec4c6696f>:0)
9. System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) (at <567df3e0919241ba98db88bec4c6696f>:0)
10. UnityEditor.HostView.Invoke (System.String methodName, System.Object obj) (at /Users/builduser/buildslave/unity/build/Editor/Mono/HostView.cs:359)
11. UnityEditor.HostView.Invoke (System.String methodName) (at /Users/builduser/buildslave/unity/build/Editor/Mono/HostView.cs:353)
12. UnityEditor.HostView.InvokeOnGUI (UnityEngine.Rect onGUIPosition, UnityEngine.Rect viewRect) (at /Users/builduser/buildslave/unity/build/Editor/Mono/HostView.cs:329)
13. UnityEditor.MaximizedHostView.OldOnGUI () (at /Users/builduser/buildslave/unity/build/Editor/Mono/GUI/DockArea.cs:1092)
14. UnityEngine.UIElements.IMGUIContainer.DoOnGUI (UnityEngine.Event evt, UnityEngine.Matrix4x4 parentTransform, UnityEngine.Rect clippingRect, System.Boolean isComputingLayout, UnityEngine.Rect layoutSize) (at /Users/builduser/buildslave/unity/build/Modules/UIElements/IMGUIContainer.cs:281)
15. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr) (at /Users/builduser/buildslave/unity/build/Modules/IMGUI/GUIUtility.cs:179)

When I was capturing the memory I also got a lot of this:

Code (CSharp):

1. Missing Profiler.EndSample (BeginSample and EndSample count must match): Texture.ReleaseTextureUploadMemory
2. Previous 5 samples:
3.     AsyncUploadManager.gScheduleAsyncRead
4.     AsyncUploadManager.ScheduleAsyncCommand
5.     AsyncUploadManager.gScheduleAsyncRead
6.     AsyncUploadManager.ScheduleAsyncCommand
7.     AsyncUploadManager.gScheduleAsyncRead
8. In the scope:
9.     Texture.ReleaseTextureUploadMemory
10.     AsyncUploadManager.AsyncResourceUpload
11.     Render Thread

I'm attaching the snapshot that makes the problem happen, my colleague also gets the exception when loading this:
https://drive.google.com/file/d/1KVwIPTRqBEZg-K64XBNCF2yhEKidEArt/view?usp=sharing

I'm using 2019.2.21f1.

----
EDIT:
Now I found out that when I profile the editor it works. I was profiling an iPad Mini 2 before.

 

@MartinTilo @alexrvn

It seems the

m_InstanceID

field in

UnityEngine.Object

does not exist in a Player.

How can I get the instanceID of an object in a Player?

The UnityEngine.Object type captured from an iPad Mini 2 with 2019.2.21f1 looks like this (notice no m_InstanceID):



The UnityEngine.Object type captured from the Editor with 2019.3.7f1 looks like this:


I use the m_InstanceID field during the "empty managed shell objects detection" (link) and only consider objects where the m_InstanceID value isn't 0 (link).

I implemented it like this, because the editor contains various empty shell objects whose instanceID all contain 0. I assumed it would be some kind of special object or so.

But because the m_InstanceID field does not exist in the Player snapshot, it skips them all

@roberto_sc Thanks for the snapshot! I fixed the null-ref issue, but I'm now stuck with the issue described above. I'll post if I have an update.

 

Does GetInstanceID() work? At least in the editor it's equivalent to m_InstanceID looking at the decompilation

 

@Peter77 do you usually get snapshots from playing in the Editor? I thought they were not reliable, I thought they would contain not only game memory but stuff from the Editor itself, since Editor snapshots are much bigger in size in the Memory Profiler. I always get from the device.

 

It totally depends and I'm sorry in advance to point out the obvious

• If I want to check memory usage in a build, capturing a build is required.
• If I want to check memory usage in the editor, capturing the editor is required.
  
• If I want to test something in Heap Explorer, capturing a snapshot in the editor is most of the time sufficient.

 

So only checking

m_CachedPtr

would be sufficient?

This was my first implementation, but then found there are a lot, what seems to be Unity internal, objects in the editor that all have

m_CachedPtr = 0

, so I added the additional filter to check for rm_InstanceID as well, because those objects all have m_InstanceID=0.

 

This tools looks exactly what I needed, but unfortunately Editor freezes on capturing the snapshot from iOS dev build.
Tried different Unity versions: 2019.3, 2019.4, same result.
Furthermore Memory Profiler doesn't work as well, same freeze.
I am on Mac, developing for iOS/Android

The log says:

Code (CSharp):

1. Creating memory snapshot file: {path}

Creating memory snapshot file: {path}

UPDATE
Seems that relates to Memory Profiler bug in 2019.3.13 and 2013.4.4
Migrated to 2019.3.15 and that works fine

 

you are right, just tested it again on 2019.4.4 and that worked

 

hi,
I just downloaded the package using package manager with the git URL and I got this error:

Library/PackageCache/com.oddworm.heapexplorer@e29645c02f1e04d53ce91202330bc0d0d79caa0c/Editor/Scripts/TestVariables.cs(9,7): error CS0246: The type or namespace name 'NUnit' could not be found (are you missing a using directive or an assembly reference?)

Apparently no-one else have this issue. I am using Unity 2019.3.06f. Is there anything I am missing?

Thanks,
Sohaib

 

@Sohaib_techverx you need to include manually the "Test Framework" package as well if you removed from your project. @Peter77 could not this be marked as a dependency when installing your package via git package?

@Peter77 one thing that I did not understood and it is not in the documentation is what GCHandles indicates and how can I solve any data that is remaining and referenced in it. It would help me a lot understand this.

Also, I'm trying to find here some memory that is remaining between scenes and it is strange that several delegates of mine appears as static when in theory they are not static.. I mean I'm creating lambda expressions and anonymous functions and placing them in some functions and referencing this delegates into actions/events in classes, is this expected? Am I understading something wrong?

I'm not used to handle memory leaks, so I'm not sure also if the issue is that the class which contains this event/action is not being cleared due to being referenced by someone which is not being cleared and so on.. Is it how it usually goes?

 

Thanks for letting me know. I believe that should be a simple fix, will post an update here when I fixed it.

To be honest, I don't know how useful the gchandles view actually is. As far as I remember, gchandles are used to prevent the garbage collector from collecting a managed object, which is just referenced on the native side. It's a while ago, my memory sucks, I'd have to look it up to provide a better explanation, but perhaps @MartinTilo could help me out here

I've never looked at how lambda expressions and anonymous functions appear in a memory snapshot and whether it's correct. Good question Do you know if they appear as static in Unity's memory profiler as well?
I'm asking, because I'm wondering:

• Perhaps that is just correct
• Perhaps it's a bug in heap explorer, in this case unity's memory profiler would not display it
• Perhaps it's a bug in the memory profiler api, in which case heap explorer and unity's memory profiler would be affected

@MartinTilo you probably know if lambda expressions and anonymous functions should appear as static delegates?

 

@Peter77, have not checked the delegates in Memory Profiler in Unity. I'm using Memory Profiler mostly for the TreeMap and such, but I was finding it easier to use Heat Explorer to find references and filter between the different data types, but I will take a look later and let you know! And thanks for the quick answer

 

Honestly I could not find myself in Unity Memory profiler to find as easily the same anonymous functions and lambda expressions as I have found in Heap explorer hehe If you want I can send via direct message a snapshot from both heap and memory profiler if you want to check later.

 

Heap Explorer 3.3.0 is available on github now
https://github.com/pschraut/UnityHeapExplorer

FIXED

• "The type or namespace name 'NUnit' could not be found" (thanks Sohaib and andre)
  
• "You need Unity 2017.4 or newer" dialog when running Heap Explorer in Unity 2020.2 (thanks Martin)

 

Thank you for keeping Heap Explorer up to date! In my experience it's a hell of a lot faster and easier to use than Unity's Memory Profiler (although that has some useful features on its own, especially when looking at individual assets in the Map view). Most importantly, I can't find any function in Memory Profiler that compares to your Paths to Root feature.

Speaking of, I've found that Paths to Root sometimes throws an exception and freezes mid-way through analyzing objects in my captures. I get an IndexOutOfRangeException thrown on line 340 of RootPathControl:

Code (CSharp):

1. value.Add(new ObjectProxy(obj.snapshot, obj.snapshot.gcHandles[c.from]));

Have you encountered this before?

 

Yes, I encountered it just recently when I was testing Unity 2020.2.0a17. What version did you use?

I believe it's related to a change in the Memory Profiling API they did a while ago and Unity staff were actually kind enough to tell me I should take a look at it, because they expected it would cause issues with Heap Explorer (they actually made me aware of it twice, so shame on me for not listening):
https://forum.unity.com/threads/hea...analyzer-for-unity.527949/page-5#post-5821303

When I checked with 2019.3 and 2020.1, I didn't see an issue. The documentation doesn't seem to be up to date (1, 2).

Sooo... not really sure what to change exactly, but I'll take a look at it at the weekend and hopefully figure out what to do.

 

Hi @MartinTilo! Thanks for answering

Hmm, interesting, not sure I understood everything about the lambd and anonymous methods yet.
Let me give a few simple examples to understand if there is any risk in these:

Code (CSharp):

1.  
2. public class MyClass : MonoBehaviour
3. {
4.     private List<Texture2D> textures;
5.     private Action myAction;
6.  
7.     private void Function1()
8.     {
9.         // Does this where usage function will be considered static?
10.         // If so, will these sprites stay in the memory? Even after my mono is destroyed? Considering that my function is not static neither my class.
11.         textures = GetComponentsInChildren<Texture2D>().ToList().Where(__texture => __texture.name.StartsWith("filter")).ToList();
12.     }
13.  
14.     private void Function2()
15.     {
16.         // Does this where usage function will be considered static?
17.         // If so, will these sprites stay in the memory? Even after my mono is destroyed? Considering that my function is not static neither my class.
18.         CustomFunctionWithCallback(delegate (Texture2D p_texture)
19.         {
20.             textures.Add(p_texture);
21.         });
22.     }
23.  
24.     private void Function3()
25.     {
26.         Texture2D __texture = new Texture2D(1, 1);
27.         // Does this where usage function will be considered static?
28.         // Is there any change in the behaviour considering this delegate is using a local variable?
29.         CustomFunctionWithCallback(delegate ()
30.         {
31.             textures.Add(__texture);
32.         });
33.     }
34.  
35.     private void Function4()
36.     {
37.         // Is this delegate also considered static?
38.         // Will I need to manually set it to null when my mono is destroied?
39.         // Or considering the action is not static this is not necessary?
40.         myAction += delegate ()
41.         {
42.             textures.Add(new Texture2D(1, 1));
43.         };
44.     }
45.  
46.     private void Function5()
47.     {
48.         // Is there any difference in this delegate where no private variable from the script is used?
49.         // Will this be static and require some null setting to clear the texture created inside?
50.         myAction += delegate ()
51.         {
52.             Texture2D __texture = new Texture2D(1, 1);
53.             Debug.Log("Creating the texture for nothing");
54.         };
55.     }
56. }
57.  

Consider that myclass can also not be mono necessarily but being referenced my someone that is monobehaviour which is also cleared..

I'm trying to understand if there is something wrong with these lambda and anonymous functions since they appear to be remaining in memory or if they are remaining in memory because of another issue in my game logic which can be holding a reference which I still not found that can lead to these stay in the memory later..

 

We're using 2019.4.2f1, so perhaps the change came downstream to both the 2019 LTS and 2020 branches?

 

Heap Explorer 3.4.0 is available on github now
https://github.com/pschraut/UnityHeapExplorer

Please thank @MartinTilo for explaining to me the new

connections

format in a memory snapshot. Without his help, this update would not have been possible. Thank you Martin!

I've updated Heap Explorer to use the new Memory Profiling API, that was introduced in Unity 2019.3 and is also the back-end of Unity's own Memory Profiler. This will hopefully get rid of a few issues.

The migration wasn't too bad, because I was lucky enough to had abstracted the memory snapshot already and only needed to rewrite the "Unity to HeapExplorer snapshot conversion", rather than every location where it's used. It's this commit if anybody is interested.

However, while migrating to the new API, I also found this bug:
(Case 1269293) 2019.4: PackedMemorySnapshot: nativeObjects.gcHandleIndex contains -1 always

Which means Heap Explorer will not display "native object to/from gchandle" connections. Martin told me via PM how I can workaround it myself, but I don't have time to work on it anymore this weekend. And to be honest, I hope Unity can just fix it. If I find time next weekend, I might look into the workaround.

I've also added checks to detect invalid array-indices during "Searching Root Paths", it will now handle this gracefully with an error message rather than an exception.

Please let me know whether 3.4.0 works for you.

 

That's excellent news, thank you!

LOL, so I didn't even have to update Heap Explorer to use the new API. I guess it was the peer pressure that made me act fast.

 

Hey @Peter77
CaputureAndSaveHeap fails in 3.4.0 due to sharing violation. I worked around it by replacing

Code (csharp):

1. UnityEngine.Profiling.Memory.Experimental.MemoryProfiler.TakeSnapshot(path, OnHeapReceivedSaveOnly);

with

Code (csharp):

1. UnityEngine.Profiling.Memory.Experimental.MemoryProfiler.TakeSnapshot(FileUtil.GetUniqueTempPathInProject(), OnHeapReceivedSaveOnly);

 

Thank you for the info! I'm going to take a look at the issue at the weekend.

 

I've created a pull request with changes inspired by this, but slightly different to allow for the file name authoring in the original code, and to fix a follow up violation when trying to write to the heap file to the snapshot file:
https://github.com/pschraut/UnityHeapExplorer/pull/1

Thanks for this tool! I have to get the diff working, as the diff of MemoryProfiler is missing hundreds of MB that I was hoping HeapExplorer would catch, but even if not, being able to walk references to the asset bundles is invaluable

 

That's great, thank you!

 

Heap Explorer 3.5.0 is available on github now
https://github.com/pschraut/UnityHeapExplorer

This version fixes the "CaputureAndSaveHeap fails in 3.4.0 due to sharing violation". Thanks @chrismarch for the fix!

Please see the changelog for details:
https://github.com/pschraut/UnityHeapExplorer/blob/master/CHANGELOG.md

 

Hey Peter77 !
Thank you very much for sharing this tool, really saved my day. Still much better than Unity's own mem profiler.

 

Thank you for the kind words. I'm glad it still proves useful

 

I released Heap Explorer 3.7.0:

• Fixed "Count" display in Compare Snapshots not updating correctly after swapping two snapshots. Thanks to niqibiao for the fix. See PR#3 for details.
• Fixed slow performance and high memory usage in rendering large lists, in particular when resizing a column or the window. Thanks to niqibiao for the fix. See PR#3 for details.

Download or install package from github:
https://github.com/pschraut/UnityHeapExplorer

 

Heap Explorer 4.0.0 is available now! This version fixes some bugs and comes with a few performance improvements. Thanks to jojo59516 for pull requests!

• Fixed search regression introduced in 3.9.0. Thanks to jojo59516 for the fix. See PR#10 for details.
• Fixed "ArgumentException: Getting control 2's position in a group with only 2 controls when doing repaint" error that occurred very often when loading a memory snapshot.
• Fixed high CPU utilization caused by repainting the "analyzing in progress" GUI. Thanks to jojo59516 for the fix. See PR#11 for details.
• Fixed that closing the Heap Explorer window while analyzing a memory snapshot is in progress sometimes didn't close the window immediately. Thanks to jojo59516 for the fix. See PR#11 for details.

https://github.com/pschraut/UnityHeapExplorer

 

@MartinTilo @Peter77

Hi, a question regarding these managed empty shells. What causes them to leak? Will the following piece of code produce a leak of an empty shell of a static testObject? I suppose, that it will leak as it's a static reference and it will be there even after object is destroyed, right?

Code (CSharp):

1. private static GameObject testObject;
2.  
3. private void Awake()
4. {
5.   testObject = new GameObject();
6.   Destroy(testObject);
7. }

Now, what will happen if I assign null to testObject after I destroy it? Will it still leak or not?

Code (CSharp):

1. private static GameObject testObject;
2.  
3. private void Awake()
4. {
5.   testObject = new GameObject();
6.   Destroy(testObject);
7.  
8.   testObject = null;
9. }

If it won't leak, does it mean that all references to UnityEngine.Object that are somewhat persistent or kept in any cache should be explicitly null-ed after being destroyed?

Thank you.

 

Yes. You need to clear any references to destroyed objects, because as far as the CLR is concerned they are just plain old managed objects that are still referenced.

Then the GC runs, managed wrappers around objects that are alive will be treated as roots and will also be referenced by other things. When the native objects are destroyed, the objects are no longer roots but that still leaves them reachable by any references to them that are still held by other managed objects.

If I'm honest it's one of the reasons I dislike the overloaded equality operator that makes destroyed objects act as if they are == null in Unity...

 

Ask two questions. The first question is: can unityhepexplorer support the real machine (Android, IOS, exe) of unity build? Sometimes the mobile platform is different from the editor platform. Some memory leaks are only available on the mobile platform. What should I do? Second question: I see that the mono heap memory on the unity profiler is 500 megabytes, but the managed memory usage in the unityhepexplorer is only 100 megabytes, but the native memory usage is 1.9g. Which item of the mono heap memory corresponds to the mono heap memory in the unity profiler, Is the unit incorrect, or do both managed memory usage and native memory usage in unityhepexplore belong to mono heap memory?

 

Yes. You need to connect to the build using Unity's Profiler. Heap Explorer will display the connected Player in the "Capture" drop-down.

What Unity version do you use? I remember there were always issues that the overall memory usage doesn't match with what Unity or Xcode reports. Perhaps @MartinTilo knows more on this topic?

I submitted a bug-report a while ago reporting:
(Case 987987) PackedMemorySnapshot: "managedHeapSections" do not match stats in Profiler

Here is the reply from Unity to the bug-report: