Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Utilities ░ ▒ ▓ █ Heap Explorer - Memory Profiler, Debugger and Analyzer for Unity

Discussion in 'Tools In Progress' started by Peter77, Apr 22, 2018.

  1. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    Yes, that's the one. I've raised the question about the NDA already after the post by @GloriaVictis and was told it's unlikely we'd sign an NDA with non enterprise support customers. Then again, we're already all under an NDA by Unity Technologies that sort of covers this I guess. Anyways, I've raised the issue and hopefully somebody will outline how to upload the repro project.
     
    Peter77 likes this.
  2. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Thank you so much for the help!
     
  3. nukeustwo

    nukeustwo

    Joined:
    Sep 4, 2015
    Posts:
    51
    Hey, impressive stuff! I had a question about interpreting some of this information, in particular the managed heap fragmentation visualisation on the overview.

    I am right in the assumption that in your example screenshot in the documentation, the 443mb is the total amount allocated by the OS and includes native and memory allocations, and the 10.9mb of fragmented allocation is purely managed?

    In which case does the visualisation given any "help" as to whether or not the fragmentation is a potential issue? Or is it just not enough information to really interpret?

    Thanks, and great work!
     
  4. GloriaVictis

    GloriaVictis

    Joined:
    Sep 1, 2016
    Posts:
    133
    @MartinTilo, I had a contact from your QA about uploading the project, but still, we try to evade that without NDA.

    Actually we could do that if we would remove a part of our project which we call "server scripts" - those are scripts we remove on building a client, which is most NDA part of it. But when we do that, profiler no more hangs/crashes.

    I have tried once more launching server from editor and trying to do a heap snapshot from here, having a hope it won't crash (like doing heap sample from built server is doing now) but we are stuck on this error:
    https://forum.unity.com/threads/wip...analyzer-for-unity.527949/page-2#post-3597619

    Could you actually ask your Unity mate @alexrvn if he had a chante to look on that? Actually this is happening even on empty scene of our project. I have randomly found this: https://github.com/Unity-Technologies/mono/pull/920/files and tried to put all .dlls into one folder but it didnt helped either sadly.

    Edit: Actually after debuging crash report and downloading unity_memory_info.c from that link it crashes on that line: https://gyazo.com/444714f07bfc0ab0fe0216234e27b142


    @Peter77 do you think there's a chance to somehow launch the server on editor, and save memory sample we would take via Unity Profiler and open at your Heap Explorer later?
     
    Last edited: Oct 24, 2018
  5. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    @GloriaVictis That change in mono seems unrelated to the crash. It fixed a case where managed objects where reported wrongly. This change landed in Unity 2017.4.6f1 so that's why putting those dlls in didn't change anything.

    And yes, the fastest way to get to the bottom of this would be by avoiding the need for an NDA.
    So it crashes in an empty scene, but not in an empty project? So maybe something in your project puts stuff in memory through static memory or static constructors, InitializeOnLoadAttribute/InitializeOnLoadMethodAttribute, Resources or Editor code. The later seems unlinkely since it also happens on builds. Maybe you could try narrowing down the project needed to reproduce based on that, i.e. looking at static / initialized on load stuff in the server scripts? Or deep profile what happens in an empty scene and go from there?

    Sorry I'm poking around in the dark here a little. The mono code to me looks like it might only reasonably crash because it failed to get the mono class but I'm nearly completely in the dark about all of the mono code base...
     
  6. OndrejP

    OndrejP

    Joined:
    Jul 19, 2017
    Posts:
    302
    What a GREAT tool! Fantastic work!

    .NET4 runtime is working for me with heap explorer, I'm using Unity 2018.3.0b9
     
    Peter77 likes this.
  7. z3nth10n

    z3nth10n

    Joined:
    Nov 23, 2013
    Posts:
    55
    I STRONGLY recommend to update to the latest Unity Editor version (2018.3.0b9) if you want to use .NET 4.x Runtime version. Because for some reason when I was debugging some big arrays with Beta 6 my Editor crashes when I analyzed the Heap, with this version works well.
     
  8. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    @Ikillnukes the reason for that is basically that we're still fixing bugs left and right in the new experimental Memory Profiling backend introduced in 2018.x. As .NET 4.x just came out of experimental in 2018.x and with that got memory capture capabilities, it is still getting stabilized (which is what the beta period is for ;)). I'd recommend staying to the bleeding edge of the beta if memory capturing is important for you. There are more fixes for crashes coming in the upcoming versions.

    However, please do report any crashes you find through the bug reporter tool. While we hope we're getting close to a final stable version of the capture code in Mono, IL2CPP and the engine's memory management, there might be permutations we have overlooked.

    (BTW: the old memory snapshot API has been rerouted to use the new backend so this goes for either API)

    Good to hear that b9 works well for you though :)
     
    z3nth10n likes this.
  9. petrse

    petrse

    Joined:
    Oct 29, 2014
    Posts:
    3
    Great work on the asset, thanks so much for sharing!
    It nicely shows what is possible with decent memory introspection API - kudos to Unity for recognizing this as a critical feature!

    Of course our project being locked into 2018.x and 4.x runtime, in its current state, the api and this asset is so tantalizing yet not practically useful due to the bug with missing generic typed fields. Can't crawl missing fields :[

    @MartinTilo
    On 2013.3.b10 simple dump of PackedMemorySnapshot.typeDescriptions from attached debug mono build running 4.x runtime yields (excerpt for illustration)

    [Simple generic type argument - perfect]
    System.Collections.Generic.List<System.String>
    [field] System.String[] _items
    [field] System.Int32 _size
    [field] System.Int32 _version
    [field] System.Object _syncRoot
    [field] System.String[] _emptyArray <static>

    [Nested generic type arguments - missing fields]
    System.Collections.Generic.List<System.Collections.Generic.KeyValuePair<UnityEngine.AnimationClip,UnityEngine.AnimationClip>>
    [field] System.Int32 _size
    [field] System.Int32 _version
    [field] System.Object _syncRoot

    [Array type generic argument - missing fields]
    System.Collections.Generic.List<System.Byte[]>
    [field] System.Int32 _size
    [field] System.Int32 _version
    [field] System.Object _syncRoot

    --------------------------------------------------

    Really hope Unity team does lot let 2018.3 leave beta before resolving this.

    Edit: formatting
     
  10. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    Hi @petrse,

    generally, the release machinery here only holds a version in a longer-than-planned beta if there are any release stopping bugs left over. So the only way for us developers to make sure that a certain bug that isn't too critical (which this isn't) is to find it and fix it in time.

    Luckily, we did and 2018.3.0b11 will ship with this bug fixed.

    On that note, thank you all for testing the API and the beta, helping us cover as much ground as possible on this before the release. Taking a memory capture is a complex and delicate process with lots of moving parts so the more a variety of projects and situations it gets tested in, the more stable we can make it and ensure that the data we capture is valid and proper.

    Fixing the issues discovered this way has been our (the memory profiler team) primary concern for the entire beta period and before and even if something should be slipping by into the final release: this is a release that is going to be an LTS version that will see continuous updates for quite a while still. Similarly, we're discovering issues that go back to 2017.4 and those solutions will get backported (and obviously, others that are only in 2017.4 will get fixed there as well) as we get to it. (Maybe with the exception of issues with the then experimental 4.x scripting runtime)
     
    GloriaVictis likes this.
  11. petrse

    petrse

    Joined:
    Oct 29, 2014
    Posts:
    3
    @MartinTilo

    Thanks for the info. I understand this is not really a tent-pole feature in the grand scheme of things, makes me that much more glad it is getting the attention it needs.
     
  12. Zhentar

    Zhentar

    Joined:
    Nov 18, 2018
    Posts:
    2
    Thank you for providing this tool! It's better than any platform vendor provided memory profiling tool I've used. But I have a bunch of feedback for you (and Unity) to make it better!
    • Memory "sections" as contiguous regions of committed bytes are quite useful from an API perspective, but from the end user perspective, they suck: it's a vague, ambiguous term so it's not clear what it actually means, they're much too large in number to large to digest, and their characteristics don't provide any actionable feedback to developers. The GC's internal version of "heap sections" - a fairly small number of fairly large blocks (on 32-bit Windows, there are no more than 512 of them, and they seem to vary from 256KB to 8MB in size) - are far better representations. The total virtual memory reserved size of these blocks is much more relevant to users than precisely which pages amongst them aren't committed (and unfortunately this seems to be both absent from the API and difficult to infer), and the number is much more manageable - and their total size much more stable (and changes in said size more meaningful).
    • A large number of the "managed heap sections" are not on the managed heap. (@Peter77 : the API returns all of the sections that actually managed heap first, and they are always page aligned (4096 bytes), while the following sections will never be page aligned, so you can fairly easily detect which ones are as advertised). The not-managed heap sections include the static fields as Peter has noted, but it also looks like they contain Mono runtime internal data. Separating these out by type would provide a better high level summary for users (but this also seems absent from the API).
    • "fragmented across X.XX GB from the Operating System" sounds alarming, but there's nothing actionable about it. The spread of virtual memory allocations is effectively arbitrary and decided by the OS, and there's no way not to have it fragmented.
    • The graph showing the positions of the memory sections is also not particularly helpful; again it's not something users have any influence over and with only a small subset of the total virtual memory allocations present, it's far too incomplete to communicate anything. I think a stacked bar graph-style visualization showing relative sizes of the high level sections (like in SysInternal's VMMap) could be a very useful visualization to have in that place.
    • I'm pretty sure linking "Understanding the managed heap" is harmful to users' understanding the managed heap. It's horribly inaccurate and misleading.
      • Unity is using a low fragmentation heap allocator now, so the introduction to fragmentation is just plain wrong. (A low fragmentation allocator reserves blocks of memory to group together similarly sized objects, e.g. when it needs to allocate a 16-byte object it actually reserves a 4096 byte block with 256 16-byte object slots in it, which means you never have the problem of accumulating small, unusable slots from fragmentation. It does have other interesting, relevant implications but fragmentation is not one of them)
      • "Key problems with the heap" gives a particularly convoluted explanation of "you only get so much memory before you crash" and makes it sound like there's a memory leak that will get you after too many temporary allocations.
        • One could also point out that it uses five bullet points to cover "twofold" problems or that the mention of 64-bit address space exhaustion being impossible is factually incorrect because current x86-64 processors have a mere 48 bits of address space, but I'll refrain since would be somewhat less constructive criticism ;).
      • "Basic memory conservation" does not offer any advice on conserving memory at all (it does offer reasonable advice on reducing object churn/GC frequency, but that's rather different).
    • An 'export to csv' option would be nice for doing analysis not supported by the UI when necessary with fairly little implementation effort.
    • dotMemory (JetBrain's memory profiler) has a number of nice features around analyzing string content, comparing individual objects within dumps, grouping objects by retention paths... you should copy all of them ;)
     
    Last edited: Nov 20, 2018
    Peter77 likes this.
  13. Joerg-Boehnel

    Joerg-Boehnel

    Joined:
    Jun 24, 2013
    Posts:
    30
    Hello,

    using 2018.3.0b11 I've made a snapshot that cannot be loaded. It seems there is an infinite loop in HeapExplorer during the "Analyzing Managed Objects\n{0} crawled, {1} extracted" step. {0} keeps counting upwards while {1} wont change anymore. I've let it run overnight but even 18 hours and over 22 billion crawled objects later it wouldn't finish.

    When I start the game and make a snapshot while still in main menu, I can load it perfectly fine. But once I've started a map, all made snapshots will stay in an infinite analyzing step, even when I go back to the main menu and drop alot of memory references. Probably some circular linking we're doing causes HeapExplorer to enter an infinite loop. The snapshot is about 150mb big (12mb zipped). Unfortunatelly this seems too big for this forum to upload. If you're interested I could email it to you.

    Best wishes
    Joerg Boehnel
     
    Peter77 likes this.
  14. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Thanks for the feedback! That's a lot to read :)

    Do you refer to the "Managed Heap Sections" view, where the tool displays a list of all heap sections? Do you know of any documentation about the non-page aligned memory? I wonder if it's something that might change between .net versions or mono/il2cpp. Just removing non-aligned entries from there would be trivial i guess.

    What do you suggest? Should I just rename "fragmented" to "spreaded" and it doesn't sound alarming anymore?

    I guess that's true. While not directly helpful, I hoped it's nice to see where in memory that particular section is located.

    Do you have a screenshot of that one? There are quite a few matches on google, but not sure which one.

    If removing that link makes the tool better, then it's basically gone already.

    What exactly should be exported? Exporting the native objects is simple, because Unity provides just a few fields which are the same for every obejct. But .net objects on the other hand have an arbitrary number of fields and I don't know how to reflect this in csv.

    I don't have dotMemory, but perhaps their documentation is useful. I will take a look.

    Thanks for your feedback!
     
  15. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Yes, I'm very interested in a snapshot to reproduce the issue. Please send it to unitytools@console-dev.de

    Thanks for the feedback!
     
  16. Zhentar

    Zhentar

    Joined:
    Nov 18, 2018
    Posts:
    2
    Yeah, that part (and the total # of sections show in the summary area). I don't know that it is documented anywhere; but it's a natural consequence of the implementation (most of the non-'heap section' sections are "MonoMemPool", which are themselves aligned, but they start with a header that is not included in the section, so the section will always be offset from the MemPool alignment), so I would expect it to be fairly stable across versions.

    It could be phrased "within an X.XXGB address space" (though my preference would be to just drop that bit entirely).


    Like this: (with a legend, of course)
    upload_2018-11-22_19-2-19.png

    Just the same basic things in the main grid would be enough, I think (c# type, c++ name, size, address, assembly).

    Hmm, although that makes me think - a detail view that is restricted to just a single type and includes that type's fields as columns could be quite useful!

    They do have a free trial :)
     
    Peter77 likes this.
  17. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    I uploaded a new version that implements some of the recent feedback, but not everything just yet. Find the download link here.

    2.8 Changelog
    • Added functionality to stop/cancel a processing step when Heap Explorer is analyzing a memory snapshot. This is a workaround for an issue that sounds like an infinite loop as reported here. I’m still interested in the memory snapshot that causes the issue to actually fix it.
    • C# Memory Sections View: Added raw memory view to inspect the actual bytes of a memory section.
    • C# Memory Sections View: Changed phrase "N memory sections fragmented across X.XXGB" to "N memory sections within an X.XXGB address space", based on this feedback.
    • Overview View: Removed memory sections graph, based on this feedback.
    • Added functionality to export parts of the memory snapshot to CSV. You find it it the “Views” popup menu, named “CSV Export”. Based on this feedback.
    @Zhentar @Joerg-Boehnel
     
    GloriaVictis likes this.
  18. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Thanks for the memory snapshot. I'm able to reproduce the issue and it seems it's caused by a bug in Unity's Memory Profiling API.

    Unity contains this struct:
    Code (CSharp):
    1. public struct PlayerLoopSystem // Located in namespace UnityEngine.Experimental.LowLevel
    2. {
    3.    public Type type;
    4.    public PlayerLoopSystem[] subSystemList; // this field is causing issues
    5.    public UpdateFunction updateDelegate;
    6.    public IntPtr updateFunction;
    7.    public IntPtr loopConditionFunction;
    8. }
    The "subSystemList" field is not detected as PlayerLoopSystem[] array, but as PlayerLoopSystem ValueType, in the memory snapshot. This makes the "PlayerLoopSystem" struct look like it contains itself in an infinite nesting fashion. Thus Heap Explorer is stuck in an endless loop.

    This issue seems to occur with ScriptingBackend .NET 4. I also tested .NET 3.5, but Unity is missing the "subSystemList" field entirely there. It seems both Unity ScriptingBackends have issues with this sort of struct, at least in Untiy 2018.3 beta.

    I reported these issues and created forum threads:

    (Case 1104590) PackedMemorySnapshot: "subSystemList" detected with wrong type (.NET 4)
    https://forum.unity.com/threads/cas...ubsystemlist-detected-with-wrong-type.589420/

    (Case 1104581) PackedMemorySnapshot: Missing "subSystemList" field in PlayerLoopSystem TypeDescription (.NET 3.5)
    https://forum.unity.com/threads/cas...eld-in-playerloopsystem-typedescripti.589405/

    It seems we have to wait for Unity Technologies to fix this issue. Or perhaps I find a workaround.
     
  19. GloriaVictis

    GloriaVictis

    Joined:
    Sep 1, 2016
    Posts:
    133
    Hello,

    I have recently started having issues with seeing RootPath of samples. It is not working on both new and old version of Heap Explorer.

    I see that I have lots of those warnings:
    guard kicked in
    UnityEngine.Debug:LogWarning(Object)
    HeapExplorer.RootPathUtility:Find(ObjectProxy) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/RootPathView/RootPathControl.cs:234)
    HeapExplorer.RootPathJob:ThreadFunc() (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/RootPathView/RootPathView.cs:115)
    HeapExplorer.HeapExplorerWindow:ThreadLoop() (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:711)


    Any chance I could send you the snap and you could take a look on that? As the RootPath was our ultimate weapon of finding leaks it would be very usefull.
     
  20. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    If the snapshot is rather small, like a 10mb .zip file, just send it to unitytools@console-dev.de. If it's bigger, please send me a download link to the file, either via email or private forum message.

    Please also let me know which Unity version you use and what the ScriptingBackend setting in your Player Settings is.

    Thanks in advance!
     
    Last edited: Nov 28, 2018
  21. GloriaVictis

    GloriaVictis

    Joined:
    Sep 1, 2016
    Posts:
    133
    Sent it, looking forward for your reply, thanks!
     
    Peter77 likes this.
  22. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    I uploaded a new version that implements some of the issues that were recently reported. Find the download link here.

    2.9 Changelog
    • Added workaround for Unity bug Case 1104590. Heap Explorer now ignores what appears as nested struct instances in a memory snapshot, which was causing an endless loop in Heap Explorer. This issue seems to occur with .NET4 ScriptingRuntime only. You can find an option to toggle this behavior in HeapExplorer's toolbar "File > Settings > Ignore Nested Structs". Based on this feedback. Heap Explorer log's it to the Console window if any nested structs are ignored.
    • Increased loop guard limit, when trying to find root paths from 100000 to 300000 iterations. This avoids this issue.
    • Added functionality to cancel a “finding root paths” operation. A “Cancel” button is now available in the root paths panel.
    • When changing the selection in the C#/C++ objects view, any running “finding root paths” job is aborted, rather than waited for completion. This makes the UI feel more responsive.
    • Changed order in which various jobs run in Heap Explorer, which makes the UI slightly more responsive.

    As I wrote earlier, this seems to be an Unity bug. In Heap Explorer 2.9, I implemented a workaround for this issue.

    The tool now ignores all those "nested struct instances", which avoids running in an endless loop. However, it seems there are still various other issues, perhaps also related to .NET4 ScriptungRuntime or perhaps 2018.3 beta, such as heap sections that cannot be found. Heap Explorer outputs a warning for these issues.

    You can toggle this workaround in Heap Explorer's toolbar > "File > Settings > Ignore Nested Structs".


    Thank you for the memory snapshot. I was able to reproduce the issue and came up with a fix.

    In order to find all root paths, Heap Explorer crawls a lot of objects. I implemented a loop-guard, to avoid that the tool is stuck in an infinite loop there. However, the number of iterations wasn't enough for your snapshot. Thus I increased the limit from 100000 iterations to 300000.

    I also implemented functionality to cancel a "finding root paths" operation at any point now. The panel displays a "Cancel" button while it's busy. With that new "cancel" functionality, I was also able to improve the general UI responsiveness a little, by always canceling a running "finding root paths" operation if you select a different object. Previously, Heap Explorer waited for the operation to complete.
     
    Last edited: Nov 28, 2018
  23. nukeustwo

    nukeustwo

    Joined:
    Sep 4, 2015
    Posts:
    51
    Wyman036_KFF, Lipoly and Peter77 like this.
  24. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Nope, I don't. Thanks for asking , makes me believe Heap Explorer is appreciated :)
     
  25. nukeustwo

    nukeustwo

    Joined:
    Sep 4, 2015
    Posts:
    51
    Absolutely it is, well if you ever get round to setting one up give me a ping
     
    Peter77 likes this.
  26. huaiyao

    huaiyao

    Joined:
    Nov 20, 2012
    Posts:
    4
    thanks for Heap Explorer, it's realy a nice tool ! After I use It server times, Now When I Open Window > Heap Explorer, nothing happened , I can't open it now.I restart my computer and create new project, but useless
     
  27. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Thanks!

    1. What Unity version do you use?
    2. What Heap Explorer version do you use?
    3. Do you get any error message when you try to open Heap Explorer?
    4. Have you tried in Unity main menu "Window > Layouts > Default"?
    If the window position or size is somehow screwed up, you can try go into the Windows Registry and delete those entries, but I believe "Window > Layouts > Default" is doing that too:
    1. Close Unity
    2. Open Windows Registry Editor (regedit.ext)
    3. Navigate to HKEY_CURRENT_USER\Software\Unity Technologies\Unity Editor 5.x\
    4. Remove HeapExplorer.HeapExplorerWindowh_h516731407
    5. Remove HeapExplorer.HeapExplorerWindoww_h516731408
    6. Remove HeapExplorer.HeapExplorerWindowx_h516731423
    7. Remove HeapExplorer.HeapExplorerWindowy_h516731422
    Not sure how/where it's stored on OSX.
     
  28. huaiyao

    huaiyao

    Joined:
    Nov 20, 2012
    Posts:
    4
    Thanks very much! I have Solved It.Thanks Again
     
  29. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    How did you solve it? What did you do?
     
  30. huaiyao

    huaiyao

    Joined:
    Nov 20, 2012
    Posts:
    4
    It's my own mistakes 。I add a computer monitor recently ,so I have double computer monitor now , but I forget it.When I open unity again,I don't know why the Heap Explorer is on the monitor which I do not open.Now I can use the heap explorer happily
     
    Peter77 likes this.
  31. XRA

    XRA

    Joined:
    Aug 26, 2010
    Posts:
    265
    great work with the heap explorer!
    I'm attempting to track down why some objects + meshes, textures, materials are not being unloaded.
    The objects that never get freed up in memory have 0 path(s) to root, (but they are not static), which seems unusual to me.
    Is this a potential issue with heap explorer not finding all the root paths? These are prefabs loaded via the Addressables package.
     
  32. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Maybe. Can you send me that memory snapshot? :)

    If the snapshot is rather small, like a 10mb .zip file, just send it to unitytools@console-dev.de. If it's bigger, please send me a download link to the file, either via email or private forum message.

    Please also let me know which Unity version you use and what the ScriptingBackend setting in your Player Settings is and the target platform from which you captured the snapshot.
     
  33. XRA

    XRA

    Joined:
    Aug 26, 2010
    Posts:
    265
    @Peter77 actually i realized it was because the asset bundle was still in use by other assets (so it wasn't unloading)

    I'm going to check and see if I get the 0 roots again & save a capture if it occurs, it is showing the roots currently.
     
    Peter77 likes this.
  34. z3nth10n

    z3nth10n

    Joined:
    Nov 23, 2013
    Posts:
    55
    In version 2.9, with Unity 2018.3.0f2 the following two errors shows:

    ERROR: 'System.Collections.Generic.Dictionary.Entry<System.Type,System.Collections.Generic.List<System.Int32>>[].baseOrElementTypeIndex' = -1 is out of range, ignoring. Details in second line
    arrayRank=1, isArray=True, typeInfoAddress=4A72E258, address=42EA5A80, memoryreader=MemoryReader, isValueType=False
    UnityEngine.Debug:LogErrorFormat(String, Object[])
    HeapExplorer.AbstractMemoryReader:ReadObjectSize(UInt64, PackedManagedType) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/MemoryReader.cs:443)
    HeapExplorer.PackedManagedObjectCrawler:SetObjectSize(PackedManagedObject&, PackedManagedType) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/PackedTypes/PackedManagedObjectCrawler.cs:590)
    HeapExplorer.PackedManagedObjectCrawler:CrawlManagedObjects() (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/PackedTypes/PackedManagedObjectCrawler.cs:330)
    HeapExplorer.PackedManagedObjectCrawler:Crawl(PackedMemorySnapshot, List`1) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/PackedTypes/PackedManagedObjectCrawler.cs:72)
    HeapExplorer.PackedMemorySnapshot:Initialize(String) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/PackedTypes/PackedMemorySnapshotEx.cs:1083)
    HeapExplorer.HeapExplorerWindow:ReceiveHeapThreaded(Object) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:968)
    HeapExplorer.ReceiveThreadJob:ThreadFunc() (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:1019)
    HeapExplorer.HeapExplorerWindow:ThreadLoop() (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:768)
    System.Threading.ThreadHelper:ThreadStart()

    'System.Collections.Generic.Dictionary.Entry<System.Type,System.Collections.Generic.List<System.Int32>>[].baseOrElementTypeIndex' = -1 at address '42EA5A80', ignoring managed object.
    UnityEngine.Debug:LogError(Object)
    HeapExplorer.PackedMemorySnapshot:Error(String, Object[]) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/PackedTypes/PackedMemorySnapshotEx.cs:95)
    HeapExplorer.PackedManagedObjectCrawler:CrawlManagedObjects() (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/PackedTypes/PackedManagedObjectCrawler.cs:172)
    HeapExplorer.PackedManagedObjectCrawler:Crawl(PackedMemorySnapshot, List`1) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/PackedTypes/PackedManagedObjectCrawler.cs:72)
    HeapExplorer.PackedMemorySnapshot:Initialize(String) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/PackedTypes/PackedMemorySnapshotEx.cs:1083)
    HeapExplorer.HeapExplorerWindow:ReceiveHeapThreaded(Object) (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:968)
    HeapExplorer.ReceiveThreadJob:ThreadFunc() (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:1019)
    HeapExplorer.HeapExplorerWindow:ThreadLoop() (at C:/Users/crash/Documents/unityheapexplorer/Unity/Assets/HeapExplorer/Editor/Scripts/HeapExplorerWindow.cs:768)
    System.Threading.ThreadHelper:ThreadStart()

    For about 46 times.

    This didn't happened to me with version 2.7 in Beta stages of Unity Editor.

    Happy new year!

    EDIT: Will you create a Package from this?
     
    Last edited: Jan 8, 2019
  35. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Looking at the snapshot, all objects where "Area2_Entrance" is "Referenced by", have no root-paths too, therefore Area2_Entrance has no root-path as well. I don't claim that's correct, but it's what the data in the memory snapshot is saying.

    There seems to be an issue, where Unity seems to miss some objects in a memory snapshot. The forum post of this issue can be found here. Maybe it's caused by that issue, I don't know.

    Unfortunately, I can't help with problem. :(

    You can try to use Unity's new MemoryProfiler, perhaps it works better (it does have the same "some objects are missing" issue though):
    https://forum.unity.com/threads/new...ew-package-available-for-unity-2018-3.597271/
     
  36. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    This seems to be an issue with ScriptingRuntime = .NET4. I was able to reproduce this issue with Unity 2018.3.0f2 and .NET4. The issue does not seem to exist with .NET3 (.NET3 has other issues).

    I've submitted a bug-report to allow Unity Technologies to fix this issue. The related forum post can be found here:
    https://forum.unity.com/threads/cas...ex-does-not-point-to-its-element-type.609922/

    My last information is that Unity Technologies will allow 3rd party developers to provide tools through their package manager some when in the future. Is this possible already?
     
  37. z3nth10n

    z3nth10n

    Joined:
    Nov 23, 2013
    Posts:
    55
    Yes, as you can see I already figured this out: https://forum.unity.com/threads/zios-editor-theme-support.411818/page-4#post-4079998

    And I want to add your package on a program I will starting woking in soon: https://github.com/uta-org (look at LiveUEPM)

    I will add a FileSystemWatcher and a ProcessSeeker to create symbolic links of Packages that users requires when they create a new project. So, for example, by default you can configure a project to start with a custom number of selected packages.
     
  38. rubeng

    rubeng

    Joined:
    Apr 20, 2013
    Posts:
    58
    Hi Peter, I am trying to use the Heap Analyzer, and when it works it is really awesome, but I am having problems trying to take snapshots.

    It usually crashes the app (or the editor if I try to profile the editor), and the app log shows


    Can not send network message. Receiver can not keep up with the amount of data sent



    (Filename: ./Runtime/Network/PlayerCommunicator/GeneralConnection.cpp Line: 424)


    Can not send network message. Receiver can not keep up with the amount of data sent



    (Filename: ./Runtime/Network/PlayerCommunicator/GeneralConnection.cpp Line: 424)


    Can not send network message. Receiver can not keep up with the amount of data sent



    (Filename: ./Runtime/Network/PlayerCommunicator/GeneralConnection.cpp Line: 424)


    test(26755,0x7fff932d2340) malloc: *** error for object 0x14f880330: pointer being freed was not allocated

    *** set a breakpoint in malloc_error_break to debug




    Sometimes it works, so I am trying to go to the same point where I need to generate the dump a ton of times, and eventually it works.

    Do you have any idea of how to improve this situation?

    Thanks

    Rubén
     
  39. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    I'm afraid I can't help with that.

    What Unity version do you use? Unity Technologies reworked the MemoryProfiling API in 2018.3, maybe this issue is fixed in that version already. They also work on a new MemoryProfiler, might be worth a try:
    https://forum.unity.com/threads/new...ew-package-available-for-unity-2018-3.597271/

    In either case I recommend to submit a bug-report, with a project to reproduce the issue, as described on this page:
    https://unity3d.com/unity/qa/bug-reporting
     
  40. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Hi Peter,

    I only found this asset yesterday ("by accident", was only looking for more recent information on Unity's memory profiler) but I already really like this tool, very much appreciated. For now I got two points of feedback:

    1. I love that you can have multiple windows of it open to look at different snapshots simultaneously. It would be nice though if it somewhere stated what file you are currently looking at ;) Ideally at some prominent place that's visible no matter what view you're looking at.

    2. Currently if looking at the compare view I can only see what types account for what difference in memory usage. It would be cool if it actually could show instances that were added. Not sure if that's the case but I'd assume that a single instance (both for Unity native types as well as C# objects) would have a fixed internal object ID that doesn't change during runtime, so those could be filtered out and only show the removed/added entries with their sizes? Ideally with a way to switch to the corresponding C++/C# objects view of those entries for further inspection.

    Regards,
    Chris
     
    Peter77 likes this.
  41. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    In Heap Explorer 3.1 (I'm going to release it in a minute), the snapshot path is displayed in the lower-right area of the main window. Let me know if this works for you.

    That would be cool indeed. I remember I did have a reason to not implementing this, but I can't think of it anymore.

    Your GetInstanceID() probably works for UnityEngine.Object instances, but not "regular C# objects". I think just using the memory address to identify whether it's a new object should do the trick too. Adding this feature is most likely not a trivial task, it's nothing I could add quickly. I've added it to my TODO list.

    Thank you for the feedback!
     
  42. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    I uploaded a new version that implements a tiny new "feature". Find the download link here.

    3.1 Changelog
    • The bottom status bar now display the loaded snapshot filepath. (suggested here).
     
  43. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Hi Peter,

    that was one of a fast reply :D

    thanks, this is perfect :)

    Hm, don't objects in Unity's Mono have a unique identifier that they are referred to instead of referring them by address directly? I know at least a lot of JVMs use this approach so address changes (e.g. for compacting the heap) do it this way. Though even if they have I obviously have no idea if you can get that stuff from Unity's memory API :)

    Suppose you might have to also check the object size or something, as there could be a new object at the same address maybe?

    Regards,
    Chris
     
    Peter77 likes this.
  44. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    I thing the GC handles should be unique and stable. Anyways, the New Memory Profiler has a diff view that shows old vs new vs same on a per instance basis.

    Though, admittedly it doesn't allow you to open more than one window yet. It's been a recurring bit of feedback about the CPU Profiler as well though so we'll be looking into solving this in a somewhat holistic way.
     
    Peter77 likes this.
  45. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Yeah right, good catch! Just the address is not a robust solution. It's probably possible that an object of the same type and size could also be located at the same address. Just knowing the address doesn't seem to be enough indeed.

    According to various memory snapshots I looked at, a lot of managed objects don't have a GC handle. How do you tell if a managed object is the same? Maybe some bits in the object header?

    I believe @Alloc 's idea to use GetInstanceID() to tell it's the same UnityEngine object (native and/or managed, or just native?) sounds feasible. But how would you handle all non-UnityEngine objects?
     
  46. emongev

    emongev

    Joined:
    Jul 5, 2012
    Posts:
    36
    This is great Peter, youre a true lifesaver. I had done sometime ago a tool that took the detailed sample window and generated a CSV with the info, but yours is a much more complete tool and im lucky i lost my old tool and found this instead :D
     
    Peter77 likes this.
  47. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Heap Explorer source code available!

    Making the source code available was a big deal to me, which I didn't take lightly.

    I started to work on Heap Explorer in 2017. I hoped I could provide it through the Unity Asset Store at some point. Perhaps even work on it full-time, since, at this time, there was no better tool available.

    I thought there could be quite some demand for the tool, because every game developer needs to know what's going on in his/her application from a memory point of view. I certainly do need such tool.

    In 2018, at some Unite Conference, Unity Technologies announced they're working on a memory profiler as well. This crushed my plans with what I had in mind for Heap Explorer. It was no longer a legit option for me to put a lot of time in the tool, as Unity Technologies tool is at least as good as what I'm able to come up with.

    After a lot of back-and-forth what to do with Heap Explorer now, I came to the conclusion that the best option is to provide the source code and (mentally) move on.

    I hope having source code access to Heap Explorer allows developers to adapt it to their specific needs, which in turn helps them to make their applications run more robust. Something pretty much everybody benefits from.

    Get the C# source code at https://github.com/pschraut/UnityHeapExplorer
     
    Last edited: Apr 24, 2021
  48. MOKII

    MOKII

    Joined:
    Jan 10, 2017
    Posts:
    1
    Excellent job
     
    Peter77 likes this.
  49. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,586
    Thank you! :)
     
  50. print_helloworld

    print_helloworld

    Joined:
    Nov 14, 2016
    Posts:
    231
    Is there a particular reason for using bitbucket to host the source over github?