Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only.

    Please, do not make any changes to your username or email addresses at id.unity.com during this transition time.

    It's still possible to reply to existing private message conversations during the migration, but any new replies you post will be missing after the main migration is complete. We'll do our best to migrate these messages in a follow-up step.

    On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live.


    Read our full announcement for more information and let us know if you have any questions.

Expected times for capturing and processing snapshots?

Discussion in 'Profiler Previews' started by pavelkouril, Dec 19, 2018.

  1. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Hello,

    I'm trying to capture a snapshot of a running game (reports around 2.2GB in task manager), and the snapshot takes forever (approximately 30 minutes) to capture and then to load (around ~15 minutes). Also, the snapshots take like 10GB on disk, and the snapshot contains around 1 million of objects (~750k managed and 250k native ones).

    Do you consider this as an expected/reasonable time for the memory profiler workflows, or is this something that you intend to work on in the future and make it considerably faster?
     
    idyakonov and phobos2077 like this.
  2. G_Wojo

    G_Wojo

    Unity Technologies

    Joined:
    Feb 8, 2017
    Posts:
    44
    Hello pavelkouril.

    Speed of memory profiling is one of really important topics for us. Could you open ticket for us and attach there your compressed snapshot ? We would use it for investigation why it took so long to open, try improve code to handle it better and investigate why it taking so much on your hard drive.
     
  3. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Hello, done.

    Reported it as a Case 1111451.
     
  4. MartinV

    MartinV

    Joined:
    Nov 12, 2013
    Posts:
    3
    I have similar issue. Capture took about 30 minutes. This was in Editor while game was running, and so I assume capturing on device could be faster due to Editor contributing a lot of allocations. But as I'm not able to run on device (separate ticket, linker crash when building Android), that's all I have at the moment.
    The larger issue is that when I then Open the capture, Unity crashes about 5 minutes into opening it. I've uploaded compressed capture in the ticket (500MB uncompressed size). Case 1111647
     
  5. samfisher-cf

    samfisher-cf

    Joined:
    Jun 13, 2016
    Posts:
    5
    The latest memory profiler preview 4 contains a fix for a stack overflow crash, which I have also noticed and fixed locally before their fix came out.

    My snapshots also take between 15-45 mins to open and diff, so I added a progress bar tracking the number of objects processed to know if the profiler is still doing something or has just frozen/hung due to some error.
     
  6. ScottyB

    ScottyB

    Joined:
    Apr 4, 2011
    Posts:
    146
    Hi @Greg_Wojciechowski, I recently submitted a new bug report (Case 1126009) where the memory snapshot crawling process either gets stuck or takes an unreasonably long time (I waiting 30 mins+ before force closing Unity). Just thought you would like to know in case it helps fixing and/or speeding up the snapshot crawling code.
     
    MartinTilo likes this.
  7. Rohansi

    Rohansi

    Joined:
    Jun 14, 2017
    Posts:
    3
    I just submitted two bug reports related to this (1142394 and 1142384) with solutions included. The first one is when capturing a snapshot it will take a long time (depending on size) to actually start dumping the snapshot, and the fix is a patch to Unity's fork of Mono. The second one is an optimization to the memory profiler package to speed up one of the last steps in opening the snapshots.

    The second issue may be better solved by introducing a processing phase after capturing snapshots so it can be cleaned up and sorted for quick access.
     
    MartinTilo likes this.
  8. Rohansi

    Rohansi

    Joined:
    Jun 14, 2017
    Posts:
    3
    Just submitted another two for processing snapshots, one for performance (1146248) and one integer overflow (1146246). This one helps with the loading when you see the progress bar at "Crawling GC handles". My solutions are included with them as well. With all my fixes I got a snapshot that's 117 GB large and took like 18 hours to capture, loading it didn't take an unreasonable amount of time though.
     
    Last edited: Apr 15, 2019
    noio and MartinTilo like this.
  9. MadeFromPolygons

    MadeFromPolygons

    Joined:
    Oct 5, 2013
    Posts:
    4,005
    They take about 30 seconds to 1 minute to capture, and a 2-3 seconds to open for us. This is on a relatively large client project for HoloLens.
     
  10. noio

    noio

    Joined:
    Dec 17, 2013
    Posts:
    244
    What is 'unreasonable' ? I have a weird memory leak that sucks up 40GB, but I've been waiting almost an hour for the snapshot to load..
     
  11. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    @noio what Unity and package version are you using? If player and editor versions differ then please state both versions.

    Does it take long to
    1. capture (i.e. player freezes),
    2. process the capture on the editor (player is unfrozen but editor isn't or doesn't yet show the snapshot)
    3. or to open it once it appears in the snapshot list?
    1 and 2 should be dramatically faster in +2019.3 versions, 3 has been sped up across the board (remember to update to the latest package versions, it'll pay off) but would still be faster with +2019.3 versions due to having less superfluous native connections to crawl through.

    For +2019.3 Unity versions I'd venture to say that anything beyond 5-10 minutes at any of these points or across all of them would be unreasonable (and worthy of a bug report) at this point, and even that's somewhat conservative.

    I guess 40GB of Memory used by the app being captured might mean that the capture process would need to access swap memory on most systems? I guess that could make it slower but almost an hour of waiting times still sounds too long for +2019.3 versions.

    For 2018.3-2019.2 versions there is little we can do about these timings, durations south of an hour are not unheard of and it can also just have hung up at some point. We have a bug report for it but fixing it in 2018.4 in the way we did for 2019.3 is still considered too complex and risky a change for an LTS version (and the others are no longer supported). :(
     
    Last edited: Aug 1, 2020
  12. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Also, please note that the snapshot size is not very related to the amount of Memory taken up by the target of the capture since it is not a full memory dump (only the managed heap is dumped, Native Memory info is gathered and reported as data points only. In versions pre-2019.3 that info could be hugely inflated due to the way native connections were reported though). So being clear in what sizes we are talking about could be benefitial.
     
  13. noio

    noio

    Joined:
    Dec 17, 2013
    Posts:
    244
    Hi Martin!

    I'm using Unity 2019.4.6, with the latest package version available currently.

    It is step 3: "opening" that took so long, and it is the dump itself that is 40GB. Maybe something went wrong during the opening, and Unity just became entirely unresponsive, I was not patient enough to find out whether it eventually opened the snapshot. :oops:

    I did pinpoint the cause of the memory leak through the regular Profiler: I managed to find the exact frame where the allocations happened and figure out what caused them that way.
     
    MartinTilo likes this.
  14. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Glad you found the leak anyways :), still would be better if the Memory Profiler would have worked well enough to help with that.
    Interesting. If you still have the snapshot and would be willing to share that with us via a bug report, we'd be very interested in checking out what too so long or if anything failed in the process of opening it.
    We'd very likely only need the snapshot, which you could compress before uploading. Just let me know the issue ID, in this thread or as a pm if you do report it. :)
     
  15. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Also, did you try taking and opening a capture now that the leak is gone and does that work?
     
  16. noio

    noio

    Joined:
    Dec 17, 2013
    Posts:
    244
    I just tried, and the MemoryProfiler works fine now that the snapshot is 200MB :)

    I tried it with one of the other Snapshots (of 2GB, the really huge ones I have deleted already for diskspace).

    And—now I notice—as soon as I try to open it, Unity throws this exception:

    OverflowException
    Unity.MemoryProfiler.Editor.Format.MemorySnapshotFileReader.GetDataByteArray (Unity.MemoryProfiler.Editor.Format.EntryType entryType, System.UInt32 entryIndex, System.UInt32 numEntries, System.Byte[][]& dataOut) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/MemorySnapshot/MemorySnapshotFileReader.cs:158)
    Unity.MemoryProfiler.Editor.Format.ByteArrayEntries.GetEntries (System.UInt32 indexStart, System.UInt32 numEntries, System.Byte[][]& dataOut) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/MemorySnapshot/ArrayEntries.cs:57)
    Unity.MemoryProfiler.Editor.CachedSnapshot+ManagedMemorySectionEntriesCache..ctor (Unity.MemoryProfiler.Editor.Format.ManagedMemorySectionEntries sectionEntries) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/Profiler/CachedSnapshot.cs:535)
    (etcetera...)


    Perhaps it will help you to know that the cause of my memory leak was me using the answer here to read all bytes from a BinaryReader:

    Code (CSharp):
    1. byte[] allData = read1.ReadBytes(int.MaxValue);
    But what that actually does is allocate an array of size
    int.MaxValue
    (2GB)

    Maybe the size of this entity is causing the OverflowException too?
     
    Last edited: Aug 1, 2020
    MartinTilo likes this.
  17. noio

    noio

    Joined:
    Dec 17, 2013
    Posts:
    244
    Alright: submitted a bug report, it's case 1267508
     
    MartinTilo likes this.
  18. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Thanks for the bug report :)
    And yes that ReadBytes(int.MaxValue) seems to cause this Overflow exception. We'll check out whats up there. :)
     
  19. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    343
    upload_2021-1-13_10-44-31.png

    Here is me, on Unity 2019.4.

    - Taking the snapshot took 5 seconds
    - Unity is stuck on this GC Handles thing for the past 13 hours. I'm terminating the process now.

    Is that a memory leak preventing me from seeing what my memory leak is?
     
  20. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    A memory leak would have crashed your editor out of memory after that duration, but please report a bug for this with the snapshot file. We should get that fixed. :)
     
  21. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Also please let me know the issue ID once you have it and we can start looking at it a bit quicker.
     
  22. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    343
    Thank you so much! I submitted a report but haven't got a confirmation email yet. I'll give it some time and get back to you.
     
    MartinTilo likes this.
  23. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    343
    There it is
    1306088
     
    MartinTilo likes this.
  24. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Thank you for reporting the bug, the fix for it has been published as 0.2.8-preview.1
     
  25. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    343
    Talk about a great customer service! Thank you so much!
     
    MartinTilo likes this.
  26. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Thank you but the credit kinda goes to @alexrvn for actually fixing and publishing it this quickly :)
     
  27. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    343
    Well then, @alexrvn, I'm sending you virtual beers!

    Kidding aside, I tried it at it's indeed working now :) Cheers
     
    alexrvn and MartinTilo like this.
  28. lacas8282

    lacas8282

    Joined:
    Apr 25, 2015
    Posts:
    139
    Memory profiler not working under Unity 2022 LTS

    upload_2023-6-14_16-15-24.png

    my game started from 4 GB, when take a snapshot growing the memory up to 24 GB and then die the machine
    'Hold on'" forever

    any idea? I need to close from win11, also I am in 4K, this is a probleme?
     
  29. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Hmm, are you capturing the Editor or a built player? If so, could you try updating the package to version 1.1.0-exp.1 (probably only possible via changing the manifest)?

    And if a built Player, are you simultaneously CPU Profiling it? If so, does this still occur if you stop the CPU profiler recording before taking a capture?
     
    lacas8282 likes this.
  30. lacas8282

    lacas8282

    Joined:
    Apr 25, 2015
    Posts:
    139
    I am on the editor yet.
    Weirdest thing is the memory profiler working on a new project. But on this large 100 GB not. I tried on a large scene. But also created a new scene what is completely empty in that large project. The same behaviour, crash after a while. Is it normal on an empty scene?
     
  31. lacas8282

    lacas8282

    Joined:
    Apr 25, 2015
    Posts:
    139
    I can send you a log file. I am using Vulkan as a backend, lot of Vulkan memory low on that file
     
  32. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Do you open any snapshots while you keep taking them? The 1.0 version still loads a lot of the snapshot content into managed containers, contributing to managed memory fragmentation. It also doesn't fully clear some data out unless the window is closed between captures, leading to recursively bigger Editor snapshots as that new data used by the Memory Profiler is included in the next capture.

    1.1 addresses most of these and will address even more of the managed containers in the next version. So it'd be good to see if updating to that version helps at all.
     
    lacas8282 likes this.
  33. lacas8282

    lacas8282

    Joined:
    Apr 25, 2015
    Posts:
    139
    Updated the manifest json with 1.1.0-exp.1

    And restarted the project and opened the memory profiler again.

    Snapshot is taken 1 GB

    Memory still goes from 7 GB to 22 GB, 100%, than go to 8 GB, AND 20 AGAIN,

    AFTER 5 minutes it is crashing
     
  34. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Are you opening the snapshots in-between of captures or does the memory increase just happen while the capturing is in progress? When does it reset down?
    And do you know if this was an issue in earlier Unity versions?
     
  35. lacas8282

    lacas8282

    Joined:
    Apr 25, 2015
    Posts:
    139
    No, I don't play the game at all. It is on stopped state. Also I open the Memory profiler and than Press the "Capture SNAPSHOT" btn. After that the thumbnail appears, I click on the thumbnail. Thats it.
     
  36. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Hmm so you just take one snapshot and then open it and both during capture and during opening the memory spikes up from ~7 to around 20GB for each of these events?

    The thing is, I'm not sure why the memory should spike up during capture as the data should be streamed to the file on disk, so the memory usage there shouldn't really increase that dramatically.

    During opening is a different issue and something we tried to address for 1.1.0-exp.1 but there's also another fix for a crash on bigger captures pending for 1.1.0-pre.2. maybe that would at least fix the crash.

    I just feel like that spike during capture shouldn't happen and might be due to an issue that then also means that opening the capture requires a good deal more memory...
     
    Last edited: Jul 17, 2023
    idyakonov likes this.
  37. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Is it better if you take a snapshot from a development build?
     
  38. idyakonov

    idyakonov

    Joined:
    Dec 6, 2021
    Posts:
    11
    Your post is from June 15, but 1.1.0-exp.2 is dated 2023-04-20 according to package changelog. Did you mean a newer version past 1.1.0-pre.1? Is there any ETA on this fix?
     
  39. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Ah, no you're right, I meant to write 1.1.0-pre.2. I fixed that mistake in the post now.
    The fix is still pending and so is that version. I'll hopefully get back to that fix and land it this week but I've got no ETA for you on the pre.2 release yet. Hopefully soon (TM).

    That said. Without context (e.g a bug report or stack trace of a crash) I can not guaranteed that this fix will fix exactly what you are facing. So while I hope not, there is a chance that it wouldn't fix whatever you're experiencing.
     
    idyakonov likes this.
  40. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    383
    This is still a massive issue for us. I can't really use the Memory Profiler at all anymore as it gets stuck at "Crawling GC Handles" for I assume ever. I've waited 30+ mins. Once or twice I have managed to get it to process with within 6 mins. So it would seem it is getting stuck on something.

    I've tried it in the editor and our development builds (usually this).

    Note that I swear I used this tool flawlessly 1-3 months ago quite a lot without issue. I don't know what's changed to make it unfortunately useless now.
     
    Last edited: Jan 4, 2024
  41. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,517
    Hi Scott,
    I assume your issue is a new one introduced in 1.1.0-pre.3 and I answered to your thread here.