Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. The 2023.1 beta is now available for testing. To find out what's new, have a look at our 2023.1 feature highlights.
    Dismiss Notice

Expected times for capturing and processing snapshots?

Discussion in 'Profiler Previews' started by pavelkouril, Dec 19, 2018.

  1. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Hello,

    I'm trying to capture a snapshot of a running game (reports around 2.2GB in task manager), and the snapshot takes forever (approximately 30 minutes) to capture and then to load (around ~15 minutes). Also, the snapshots take like 10GB on disk, and the snapshot contains around 1 million of objects (~750k managed and 250k native ones).

    Do you consider this as an expected/reasonable time for the memory profiler workflows, or is this something that you intend to work on in the future and make it considerably faster?
     
    phobos2077 likes this.
  2. G_Wojo

    G_Wojo

    Unity Technologies

    Joined:
    Feb 8, 2017
    Posts:
    32
    Hello pavelkouril.

    Speed of memory profiling is one of really important topics for us. Could you open ticket for us and attach there your compressed snapshot ? We would use it for investigation why it took so long to open, try improve code to handle it better and investigate why it taking so much on your hard drive.
     
  3. pavelkouril

    pavelkouril

    Joined:
    Jul 22, 2016
    Posts:
    129
    Hello, done.

    Reported it as a Case 1111451.
     
  4. MartinV

    MartinV

    Joined:
    Nov 12, 2013
    Posts:
    3
    I have similar issue. Capture took about 30 minutes. This was in Editor while game was running, and so I assume capturing on device could be faster due to Editor contributing a lot of allocations. But as I'm not able to run on device (separate ticket, linker crash when building Android), that's all I have at the moment.
    The larger issue is that when I then Open the capture, Unity crashes about 5 minutes into opening it. I've uploaded compressed capture in the ticket (500MB uncompressed size). Case 1111647
     
  5. samfisher-cf

    samfisher-cf

    Joined:
    Jun 13, 2016
    Posts:
    5
    The latest memory profiler preview 4 contains a fix for a stack overflow crash, which I have also noticed and fixed locally before their fix came out.

    My snapshots also take between 15-45 mins to open and diff, so I added a progress bar tracking the number of objects processed to know if the profiler is still doing something or has just frozen/hung due to some error.
     
  6. ScottyB

    ScottyB

    Joined:
    Apr 4, 2011
    Posts:
    146
    Hi @Greg_Wojciechowski, I recently submitted a new bug report (Case 1126009) where the memory snapshot crawling process either gets stuck or takes an unreasonably long time (I waiting 30 mins+ before force closing Unity). Just thought you would like to know in case it helps fixing and/or speeding up the snapshot crawling code.
     
    MartinTilo likes this.
  7. Rohansi

    Rohansi

    Joined:
    Jun 14, 2017
    Posts:
    3
    I just submitted two bug reports related to this (1142394 and 1142384) with solutions included. The first one is when capturing a snapshot it will take a long time (depending on size) to actually start dumping the snapshot, and the fix is a patch to Unity's fork of Mono. The second one is an optimization to the memory profiler package to speed up one of the last steps in opening the snapshots.

    The second issue may be better solved by introducing a processing phase after capturing snapshots so it can be cleaned up and sorted for quick access.
     
    MartinTilo likes this.
  8. Rohansi

    Rohansi

    Joined:
    Jun 14, 2017
    Posts:
    3
    Just submitted another two for processing snapshots, one for performance (1146248) and one integer overflow (1146246). This one helps with the loading when you see the progress bar at "Crawling GC handles". My solutions are included with them as well. With all my fixes I got a snapshot that's 117 GB large and took like 18 hours to capture, loading it didn't take an unreasonable amount of time though.
     
    Last edited: Apr 15, 2019
    noio and MartinTilo like this.
  9. MadeFromPolygons

    MadeFromPolygons

    Joined:
    Oct 5, 2013
    Posts:
    3,665
    They take about 30 seconds to 1 minute to capture, and a 2-3 seconds to open for us. This is on a relatively large client project for HoloLens.
     
  10. noio

    noio

    Joined:
    Dec 17, 2013
    Posts:
    201
    What is 'unreasonable' ? I have a weird memory leak that sucks up 40GB, but I've been waiting almost an hour for the snapshot to load..
     
  11. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    @noio what Unity and package version are you using? If player and editor versions differ then please state both versions.

    Does it take long to
    1. capture (i.e. player freezes),
    2. process the capture on the editor (player is unfrozen but editor isn't or doesn't yet show the snapshot)
    3. or to open it once it appears in the snapshot list?
    1 and 2 should be dramatically faster in +2019.3 versions, 3 has been sped up across the board (remember to update to the latest package versions, it'll pay off) but would still be faster with +2019.3 versions due to having less superfluous native connections to crawl through.

    For +2019.3 Unity versions I'd venture to say that anything beyond 5-10 minutes at any of these points or across all of them would be unreasonable (and worthy of a bug report) at this point, and even that's somewhat conservative.

    I guess 40GB of Memory used by the app being captured might mean that the capture process would need to access swap memory on most systems? I guess that could make it slower but almost an hour of waiting times still sounds too long for +2019.3 versions.

    For 2018.3-2019.2 versions there is little we can do about these timings, durations south of an hour are not unheard of and it can also just have hung up at some point. We have a bug report for it but fixing it in 2018.4 in the way we did for 2019.3 is still considered too complex and risky a change for an LTS version (and the others are no longer supported). :(
     
    Last edited: Aug 1, 2020
  12. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    Also, please note that the snapshot size is not very related to the amount of Memory taken up by the target of the capture since it is not a full memory dump (only the managed heap is dumped, Native Memory info is gathered and reported as data points only. In versions pre-2019.3 that info could be hugely inflated due to the way native connections were reported though). So being clear in what sizes we are talking about could be benefitial.
     
  13. noio

    noio

    Joined:
    Dec 17, 2013
    Posts:
    201
    Hi Martin!

    I'm using Unity 2019.4.6, with the latest package version available currently.

    It is step 3: "opening" that took so long, and it is the dump itself that is 40GB. Maybe something went wrong during the opening, and Unity just became entirely unresponsive, I was not patient enough to find out whether it eventually opened the snapshot. :oops:

    I did pinpoint the cause of the memory leak through the regular Profiler: I managed to find the exact frame where the allocations happened and figure out what caused them that way.
     
    MartinTilo likes this.
  14. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    Glad you found the leak anyways :), still would be better if the Memory Profiler would have worked well enough to help with that.
    Interesting. If you still have the snapshot and would be willing to share that with us via a bug report, we'd be very interested in checking out what too so long or if anything failed in the process of opening it.
    We'd very likely only need the snapshot, which you could compress before uploading. Just let me know the issue ID, in this thread or as a pm if you do report it. :)
     
  15. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    Also, did you try taking and opening a capture now that the leak is gone and does that work?
     
  16. noio

    noio

    Joined:
    Dec 17, 2013
    Posts:
    201
    I just tried, and the MemoryProfiler works fine now that the snapshot is 200MB :)

    I tried it with one of the other Snapshots (of 2GB, the really huge ones I have deleted already for diskspace).

    And—now I notice—as soon as I try to open it, Unity throws this exception:

    OverflowException
    Unity.MemoryProfiler.Editor.Format.MemorySnapshotFileReader.GetDataByteArray (Unity.MemoryProfiler.Editor.Format.EntryType entryType, System.UInt32 entryIndex, System.UInt32 numEntries, System.Byte[][]& dataOut) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/MemorySnapshot/MemorySnapshotFileReader.cs:158)
    Unity.MemoryProfiler.Editor.Format.ByteArrayEntries.GetEntries (System.UInt32 indexStart, System.UInt32 numEntries, System.Byte[][]& dataOut) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/MemorySnapshot/ArrayEntries.cs:57)
    Unity.MemoryProfiler.Editor.CachedSnapshot+ManagedMemorySectionEntriesCache..ctor (Unity.MemoryProfiler.Editor.Format.ManagedMemorySectionEntries sectionEntries) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/Profiler/CachedSnapshot.cs:535)
    (etcetera...)


    Perhaps it will help you to know that the cause of my memory leak was me using the answer here to read all bytes from a BinaryReader:

    Code (CSharp):
    1. byte[] allData = read1.ReadBytes(int.MaxValue);
    But what that actually does is allocate an array of size
    int.MaxValue
    (2GB)

    Maybe the size of this entity is causing the OverflowException too?
     
    Last edited: Aug 1, 2020
    MartinTilo likes this.
  17. noio

    noio

    Joined:
    Dec 17, 2013
    Posts:
    201
    Alright: submitted a bug report, it's case 1267508
     
    MartinTilo likes this.
  18. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    Thanks for the bug report :)
    And yes that ReadBytes(int.MaxValue) seems to cause this Overflow exception. We'll check out whats up there. :)
     
  19. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    314
    upload_2021-1-13_10-44-31.png

    Here is me, on Unity 2019.4.

    - Taking the snapshot took 5 seconds
    - Unity is stuck on this GC Handles thing for the past 13 hours. I'm terminating the process now.

    Is that a memory leak preventing me from seeing what my memory leak is?
     
  20. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    A memory leak would have crashed your editor out of memory after that duration, but please report a bug for this with the snapshot file. We should get that fixed. :)
     
  21. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    Also please let me know the issue ID once you have it and we can start looking at it a bit quicker.
     
  22. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    314
    Thank you so much! I submitted a report but haven't got a confirmation email yet. I'll give it some time and get back to you.
     
    MartinTilo likes this.
  23. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    314
    There it is
    1306088
     
    MartinTilo likes this.
  24. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    Thank you for reporting the bug, the fix for it has been published as 0.2.8-preview.1
     
  25. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    314
    Talk about a great customer service! Thank you so much!
     
    MartinTilo likes this.
  26. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,898
    Thank you but the credit kinda goes to @alexrvn for actually fixing and publishing it this quickly :)
     
  27. marcrem

    marcrem

    Joined:
    Oct 13, 2016
    Posts:
    314
    Well then, @alexrvn, I'm sending you virtual beers!

    Kidding aside, I tried it at it's indeed working now :) Cheers
     
    alexrvn and MartinTilo like this.