Expected times for capturing and processing snapshots?

Hello,

I'm trying to capture a snapshot of a running game (reports around 2.2GB in task manager), and the snapshot takes forever (approximately 30 minutes) to capture and then to load (around ~15 minutes). Also, the snapshots take like 10GB on disk, and the snapshot contains around 1 million of objects (~750k managed and 250k native ones).

Do you consider this as an expected/reasonable time for the memory profiler workflows, or is this something that you intend to work on in the future and make it considerably faster?

 

Hello, done.

Reported it as a Case 1111451.

 

I have similar issue. Capture took about 30 minutes. This was in Editor while game was running, and so I assume capturing on device could be faster due to Editor contributing a lot of allocations. But as I'm not able to run on device (separate ticket, linker crash when building Android), that's all I have at the moment.
The larger issue is that when I then Open the capture, Unity crashes about 5 minutes into opening it. I've uploaded compressed capture in the ticket (500MB uncompressed size). Case 1111647

 

The latest memory profiler preview 4 contains a fix for a stack overflow crash, which I have also noticed and fixed locally before their fix came out.

My snapshots also take between 15-45 mins to open and diff, so I added a progress bar tracking the number of objects processed to know if the profiler is still doing something or has just frozen/hung due to some error.

 

Hi @Greg_Wojciechowski, I recently submitted a new bug report (Case 1126009) where the memory snapshot crawling process either gets stuck or takes an unreasonably long time (I waiting 30 mins+ before force closing Unity). Just thought you would like to know in case it helps fixing and/or speeding up the snapshot crawling code.

 

I just submitted two bug reports related to this (1142394 and 1142384) with solutions included. The first one is when capturing a snapshot it will take a long time (depending on size) to actually start dumping the snapshot, and the fix is a patch to Unity's fork of Mono. The second one is an optimization to the memory profiler package to speed up one of the last steps in opening the snapshots.

The second issue may be better solved by introducing a processing phase after capturing snapshots so it can be cleaned up and sorted for quick access.

 

Just submitted another two for processing snapshots, one for performance (1146248) and one integer overflow (1146246). This one helps with the loading when you see the progress bar at "Crawling GC handles". My solutions are included with them as well. With all my fixes I got a snapshot that's 117 GB large and took like 18 hours to capture, loading it didn't take an unreasonable amount of time though.

 

They take about 30 seconds to 1 minute to capture, and a 2-3 seconds to open for us. This is on a relatively large client project for HoloLens.

 

What is 'unreasonable' ? I have a weird memory leak that sucks up 40GB, but I've been waiting almost an hour for the snapshot to load..

 

Hi Martin!

I'm using Unity 2019.4.6, with the latest package version available currently.

It is step 3: "opening" that took so long, and it is the dump itself that is 40GB. Maybe something went wrong during the opening, and Unity just became entirely unresponsive, I was not patient enough to find out whether it eventually opened the snapshot.

I did pinpoint the cause of the memory leak through the regular Profiler: I managed to find the exact frame where the allocations happened and figure out what caused them that way.

 

I just tried, and the MemoryProfiler works fine now that the snapshot is 200MB

I tried it with one of the other Snapshots (of 2GB, the really huge ones I have deleted already for diskspace).

And—now I notice—as soon as I try to open it, Unity throws this exception:

OverflowException
Unity.MemoryProfiler.Editor.Format.MemorySnapshotFileReader.GetDataByteArray (Unity.MemoryProfiler.Editor.Format.EntryType entryType, System.UInt32 entryIndex, System.UInt32 numEntries, System.Byte[][]& dataOut) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/MemorySnapshot/MemorySnapshotFileReader.cs:158)
Unity.MemoryProfiler.Editor.Format.ByteArrayEntries.GetEntries (System.UInt32 indexStart, System.UInt32 numEntries, System.Byte[][]& dataOut) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/MemorySnapshot/ArrayEntries.cs:57)
Unity.MemoryProfiler.Editor.CachedSnapshot+ManagedMemorySectionEntriesCache..ctor (Unity.MemoryProfiler.Editor.Format.ManagedMemorySectionEntries sectionEntries) (at Library/PackageCache/com.unity.memoryprofiler@0.2.5-preview.1/Editor/Profiler/CachedSnapshot.cs:535)
(etcetera...)


Perhaps it will help you to know that the cause of my memory leak was me using the answer here to read all bytes from a BinaryReader:

Code (CSharp):

1. byte[] allData = read1.ReadBytes(int.MaxValue);

But what that actually does is allocate an array of size

int.MaxValue

(2GB)

Maybe the size of this entity is causing the OverflowException too?

 

Alright: submitted a bug report, it's case 1267508

 

Here is me, on Unity 2019.4.

- Taking the snapshot took 5 seconds
- Unity is stuck on this GC Handles thing for the past 13 hours. I'm terminating the process now.

Is that a memory leak preventing me from seeing what my memory leak is?

 

Thank you so much! I submitted a report but haven't got a confirmation email yet. I'll give it some time and get back to you.

 

There it is
1306088

 

Talk about a great customer service! Thank you so much!

 

Well then, @alexrvn, I'm sending you virtual beers!

Kidding aside, I tried it at it's indeed working now Cheers

 

Memory profiler not working under Unity 2022 LTS



my game started from 4 GB, when take a snapshot growing the memory up to 24 GB and then die the machine
'Hold on'" forever

any idea? I need to close from win11, also I am in 4K, this is a probleme?

 

I am on the editor yet.
Weirdest thing is the memory profiler working on a new project. But on this large 100 GB not. I tried on a large scene. But also created a new scene what is completely empty in that large project. The same behaviour, crash after a while. Is it normal on an empty scene?

 

I can send you a log file. I am using Vulkan as a backend, lot of Vulkan memory low on that file

 

Updated the manifest json with 1.1.0-exp.1

And restarted the project and opened the memory profiler again.

Snapshot is taken 1 GB

Memory still goes from 7 GB to 22 GB, 100%, than go to 8 GB, AND 20 AGAIN,

AFTER 5 minutes it is crashing

 

No, I don't play the game at all. It is on stopped state. Also I open the Memory profiler and than Press the "Capture SNAPSHOT" btn. After that the thumbnail appears, I click on the thumbnail. Thats it.

 

Your post is from June 15, but 1.1.0-exp.2 is dated 2023-04-20 according to package changelog. Did you mean a newer version past 1.1.0-pre.1? Is there any ETA on this fix?

 

This is still a massive issue for us. I can't really use the Memory Profiler at all anymore as it gets stuck at "Crawling GC Handles" for I assume ever. I've waited 30+ mins. Once or twice I have managed to get it to process with within 6 mins. So it would seem it is getting stuck on something.

I've tried it in the editor and our development builds (usually this).

Note that I swear I used this tool flawlessly 1-3 months ago quite a lot without issue. I don't know what's changed to make it unfortunately useless now.