Unable to take a memory sample from bigger scenes. (Infinity loop)

Hello,

I'd think that the memory snapshot process for the old Memory Profiler (the one in the Profiler Window's Memory view under detailed) should work. There used to be a lockup in the connection that was fixed earlier this year. So the question is in which part of the process it hangs. If you could use this API to request the snapshot by code and just see if you actually get a snapshot, that would clear up if it's an issue in the UI or in the backend.

However, the focus is currently on the new Memory Profiler that's about to ship with 2018.3 so I'd wait for that if possible.

 

The doc link was for 2018.2 and switching version does not carry over the opened API, but the API is available in 2017.4

With what you described it sounds to me like you should be looking at the CPU profiler and find out where all those GC.Allocs are coming from and reducing them.
Have you looked at this doc yet?

How is your managed heap growing over time? What exact version of 2017.4 are you on? Have you had a look at this tool?

Also yes, you could just file a bug about this.

 

Hi again,

going over your issues a bit more closely again, taking a memory snapshot might not be the only thing you can do to address this. It might not even help you all that much. What it would show you is: native objects (like textures or other assets) and managed objects.

Sure, if you're leaking memory, then it would help you discover that. But I'm guessing you're not using the unsafe keyword in C# to allocate unmanaged memory? Maybe you're using RenderTextures or something like that which could leak native memory because it is not getting disposed but I'd assume you'd crash out of memory way faster if that was the case. Maybe you're keeping static references to managed memory so stuff does not get cleaned up by the GC. However, this should also show up in a smaller scale. Mabye you're just fragmenting your Mono Memory by constantly allocating on the managed heap and letting the GC clean most of it up but not all of it. Then, the GC has to go over larger and more fragmented pieces of memory with each iteration taking longer and longer.

To check if you're actually leaking and if it is native or managed, you don't necessarily need to take a snapshot. You don't need deep profiling or potentially even attaching the profiler.

In a debug build of your server, you could periodically, (e.g. every 15 min or half hour) turn on Profiler.enableBinaryLog (check the linked docs for more detailed instructions) and specify a different log file location each time. Since the Profiler Window currently only allows for showing the last 300 frames of that capture, there is no need to capture more frames than that each time. Now, when loading these captures in to the Profiler Window, you can check the stats in the Memory Profiler Chart of the Profiler Window. Do the numbers there go up or do they stay mostly constant over time? Also, if some go up, which ones?

You can also use these methods to get an idea of your memory usage:

// Reserved Mono Memory
UnityEngine.Profiling.Profiler.GetMonoHeapSizeLong()

// Used Mono Memory
UnityEngine.Profiling.Profiler.GetMonoUsedSizeLong()

// Reserved Native Memory
UnityEngine.Profiling.Profiler.GetTotalReservedMemoryLong()

// Used Native Memory
UnityEngine.Profiling.Profiler.GetTotalAllocatedMemoryLong()

Remember, If you use Debug.Log to output those, you're aggravating your GC.Alloc problem so use that sparingly and maybe not while capturing the binary log. If your server is not running headless, maybe you can display it on the screen. However, converting numbers to text and changing the UI frequently is not usually a very performant and GC.Alloc free operation either. There are suggestions for how to implement a StringBuilder that does this in a GC.Alloc free way out there, e.g.:
http://www.gavpugh.com/2010/03/23/xnac-stringbuilder-to-string-with-no-garbage/
http://www.gavpugh.com/2010/04/01/xnac-avoiding-garbage-when-working-with-stringbuilder/
http://www.gavpugh.com/2010/04/05/xnac-a-garbage-free-stringbuilder-format-method/

Also, you might want to use TextMeshPro, since the default uGUI Text fields regenerate the text mesh everytime, Allocating Garbage.

If you want to know more in detail, where your GC.Allocs come from, but can't or don't want to rely on deep profiling, you can use a Divide and Conquer methodology to narrow it down. Checkout where most of the allocations happen in the profiles you captured with the Binary log method and then divide the methods that you can see there up with CustomSamplers. For example, you could put a CustomSampler into every bigger method you're calling from the one you can see in the profiler, basically selectively going deeper in the callstack for where you need to investigate. If that doesn't help narrowing it down, go even deeper and delete those Samplers again under which there where no allocations to reduce overhead. This is a bit more of a manual process than what we'd like it to be but at least it should help you get somewhere with the investigation.

Maybe you can't reduce your GC.Allocations to zero. Maybe you need to consider periodically spinning up a new server instance of the game to take over with a clean memory slate. But reducing you're allocations should help the server run for longer before it runs into problems.

Also, if we can figure out a way for you to share the project with us, we still do want to fix the bug preventing you from capturing and maybe this will help figuring out what happens. What I'm suggesting is alternatives and things you could try in the meantime

 

Right, that does look like you're leaking managed memory. I've replied to the other thread as well and am poking around a bit. Thanks for bearing with us as we're trying to figure this out.

To your questions:
There is overhead in development players as the code can only be parsed out with a recompile. So whether the profiler is attached or not, having those calls in your code does come with a bit of overhead. That said, when the profiler is disabled/not attached the overhead is reduced. The CustomSampler API comes with less overhead than Profiler.BeginSample, since (among other things) the later one has to emit the name as meta data and the former just creates a sample id.

For release players, the overhead is very minimal as Profiler.BeginSample(), Profiler.EndSample(), CustomSampler.Begin() and CustomSampler.End() calls are compiled out as they are marked by the Conditional attribute. Creation in a release build is no more than the method call overhead for calling into the bindings layer, one call to native and an IntPtr compare.
There is a small memory overhead of the CustomSampler API of one pointer size wherever a created sample is stored, plus the name string (which isn't stored in the CustomSampler in release).

 

The performance difference is real though I can't tell you by how much. One other thing that can have an impact is the option to Attach for Script Debugging (When building a player, the debug build option "Script Debugging", for the Editor Preferences>External Tools>Editor Attaching) as this injects code that checks if an attempt to connect has been made and if so, accept it. For better benchmarking results when profiling a development player, that option should be turned off.

 

Hey there,
at first we assumed the connection might time out, which would have needed a very involved fix, but it turns out that it doesn't.
The status right now is threefold:

• Regarding the crash: a fix for that is in the process of landing.
• Regarding the infinite loop: we've been able to capture, which took 2 1/2 hours due to the sheer amount of managed objects and the connections between them.
• Regarding the out of memory crash: we've managed to capture above mentioned snapshot on a laptop with 32 GB RAM, It might well be that a system with less RAM or a non stripped version of the project might run out of memory.

In 2018.3, we've changed the backend to be streaming the native memory capture to reduce the RAM usage during capture. Since capturing managed memory seems to be the biggest issue, that might not be enough. We'll be looking at reducing RAM usage even further by streaming the managed memory as well in future versions of the backend, as well as streaming to disk and nearly entirely avoiding RAM in especially dire circumstances. Sadly there is no way to backport these backend changes to 2017.4.

 

Hey!
When running the tests(2017.4) I grabbed two snapshots first one took around 2hr:45m second around 3hr+ I didn't time that one. So the main offenders here are the sheer number of objects in managed and native. Part of the aforementioned time will be spent crawling handles which at least on my tests(with the repo you guys provided, which was stripped) was approx 30-45 min. The largest amount tho was just going through the native objects (think anything that inherits UnityEngine.Object) liveness states, We're planning to take a look at that for both 2018.4 and 2017.4 as improving it should speed up the capture on massive projects. On the memory overhead side of things, I doubt we can do anything to get the capture streaming into 2017.4, I'm hopeful for 2018.4.
My advice in getting snapshots out of the current system would be chunking by areas and then by systems when testing ( if possible). I'd also suggest trying to use the "Memory" tab from the Profiler and attempt to take memory samples (while somewhat limited in information compared to the memory profiler captures it should provide at least some minimal amount of data to work on).
As a side note main issue when taking a capture (where the application pushes the limits on memory available on the device) is the copying of the managed heap(gets duplicated and written to the snapshot object). This is problematic to get around of since pretty much most allocations in your scripts are usually managed. On your stripped down repo project this was not really the issue as not that many managed allocations were in play, I suppose on the actual non-stripped project this is completely different.

Last but not least, reading this thread made it sound that you guys deployed your server under development build, that is not alright as development builds add quite a bit of overhead versus release builds, I'd also suggest trying to do IL2CPP builds for the server as should give a performance boost ( in general, but I do not know the code base in question here so it might not have too much of an impact).

 

Not as of yet, that sounds really weird ... having a mesh with references to [many] instead of being referenced by [many] :-/.

What I'd expect to see in such a case would be:
We have a MeshFilter now this MeshFilter is referenced by the GameObject it is attached to, where the GameObject is also referenced by say an AudioSource(on the GameObject). The pattern here is each Component has a ref to it's GameObject and each GameObject has an array of components. Now lets walk the hierarchy the AudioSource has a reference to the GameObject it is attached to which in turn references your MeshFilter which finally references the Mesh.

Edit:
We will be improving the way we report references so that it is easier to understand.

 

https://docs.unity3d.com/2017.4/Documentation/Manual/IL2CPP-BuildingProject.html IL2CPP is available afaik, it needs to be installed when you install/change Unity

 

@GloriaVictis We haven't used the 2018.3 memory profiler for anything with a memory footprint as large as yours. What I'd love to do is find out how it crashed on your end when de-serializing as that might be yet another issue with the old back-end. I would suggest moving to 2018.3 either way and switching to 2018.4 LTS would be way better than going to the 2019 tech stream as you already have a live game. The only question would probably be how painful would the migration be in either case.

 

I highly doubt it will get backported to 2017.4 as it was introduced as experimental. Bringing an experimental change into a stabilizing and released version would be too risky.

Depending on how frequently you get GC.Collect calls, I also doubt that it would solve the problem. If you accumulate allocations faster than they can be processed by splitting the GC.Collect over multiple frames, it will just fall back to processing them all at once instead of time-sliced. The addition of the Incremental GC feature does not suddenly make all issues with managed allocations go away. You'd still want to reduce the amount you allocate frame over frame, ideally to 0.

That said, if you are okay to be on the Tech Stream and update through to 2019.LTS, it might be worth the risk/a try.

 

yeah, I guess that should leave enough time. And yes, some APIs sadly make it really hard to get the amount of allocations down...