Official Debugging Editor freezes and hangs [Windows Only]

Hi,
We’d like to share a debugging technique with you that will allow you to capture call stacks of the Editor when it is hung up and unresponsive. Attaching these call stacks to your bug reports can in some instances help us figure out where and why things are getting stuck, even if you can’t share your projects or other details with us.

For some time we've been testing out an approach that will help show where the editor gets stuck and is compatible with old and new versions of Unity as well.

What, you'll need:

1. Process Explorer from SysInternals
2. The right dbghelp.dll can be installed from here: Debugging Tools for Windows

Then, open up Process Explorer, and in there find Unity.exe with the right parameters (i.e. the one that contains -projectpath [your_project_name])

Inside Process Explorer click on Options -> Configure Symbols and point to the right dbghelp.dll from step 2.

The Symbols Path should be the path to the Editor folder for the version of Unity you want to debug (i.e. C:\Program Files\Unity\Hub\Editor\[UNITY_VERSION]\Editor\ ). It's important to select the right version of the Editor because it will use the symbols supplied by the PDB files to figure out the name of the functions being executed. Unfortunately, it cannot resolve call stack frames that are using Mono in them.




Once that is set up, double click Unity.exe in ProcessExplorer, go to the Threads tab, and double click the one that says Unity.exe!WinMainCRTStartup

That will show the current call stack on the main thread in Unity (like in the picture below). Using this call stack can help us figure out what's going on with the Editor faster, and it might speed up our debugging investigations as well.



Looking forward to helping out and resolving these issues of the editor getting stuck. Please feel free to use this thread to ask questions and discuss the debugging process. Please don’t use it to report your issues. Use the bug reporter for that.

 

Hey, I'll update the post with more instructions shortly, but from the link you gave me, the winsdksetup.exe is the file that gets downloaded, and opening it up gives you the choice to install a few things.
If you select the "Debugging Tools For Windows" (refer to pic below) it should then install it in the following location by default:
C:\Program Files (x86)\Windows Kits\10



Then, if you can go to the bin\x64 folder you should find DbgHelp.dll and you can then point Process Explorer to use that: C:\Program Files (x86)\Windows Kits\10\bin\x64\DbgHelp.dll

This looks almost like what you wrote up there, so please let me know if it doesn't work and I can look into it some more.

 

Symbols are located next to Unity.exe in Unity installation directory. You can also point the debugger to our symbol server (https://symbolserver.unity3d.com/). Dumps never include symbols.

 

In this callstack, it looks like something is preventing the domain unload from happening.
Do you have some IDE or a debugger attached? This might be preventing the call from completing.
If you do have a debugger attached, I would recommend updating the package for the debugger in your project to the latest version, since the Visual Studio team has their eye on this sort of race conditions.

Does that help?

 

I'd also suggest seeing what other threads do. Usually callstack like this indicates that a thread has been started and refuses to finish and Unity must wait for all other C# threads to exit before continuing.

 

That's what the editor does, tries aborting them. However, if they are stuck executing native or plugin code, aborting threads may not work (and that's usually the reason for these complete hangs on domain reloads).

 

Hi @DVelt , thanks for the stack trace.

From what I can see, the editor is trying do to a domain reload but is getting stuck doing it.
What I've seen in the past is that some native code, which has been called from managed code, may be doing a long running operation or may be stuck itself too. This prevents Mono from unloading threads as it has to wait for the Native code to come back (Mono can't terminate native code).

Have you got other threads that are also stuck on "WaitForSingleObject" ? That might help shed some more light on this issue.

 

Hi @Gaitorade3D
If you could share the stack traces then we might be able to determine what is creating this apparent deadlock. To be honest, probably the quickest thing for you to do would be to figure out a certain set of actions that lead to the deadlock and avoid doing them, while we get a fix out into the Editor (for whatever we discover in this convo).

 

@okluskyond thanks for supplying the stack traces. I can see that they are for 2 separate processes (2952 and 15724) was this during the same session, and if so, was one of them an asset import worker?

Also, the stack traces for the WinMainCRTStartup thread seem incomplete. Usually we would see a bunch of Unity function calls before the Mono. Did you add the path to the editor inside Process Explorer, and set the path to the dbghelp.dll? That would help to understand where in the editor the code is coming from

 

Ok, so I've had a look around for the try_enter_with_atomic_var stack frame that shows up on your callstack.
Seems like Microsoft knows about this issue: https://github.com/dotnet/runtime/issues/62419 and there's a parent case here: https://github.com/dotnet/runtime/issues/51864

Based on the different information provided on the 2nd link, its related to loading a type.

I still think it would be great to try and get the rest of the callstack, so I could be able to perhaps offer a workaround. Maybe you can try running Process Explorer as Admin and see if that resolves the rest of the callstack?

 

Thanks for the callstack @Gaitorade3D ,

From what I can see you're trying to enter play mode and then the Editor gets stuck doing a domain unload. Is this happening everytime, or under specific circumstances?

Props for using procmon too! The registry key access isn't something I have seen as a problem before, and I generally filter them out anyways since it causes too much spam. I know at least that Preferences are stored in the registry, so we will read it every so often to figure out various settings, but I would not be surprised if it contributed to a deadlock on Mono.

I have been talking to @okluskyond about the problems in their project and after a bit of back and forth on the DMs it seems like we're making some progress. Will update once we find out more.

As for your problem once you can say under which circumstances it happens we can try and fix it up!

 

When this happens (editor getting completely stuck), can you find Unity.exe in task manager details tab, right click it, press "Create Dump File", put it in a zip/7z file and send it to me via a DM?

 

This is a symbolicated callstack:

Code (csharp):

1.      ntdll.dll!NtWaitForSingleObject()   Unknown
2.      KERNELBASE.dll!WaitForSingleObjectEx()   Unknown
3.      [Inline Frame] mono-2.0-bdwgc.dll!win32_wait_for_single_object_ex_interrupt_checked(_MonoThreadInfo *) Line 127   C
4.      mono-2.0-bdwgc.dll!win32_wait_for_single_object_ex(void * handle, unsigned long timeout, int alertable, int cooperative) Line 141   C
5.      [Inline Frame] mono-2.0-bdwgc.dll!mono_coop_win32_wait_for_single_object_ex(void *) Line 160   C
6.      [Inline Frame] mono-2.0-bdwgc.dll!mono_w32handle_wait_one(void *) Line 849   C
7.      mono-2.0-bdwgc.dll!mono_monitor_wait(MonoObjectHandle obj_handle, unsigned int ms, unsigned char) Line 1445   C
8.      [Inline Frame] mono-2.0-bdwgc.dll!ves_icall_System_Threading_Monitor_Monitor_wait(MonoObjectHandle) Line 1506   C
9.      mono-2.0-bdwgc.dll!ves_icall_System_Threading_Monitor_Monitor_wait_raw(volatile void * a0, unsigned int a1) Line 1028   C
10.      0000017c963a4839()   // System.Threading.Monitor.Monitor_wait
11.      0000017c963a3e83()   // System.Threading.Monitor.ObjWait
12.      0000017c963a3903()   // System.Threading.Monitor.Wait
13.      0000017d7efcddbb()   // System.Threading.Monitor.Wait
14.      0000017d7efcb99b()   // System.Threading.ManualResetEventSlim.Wait
15.      0000017d7efc4873()   // System.Threading.Tasks.Task.SpinThenBlockingWait
16.      0000017d7efc37ab()   // System.Threading.Tasks.Task.InternalWait
17.      0000017d7efd2c6b()   // System.Threading.Tasks.Task.Wait
18.      0000017d7efd2a33()   // System.Threading.Tasks.Task.Wait
19.      0000017d7efd1a7b()   // System.Net.ServicePointScheduler/AsyncManualResetEvent.Set
20.      0000017d7efcf973()   // System.Net.ServicePointScheduler.Run
21.      0000017c6a58dbdb()   // System.Net.ServicePointScheduler.CloseConnectionGroup
22.      0000017c6a58d70b()   // System.Net.ServicePoint.CloseConnectionGroup
23.      0000017c6a58c4cb()   // System.Net.ServicePointManager.CloseConnectionGroup
24.      0000017c6a58c1d3()   // System.Net.Http.MonoWebRequestHandler.Dispose
25.      0000017c6a58c072()   // System.Net.Http.MonoWebRequestHandler.Dispose
26.      0000017c6a58beef()   // System.Net.Http.HttpClientHandler.Dispose
27.      0000017c6a58bdff()   // System.Net.Http.HttpMessageHandler.Dispose
28.      0000017c6a58bce3()   // System.Net.Http.HttpMessageInvoker.Dispose
29.      0000017c6a58bb7b()   // System.Net.Http.HttpClient.Dispose
30.      0000017c6a58b9bf()   // System.Net.Http.HttpMessageInvoker.Dispose
31.      0000017c6a58b833()   // GoldenGait.WebRequests.HttpClientHandler.~HttpClientHandler
32.      0000017d7a5c073c()   // System.Object.runtime_invoke_virtual_void__this__
33.      mono-2.0-bdwgc.dll!mono_gc_run_finalize(void * obj, void * data) Line 383   C
34.      mono-2.0-bdwgc.dll!finalize_domain_objects() Line 944   C
35.      [Inline Frame] mono-2.0-bdwgc.dll!mono_runtime_do_background_work() Line 978   C
36.      mono-2.0-bdwgc.dll!finalizer_thread(void * unused) Line 1036   C
37.      mono-2.0-bdwgc.dll!start_wrapper_internal(StartInfo * start_info, unsigned __int64 * stack_ptr) Line 1272   C
38.      mono-2.0-bdwgc.dll!start_wrapper(void * data) Line 1348   C
39.      kernel32.dll!BaseThreadInitThunk()   Unknown
40.      ntdll.dll!RtlUserThreadStart()   Unknown

Basically what happens is that Unity asks Mono to unload the domain, which causes all finalizers for all objects fire. One of those objects is GoldenGait.WebRequests.HttpClientHandler, which is in your project. It calls HttpMessageInvoker.Dispose, which eventually ends up in System.Net.ServicePointScheduler.Run and it seems to spawn an async Task, and then waits for it to complete. Unfortunately, thread pool is shut down at this point and the task will never execute, thus you get a deadlock.

To fix this, move the System.Net.Http dispose calls from GoldenGait.WebRequests.HttpClientHandler finalizer to the main thread so that it gets disposed before the threadpool is shut down.

Alternatively, I can also recommend using UnityWebRequest instead as it's aware of Unity editor domain lifecycle.

 

Hi @tamano_takashi ,
Seems like the PackageManager is stuck invoking NPM in this case.
I've had a look through our internal slack conversations and found that the issue was fixed in 2020.1.0a3
The issue was: " ... reading from stdout and then everything from stderr will result in deadlocks in certain cases."

The error may have been triggered if there were errors while downloading packages, so perhaps try and make a new project and add the packages in your current project and check one by one which of them generates an error when being downloaded from NPM. That could be one way to work around this problem.

Also, Unity 2018 already reached its end of life support around 2021, so it might also be a good idea to consider upgrading

 

Hey @unity_52F8E92C14ACB015AF43
This appears to be a hang inside of Mono.
It may be possible to do the same thing that @Tautvydas-Zilys did a few posts above and we could try to symbolicate the callstack from a process dump.
Here's the steps he mentioned:

You are also free to send me a DM, so we can try and figure this out.