Assembly Reload Freezing problem

Hi, I'm really excited about the option to skip Assembly Reload in 2019.3, however, there is one problem remaining.


Traditionally, Unity will reload all its domain assemblies and it will cause some problem using native DLLs
If you are using a native DLL with pinvoke, it cannot be unloaded.
Many frameworks are written this way because pinvoke allows the fastest access.

Unity will load it once but if you run for the second time, Unity will freeze trying to unload the native assembly.
This is causing some huge grief that we cannot use many frameworks out there.
Unity has been saying that there is nothing that they cannot do about the freeze. It's not true.

While we are addressing the Assembly Reload issue, I think it's a good time to fix the freezing problem above.

Yeah, enable on skipping Assembly Reload at "Enter to Play" will help not to reload the native assemblies, but if you compile a project, Unity will try to reload all its assemblies again and it freezes.

If the assemblies are external frameworks and we are just using it without modifications, why reload every time?

What I'm proposing is that we add a FLAG to certain assemblies so that it skips reloading.

It will enable us to use many frameworks out there that are currently not useable.

Unfortunately, I don't have Unity source code but I suppose it's not really that complicated to skip the flagged assembly when reloading.
I believe it will be a huge win if you can make this happen.

You already added a nice feature to skip reloading assemblies already. It's probably the best feature I've ever seen the past a couple of years and please help to support this feature to make it complete.

Cheers!

 

@alexeyzakharov hi, Alex, I have no one to turn to but to ask you for the help.
There is an external native library with PInvoke that I really want to use but it will cause the freeze as I mention it above. I gave up on using it until now but the skipping Assembly Reloading gives me a hope to ask Unity again.

I would really appreciate it if you can help how to skip unloading an external library by Unity.
I believe it won't take much time to implement once you figure out the proper way to flag the assembly.
Many thank in advance.

Cheers!

 

I found something interesting today.

Clicking on the assembly shows an option to "Load on startup". I verified that it allows the assembly to load at the Editor startup instead on "Play"
I believe the purpose of this checkbox is to skip reloading to prevent the freezing problem, however it still causing the freeze.


It even shows a warning that you will have to restart Unity to unload. It's aware that it's a native and it won't unload. But why is it trying to unload during Domain Reload??

Since we already have the "flag", we just need to fix it.
To me it's more like a bug than a feature, therefore I reported a bug for better tracking.

(Case 1190509) Unity Freezes during Domain Assembly Reload

Can someone from Unity please follow up as it's causing a roadblock in my project?

Thank you very much for the help.

 

@alexeyzakharov provided a nice overview of assembly reloading in his
Entering PlayMode faster without domain and scene reloads document under "Domain Reload" section.

What happens with regards to native .dlls that are used from managed .dlls, is that Mono loads them on first access and never unloads them again. This is why you have to restart Unity if you are working on a native .dll and want to test a newer version.

We only reload managed .dlls in the editor. We cannot only reload some of them because what really happens is that we unload all managed .dlls in the current .NET AppDomain by destroying the current .NET AppDomain and then create a new .NET AppDomain and load all managed .dlls into the new .NET AppDomain. If we were to skip loading of any managed .dlls, you would have an incomplete collection of assemblies. .NET AppDomain does not support unloading of individual managed .dlls.

Also, Mono lazily loads any managed .dlls it can find from references of already loaded managed .dlls. It does so by looking in search paths that we provide and we provide search paths for all managed .dlls in the Assets folder and in Library/ScriptAssemblies (compiled assemblies).

Even if we were to skip loading an assembly (which we can't), Mono would just load it on first access, similar to how it does with native .dlls.

The reason why you could be seeing freezes with native .dlls in Unity is because they do not take assembly reloading into account, as it is not a commonly used feature in .NET and we have a very specific usage of it.

One of the specific reasons why native .dlls can freeze is because when the .NET AppDomain is unloaded/destroyed, it has to stop all managed threads in the domain. As also documented in AppDomain.Unload

"The dedicated thread attempts to unload the domain, and all threads in the domain are aborted. If a thread does not abort, for example because it is executing unmanaged code, or because it is executing a finally block, then after a period of time a CannotUnloadAppDomainException is thrown in the thread that originally called Unload. If the thread that could not be aborted eventually ends, the target domain is not unloaded. Thus, in the .NET Framework version 2.0 domain is not guaranteed to unload, because it might not be possible to terminate executing threads."

One key difference between this description and how Mono is implemented, is that it does not have a timeout for unloading the domain and never throws CannotUnloadAppDomainException.

If you have a managed thread in the domain that is calling into native (unmanaged) code and Mono then tries to stop the thread on domain unload (which happens with Asynchronous Procedure Calls (APC) on Windows) and fails, then Mono will just get stuck in an infinite loop wait for the thread to stop. The reason is that the managed/native .dlls do not take into account that a thread abort could happen at any time. You also have to be very diligent when dealing with it in the native .dll (handle APC on Windows correctly) and ensure that everything is reset correctly when the managed .dll accesses the native .dll in the new AppDomain.

TL;DR: We cannot just not reload some managed assemblies, native assemblies are never reloaded and authors of third party libraries that use native .dlls have to add explicit support .NET AppDomain unloads/loads to make them not freeze in Unity.

 

@lukaszunity
Hi, thanks for the detailed explanations. I think I understand how AppDomain works and how it manages its dlls. The freezing problem has been an issue for a very long time. I've been saving it until Unity is working on Assembly Reload and I think it's a good time to talk about it.

In this case, the native dll does not change and I don't want to test "a newer version", instead, I want Unity to skips unloading the dll to avoid the freeze.

If the managed dll calls the native dll through PInvoke, and later the managed dll is trying to unload itself, it also tries to unload the native dll of which cannot be unloaded. Just to clarify, I'm not trying to selectively skip unloading any managed dll but to skip the native dll unloading referenced by a managed dll.

I think the problem is that Unity is designed a way that it reloads all its dlls through AppDomain.
If you cannot change how AppDomain works, perhaps the right answer is to manage dlls yourself instead of relying on AppDomain.

Honestly, I think AppDomain.Reload is lazy way to reset the state. We should move away from AppDomain.Reload in the future and manage reloading modified assemblies by Unity individually. This will save huge time not reloading many unchanged assemblies.

Unity has been ignoring this freeze problem until now, but I think it's simply not acceptable anymore. Think about how the opportunities are lost because we can't use so many libraries out there? and the time we all have to wait until for the AppDomain reload?

If we cannot move away from the AppDomain.Reload right away, the workaround is to modify the mono.
I'm sure you have full access to Mono and you can make Unity.AppDomain.Unload("FreezeFreeDomain") and save us from the freezing hell until Unity can completely managing the assemblie.

Cheers!

 

Here is a side note about moving away from AppDomain.Reload

We are recommended to use AsmDef to reduce the compile time. At first, I thought it was a really good idea and many developers started using them including myself. If you are using many assets but you are changing your own code, you can save a huge amount of compile time if you put your code into your own AsmDef. It makes total sense.

However, the majority of time spent is not the compiling time but AppDomain.Reload. As you said, it tries to unload all its assembly and load them again every single time. This includes everything, Unity assemblies, 3rd party assemblies, and your own assemblies. This is not so smart in my opinion.

I did a test if having many assemblies will save time compared to having a single Assembly-CSharp.dll. It turns out that having a single Assembly-CSharp.dll is faster. C# Compile time is already fast enough but the Domain Reload time is dominating trying to unload/load many small assemblies.

The conclusion is that, we should move away from AppDomain.Reload, period! The good news is that 2019.3 provides some options and we are a step closer.

From what I heard, sub-500ms update is still on the plan according to Joachim.
I believe it's not possible with AppDomain.Reload and I assume that you are already looking into manually managing assemblies yourself. I hope I'm right.

Thanks.

 

As I mention in my previous response. Unloading of native .dlls does not exist as a concept. The problem is not that we are trying to unload the native .dll, but that the managed/native .dll is not handling the unload event correctly. E.g. when the managed .dll is unloaded and threads are aborted, the managed .dll should notify the native .dll of this and perform the right actions. If the native .dll is interrupted, then it should also handle it correctly.

Managed .dlls always have to be load into a domain. There always exists at least one. You can get it with AppDomain.CurrentDomain.

While it could be considered lazy, the alternative is that you have handle resetting of state yourself. Also, it simulates the behaviour that happens in the player on startup, e.g. completely reset of everything to an initial state.

We can't manage .dlls individually, because they have to be in a domain and we use the domains because it is only way you can unload managed .dlls in .NET. Also there is no access between .dlls across domain boundaries. Domains were originally developed as a security isolation mechanism.

If there is a bug, please report it. All issues we have seen with native .dlls is that the native .dlls are not compatible with AppDomain reloads.

And again, there is no other way besides AppDomains to provide a mechanism for reloading assemblies in the .NET Framework and there is no way for us to modify Mono to fix this and be compatible with .NET. If we tell user code that we are about to unload and the user code does not comply with this, then there is nothing we can do. This is similar to if you use a third party C# library and there is an infinite loop/recursion bug in a method, there is nothing you can do about in your code, the issue has to be fixed in the third party library.

 

We are looking into other options than AppDomain reload for reloading code after recompiling assemblies. Such as using AssemblyLoadContext from .NET Core in Mono, which allows for reloading of individual assemblies. But this is a longer term plan and there are many details around how this would work that are currently unclear. Stay tuned for updates in the future ;-)

 

That's good to know but that's sounds like a really a long term project.

Anyway, the freezing happens if you P/Invoke into the native libary. No matter what, the native dll will not unload.

 

Wow, I thought AssemblyLoadContext was .NetCore v3 feature but it's been there since v1!

I'm really mad why it wasn't taken advantage of earlier.

If I were Unity, I'll put the highest priority on removing AppDomain reloading.
It's not only causing problems with the freeze but it just doesn't make sense to reload the whole assemblies if just a single line changes in any assembly.

My project is suffering from very long AppDomain reloading.. It currently takes 15+ seconds and it's so painful to work every day. I've been waiting for optional AppDomain reloading for "Enter to Play" but it won't help for the compilation.

Again, If I were Unity and I know if there is a way to save time tremendously, I'll do anything to save the precious time. Not just for some users but for the sake of the whole Unity users. It can add up to months and years of wasted time quite easily.

Anyway, instead of putting this into the research project, Unity should really sitdown and do something about it right now.
Joachim already tried told us sub 500ms compilation(but failed) and he mentioned it's still on the list while we are continue to suffer from the lazy AppDomain reloading.

I'm not sure what project Unity has on its plate but I can argue removing AppDomain reloading is much more important than the many projects. Everyone can benefit immediately unlike many project that you have to adopt.

I'll wait for the good news. If there is a better person to talk to, please PM me. I'm determined to see this happen.

Thanks.

 

We use .NET Framework in Unity and it does not support AssemblyLoadContext. We have to make changes to the internals of Mono to make it work in a .NET Framework scenario. So AssemblyLoadContext wasn't available to us and we couldn't use it.

I suggest using the profiler in Unity and enabling editor profiling and then finding the frame where we reload the assemblies and dig into profiling data. One of the main causes for slow reloads is slow user C# code that we call into during reload. E.g. MonoBehaviour.OnDisable/OnEnable/Awake, InitializeOnLoad, etc.

We seen third party plugins slow down projects by 30+ seconds because they perform operations that do not scale with a large project.

 

here are some facts:

reload time on empty project (disable all built-in packages) was 2~3s , entering play mode was about 0s
reload time on HDRP template project (create by UnityHub) was ~10s, entering play mode was about 2~3s

in a foreseeable future, more and more Unity's package would be the source of slow down problem, doesn't expect sub-500ms, sub-5000ms for real project would be fine, hope the "longer term plan" wouldn't be too long

 

@lukaszunity

Do these problems go away if using IL2CPP instead of mono?

 

No. The editor uses Mono always, regardless whether the Scripting Backend is set to IL2CPP.

 

Hi. I'm facing a similar issue: My native C library has its own thread with an event loop, which calls into C# through a registered callback. But it seems this causes the C# to register the thread and wait forever for it to stop on unloading.
For reasons I cannot delete and recreate this thread between runs, it has to persist, so my question is: Is there any way to unregister the thread so that unloading won't wait for it?

 

Generally mono should terminate c# threads on domain unload, the problem might be when a thread is created during domain unload. Could you check/ensure that this is not the case?

non-trivial and platform-dependent I can think of would be to use e.g. EasyHook or similar libraries to hijack the system's (e.g. CreateThread) thread creation function and record thread ids of threads which were created from your library, then on domain reload you can terminate those forcibly.

 

this happen to me today suddenly, its so annoying where you wait fews seconds, where you have this keep saving of codes this never happen before

 

Are you sure the problem is not just domain reload itself? I find the issue scales almost completely with the filecount in my project, not class files, but total files period. Disabling the "Domain Reload" option when entering Play, always seems to shave off roughly the same amount of time as my "reload" times on script compile... these 2 times seem extremely tightly coupled.

When I try and log assembly compile times, it's never more than a couple of seconds. Where is the other 8s+ going? Sure seems like it's general parsing of the project files... Every. Single. Time. We. Save. A. File.

If there was some way that we could mark a folder as "manual domain reload", we could stick our 6000+ plugin files there? Then, the few hundred or so actual project files could scale and continue to be extremely fast for iteration.

This would be a complete game-changer. Currently, the minute your project reaches and production scale, your scripting reload times are 10seconds plus, and your ability to quickly iterate goes out the window.

Could that be possible? A domain reload that had the ability to ignore specific folders, and do either a Full or Partial reload? This would be an absolutely massive productivity booster.

@alexeyzakharov @lukaszunity any thoughts on this?

 

Happens once a few days.

 

Still there in 2020.3 lts

 

I'm there too, 8min so far

 

Same issue here, and it happens to me at least a couple of times per day. Too bad it doesn't tell you specifically what script native code dll it is stuck on after some duration of waiting.

 

You could try deleting the 'Library' folder from the project in the file explorer. After that it will rebuild the Library anyway and possibly solve the issue. Worked for me

 

2020.3.16f1c1, happen again in today....

in 2019 version, I never see the hold error, what happen?

 

Hello, I am finding similar issue, stuck forever showing "importing".

This is the point until editor keeps login (after me closing the application via task manager):

Spoiler

...
Platform modules already initialized, skipping
Validating Project structure ... 0.019175 seconds.
Shader import version has changed; will reimport all shaders...
Upgrading shader files ...0.322636 seconds.
Application.AssetDatabase Initial Script Refresh Start
Registering precompiled user dll's ...
Registered in 0.656992 seconds.
AssetDatabase: script compilation time: 0.038084s
Begin MonoManager ReloadAssembly
Native extension for WindowsStandalone target not found
Native extension for Android target not found
Refreshing native plugins compatible for Editor in 51.66 ms, found 3 plugins.
Preloading 0 native plugins for Editor in 0.00 ms.
--My editor log is written until here when ocasionally Unity gets stuck launching my project and showing "Importing" on screen indefinitely.


From previous log that worked properly, the log continue after the line above with:

Mono: successfully reloaded assembly
- Completed reload, in 4.934 seconds
Domain Reload Profiling:
ReloadAssembly (4970ms)
BeginReloadAssembly (107ms)
ExecutionOrderSort (0ms)
DisableScriptedObjects (3ms)
BackupInstance (0ms)
ReleaseScriptingObjects (0ms)
CreateAndSetChildDomain (14ms)
EndReloadAssembly (4820ms)
LoadAssemblies (2586ms)
RebuildTransferFunctionScriptingTraits (0ms)
SetupTypeCache (605ms)
ReleaseScriptCaches (1ms)
RebuildScriptCaches (55ms)
SetupLoadedEditorAssemblies (1444ms)
LogAssemblyErrors (0ms)
InitializePlatformSupportModulesInManaged (4ms)
SetLoadedEditorAssemblies (0ms)
RefreshPlugins (50ms)
BeforeProcessingInitializeOnLoad (79ms)
ProcessInitializeOnLoadAttributes (1178ms)
ProcessInitializeOnLoadMethodAttributes (131ms)
AfterProcessingInitializeOnLoad (2ms)
EditorAssembliesLoaded (0ms)
ExecutionOrderSort2 (0ms)
AwakeInstancesAfterBackupRestoration (5ms)
Platform modules already initialized, skipping
...

 

The issue just continues randomly on each pc restart. It just get stuck forever on Importing.

 

Are you using git or something? We also had that problem and in git, we discarded the project settings and then it works fine again

 

Using 2020.3.22f1 (LTS), this happens frequently. Perhaps once an hour. More often then not its when I've edited some code and return to Unity to have it compile. It also happens about once every 3 or 4 times I load the project. It sort of makes the task of working in Unity an annoying chore. I'm constantly killing it and restarting.

Other than Unity Assets from the store, I don't have anything else fancy going on.

 

Yes, I am using Git. Thanks for pointing out, I will check

 

Still happening on the 2022.1.0b7 builds.

These modals really need to be cancelable.

 

Here's a full log of the last crash.

 

No updates about this issue?

 

Been searching all over for a solution to this problem...

It happens many times a day... sometimes without even opening Visual Studio or editing any line of code... just creating a new scene, moving a few prefabs around, not even adding any scripts to the inspector... then trying to play-test... it just stalled at "reload script assemblies" for 20+ mins until i gave up and forced quit unity...

IF i have Visual Studios opened, i can simply add/remove a space to any file, save over it, and it fixes the issue... but its more of a frustrating work-around than an actual fix...
we are working on 2021.2.11.f1 but to get an actual solution, i'd be willing to push for an upgrade

 

Same issue with 2021.3 and Visual Studio 2022.
Bug is more and more frequent...
Same remark as RireHawkX

 

It seems more stable on 2021.3.1

 

Same issue on Unity 2020.3.33f1, frequently. It started when we updated from 2020.3.20f1, so the regression is somewhere between those versions.

 

By modifying the code under VS and recompiling I get, some times, the error message:
"An abnormal situation has occurred: the PlayerLoop internal function has been called recursively."
Most of the time it is enough to relaunch Unity to return to normal.
Very difficult to isolate the issue. Some projects are concerned and not others !
I confirm what Ludiq says: it seems that issue appeared with the 2020 version.

 

Don't know if this will be of use to anyone, but when I was getting frozen on reload script assemblies, I did a LOT of guess-and-checking, and found that deleting the UserSettings/Layouts/default-2021.dwlt in my project's folder solved the problem. I have no idea what that file is for or if I'm doing something bad by deleting it, but given that it takes my Unity from useless to functional, I'm okay with it.

 

Thank you racetrack !
Issue is clearly related to this file. For me it happened again once or twice, and it has been stable since....

 

Thanks a lot! It worked for me too! Sometimes it happens again after I restart Unity, however I just delete the file again and it works normally =)

(Edit: It only works sometimes)

 

I found another super weird work-around...

If i press play and it freezes on "reloading assembly", I now simply open up explorer.exe (my computer), then navigate to the Assets folder of the opened project... before I am done, unity unfreezes and keeps going... I have tried letting it stalled for a few mins, then opened windows explorer and before I finished opening my project\assets folder it unfroze! No need to even edit or delete anything, simply opening the folder location fixes it!! (and I confirmed with 3 other team members and it worked for them as well!)

 

Works for me aswell. Thank you.

 

Sorry for the res but I'm encountering this with latest 2021.3.29f1 (and previous version, Mac).

The Mac Icon has the 'white circle' as per start of compiling but no progress. Touching the file system under Assets (e.g. create new folder or deleting an empty one) seems to unlock the Reload Script Assemblies/Compiling hang.

Did any one find a more satisfactory solution that this manual hack which is kinda crazy?? Compile and reload times are otherwise no problem. This is killing productivity.... is there a bug report? Searching for this issue gets tangled up in many other different issues related to compile speed, but that isn't the problem here. Also it 'seems' to be a bit flaky as to when it happens...but once it starts it seems to continue to be a problem. The add folder/remove folder appears to be a reliable way of unblocking the process.

I'm using Unity Version Control which might be interfering with file system? Asset Pipeline auto refresh is off, as well as Reload Domain.

EDIT:

I found a more recent thread and removing PlasticSCM package seems (at the moment!!), to have removed the compile hang. The touching the file system fix did seem to point strongly to the VCS.

https://forum.unity.com/threads/con...-crashing-hanging.1057568/page-3#post-8693079

 

Same here. 2021.3.11 LTS, Windows.
Deleting UserSettings/Layouts/default-2021.dwlt not working