Search Unity

Assembly Reload Freezing problem

Discussion in '2019.3 Beta' started by chrisk, Oct 8, 2019.

  1. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    699
    Hi, I'm really excited about the option to skip Assembly Reload in 2019.3, however, there is one problem remaining.


    Traditionally, Unity will reload all its domain assemblies and it will cause some problem using native DLLs
    If you are using a native DLL with pinvoke, it cannot be unloaded.
    Many frameworks are written this way because pinvoke allows the fastest access.

    Unity will load it once but if you run for the second time, Unity will freeze trying to unload the native assembly.
    This is causing some huge grief that we cannot use many frameworks out there.
    Unity has been saying that there is nothing that they cannot do about the freeze. It's not true.

    While we are addressing the Assembly Reload issue, I think it's a good time to fix the freezing problem above.

    Yeah, enable on skipping Assembly Reload at "Enter to Play" will help not to reload the native assemblies, but if you compile a project, Unity will try to reload all its assemblies again and it freezes.

    If the assemblies are external frameworks and we are just using it without modifications, why reload every time?

    What I'm proposing is that we add a FLAG to certain assemblies so that it skips reloading.

    It will enable us to use many frameworks out there that are currently not useable.

    Unfortunately, I don't have Unity source code but I suppose it's not really that complicated to skip the flagged assembly when reloading.
    I believe it will be a huge win if you can make this happen.

    You already added a nice feature to skip reloading assemblies already. It's probably the best feature I've ever seen the past a couple of years and please help to support this feature to make it complete.

    Cheers!
     
    Romenics and DrummerB like this.
  2. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    699
    @alexeyzakharov hi, Alex, I have no one to turn to but to ask you for the help.
    There is an external native library with PInvoke that I really want to use but it will cause the freeze as I mention it above. I gave up on using it until now but the skipping Assembly Reloading gives me a hope to ask Unity again.

    I would really appreciate it if you can help how to skip unloading an external library by Unity.
    I believe it won't take much time to implement once you figure out the proper way to flag the assembly.
    Many thank in advance.

    Cheers!
     
  3. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    699
    I found something interesting today.

    Clicking on the assembly shows an option to "Load on startup". I verified that it allows the assembly to load at the Editor startup instead on "Play"
    I believe the purpose of this checkbox is to skip reloading to prevent the freezing problem, however it still causing the freeze.
    upload_2019-10-10_23-32-27.png

    It even shows a warning that you will have to restart Unity to unload. It's aware that it's a native and it won't unload. But why is it trying to unload during Domain Reload??

    Since we already have the "flag", we just need to fix it.
    To me it's more like a bug than a feature, therefore I reported a bug for better tracking.

    (Case 1190509) Unity Freezes during Domain Assembly Reload

    Can someone from Unity please follow up as it's causing a roadblock in my project?

    Thank you very much for the help.
     

    Attached Files:

  4. lukaszunity

    lukaszunity

    Unity Technologies

    Joined:
    Jun 11, 2014
    Posts:
    461
    @alexeyzakharov provided a nice overview of assembly reloading in his
    Entering PlayMode faster without domain and scene reloads document under "Domain Reload" section.

    What happens with regards to native .dlls that are used from managed .dlls, is that Mono loads them on first access and never unloads them again. This is why you have to restart Unity if you are working on a native .dll and want to test a newer version.

    We only reload managed .dlls in the editor. We cannot only reload some of them because what really happens is that we unload all managed .dlls in the current .NET AppDomain by destroying the current .NET AppDomain and then create a new .NET AppDomain and load all managed .dlls into the new .NET AppDomain. If we were to skip loading of any managed .dlls, you would have an incomplete collection of assemblies. .NET AppDomain does not support unloading of individual managed .dlls.

    Also, Mono lazily loads any managed .dlls it can find from references of already loaded managed .dlls. It does so by looking in search paths that we provide and we provide search paths for all managed .dlls in the Assets folder and in Library/ScriptAssemblies (compiled assemblies).

    Even if we were to skip loading an assembly (which we can't), Mono would just load it on first access, similar to how it does with native .dlls.

    The reason why you could be seeing freezes with native .dlls in Unity is because they do not take assembly reloading into account, as it is not a commonly used feature in .NET and we have a very specific usage of it.

    One of the specific reasons why native .dlls can freeze is because when the .NET AppDomain is unloaded/destroyed, it has to stop all managed threads in the domain. As also documented in AppDomain.Unload

    "The dedicated thread attempts to unload the domain, and all threads in the domain are aborted. If a thread does not abort, for example because it is executing unmanaged code, or because it is executing a finally block, then after a period of time a CannotUnloadAppDomainException is thrown in the thread that originally called Unload. If the thread that could not be aborted eventually ends, the target domain is not unloaded. Thus, in the .NET Framework version 2.0 domain is not guaranteed to unload, because it might not be possible to terminate executing threads."

    One key difference between this description and how Mono is implemented, is that it does not have a timeout for unloading the domain and never throws CannotUnloadAppDomainException.

    If you have a managed thread in the domain that is calling into native (unmanaged) code and Mono then tries to stop the thread on domain unload (which happens with Asynchronous Procedure Calls (APC) on Windows) and fails, then Mono will just get stuck in an infinite loop wait for the thread to stop. The reason is that the managed/native .dlls do not take into account that a thread abort could happen at any time. You also have to be very diligent when dealing with it in the native .dll (handle APC on Windows correctly) and ensure that everything is reset correctly when the managed .dll accesses the native .dll in the new AppDomain.

    TL;DR: We cannot just not reload some managed assemblies, native assemblies are never reloaded and authors of third party libraries that use native .dlls have to add explicit support .NET AppDomain unloads/loads to make them not freeze in Unity.
     
    alexeyzakharov likes this.
  5. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    699
    @lukaszunity
    Hi, thanks for the detailed explanations. I think I understand how AppDomain works and how it manages its dlls. The freezing problem has been an issue for a very long time. I've been saving it until Unity is working on Assembly Reload and I think it's a good time to talk about it.

    In this case, the native dll does not change and I don't want to test "a newer version", instead, I want Unity to skips unloading the dll to avoid the freeze.

    If the managed dll calls the native dll through PInvoke, and later the managed dll is trying to unload itself, it also tries to unload the native dll of which cannot be unloaded. Just to clarify, I'm not trying to selectively skip unloading any managed dll but to skip the native dll unloading referenced by a managed dll.

    I think the problem is that Unity is designed a way that it reloads all its dlls through AppDomain.
    If you cannot change how AppDomain works, perhaps the right answer is to manage dlls yourself instead of relying on AppDomain.

    Honestly, I think AppDomain.Reload is lazy way to reset the state. We should move away from AppDomain.Reload in the future and manage reloading modified assemblies by Unity individually. This will save huge time not reloading many unchanged assemblies.

    Unity has been ignoring this freeze problem until now, but I think it's simply not acceptable anymore. Think about how the opportunities are lost because we can't use so many libraries out there? and the time we all have to wait until for the AppDomain reload?

    If we cannot move away from the AppDomain.Reload right away, the workaround is to modify the mono.
    I'm sure you have full access to Mono and you can make Unity.AppDomain.Unload("FreezeFreeDomain") and save us from the freezing hell until Unity can completely managing the assemblie.

    Cheers!
     
  6. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    699
    Here is a side note about moving away from AppDomain.Reload

    We are recommended to use AsmDef to reduce the compile time. At first, I thought it was a really good idea and many developers started using them including myself. If you are using many assets but you are changing your own code, you can save a huge amount of compile time if you put your code into your own AsmDef. It makes total sense.

    However, the majority of time spent is not the compiling time but AppDomain.Reload. As you said, it tries to unload all its assembly and load them again every single time. This includes everything, Unity assemblies, 3rd party assemblies, and your own assemblies. This is not so smart in my opinion.

    I did a test if having many assemblies will save time compared to having a single Assembly-CSharp.dll. It turns out that having a single Assembly-CSharp.dll is faster. C# Compile time is already fast enough but the Domain Reload time is dominating trying to unload/load many small assemblies.

    The conclusion is that, we should move away from AppDomain.Reload, period! The good news is that 2019.3 provides some options and we are a step closer.

    From what I heard, sub-500ms update is still on the plan according to Joachim.
    I believe it's not possible with AppDomain.Reload and I assume that you are already looking into manually managing assemblies yourself. I hope I'm right.

    Thanks.
     
  7. lukaszunity

    lukaszunity

    Unity Technologies

    Joined:
    Jun 11, 2014
    Posts:
    461
    As I mention in my previous response. Unloading of native .dlls does not exist as a concept. The problem is not that we are trying to unload the native .dll, but that the managed/native .dll is not handling the unload event correctly. E.g. when the managed .dll is unloaded and threads are aborted, the managed .dll should notify the native .dll of this and perform the right actions. If the native .dll is interrupted, then it should also handle it correctly.

    Managed .dlls always have to be load into a domain. There always exists at least one. You can get it with AppDomain.CurrentDomain.

    While it could be considered lazy, the alternative is that you have handle resetting of state yourself. Also, it simulates the behaviour that happens in the player on startup, e.g. completely reset of everything to an initial state.

    We can't manage .dlls individually, because they have to be in a domain and we use the domains because it is only way you can unload managed .dlls in .NET. Also there is no access between .dlls across domain boundaries. Domains were originally developed as a security isolation mechanism.

    If there is a bug, please report it. All issues we have seen with native .dlls is that the native .dlls are not compatible with AppDomain reloads.

    And again, there is no other way besides AppDomains to provide a mechanism for reloading assemblies in the .NET Framework and there is no way for us to modify Mono to fix this and be compatible with .NET. If we tell user code that we are about to unload and the user code does not comply with this, then there is nothing we can do. This is similar to if you use a third party C# library and there is an infinite loop/recursion bug in a method, there is nothing you can do about in your code, the issue has to be fixed in the third party library.
     
  8. lukaszunity

    lukaszunity

    Unity Technologies

    Joined:
    Jun 11, 2014
    Posts:
    461
    We are looking into other options than AppDomain reload for reloading code after recompiling assemblies. Such as using AssemblyLoadContext from .NET Core in Mono, which allows for reloading of individual assemblies. But this is a longer term plan and there are many details around how this would work that are currently unclear. Stay tuned for updates in the future ;-)
     
  9. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    699
    That's good to know but that's sounds like a really a long term project.

    Anyway, the freezing happens if you P/Invoke into the native libary. No matter what, the native dll will not unload.
     
  10. chrisk

    chrisk

    Joined:
    Jan 23, 2009
    Posts:
    699
    Wow, I thought AssemblyLoadContext was .NetCore v3 feature but it's been there since v1!

    I'm really mad why it wasn't taken advantage of earlier.

    If I were Unity, I'll put the highest priority on removing AppDomain reloading.
    It's not only causing problems with the freeze but it just doesn't make sense to reload the whole assemblies if just a single line changes in any assembly.

    My project is suffering from very long AppDomain reloading.. It currently takes 15+ seconds and it's so painful to work every day. I've been waiting for optional AppDomain reloading for "Enter to Play" but it won't help for the compilation.

    Again, If I were Unity and I know if there is a way to save time tremendously, I'll do anything to save the precious time. Not just for some users but for the sake of the whole Unity users. It can add up to months and years of wasted time quite easily.

    Anyway, instead of putting this into the research project, Unity should really sitdown and do something about it right now.
    Joachim already tried told us sub 500ms compilation(but failed) and he mentioned it's still on the list while we are continue to suffer from the lazy AppDomain reloading.

    I'm not sure what project Unity has on its plate but I can argue removing AppDomain reloading is much more important than the many projects. Everyone can benefit immediately unlike many project that you have to adopt.

    I'll wait for the good news. If there is a better person to talk to, please PM me. I'm determined to see this happen.

    Thanks.
     
    jdtec and alexeyzakharov like this.
  11. lukaszunity

    lukaszunity

    Unity Technologies

    Joined:
    Jun 11, 2014
    Posts:
    461
    We use .NET Framework in Unity and it does not support AssemblyLoadContext. We have to make changes to the internals of Mono to make it work in a .NET Framework scenario. So AssemblyLoadContext wasn't available to us and we couldn't use it.

    I suggest using the profiler in Unity and enabling editor profiling and then finding the frame where we reload the assemblies and dig into profiling data. One of the main causes for slow reloads is slow user C# code that we call into during reload. E.g. MonoBehaviour.OnDisable/OnEnable/Awake, InitializeOnLoad, etc.

    We seen third party plugins slow down projects by 30+ seconds because they perform operations that do not scale with a large project.
     
  12. filod

    filod

    Joined:
    Oct 3, 2015
    Posts:
    163
    here are some facts:

    reload time on empty project (disable all built-in packages) was 2~3s , entering play mode was about 0s
    reload time on HDRP template project (create by UnityHub) was ~10s, entering play mode was about 2~3s

    in a foreseeable future, more and more Unity's package would be the source of slow down problem, doesn't expect sub-500ms, sub-5000ms for real project would be fine, hope the "longer term plan" wouldn't be too long
     
    TextusGames likes this.
  13. PudgePacket

    PudgePacket

    Joined:
    Feb 27, 2014
    Posts:
    8
    @lukaszunity

    Do these problems go away if using IL2CPP instead of mono?
     
  14. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,090
    No. The editor uses Mono always, regardless whether the Scripting Backend is set to IL2CPP.
     
  15. tidalwave

    tidalwave

    Joined:
    Oct 4, 2014
    Posts:
    1
    Hi. I'm facing a similar issue: My native C library has its own thread with an event loop, which calls into C# through a registered callback. But it seems this causes the C# to register the thread and wait forever for it to stop on unloading.
    For reasons I cannot delete and recreate this thread between runs, it has to persist, so my question is: Is there any way to unregister the thread so that unloading won't wait for it?
     
  16. alexeyzakharov

    alexeyzakharov

    Unity Technologies

    Joined:
    Jul 2, 2014
    Posts:
    424
    Generally mono should terminate c# threads on domain unload, the problem might be when a thread is created during domain unload. Could you check/ensure that this is not the case?

    non-trivial and platform-dependent I can think of would be to use e.g. EasyHook or similar libraries to hijack the system's (e.g. CreateThread) thread creation function and record thread ids of threads which were created from your library, then on domain reload you can terminate those forcibly.
     
  17. domdev

    domdev

    Joined:
    Feb 2, 2015
    Posts:
    360
    this happen to me today suddenly, its so annoying where you wait fews seconds, where you have this keep saving of codes this never happen before
     
  18. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    323
    Are you sure the problem is not just domain reload itself? I find the issue scales almost completely with the filecount in my project, not class files, but total files period. Disabling the "Domain Reload" option when entering Play, always seems to shave off roughly the same amount of time as my "reload" times on script compile... these 2 times seem extremely tightly coupled.

    When I try and log assembly compile times, it's never more than a couple of seconds. Where is the other 8s+ going? Sure seems like it's general parsing of the project files... Every. Single. Time. We. Save. A. File.

    If there was some way that we could mark a folder as "manual domain reload", we could stick our 6000+ plugin files there? Then, the few hundred or so actual project files could scale and continue to be extremely fast for iteration.

    This would be a complete game-changer. Currently, the minute your project reaches and production scale, your scripting reload times are 10seconds plus, and your ability to quickly iterate goes out the window.

    Could that be possible? A domain reload that had the ability to ignore specific folders, and do either a Full or Partial reload? This would be an absolutely massive productivity booster.

    @alexeyzakharov @lukaszunity any thoughts on this?
     
    Last edited: Sep 28, 2020
  19. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    725


    Happens once a few days.
     
    Babiole and Anisoft like this.
  20. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    725
    Still there in 2020.3 lts

     
  21. GilJoWal

    GilJoWal

    Joined:
    Mar 28, 2020
    Posts:
    15
    I'm there too, 8min so far
     
  22. Tymac

    Tymac

    Joined:
    Nov 12, 2015
    Posts:
    3
    Same issue here, and it happens to me at least a couple of times per day. Too bad it doesn't tell you specifically what script native code dll it is stuck on after some duration of waiting.
     
  23. brollsroyce

    brollsroyce

    Joined:
    Jun 7, 2021
    Posts:
    1
    You could try deleting the 'Library' folder from the project in the file explorer. After that it will rebuild the Library anyway and possibly solve the issue. Worked for me
     
  24. UioSun

    UioSun

    Joined:
    Mar 22, 2020
    Posts:
    1
    20210901152522.jpg

    2020.3.16f1c1, happen again in today....

    in 2019 version, I never see the hold error, what happen?
     
  25. Jorhoto

    Jorhoto

    Joined:
    May 7, 2019
    Posts:
    98
    Hello, I am finding similar issue, stuck forever showing "importing".

    This is the point until editor keeps login (after me closing the application via task manager):
    ...
    Platform modules already initialized, skipping
    Validating Project structure ... 0.019175 seconds.
    Shader import version has changed; will reimport all shaders...
    Upgrading shader files ...0.322636 seconds.
    Application.AssetDatabase Initial Script Refresh Start
    Registering precompiled user dll's ...
    Registered in 0.656992 seconds.
    AssetDatabase: script compilation time: 0.038084s
    Begin MonoManager ReloadAssembly
    Native extension for WindowsStandalone target not found
    Native extension for Android target not found
    Refreshing native plugins compatible for Editor in 51.66 ms, found 3 plugins.
    Preloading 0 native plugins for Editor in 0.00 ms.
    --My editor log is written until here when ocasionally Unity gets stuck launching my project and showing "Importing" on screen indefinitely.


    From previous log that worked properly, the log continue after the line above with:

    Mono: successfully reloaded assembly
    - Completed reload, in 4.934 seconds
    Domain Reload Profiling:
    ReloadAssembly (4970ms)
    BeginReloadAssembly (107ms)
    ExecutionOrderSort (0ms)
    DisableScriptedObjects (3ms)
    BackupInstance (0ms)
    ReleaseScriptingObjects (0ms)
    CreateAndSetChildDomain (14ms)
    EndReloadAssembly (4820ms)
    LoadAssemblies (2586ms)
    RebuildTransferFunctionScriptingTraits (0ms)
    SetupTypeCache (605ms)
    ReleaseScriptCaches (1ms)
    RebuildScriptCaches (55ms)
    SetupLoadedEditorAssemblies (1444ms)
    LogAssemblyErrors (0ms)
    InitializePlatformSupportModulesInManaged (4ms)
    SetLoadedEditorAssemblies (0ms)
    RefreshPlugins (50ms)
    BeforeProcessingInitializeOnLoad (79ms)
    ProcessInitializeOnLoadAttributes (1178ms)
    ProcessInitializeOnLoadMethodAttributes (131ms)
    AfterProcessingInitializeOnLoad (2ms)
    EditorAssembliesLoaded (0ms)
    ExecutionOrderSort2 (0ms)
    AwakeInstancesAfterBackupRestoration (5ms)
    Platform modules already initialized, skipping
    ...
     
  26. Jorhoto

    Jorhoto

    Joined:
    May 7, 2019
    Posts:
    98
    The issue just continues randomly on each pc restart. It just get stuck forever on Importing.
     

    Attached Files:

  27. DriesVrBase

    DriesVrBase

    Joined:
    Mar 24, 2020
    Posts:
    40
    Are you using git or something? We also had that problem and in git, we discarded the project settings and then it works fine again
     
  28. Mantic

    Mantic

    Joined:
    Jan 5, 2013
    Posts:
    3
    Using 2020.3.22f1 (LTS), this happens frequently. Perhaps once an hour. More often then not its when I've edited some code and return to Unity to have it compile. It also happens about once every 3 or 4 times I load the project. It sort of makes the task of working in Unity an annoying chore. I'm constantly killing it and restarting.

    Other than Unity Assets from the store, I don't have anything else fancy going on.
     
  29. Jorhoto

    Jorhoto

    Joined:
    May 7, 2019
    Posts:
    98
    Yes, I am using Git. Thanks for pointing out, I will check ;)
     
unityunity