Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice

Why is "Domain Reload" sooo slow, and will this ever be fixed?

Discussion in 'Editor & General Support' started by shawnblais, Jun 30, 2020.

  1. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    We're currently on a project, where a simple script change locks up the editor for 12-14s.

    When we analyze compilation times it's 2.18s.

    Where are the other 12 seconds going?? Does this ENTIRE thing need to happen every-time a script is changing?? It's such a massive productivity killer. Compare this to a 1000 class app in Flutter, where script-reload time is ~200ms w/ Live Reload, this hurts to watch.

    Unity recently added this:


    This removes the "domain reload" when entering Play Mode, and it seems to shave off exactly about 10-12s in our project.

    This leads me to believe that all of the "re-compile" time is actually this "Domain Reload", and this is the real root of the poor iteration times and lack of scalability in Unity.

    My question is, will this ever be addressed? Is it even possible to address? Is Unity planning to get at the heart of the issue, which is speeding up Domain Reload, or are they just going to continue with band-aids like allowing us to turn it off. Curious what the roadmap is here...

    Seems like some intelligent system that only updates things that have changed is what is really needed here? Currently you change 1 file, and 5000 are reloaded into domain, that can't _really_ be the desired behavior here.
     
    Last edited: Jun 30, 2020
    lacas8282, RSH1, tonytopper and 10 others like this.
  2. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,697
    It won't happen any time soon, it at all. They were talking about having a 500ms goal for compilation a few years ago. Have you heard it again recently?
     
  3. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    Bump. Can one of the engineers chime in here, is this something that is even on the radar?
     
    tonytopper likes this.
  4. jamespaterson

    jamespaterson

    Joined:
    Jun 19, 2018
    Posts:
    398
    Hi. There were a number of threads about this and related issues. There were a number of people who documented the things they tried to do to improve domain reload time. Some things that helped me gain back a few seconds, of course your mileage will vary:

    Close all unity editor sub-windows that are not needed.
    Reduce complexity of the loaded scene
     
    lacas8282 and shawnblais like this.
  5. jamespaterson

    jamespaterson

    Joined:
    Jun 19, 2018
    Posts:
    398
  6. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    Thanks James. I've replied in the linked thread with some thoughts. Great to know that some form of script patching might be on the horizon (however distant)
     
    jamespaterson likes this.
  7. KarambitWithAK

    KarambitWithAK

    Joined:
    Jun 1, 2023
    Posts:
    1
    I recently started Learning Unity the the flappy bird video by GMTK youtube channel, and i am sad to say that this issue still persists as of June 2023, I only had a basic project so my "domain reloading" time wasnt as bad but it sure is a flow/productivity killer for me (8 sec - 3 Sec) even tho i have a pretty good gaming laptop (rtx3050, Ryzen 5 5600H, 16GB ram)
     
    tonytopper likes this.
  8. SpicyCatGames

    SpicyCatGames

    Joined:
    Sep 30, 2019
    Posts:
    14
    I used to run upto unity 2019 on a potato PC with a 55nm pentium cpu, 4GB ram and no SSD. This issue wasn't there. Now it's slow as heck even on much better computers.
     
  9. jRocket

    jRocket

    Joined:
    Jul 12, 2012
    Posts:
    700
    That's nothing. I've seen domain reload take minutes.

    upload_2023-7-7_16-0-0.png
     
    zhuchun likes this.
  10. Ryiah

    Ryiah

    Joined:
    Oct 11, 2012
    Posts:
    21,095
  11. halley

    halley

    Joined:
    Aug 26, 2013
    Posts:
    2,410
    I tried 2022.3 LTS, saw that my iteration time doubled over 2021.3 LTS, and decided not to move forward.
     
    Unifikation, Ruslank100 and Neiist like this.
  12. CodeRonnie

    CodeRonnie

    Joined:
    Oct 2, 2015
    Posts:
    529
    You can also disable Auto Refresh. I imagine you could make sure it refreshes when you press play using an editor script, if it doesn't already do that automatically. However that won't apply to changes made to editor scripts. For those you have to manually refresh. I may try that on Monday.

    https://docs.unity3d.com/2021.1/Documentation/Manual/Preferences.html#General

    I wonder if you can ensure it will refresh all of your other assets normally, and only manually refresh scripts.
     
  13. samualpaquin

    samualpaquin

    Joined:
    Mar 19, 2021
    Posts:
    1
    upload_2023-7-9_21-4-55.png

    Mine is still reloading...

     
  14. Neiist

    Neiist

    Joined:
    Sep 18, 2012
    Posts:
    31
    Just updated a package (Odin Inspector), took more than 10 minutes between domain reloads, import iterations, compilations, etc. then that package generated stuff and it all started again. In the end it took about 17 minutes x) I wonder if some third party packages can make the domain reload time much worse.

    Taking some time to update a package I can understand, but compilation + domain reload now takes about 1 minute for me.

    Many years ago it was one of the factors that pushed me to stop working on a big project and I stopped using unity for several years, because I spent my life looking at a progress bar. Nowadays I find myself holding back from testing what I just coded because I know I ll lose several minutes of my life... so many times a day already.

    I may be wrong here but disabling the domain reload seems to be just a fake/half solution, who knows what third party packages are doing in regards to it being turned off, and what problems it might silently cause in general. It only seems to exist to be able to say "you can disable it, you can't complain now can you?". If it's optional then how can this become a standard? if i'm missing something feel free to enlighten me.
     
    Last edited: Jul 11, 2023
    tonytopper likes this.
  15. CodeRonnie

    CodeRonnie

    Joined:
    Oct 2, 2015
    Posts:
    529
    I wouldn't call it a fake solution. It can reduce the number of domain reloads you have to endure, under the right conditions. That's not nothing, and it's helpful to point out if it might provide a benefit to anyone. However, calling it a partial solution is valid because you will have to reload the domain eventually. If those domain reloads are taking an excessively long amount of time I think you have every right to be displeased. Just because someone points out a way to reduce the number of domain reloads doesn't mean that the entire issue of how long each reload takes is null and void.

    As far as whether or not disabling domain reload on entering play mode is compatible with 3rd party plugins, or whether it will ever become the standard, that's also a fair concern. It has been discussed at length in other threads that are specifically about enter play mode options. The devs would love for the feature to be less experimental and more widely adopted, but they also acknowledge those hurdles that stand in the way.
     
    Neiist likes this.
  16. CodeRonnie

    CodeRonnie

    Joined:
    Oct 2, 2015
    Posts:
    529
    I just realized that the Auto Refresh option is not in 2021.3, and as I go through the release notes for 2021 I'm noticing there were a lot of changes to asset database refresh and domain reloads. I'll compile what I can scrape from the release notes.

    EDIT: The Auto Refresh setting was moved from General to Asset Pipeline in Preferences!
     
    Last edited: Jul 10, 2023
  17. jRocket

    jRocket

    Joined:
    Jul 12, 2012
    Posts:
    700
    Slow domain reload has always been an issue in the last few years, but I think it getting stuck or taking several minutes or longer to complete is a new unrelated bug. I'm on 2022.3.2
     
  18. CodeRonnie

    CodeRonnie

    Joined:
    Oct 2, 2015
    Posts:
    529
    EDIT: The Auto Refresh setting was moved from General to Asset Pipeline in Preferences!

    Here are some relevant release notes I pulled for version 2021. What's odd is that I don't see explicitly where "Auto Refresh" was removed. However, I'm on 2021.3.26 now and I don't see the option. It looks like it was removed from the manual for 2021.2. From what I gather, there were conflicts between Auto Refresh and the Script Changes While Playing hot reload options There are some hints in these notes however about possible alternate methods of controlling when a domain reload occurs, but it's not very clear. Reading through the release notes also reminded me of having seen a way to profile your domain reloads to know exactly what is causing them to be slow. I'll see if I can find more info on that again.

    2021.1.0
    Known Issues
    - Scripting: DomainReloadTests performance tests have regressed due to removal of built-in support for Visual Studio as a code editor (1336648)
    - Scripting: Increased Script Assembly reload time (1323490) https://issuetracker.unity3d.com/issues/increased-reload-time

    RESOLUTION NOTE (2021.2.X):
    Our teams are actively working at resolving this issue, but because this issue is requiring more work than a single bug fix, we are closing this issue and we will give further updates on this forum thread https://forum.unity.com/threads/improving-iteration-time-on-c-script-changes.1184446/
    Thank you for your feedback, patience and understanding.

    Recent post from that thread: https://forum.unity.com/threads/imp...c-script-changes.1184446/page-13#post-9131365
    Hi there! Since I'm the person who closed it, I wanted to chime in on the discussion around this bug:
    https://issuetracker.unity3d.com/is...-script-in-editor-became-really-slow-in-trunk
    We did an investigation for this particular bug and found several issues that have been filed internally with other teams (for example - a source control plugin issue, an issue with Mono performance degrading during an Editor session, and issues with the number of packages and source files included by default with more recent versions of Unity). I've then closed this particular bug because the changes necessary to completely fix it are too big for a single bug-fixing task.
    All that said, I'd like to emphasize that we are continuing to work on improving iteration time (and this includes ongoing work around CoreCLR). Thank you for being patient with us while we work on it!

    Features
    - Asset Import: Reduced cost of domain reloads on asset import worker process by removing unnecessary additional domain reloads.
    Improvements
    - Scripting: Script compilation now has a background task progress bar when auto-refresh is disbled.
    API Changes
    - Scripting: Removed: Previously undocumented EditorUtility.CompileCSharp() is removed. Its functionality can be achieved through the supported AssemblyBuilder class.
    Fixes
    - Package Manager: Fixed an issue where the Package Manager wasn't applying changes in the project manifest when autorefresh was disabled. (1292471)

    2021.1.6
    Fixes
    - Asset Pipeline: Objects loaded during import in initial script refresh, are now unloaded. (1296506)

    2021.1.9
    Fixes
    - Scripting: When environment variable UNITY_DIAG_ENABLE_DOMAIN_RELOAD_TIMINGS is set, write detailed timings on everything happening within domain reloads to the editor logs. (1328820)

    2021.1.13
    Fixes
    - Asset Pipeline: Fixed for loaded native assets that could get unnecessarily reloaded after a domain reload. (1323425)


    2021.2.0
    Improvements
    - Asset Pipeline: Added summary in the editor log about what happened during a refresh (import).
    Fixes
    - Asset Pipeline: All domain reloads are now done inside asset db. This fixes problem with reloading of asset objects when doing manual refresh. (1341910)
    - Asset Pipeline: Fixed issue with asset reference getting lost, if asset is modified and domain reload is done in the same refresh. (1357812)
    - Asset Pipeline: Fixed issue with missing domain reload when entering play mode and LockReloadAssemblies is set. (1367222)
    - Asset Pipeline: New version of OnPostProcessAllAssets with didDomainReload parameter added. This callback should be used for domain reload related initialization that requires asset operations such as asset loading. InitializeOnLoad method shouldn't be used for asset operations, because InitializeOnLoad is called before asset importing is completed. (1274994)
    - Asset Pipeline: Updated reload tests to cover async domain reload.

    2021.2.2
    Improvements
    - Visual Scripting: Eliminating domain reload cost for users when not using Visual Scripting

    2021.2.4
    Fixes
    - Asset Pipeline: Fixed a performance regression where projects with large numbers of precompiled assemblies might experience slower import times. (1372980)

    2021.2.12
    Fixes
    - Editor: Fixed an issue where there was no automatic filter out and ignore assemblies that were automatic included as a part of the system library references. (1363463)

    2021.2.13
    Improvements
    - Scripting: Improved AssemblyUpdater to not try to connect to VCS unless there assemblies under VCS that needs updating. (1353761)

    2021.2.17
    Improvements
    - Scripting: Optimized searching for scripts when the assembly name is not supplied, can significantly improve performance entering PlayMode amongst other areas.


    2021.3.9
    Fixes
    - Asset Pipeline: The option to 'Recompile after finished playing` did not work and has been removed. To support the intended functionallity a new option to only auto refresh outside playmode has been added. (UUM-818)

    2021.3.11
    Improvements
    - Asset Import: Reduced the time taken by refreshing the asset database, which among other things improves the performance of editing prefabs.

    2021.3.12
    Fixes
    - Package Manager: Fixed an issue where having packages with a lot of versions increases domain reload time drastically. (UUM-12670)

    2021.3.14
    Improvements
    - Editor: Reduced the time taken by AssetDatabase.FindAssets when used with a custom filter, speeding up domain reloads and other editor operations that made the poor choice of calling this function in the first place.
    - VFX Graph: Reduced the time taken by VFXGraph.CheckCompilationVersion that would previously potentially query all assets on every domain reload.

    2021.3.18
    Known Issues
    - Asset - Database: Script recompiles in Play Mode when Script Changes While Playing option is set to Recompile After Finished Playing and Auto refresh is set to enabled. (UUM-20409)
    Improvements
    - Editor: Reduced time to rebuild the CreateAsset(...) menu during a domain reload, it previously was scaling badly for projects with many assemblies.

    2021.3.23
    Improvements
    - Serialization: Improved performance of restoring managed objects during a domain reload (again!).
    - Serialization: Improved performance of restoring managed objects during a domain reload.

    2021.3.24
    Improvements
    - Editor: Reduced impact of Shader Graph package on performance of domain reloads by making node class cache populate on-demand.

    2021.3.25
    Changes
    - Editor: Removed automatic upgrade code for UniversalRendererData from alpha & beta builds of 2021.2 to current as it is no longer required and was incurring a domain reload performance cost.
    Fixes
    - Asset Pipeline: Prevent script re-compilation when 'Recompile after playmode' and 'Auto-refresh' are set. (UUM-20409)
    - Editor: Objects now keep their non-serializable state when reloaded. (UUM-7785) o_O
     
    Last edited: Jul 10, 2023
    Neiist likes this.
  19. CodeRonnie

    CodeRonnie

    Joined:
    Oct 2, 2015
    Posts:
    529
    Ahh, I found it. If you want to profile your domain reloads there are multiple options. (I haven't used these myself.)

    Editor Iteration Profiler
    https://forum.unity.com/threads/introducing-the-editor-iteration-profiler.908390/

    Compilation Visualizer
    https://openupm.com/packages/com.needle.compilation-visualizer/

    And here is a helpful article on profiling and optimizing domain reloads: https://johnaustin.io/articles/2020/domain-reloads-in-unity One intersting thing that seems to be suggested by all of this is that you may have faster domain reloads if you're in an empty scene with no inspector open because reloading the domain also reloads all of the current objects. I could be misinterpreting something there, but if so that's certainly a useful tip. EDIT: Maybe I am misinterpreting, and it needs to reload the asset database, not just what you have open. If only I could figure out how to manually control precisely when the domain reload occurs in the latest versions of 2021.3.
     
    Last edited: Jul 10, 2023
    Neiist likes this.
  20. CodeRonnie

    CodeRonnie

    Joined:
    Oct 2, 2015
    Posts:
    529
    EDIT: The Auto Refresh setting was moved from General to Asset Pipeline in Preferences! Disregard the following. Now I feel silly. :confused:

    Unfortunately, the manual page on Domain Reloading (https://docs.unity3d.com/Manual/DomainReloading.html) contains dated information. It says, "When Domain Reloading is disabled, Unity still refreshes the scripting state when you update or re-import a script, based on your auto-refresh settings." That links to the general preferences page, which no longer contains auto-refresh settings. However, it seems like you can still hack that functionality back in using EditorApplication.LockReloadAssemblies. (https://docs.unity3d.com/ScriptReference/EditorApplication.LockReloadAssemblies.html) Here is a thread on the topic: https://forum.unity.com/threads/can...after-edit-create-or-remove-a-script.1268315/ Maybe it can help someone, but after everything I've seen now I would be hesitant to try to add functionality back in that Unity seems to have removed for some reason or another. Fortunately I'm not currently having the super long domain reload times on my little projects.
     
    Last edited: Jul 10, 2023
  21. CodeRonnie

    CodeRonnie

    Joined:
    Oct 2, 2015
    Posts:
    529
    Okay, I got manual refresh working, but there was one additional step I had to perform. I'm using Visual Studio 2022 v17.5.3. Set this setting to False and restart Visual Studio and Unity:

    Tools > Options > Tools for Unity > Code Edition > Refresh Unity's AssetDatabase on save
     
  22. Ryiah

    Ryiah

    Joined:
    Oct 11, 2012
    Posts:
    21,095
    It's mostly optional for the reason that you've surmised: a great deal of code predates its availability. Since it came out I've only had a single project break when it was turned off and I'm in the habit (thanks in part to the lengthened domain reload times of DOTS) of turning it off as soon as I start or open a project.
     
    Last edited: Jul 10, 2023
    Neiist likes this.
  23. Neiist

    Neiist

    Joined:
    Sep 18, 2012
    Posts:
    31
    Thanks for the reassuring heads up, I decided to give it a go after making sure I had everything saved on git, and I realized that the domain reload isn't skipped only when entering play mode, as the settings labels strongly suggest, I mean it's written twice "Enter Play Mode Settings|Options".. but also after compilation.

    The reason why I was so skeptical in the first place is because I thought it was only made to iterate fast while entering and leaving play mode, without touching the code.

    I was convinced that modifying the code would still trigger a reload because changes made to static constructors / initialization would otherwise be left out when added or modified, eventually leading someone to believe that he made a mistake because nothing would indicate that the code is simply not called. Obviously I should have done that before talking, but that's the thing, I've grown so wary of new versions and new features, that my first instinct is to assume the worst to not be disappointed.

    Nevertheless it brought my usual 1 minute down to 15~18 seconds which, in itself, is delightful. But the question of having introduced possible invisible failure points remains, maybe I became straight up paranoid over the years x) Yet happy so far.

    EDIT
    After inspecting every single static field in my project I'm glad I avoided relying on static logic / singletons from the beginning, and already unsubscribed static events in Dispose() or OnDisable(). Only a few variables (counters in my case) had to be reset. Looking at some third party packages, the most important ones for me have already adapted their code, the only worrisome things I found were in pretty old/deprecated packages. I'm quite relieved to see that no change is needed in most cases. Glad I gave it a chance after all and sorry for the confusion.

    EDIT EDIT
    Nevermind it did cause several things to break that didn't show at first.. I ll spend time fixing these wondering why I'm always fixing Unity's problems instead of making games.

    Okay I think I'm crazy, yesterday it seemed to skip the reload after I modified code, now it takes 1 minute just like before... so scratch everything I've said so far, I give up trying to understand what this feature is about.
     
    Last edited: Jul 11, 2023
  24. Neiist

    Neiist

    Joined:
    Sep 18, 2012
    Posts:
    31
    upload_2023-7-12_23-4-25.png
    Counting my hair, come on count with me 1... 2... 3..... 4...
     
  25. Neiist

    Neiist

    Joined:
    Sep 18, 2012
    Posts:
    31
    I just tried this and I get this
    upload_2023-7-13_1-54-41.png

    Looking at this line at the bottom it looks like unity is waiting for Burst to compile, so I checked the Burst settings and both "Synchronous Compilation" and "Native Debug Mode Compilation" were enabled. I don't want to get false hopes again but disabling those does seem to remove that 30 seconds step, I don't know which one of them at this point and I had enough loading bars to look at for today, but this info might still be useful to people.

    Thanks a lot @CodeRonnie for mentioning the tool!
     
    Last edited: Jul 13, 2023
    lacas8282 and CodeRonnie like this.
  26. CodeRonnie

    CodeRonnie

    Joined:
    Oct 2, 2015
    Posts:
    529
  27. scryedzxp

    scryedzxp

    Joined:
    May 14, 2013
    Posts:
    47
    I was getting this issue too after adding a lot of scripts in my change. Rebooting my machine seemed to fix the issue.
     
  28. chikchik

    chikchik

    Joined:
    Nov 29, 2012
    Posts:
    3
    If "Completing Domain" stage takes a long time, just cleanup directory c:\Users\<UserName>\AppData\Local\Temp\.
    Of course it solve issue for Windows only.
     
  29. manuelgoellnitz

    manuelgoellnitz

    Joined:
    Feb 15, 2017
    Posts:
    397
    That folder is basically empty and it still takes 2 minutes...
     
  30. manuelgoellnitz

    manuelgoellnitz

    Joined:
    Feb 15, 2017
    Posts:
    397
    Today we found out:

    The Domain Reload time is very depended on the current open scene.
    When we have our main scene opened in the editor:
    - Playmode start time is about 1,30 min
    - Compile time also 2 minutes with about 45 seconds "domain reload" and 45 seconds "importing assets".

    When I switch to an empty scene, which has only one gameobject with a script that loads the main scene:
    - Playmode start time is < 30 seconds
    - Compile time 15 seconds

    So it seems to me that Unity does some unnecessary stuff in "Domain Reload" and "importing assets"