Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

Question Long Time stuck on Completing Domain after any code change

Discussion in 'Entity Component System' started by Occuros, Oct 2, 2022.

  1. Occuros

    Occuros

    Joined:
    Sep 4, 2018
    Posts:
    284
    Since upgrading to the new editor (2022.2.0b8) and entities 1.0.0.exp.8 iteration time is a lot slower than before in any project (Samples or my own).

    It takes a long time while the unity load bar shows: "Completing Domain".

    Does anyone experience the same?
     
    AldeRoberge and tonytopper like this.
  2. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    908
    No, and haven't heard from anyone else having that problem. At least not always. I had very few occasions where the bar just hangs and I had to close Unity.

    To find out what happens, open the Profiler and switch from Play Mode to Edit Mode. That way you can debug the profiler and you should be able to see what's taking so long.
     
  3. eterlan

    eterlan

    Joined:
    Sep 29, 2018
    Posts:
    177
    Same problem here. It takes almost half minutes to compile when I change code in sample project.
     
  4. Occuros

    Occuros

    Joined:
    Sep 4, 2018
    Posts:
    284
    Thank you @Enzi, I tried but during that time, the profile is nonresponsive (internal and as a separate process) and only starts working again once the `Completing Domain` phase is over.

    So at the moment, I am at a loss on how to investigate this further.
     
  5. Occuros

    Occuros

    Joined:
    Sep 4, 2018
    Posts:
    284
    I could Identify that it is PC architecture dependent it seems (or at least it's not project related).

    Cloning the same project on two different machines leads to \very different times needed for `Completing Domain`.

    One machine is Intel the other is AMD, but they have roughly the same speed for compilation of normal projects, yet in 2022 they differ by 3-4x
     
  6. QuebecDev

    QuebecDev

    Joined:
    May 9, 2020
    Posts:
    37
    Hi, I have this problem too!

    I just jump on Unity 2022.2.1 and ECS 1.0.0 pre.15

    I always work without the burst compiler. and my reload time is +/- 10sec.

    But sometimes I do enable the burst compiler to test some things and the next time I did a script modification and then press Ctrl+R to reload domain... It took me 10sec to reload and +/-3min of ''Completing Domain'' freeze.

    This was not happening before Unity 2022.2.1 and ECS 1.0.0 pre.15.

    Anyway, I will keep the burst compiler disabled for now :)
     
  7. LDHTL

    LDHTL

    Joined:
    May 24, 2020
    Posts:
    2
    Hi, I had the same problem.

    For me pressing Ctrl-R during the "Completing Domain" Window forces Unity The reload the domain. That sometimes fixes the problem and shortens the time to wait.
     
  8. PixelDough

    PixelDough

    Joined:
    Apr 27, 2018
    Posts:
    45
    Every Unity developer I know has encountered this, for about two years now, and it's really sad that there's not at the very least like a band-aid fix that could show a button to try reloading the domain if it goes on too long. Not a forced timeout, but if you feel it's taking longer than it should, you could have a button on the dialog box that would restart the process of reloading the domain. Instead, we just have to force close Unity and likely lose progress on whatever we were doing in editor, which gets especially aggravating when you're jumping in and out of your code editor tweaking certain things.

    Not to mention there's no clear indication that it's even stuck. I could leave it running for a few hours and it'd still be stuck, but it'd look no different than if it was just taking too long. It's unforgivable that the only solution is just "force close Unity and lose progress because there's no auto-saves or recovery process either"
     
    apkdev, jdtec and Mikaelan like this.
  9. Doomlaser

    Doomlaser

    Joined:
    Dec 9, 2013
    Posts:
    22
    Running into this problem on 2022.2.6f1 and it's quite annoying. Super long iteration times are quite bad for productivity, but that's what I am getting.
     
  10. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,626
    My iteration times vary from

    7 seconds - if I'm editing a top level system assembly that doesn't cause a subscene rebake
    to
    over 2minutes - if I edit a low level assembly that everything depends on and causes a subscene rebake

    The low level assemblies don't get changed much now that they're setup, but because of this I do have separate projects setup for them where I can work without causing long iterations.

    so TLDR: use assembly definitions, have separate projects for core/low level assemblies.
     
    tonytopper likes this.
  11. crawfis

    crawfis

    Joined:
    Jan 30, 2014
    Posts:
    103
    Any fix for this? I too am running into this problem.
     
    tonytopper likes this.
  12. Quit

    Quit

    Joined:
    Mar 5, 2013
    Posts:
    63
    It's the same here. Very slow iteration time. Every change takes a very long time to recompile even with the domain reload flags off.

    Starting the project takes like 5-10minutes to load up (that's an empty project from their URP 3D template). I'm talking about opening the project when it's been generated already.

    Recompiling any C# change takes 15-60seconds on this one as well. I have entity systems window open and subscene with one gameobject in it and the authoring script, which is empty. No job is being done, and it's horrendous unfortunately.
     
    OUTTAHERE likes this.
  13. elliotc-unity

    elliotc-unity

    Unity Technologies

    Joined:
    Nov 5, 2015
    Posts:
    228
    For me this happens badly with synchronous burst, not too bad but not amazing with asynchronous burst, and great with burst off.

    This is a per-machine setting, which would explain it being different between computers.

    It's a known issue in burst that unfortunately requires a very fundamental rejigger (moving burst out of process when compiling for the editor).
     
    Quit and Occuros like this.
  14. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,029
    I thought it's already like that at the latest burst version?
     
  15. elliotc-unity

    elliotc-unity

    Unity Technologies

    Joined:
    Nov 5, 2015
    Posts:
    228
    No, that's only for player builds.
     
  16. Quit

    Quit

    Joined:
    Mar 5, 2013
    Posts:
    63
    I'll try running it with Burst off completely and see if anything changes. Thanks.

    However, I don't know if that changes any startup project time. Though it might be doing checks/burst recompilation there as well. Cuz that's very slow too.
     
  17. elliotc-unity

    elliotc-unity

    Unity Technologies

    Joined:
    Nov 5, 2015
    Posts:
    228
    Yes, it will do bursting on startup, especially with synchronous burst on.
     
  18. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    3,983
    At least in the earlier 22.2 versions, sometimes domain reload thread-locks. I know this because I can see all threads in the Unity process idle and not consuming CPU resources (or very very little likely due to spurious wakeups). Not sure if this happens in later versions, as Pre.47 is broken for me, but my point is there might be two separate issues at play. The thread-lock issue happens regardless of if Burst is enabled. I usually have to kill the process, and then reopening I get an error about multiple instances of the same subscene, after which it seems the editor corrects itself.
     
    iamarugin, Edmario_ and OUTTAHERE like this.
  19. OUTTAHERE

    OUTTAHERE

    Joined:
    Sep 23, 2013
    Posts:
    656
    I used to have this A LOT but it has gotten much, much better.
    It must have something to do with the structure of the project/systems and baking, plus burst.
     
  20. Quit

    Quit

    Joined:
    Mar 5, 2013
    Posts:
    63
    Seems to be that disabling the Burst in the editor speeds it up a lot. I'm not really seeing any benefit to having Burst enabled in the editor during the dev time especially. It's such a drag to the whole iteration. I understand the benefits of Burst which are amazing, and it MUST be enabled when optimizing/testing a build/performance, but it's a killer of iteration unfortunately.

    Thanks for shining the light here. My iteration speed went from 30-60seconds to 5seconds max per compile. Don't know if it's just a luck, but for now it's amazing.
     
  21. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,653
    What we're using is we've enabled internals vissibility to our editor script and added additional checkbox in our menu. Now we disable burst in editor by default, and if this checkbox is enabled - burst automatically enables after pressing playmode and disables after exit playmode. It reduced compilation time a lot for us without having to disable burst manually in editor and enabling it before play mode all the time. It also prevents us from seeing multiple internal compiler errors when something locks stuff burst working on
    upload_2023-3-19_14-1-36.png
    Code (CSharp):
    1.  
    2. [InitializeOnLoad]
    3. public static class PlaymodeUtils
    4. {
    5.     private const string EnableBurstCompilationOnPlaymodeMenu = "DNO/Enable Compilation On Play";
    6.     private const string EnableBurstCompilationOnPlaymode     = "BurstCompilationOnPlayMode";
    7.     private static bool _stateOnLoad;
    8.    
    9.     [MenuItem(EnableBurstCompilationOnPlaymodeMenu, false, 0)]
    10.     public static void EnableBurstCompilation()
    11.     {
    12.         _stateOnLoad = !_stateOnLoad;
    13.         EditorPrefs.SetBool(EnableBurstCompilationOnPlaymode, _stateOnLoad);
    14.     }
    15.    
    16.     [MenuItem(EnableBurstCompilationOnPlaymodeMenu, true)]
    17.     private static bool EnableBurstCompilationValidate()
    18.     {
    19.         Menu.SetChecked(EnableBurstCompilationOnPlaymodeMenu, _stateOnLoad);
    20.         return true;
    21.     }
    22.     static PlaymodeUtils()
    23.     {
    24.         EditorApplication.playModeStateChanged += LogPlayModeState;
    25.         _stateOnLoad                           =  EditorPrefs.GetBool(EnableBurstCompilationOnPlaymode, false);
    26.     }
    27.     private static void LogPlayModeState(PlayModeStateChange state)
    28.     {
    29.         switch (state)
    30.         {
    31.            
    32.             case PlayModeStateChange.ExitingEditMode:
    33.                 BeforePlaymode();
    34.                 break;
    35.             case PlayModeStateChange.ExitingPlayMode:
    36.                 AfterPlaymode();
    37.                 break;
    38.             case PlayModeStateChange.EnteredPlayMode:
    39.             case PlayModeStateChange.EnteredEditMode:
    40.                 break;
    41.         }
    42.     }
    43.     private static void BeforePlaymode()
    44.     {
    45.         if(EditorPrefs.GetBool(EnableBurstCompilationOnPlaymode, false))
    46.             BurstEditorOptions.EnableBurstCompilation = true;
    47.     }
    48.    
    49.     private static void AfterPlaymode()
    50.     {
    51.         if(EditorPrefs.GetBool(EnableBurstCompilationOnPlaymode, false))
    52.             BurstEditorOptions.EnableBurstCompilation = false;
    53.     }
    54. }
    55.  
     
    ivaylo5ev, Occuros, Selmar and 5 others like this.
  22. vildauget

    vildauget

    Joined:
    Mar 10, 2014
    Posts:
    120
    Thanks. You post code, noobs will copy/paste (me). Can you point me in the right direction with explanation, keywords or links as to how this is done? I guess that's the part that makes "BurstEditorOptions" available.
     
  23. Sirius64

    Sirius64

    Joined:
    Jan 15, 2020
    Posts:
    7
    You could replace that line with BurstCompiler.Options.EnableBurstCompilation
     
    vildauget likes this.
  24. vildauget

    vildauget

    Joined:
    Mar 10, 2014
    Posts:
    120
    Excellent, works now. Thank you.
     
  25. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,653
    Yes, you can do that in most cases, but will miss one piece - call of StaticTypeReinit assembly attributes which is called only from BurstEditorOptions (if you have one):
    upload_2023-3-20_9-42-26.png
     
    xVergilx, Sirius64 and bb8_1 like this.
  26. Goularou

    Goularou

    Joined:
    Oct 19, 2018
    Posts:
    50
    It can take minutes in my project, even without burst ! It looks like, with code errors, it is stuck forever also... OK safe mode is made for that, but still, it shows that it is a compilation error not only related to burst.

    Copying project from one machine to another also triggers long duration on first time.
     
    Last edited: Mar 30, 2023
  27. nicloay

    nicloay

    Joined:
    Jul 11, 2012
    Posts:
    535
    APTEM59, lintoka123, Arkanad and 2 others like this.
  28. Snedden27

    Snedden27

    Joined:
    Jul 4, 2020
    Posts:
    7
    I upgraded a project done in LTS to 2022.2, got an error when opened the project:
    Failed to find entry-points
    Ignore it, tried to restart unity , now it showed me enter safe mode or ignore , I ignored , being stuck on the Completing Domain for like 10 minutes now
     
  29. yengNami

    yengNami

    Joined:
    Feb 14, 2021
    Posts:
    12
  30. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    943
    And here I thought that my PC is probably outdated when domain reload takes a long while than usual.
     
  31. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    943
    Can you share how GetGlobalOptions() is implemented?
     
  32. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    908
    Long time ago, I said I'm not having issues but now it's pretty normal. It definitely has to do with Burst regressions. Without Burst, Unity is blazing fast. That goes for recompiling, domain reloading and even starting up a project. With Burst enabled everything slows down. Starting up a project takes very long (few minutes against a few seconds) and recompiling and domain reloading too. I can't pinpoint what's going on. When I'm making changes in an assembly to NON bursted code (you read that right) it takes very long on the window, domain reloading. When I change code in a bursted job, just a few seconds. Impossible to explain really. :D

    It's pretty frustrating to have to deal with these kind of regressions that slowly creep in. Hope the next Burst release will address some of these problems. Iteration time with Burst is painful nowadays and it wasn't always like that.

    The best solution right now is to just disable Burst for heavy coding sessions.

    Oh, and I'm having the "Synchronous compilation" option enabled. Some said there lies the problem and while it's true the editor isn't blocked, the compilation still takes that long so it's not that I'm saving any time here.
    As I'm stress testing most of the time, I have to wait for Burst to finish, no matter what.
     
    Arkanad and Goularou like this.
  33. aitchest-of-dees

    aitchest-of-dees

    Joined:
    Dec 28, 2013
    Posts:
    71
    I feel like I spend as much time waiting for Unity to become responsive as I do searching in vain for a solution.

     
  34. Dylan2020c

    Dylan2020c

    Joined:
    Jun 30, 2023
    Posts:
    1
    idk why but mine has took9.29 m so far unity wont open nor the even loading domain bar
    anyone now why this happen?
     
  35. lintoka123

    lintoka123

    Joined:
    Feb 6, 2023
    Posts:
    1
  36. moatdd

    moatdd

    Joined:
    Jan 13, 2013
    Posts:
    150
    upload_2023-7-5_18-26-25.png
    It takes 41s to do an iteration and the vast bulk is stuck in AssemblyReload. This is insane and I've been dealing with this situation for months. Unity stopped being fun to work with and I hate my job now.

    The Scene Reload and Domain Reload toggles are already off.
    upload_2023-7-5_18-38-46.png

    My code makes up a TINY amount of the overall project, with all third-party code sitting in plugin folders so that when I make adjustments to my own code, it only recompiles my own code.

    upload_2023-7-5_18-35-34.png

    And, according to the iteration profiler, my code compiles in under 1.5s.

    I even spent spent some time and money to upgrade my workstation's ram from 16GB to 64GB and it has not resolved the issue. Nobody on the discord replies to my posts and I'm at my wit's end.
     
    Last edited: Jul 5, 2023
    AldeRoberge likes this.
  37. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    3,983
    Disable Burst when you need fast iteration. I couldn't tell you why it affects domain reload so much, but it does and you aren't the only one to experience it.
     
    moatdd likes this.
  38. moatdd

    moatdd

    Joined:
    Jan 13, 2013
    Posts:
    150
    Thank you, it seems that there's a lot of unnecessary burst re-compilation. I also learned that closing the Entity Hierarchy window took about 20s off of my compile time...

    ...but opening the window immediately causes the Burst compilation to occur.
     
    Last edited: Jul 6, 2023
  39. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    908
    Yesterday I did a little digging into the problem.
    There's a stage that registers every method for ISystem and for every method a FunctionPointer is compiled.
    Unity.Entities/SystemBaseRegistry.cs line 271

    This compile method triggers a JIT compilation. The unexpected thing here is that this JIT compilation takes a very long time. Profiling this in Superluminal, it turns out that the GC is stalling the process.


    Notice the many GC_methods.



    Furthermore, it seems every ISystem method triggers a synchronous burst compile. Every, single time something is recompiled. Nothing seems to be cached here. I'd also expect, at this stage, that the JIT is already compiled. Turning off Burst, the whole 10 sec block disappears. To be exact, it's still called, there's just a fallback to use the JIT entry point of the ISystem method and it's done nearly instantaneous.
    So, I don't quite get it what's going on here but I do hope it will get fixed because this is eating into iteration time drastically and it only showed up with ISystem.

    Also a notice, because I didn't realize, until yesterday and that this is a problem. Adding any package in local state adds all the tests projects. This added at least ~7seconds of recompile time to my project.
     
    Last edited: Jul 6, 2023
  40. s4lt3d

    s4lt3d

    Joined:
    Nov 21, 2014
    Posts:
    8
    Thank you! Disabling burst saves me 30-60 seconds per script change. It feels like Unity developers don't use Unity to make their own games. How would they live with these issues if they did.
     
    tonytopper likes this.
  41. elliotc-unity

    elliotc-unity

    Unity Technologies

    Joined:
    Nov 5, 2015
    Posts:
    228
    Yes we do and yes it sucks, and no I don't know how we live with it, but we do. People are actively working on the burst domain reload issue (I get the impression they'll be done in a few months ish), but if anything the iteration time for internal devs is worse than for small-to-medium game projects because the project we iterate in most commonly has way too much entities.foreach because it has the tests for that, whereas users can just not use entities.foreach and eliminate that part of the iteration time burden. (A sample game we use had 11s iteration time with burst off, whereas the commonly used entities test project had 30s.)
     
  42. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    3,983
    I think there's also something platform-specific going on, because over the weekend I was doing a game jam with some others, and one of them had an Apple M2 laptop and with Burst disabled, domain reloads were maybe 2 seconds whereas for everyone else on Windows they were closer to 20 seconds for this project.
     
    euden_one and tonytopper like this.
  43. tonytopper

    tonytopper

    Joined:
    Jun 25, 2018
    Posts:
    199
    I have Jobs -> Burst -> Enable Compilation turned off and "Completing Domain" still kills my iteration times. On Windows with 2022.3.7f1

    It's taking around 15 seconds, which sounds better than some but is way too long.
     
  44. webik150

    webik150

    Joined:
    Mar 23, 2013
    Posts:
    59
    Similar issue in 2022.3.10f1. I have burst compilation off, but every now and then after a code change, unity starts "Completing Domain" which never ends (last time I waited 25 minutes before killing the editor, just to see if it ever finishes)

    upload_2023-10-16_13-33-47.png
     
    DmitryDerybas, StefanWo and fadizant like this.