Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on Dec 8, 2022, between 7 am & 7 pm EST, in the DOTS Dev Blitz Day 2022 - Q&A forum, Discord, and Unity3D Subreddit to learn more about DOTS directly from the Unity Developers.
    Dismiss Notice
  3. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Official Improving iteration time on C# script changes

Discussion in 'Scripting' started by xoofx, Oct 18, 2021.

  1. RobertOne

    RobertOne

    Joined:
    Feb 5, 2014
    Posts:
    205
    My solution is to stick with the last unity 2019 as much as possible and think about if any of the new features (dont even ask me what i am missing, i totally lost track) are really worth THIS. If so, upgrade to 2020 but whatever you do: stay away from unity 2021
     
  2. DragonCoder

    DragonCoder

    Joined:
    Jul 3, 2015
    Posts:
    870
    Though let's not rightaway exaggerate. In my day job (machine industry) where we work with C++, it's normal to wait over 10 minutes for every recompile.
    With 30s we are still quite cushioned here xP
     
  3. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    5 seconds is about my limit. I've programmed in Unity for years and when it gets slow I move code out to Plugins folder or use assemblies.

    You guys really don't need to be accepting long compile times. What have you tried to reduce them so far?
     
    Lahcene and stonstad like this.
  4. Maria_Angelova

    Maria_Angelova

    Unity Technologies

    Joined:
    Mar 3, 2020
    Posts:
    18
    Some more info would help to respond to your complaint here: links to example bug reports you've sent would go far in elaborating about the crash you are experiencing and what response you got on the tickets would help know where the investigation on our side landed. We take crashes very seriously. Example reasons for not addressing a crash could be we are unable to reproduce at all (in which case a fix is hard to develop), a fix was implemented in a later unity version which has not been backported (yet), the crash was in user code or third party code, the crash indicates corrupt scene or other project asset - in such a case sometimes fixes are more complex and may require changing unity architecture, rewriting big subsystems, take time to develop and are usually not backportable.
     
  5. Maria_Angelova

    Maria_Angelova

    Unity Technologies

    Joined:
    Mar 3, 2020
    Posts:
    18
    This is interesting information. First I want to say that there's a lot of different stuff that happens during AssetDatabase.V2 refresh so it may appear at the bad guy but it a shell for different types of work done: adb work, script compilation, domain reload, a lot of user code (eg. init on load, OnEnable, etc.) ca execute in here. To truly see the offending code we need to investigate in depth.
    It is interesting that you say performance degrades as the Editor stays open for a while. This is a good line of attack to work on. Does it happen even if it is not being worked with? As in if you open your project, make a minor script change and capture before you go home, then in the morning undo and make a capture, would it be much slower again? Or does it only degrade as the Editor is being used? It would be very valuable for us if you made this test using Diagnostic switches turned on and post the before and after times both in the case of not using the Editor and also with using it.
     
  6. Maria_Angelova

    Maria_Angelova

    Unity Technologies

    Joined:
    Mar 3, 2020
    Posts:
    18
    There's no single change we can do to unity to make this problem go away. This requires work across many teams for a long period of time.
    Definitely watch for updates if you are invested in using Unity for your projects. Other good practices include:
    - Use as newer public minor releases as possible and update existing projects often (to get latest bugfixes and improvements on a given major version);
    - if Unity behaves unacceptable, please report the problem in as much detail as possible.
     
  7. Maria_Angelova

    Maria_Angelova

    Unity Technologies

    Joined:
    Mar 3, 2020
    Posts:
    18
    You can try disabling ADB autorefresh, and trigger it manually or using code. Fast enter play mode is another workaround which will skip doing a domain reload when you enter play mode in case there's no script changes.
    Splitting code into more assemblies using asmdefs (more modularity) and using version defines and define constraints to strip unneeded code during compilation - helps compilation times. Use the Editor Iteration Profiler to identify badly performing user code. Remove unneeded packages, sources and assets. User TypeCache APIs instead of reflection.
     
  8. wheee09

    wheee09

    Joined:
    May 21, 2018
    Posts:
    68
    Hmmm, thanks for the response, but I'm a bit confused as to how most of that would help.

    From the log I have this specifically:
    Code (CSharp):
    1. ImportOutOfDateAssets: 7629.724ms (5741.443ms without children)
    2.                 CompileScripts: 1881.188ms
    3.                 ReloadImportedAssets: 1.811ms
    4.                 EnsureUptoDateAssetsAreRegisteredWithGuidPM: 2.997ms
    5.                 InitializingProgressBar: 0.000ms
    6.                 PostProcessAllAssetNotificationsAddChangedAssets: 1.800ms
    7.                 OnDemandSchedulerStart: 0.486ms
    My code is already broken out in asmdefs and the compile time (according to the above snippet) is 1881.188ms which is fine. In fact, a good portion of the total time is taken up with loading the assemblies which will probably just get worse if I increase the number of assemblies.

    Playmode doesn't have anything to do with the autorefresh/recompilation so it's not particularly relevant here.

    Disabling ADB autorefresh just means I have to trigger it manually but it still needs to do it.

    My big question is why is it taking 7.6s. The children elements don't add up to that, and what does "5741.443ms without children" mean?

    I will try the Editor Iteration Profiler.
     
  9. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,965
    For everyone that's got ImportAndPostpro9cessOutOfDateAssets or AssetDatabase.V2.Refresh or whatever taking long - dig deeper! Turn on deep profiling and just expand a couple of more times, and you'll get to actual useful information, like what code is taking time to run.

    If it's getting slower over time, profile what's taking time when you just start up the editor, and profile what's taking time a bunch later. Compare! It's probably something like a scripted object that gets generated, but never cleaned up, that has a somewhat expensive OnEnable, but that's impossible to know without checking.

    As an example, I've got somewhat bad compile times and enter play mode times. I know why entering play mode is bad, as I have checked that earlier - it's because SpriteShape is bad and takes a million years baking data, and caching that data is hard and tedious. So we've got fixing that on the short-list of things we need to do.

    The compile/assembly reload times isn't that bad - like 4-5 seconds. But I can figure out what those seconds are pretty easily. Follow me along for a ride if you want:

    If I turn on deep profiling, compile+assembly reload takes a bit over 8 seconds. If I dig down, 10% of the time is spent in ScriptExecutionOrderInspector.OnEnable, so that's a good spot to start:

    upload_2022-2-8_13-34-8.png


    But wait! There's already an actionable bug I can report here to Unity. ScriptExecutionOrderingInspector is probably the window you see when you open Edit->Project Settings-> Script Execution order. But I have that window closed! So there's a bug here where windows that I don't have open are still getting OnEnable calls, which are slowing down the reload time somewhat.

    That being said, I wonder why that OnEnable is taking so long, though, so let's dig deeper. upload_2022-2-8_13-47-34.png

    S***, that is a lot of comparison calls! Let's look at the code. Turns out it sucks a bit:
    upload_2022-2-8_14-15-19.png

    Whoops it is horrible! If you replace
    if (GetExecutionOrder(script) == 0)
    there with just using the value found right above, my assembly reload would save ~1 million object comparisons, and... idk half a second in deep profile for that scale probably means like 100ms real time or whatever, deep profiling makes code that has many calls look worse than the reality. My profiling result is from 2020.3, but the code is still bad in the 2021.2 reference source.

    I totally see how that bug happens - somebody needed the execution order at one point and used the method named GetExecutionOrder. Private named methods are somewhat dangerous if you call them without reading them :p

    So, yeah, it's totally possible to figure out what's taking time. Sometimes it's just inside some code you don't have the source of, so you can't fix it, but at least you can point to what's being bad.

    Just going "some very high level profile marker is taking long" or "things are slow" when the profiling tools avaiable are of such high quality doesn't do nothing - you can see that @Maria_Angelova and Unity is responding. But if you actually take the time to figure out what's slow (the above info took me ~15 minutes to figure out), you'll get stuff fixed a lot faster, since you can point at what's actually bad.

    Like how I can point at two bugs:
    ScriptExecutionOrderInspector.PopulateScriptArray is taking many orders of magnitude longer than it should!
    Windows that are closed seem to be getting an OnEnable!

    So you can do it too! Just hit deep profile, change some code, pause the profiler, and expand until there's methods with names that mean something.
     
    Last edited: Feb 8, 2022
  10. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,965
    Can you please start supporting it again? It's got bugs.
     
    SugoiDev, JoNax97 and hippocoder like this.
  11. Trindenberg

    Trindenberg

    Joined:
    Dec 3, 2017
    Posts:
    304
    I tried the Deep Profile, this is what I have; (average without is about 7.5 seconds)
    upload_2022-2-8_17-51-14.png

    Application.morethanafewTicks. Seems like a lot happens editor side after reload
    upload_2022-2-8_17-46-6.png
     
  12. wheee09

    wheee09

    Joined:
    May 21, 2018
    Posts:
    68
    You need to look at the Time ms column on the right side and start expanding on the ones that are the highest - in the first image.

    ReloadAssemblies -> EndReloadAssembly -> SeupLoadedEditorAssemblies takes almost 5s for eample. Expand that and see if it's anything related to your code or assets.
     
    Lahcene likes this.
  13. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    391
    I enjoyed reading your post Baste. Although I have viewed deep iteration profile data many times I never thought to inspect the underlying source of the editor for the actual cause.

    I reviewed my project's iteration data with fresh eyes. In my scenario I see timing distributed across many methods within AssetDatabase.Refresh. I do have the issue of asset refresh being executed twice and this is a reported and acknowledged bug.

    For the 21s wait I see nothing that sticks out to me. ScriptCompilation is 2220ms. AssetImport is 9893ms. And AssemblyReload is 9893ms. I find it odd that AssetImport and AssemblyReload, which are sibling nodes in the iteration event tree, have the exact same timing down to the fraction of a millisecond.

    When I drill down I am not seeing a single leaf method that consumes more time than its peers.

    Also weird is that AssetDatabase.V2.CompileScripts seems to compile scripts, and so does ScriptCompilation. The complexity here is more than the average lay user can understand.
     
  14. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    391
    How can I help Unity in my scenario? I do report bugs ... but I have a 21s wait on a 5GHz machine with an NVME SSD. How do I get you the data you need when I don't see the bug?
     
  15. wheee09

    wheee09

    Joined:
    May 21, 2018
    Posts:
    68
    Just to follow up - I installed the Editor Iteration Profiler which is nifty as it shows the juicy details without the pain of using the Profiler (which is sluggishly slow).

    I've noticed two things:
    1. the biggest culprit is the Unity Burst compiler - which seems to be using reflection. It takes about 1-2s to do its onload method.
    Screen Shot 2022-02-08 at 2.00.18 PM.png

    2. it appears that my compilation isn't the culprit. There seems to be a blackhole somewhere sucking up time - and the Editor Iteration Profiler won't help. Perhaps the regular Profiler will.

    I have code (adapted from
    // https://gist.github.com/karljj1/9c6cce803096b5cd4511cf0819ff517b) that outputs compilation time metrics but from a more holistic viewpoint.

    Here's an example:
    1. Total Assembly Compilation Time: 3.4s
    2. "CompilationPipeline Time": 8.4s
    3. Time before Editor UI is available: 11.2s

    The times are generated as follows:
    1. Tally/Sum the time to compile all modified assemblies using CompilationPipeline.assemblyCompilationStarted and CompilationPipeline.assemblyCompilationFinished.

    2. Simply get the time elapsed between CompilationPipeline.compilationStarted and CompilationPipeline.compilationFinished.

    3. This is the time from CompilationPipeline.compilationStarted to a polling check when Editor.isCompiling == false and Editor.isUpdating == false. This is basically a hacky way to determine when the Editor UI is finally unlocked.

    To summarize:
    The Total Assembly Compilation Time (1) matches the Profiler/Editor Iteration Profiler. What I need to figure out is why making code changes takes 5-8s longer (8.4-3.4 or 11.2-3.4) before the Editor is available to be used. That's my real issue.

    Unfortunately the Editor Iteration Profiler won't help me here since it only drills into #1. Of which, the Burst Compiler onload takes up almost half of it.

    Any thoughts about that?
     
    Last edited: Feb 8, 2022
    DragonCoder and stonstad like this.
  16. jjejj87

    jjejj87

    Joined:
    Feb 2, 2013
    Posts:
    894
    Honestly, no one in their right mind is going to you send you a commercial project...it just won't happen. And while I am aware that bug hunting requires a reproduceable source, I can tell you with confidence that even fewer people are going to make a stripped version of the project that replicates that issue. It is just time consuming and Unity is not a community engine, it is a commercial engine. I would do it if I absolutely had no other workarounds and even then I would rather cut features or reduce the project than sending the project. And often times, I have no power over it.

    Also, in many times I've reported bugs, they were very replicatable, like starting a new project and click this and boom. And I have spent many times sending in an empty template to prove that something is wrong...which could have been just simply checked by the devs by starting up a project...and just checking. But always, the devs will ask for a replicatable project...and the effort has to be sent by us to prove that it exists...after few years of this idiotic dance, I have realized that I shouldn't be doing this...

    Then there is the bug reporting process. We have to submit the bug and prove it exists. And if we cant for some real world reason, then the ticket is closed...because we haven't responded in time or have failed to present a reproduceable project...and rinse and wash and repeat...

    This all happens because Unity doesn't make games. I don't really care if you guys sell the game or not, but clearly the dev team doesn't test the engine in a full commercial game environment. Every time the community mentions this the devs start to have a stroke so I won't go further, but you guys really got to work on QA...why is every dev surprised when we say the engine is very unstable at full scale projects...

    just my 2 cents.
     
  17. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,965
    We do it all the time. I'm talking with somebody from Unity right now because they asked me to send our project. There exists no reason why I shouldn't.

    If you can't due to red tape, well then your management is making bad decisions that are causing them to lose money due to bugs not getting fixed, and it's a you problem. Look for a different job, maybe?
     
  18. jjejj87

    jjejj87

    Joined:
    Feb 2, 2013
    Posts:
    894
    It seems like your trust in Unity is great but not in me. Not because Unity is a bad company or anything but because there is no protection legally and literally.

    While Unity will mostly likely try their best, in the worst case they won't be held responsible nor will any one be able to reliably hold anyone responsible. How would anyone know realistically? That is the nature of the things here. You will most likely never know. Even if we did, how would the damages done be evaluated? How would it be covered?

    Also, I understand that the choices are all open to an individual/organization, but saying that sending a source file of a commercial project with "trust" is not advise I would give. Nor is it a bad management choice but a sound choice. I would lose it if my invested company's managers made that kind decisions. The red tape is to protect the company's invested assets and I honestly fail to see how that is a "you problem...and look or a different job"

    So, Baste, while I appreciate your feedback, I can't agree with you. Unless the world we live in are vastly different, the world just does not work on "trust". And I am sure there are many users who feel the same way. Whether they are willing to publicly share that sentiment, is another story.

    Also, as we are moving away from the topic a bit...maybe we should continue this conversation somewhere else. Admins are gonna remove posts :)
     
  19. DragonCoder

    DragonCoder

    Joined:
    Jul 3, 2015
    Posts:
    870
    If you send your assets to someone that does not mean they have any rights to use them. If Unity for whatever awkward reason would then launch a game based on your assets, you could of course sue them since you can prove that you made those first...
    Yes, companies are rightfully careful about their assets, but to not trust the company that creates literally the most important component of your game - its engine - sounds a bit silly.


    This thread gave me some good hints on what to check on my own "incrementally longer compile times" in my project. Will try to contribute...
     
  20. KamilDA

    KamilDA

    Joined:
    May 21, 2020
    Posts:
    278
    This is all very moot - and outside the scope of this thread.

    You can have Unity sign any type of NDA you want and have a very open relationship with them - and have account managers / service specialists assigned to your case specifically.

    Sorry for being blunt @jjejj87, but it sounds more like a 'you' problem of not making the required efforts to do it properly, safely, legally - than a 'Unity' not being trust worthy problem.
     
    Last edited: Feb 10, 2022
    Saniell and stonstad like this.
  21. fish-rp

    fish-rp

    Joined:
    May 6, 2021
    Posts:
    9
    Is there a similar thread for "Resolving Packages". I really like the direction Unity went with the Package Manager, but wow it takes a long time to update a package. Even changing 1 line of code in a medium sized package will take several minutes to re-import it. If the package manager is using GIT in the backend you'd think it could just pull the new commits instead of cloning the entire package again.

    We have a lot of cross-project packages so we're updating hashes quite frequently. Ideally this should be easier with the Package Manager, but waiting for resolving packages each time we change a Package hash or change branches, quickly becomes cumbersome.

    I've noticed this becomes even worse the more packages you are using.
     
    andreyefimov2010 likes this.
  22. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Don't let stuff get too personal. And yes, sending a project to Unity should be something people engineer for. Everyone in dev knows debugging something without the project is a horrible exercise in guesswork.

    If I were Unity I wouldn't want to guess at problems, at this scale.
     
  23. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,965
    We haven't experienced that. How are you embedding the packages? Just straight in the packages/ folder, or through some other mechanism?
     
  24. Maria_Angelova

    Maria_Angelova

    Unity Technologies

    Joined:
    Mar 3, 2020
    Posts:
    18
    When scripts change there's several tasks performed by the editor:
    - compile the scripts incrementally, if successful do the rest:
    - call OnDisable() on managed classes
    - backup the current scene from the current mono domain
    - unload the old Mono domain
    - load a new Mono domain
    - restore the scene backup (RestoreBackups)
    - cache stuff (eg. type info MonoManager::SetupTypeCache for faster-than-reflection queries)
    - run initialization code (Awake, InitializeOnLoad, OnEnable, etc.)
    - invoke callbacks for various events during Domain reload (eg. AssemblyReloadEvents)
    A lot of this stuff has to run on the main thread too which makes it hard to speed it up by running stuff async. Several places in this process "user" code (project's and packages') gets invoked so the total cost, summarized by this list, is actually split between Unity routines and code present in your project. That is also why it is so important to dig deep in the callstacks there where the profiler shows a large cost, as you mentioned earlier.

    EIP is unfortunately not being maintained however there's plans that the unity profiler will replace it and provide similar or better functionality for diagnosing bad iteration times.

    We are also in the process of backporting a small diagnostic tool described here: https://forum.unity.com/threads/any...pt-assembly-reload-time.1117138/#post-7235969 which should make it easier for you to share perf times with us by extracting them from the editor.log.

    But as stated above, the performance can be examined also using the unity profiler with "Deep Profile" enabled, record during a domain reload, go to the profiler's hierarchy vie and filter in the search field for "UpdateAndCompileScripts" or "ReloadAssemblies" and locate the frame containing either of those by scrolling on the time line. Focus on this frame and look for code in your project which slows down the process: look for rows with a high "Time ms" value inside these nodes:
    - FinalizeReload/AwakeInstancesAfterBackupRestoration and expand until you see the list of OnEnable() calls;
    - FinalizeReload/SetupLoadedEditorAssemblies/ProcessInitializeOnLoadAttributes and expand stuff that takes a lot of time
    - FinalizeReload/RestoreBackups/.../RebuildManagedInstances - there might be some custom tools here which take a long time to reconstruct. In general if RestoreBackups is slow you can try iterating on code while having as small scene as possible loaded and as few editor subwindows open as possible.
    - FinalizeReload/didReloadMonoDomain.Invoke - check if you have any expensive code in the project subscribed to this event
    - ReloadAssemblies/LoadAllAssembliesAndSetupDomain/SetupTypeCache - this is expensive, we know. The way to influence this is by cleaning up your project from code which you do not need.
    - ReloadAssemblies/ReloadAssembly/BeginReloadAssembly/DisableScriptedObjects - this is where to look for expensive OnDisable() calls
    - ReloadAssemblies/ReloadAssembly/BeginReloadAssembly/BackupScriptedObjects/BackupInstance - this is where to look for expensive OnBeforeSerialize() calls
    - ReloadAssemblies/ReloadAssembly/BeginReloadAssembly/AssemblyReloadEvents.OnBeforeAssemblyReload/Enumerator.MoveNext - this where to look for expensive OnBeforeAssemblyReload() calls

    Note, some of the above paths may vary from one unity version to another.

    Hope this helps understand where the time goes and gives better idea on how to examine your projects for potential improvements on your side of the code barrier. :)
     
    dmaj, Lahcene, cxode and 4 others like this.
  25. Maria_Angelova

    Maria_Angelova

    Unity Technologies

    Joined:
    Mar 3, 2020
    Posts:
    18
    We are aware of this and the Burst team has it on their list to fix and backport.
     
    Lahcene and Saniell like this.
  26. fish-rp

    fish-rp

    Joined:
    May 6, 2021
    Posts:
    9
    The packages are embedded using the manifest.json file. They point to packages hosted on our private GIT repos. (Unity 2020.3)
     
  27. sheredom

    sheredom

    Unity Technologies

    Joined:
    Jul 15, 2019
    Posts:
    300
    Just a little colour for this - basically in Burst 1.6 and before we would use the BurstReflection infrastructure to work out what and where things that can be Burst compiled are, that's why we need to do this [InitializeOnLoad] call to re-walk things.

    Luckily on 1.7+ we didn't need this, and actually we had totally forgot that we didn't need to this anymore. So in 1.7 (probably 1.7.1, I think I might just have missed the 1.7.0 cut) and future packages its about 2x faster in my local tests. Note we still have a bunch of things we have to do in the BurstLoader static constructor, but BurstReflection has thankfully disappeared from the trace :)
     
    hippocoder, Lahcene, wheee09 and 2 others like this.
  28. wheee09

    wheee09

    Joined:
    May 21, 2018
    Posts:
    68
    Thank you for the awesome detail here!

    I've been digging through the profiler and have discovered a lot of what you described for myself. I only wish I was able to read all that before I dived in.

    I'll need some time to digest and analyze. Hopefully I can find some low hanging fruit but I suspect it's going to be multiple things all adding just a tiny bit of extra time.
     
  29. wheee09

    wheee09

    Joined:
    May 21, 2018
    Posts:
    68
    That's great to hear!

    I upgraded the Burst package the other day to (Version 1.7.0-pre.1 - October 28, 2021) which was what was available in the Package Manager and can confirm that it still uses reflection.

    I'll see if I can manually install 1.7.1. Be great to eliminate the 1-2 seconds.

    *Edit* doesn't look like I can install it.. unless there's a way? (using 2021.2)
     
    Last edited: Feb 11, 2022
  30. sheredom

    sheredom

    Unity Technologies

    Joined:
    Jul 15, 2019
    Posts:
    300
    Oh sorry - to be 100% clear we haven't shipped 1.7.0 yet, never mind 1.7.1. Still gotta go through the testing and verification processes internally. I'll post back here once the fix is shipped!
     
    Lahcene and wheee09 like this.
  31. wheee09

    wheee09

    Joined:
    May 21, 2018
    Posts:
    68
    I've been digging with the profiler and I also have discovered that the Addressables is a major culprit as well (even more so than the Burst library). Please see attached screenshot.

    Screen Shot 2022-02-14 at 10.12.26 PM.png

    Using the profiler (which slows down everything), I end up with a ~17 second load times (vs 11 seconds when not using the profiler). Of the 17 seconds, 6 of that is the AddressableAssetSettings.OnEnable and 1.5 is the Burst onLoad.

    Is this something that you/team are aware of?

    EDIT:
    Well that's interesting. I forgot to turn on Deep Profile this time around, but it looks like for some reason the AddressableAssetSettings is trying to ping certain ip addresses. If I have my VPN turned on, I'm guessing it times out after 6 seconds. If I turn off my VPN, then that particular OnEnable is quick.

    Screen Shot 2022-02-14 at 10.29.44 PM.png

    Why is Unity/Addressables even pinging anything especially during an editor recompile? That seems wrong and unnecessary.
     
    Last edited: Feb 15, 2022
    Ksanone and oscarAbraham like this.
  32. FeastSC2

    FeastSC2

    Joined:
    Sep 30, 2016
    Posts:
    967
    (EDIT: I'll keep editing this thread as I find new interesting ways to reduce compilation time.)

    Addendum:
    - Here's an interesting article on the subject of domain reloads in Unity.
    It's a short read that mentions a lot of what is being talked about in this thread.
    (The author of the article mentions that with proper care for optimizations, anyone can get his project's compilation time to < 5s. This is not true: he has said so himself later on in this thread.)
    - Check this out if you're on Unity 2021. (This feature will come later to Unity 2019/2020 LTS)


    Here's what were the biggest impacts to reducing my game's compilation times on Unity 2020.3.23f1.

    - Remove Burst from your project (unity package manager)
    - Remove any text editor package that you don't use as your main script editor (Visual Studio, Visual Studio Code, Rider's etc...). These things actually eat up compilation time. Use one, delete the rest.
    - Odin Inspector adds up around 2s for compilation in my project. This was on Odin's newest release, v3.0.12. If you can't delete it because you need it, definitely upgrade it to version 3+.
    - PolyBrush was causing a big slowdown as well. From 1 to 2 seconds. I think it's because PolyBrush has an OnPostProcessAllAssets call, and I have a lot of assets in my project.
    - AStarPathfinding (v 4.2.15) takes around a 1s to setup its pathfinding visualization gizmos.

    In case those high compilation numbers are surprising to you: whatever unoptimized compilation time issues are in those packages, it scales with the amount of assets that are in a project.
    So if you make your tests in project with barely no assets like the 3D template, you won't see huge differences of compilation times by removing those packages. But if you add them in a real-sized project then you will start noticing similar numbers.
    This is quite nasty because it means that you can add assets to your project and even though you tested their viability at an earlier point... You might realize later that they're actually an unoptimized liability. But at that point you might have already built on top of them quite a few systems.


    Tips:
    - Any asset with [InitializeOnLoad] contributes to making your compile time longer, this REALLY adds up. Make sure that whatever code is in there is super optimized or better yet consider finding another way than using that attribute (I haven't looked into what another way could be yet).
    - OnPostProcessAllAssets: Be careful about any OnPostProcessAllAssets call in your project. Use the profiler to see if this is what's taking up compilation time.
    Test this out for your own project: you can easily comment all the InitializeOnLoad and OnPostProcessAllAssets in all your project's scripts (except for Unity's packages).
    - Use your profiler: Unity's packages are not all optimized for compilation. But this is also true for some of the asset stores's packages (obviously). Use your profiler to detect that.
    - Use a copy of your project and delete files for tests: Make a copy of your project and remove your files step by step to see what's causing compilation slowdowns.
    I think this method should be used in combination with the profiler.
    - Don't trust the profiler's ms too much: The profiler indicates slowdowns, but sometimes they're worse than what the profiler says, it's just a guide. Your real test is to remove those assets from your project and then compile 4-5 times to see how your compilation time has been modified.

    Unsolved:
    - If I have either the game view or the hierarchy window opened, I can see that AwakeInstancesAfterBackupRestoration takes 1s in the Profiler (in an empty scene).
    If those windows are not opened, AwakeInstancesAfterBackupRestoration only takes .2s.
    I haven't been able to pinpoint what causes this.
    In comparison, in a 3D template project, it takes .2s even even though these 2 windows are opened.

    My biggest surprise was the following:
    Adding scripts is NOT the only thing that makes compilation take more time.
    It appears that anything you add to your project, whether it's a mesh or a texture makes your compilation time slightly worse. These things add up but of course, you don't have much of a choice there.

    As a test, I added 30gb of assets to a brand new 3D template project.
    And after 1h20minutes of importing those assets... It changed the script compilation time from 2.8s to 4s even though those assets were not scripts, shaders or prefabs.


    Assembly definitions:
    If you, like me, thought that by isolating a script in a assembly definition with no dependencies you are going to get a quick compilation time as quick as if's a new project, it's not the case.
    Some people are saying that assembly definitions are not significant to reduce compilation times.
    This is because anything you add to your project contributes to buffing up the load time of ReloadAssemblies (you can search for that term in your profiler).
    Unfortunately, typically it's not recompiling assemblies that takes a lot of time, it is simply LOADING them. So whatever you do with your project to optimize it to have faster compilation times, this always will be your last (somewhat) insurmountable hurdle.
    But that doesn't mean that assembly definitions won't lower your compilation time or that they are useless.
    I, despite what's been said, have not observed that having many assembly definitions creates significant slowdowns. And if, by the way, that was the case, it would be a big problem because Unity's default packages have lots of assembly definitions.
    I think you should make assembly definitions in your project, it will help.
    If you still don't want to, at least put the scripts you're not using in your game code in a folder called "Plugins", it will also help.

    But to be clear, what the bottleneck of compilation time is, is the now infamous ReloadAssemblies which is not really affected by assembly definitions.

    Slow enter playmode?
    Compilation time is slow on Unity, especially on new versions of Unity. For now, and maybe forever from what I hear from certain unity developers?...
    So like me, you're doomed to wait for the long compilation to end.
    But this is not true for entering and exitting playmode.
    At least, you can enter playmode very quickly thanks to the Enter Play Mode settings that you can find in Project Settings -> Editor. This is THE most amazing feature in Unity, it takes around 1s to enter/exit empty scenes. It will make your entering play mode very quick BUT you need to change some of your scripts to be compatible with this feature (basically your static variables's values are not reset so you need to reset them yourself in a method with an attribute). There's info on other forum threads on how to achieve that.
    Here's the setting in your project: https://i.imgur.com/IdVVapw.png
    I also had at some point my enter playmode slowed down by this issue.

    Last piece of advice:
    If you're on a version of Unity that's prior to Unity 2020 and you care about compilation time, Stay on Unity 2019-! It will compile faster (at least until Unity developers come up with a solution).

    The future:
    If you still can't bear with the compilation times you have, mention it to the unity staff.
    Try and stay positive about it, they're hoomans like us.
    Speak up, repeat, email and more importantly: report bugs.
    Thanks to the ones doing that work.
    And thanks to the people @ unity working on who just might end up literally saving years of individual people's lives if they manage to significantly improve those compilation times. I'll be frank, right now it's a pain in the a**.

    My project's performance:
    The following times are for a rather big project, that will only get bigger unfortunately.
    I tested to see what would happen if I don't follow any optimization I listed above and it takes around 25-30s to compile.
    After optimization, on an isolated script in an isolated assembly, compilation takes 7s.
    And for my player's script it takes around 2 additional seconds to compile.
    So roughly speaking: I must wait around 10s every time I change a script.
    I wanted to get this process under 5s but it looks like it's not an option until Unity makes a change. I could not remove Odin because it's too integral to my project.

    Anyway...
    If you do those things I mentioned, it should make your compilation times tolerable. Not great, but tolerable.

    Good luck.
     
    Last edited: Apr 13, 2022
    PutridEx, Extrys, Lo-renzo and 14 others like this.
  33. NWHCoding

    NWHCoding

    Joined:
    Jul 12, 2012
    Posts:
    1,459
    I took the leap and upgraded to Unity 2022 beta 8 and the compilation times went down by about half, if not more. I am assuming that most of the optimizations mentioned here then are not backported yet to Unity 2021.3 as that was quite slow. Now I am happy with the compilation times and I have not noticed any increase in bug frequency (unlike the 2018/2019 betas which were borderline unusable).
    The compilation times getting longer and longer with each compile also went completely away.
    I have been on 2022 for about 10 days now.
     
    Lahcene, cxode, JoNax97 and 2 others like this.
  34. FeastSC2

    FeastSC2

    Joined:
    Sep 30, 2016
    Posts:
    967
    That's interesting.
    I updated my project to version 2022.1.0b8 to try this out.
    However, my compilation times are in the same range as Unity 2020.3. In fact, they are a bit worse. But I never had your issue of longer compilation times on consecutive compilations.
     
    Last edited: Feb 21, 2022
  35. RobertOne

    RobertOne

    Joined:
    Feb 5, 2014
    Posts:
    205
    I was excited when I red NWHCodings post but also can't confirm that at all.
    My Compile times are between 2.8 and 2.6 seconds for a single script on a fresh and empty unity project there. (all packages removed except unity UI, textmeshpro and visual sudio editor, running an intel core i9-9900K)

    2018.4.36 : 1.3s - 1.4s
    2019.4.36 asset database v1 : 1.5s
    2019.4.36 asset database v2 : 1.9s
    2020.3.29 : 1.6s
    2021.2.12 : 2.4s - 2.6s
    2022.1.0b8 : 2.6s - 2.8s

    I honestly think 2020 is totally fine, and if 2020 would be available for m1 macs, i wouldn't care at all about this topic. I don't think we will ever come back to unity 2019(asset db 1) or even 2018 speeds. but those 2021/2022 numbers are really worrying me.
     
    Last edited: Feb 21, 2022
    hippocoder, Lahcene and sameng like this.
  36. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    391
    @xoofx @Maria_Angelova

    Could the Visual Studio Editor package remove its runtime (non-debug) dependency on the Unity Test Framework which would then also remove a dependency on Custom NUnit?
     
  37. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Yes please, I don't use those either.
     
  38. cxode

    cxode

    Joined:
    Jun 7, 2017
    Posts:
    256
    As discussed extensively in this thread, "a single script on a fresh and empty unity project" is a VERY different situation from real-world projects with a lot of assets. Your results do not contradict @NWHCoding's results.
     
    hippocoder likes this.
  39. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    I am now on latest 2022.1 beta and it's reloading from script changes a lot longer than my old version of Unity. I think it's a domain reload and not script compilation bottleneck somewhere.

    Time to delve into the profiler and so forth. Thanks for the advice given in this thread. Seems pretty common if a project like mine gets it (I tend to keep architecture super simple and direct).

    I found that it was faster removing asmdefs from my own project... and that compilation (possibly domain reload) takes longer as time goes by.
     
  40. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    9,628
    Even with the numbers Unity has posted in the previous page, it seems there is an obvious trend that every major Unity version is getting slower.
     
  41. sameng

    sameng

    Joined:
    Oct 1, 2014
    Posts:
    151
    It's true, an empty project is a very different situation. Adding any assets into your project will make it slower.
    It's very clear to me that the editor is noticeably taking longer for assembly domain reload every major version.

    I also tried the latest 2022 version on my project.
    I had high hopes. Even if it kept speed parity I would be happy.

    But 2022 is even slower than 2021 for me.
     
  42. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    9,628
    upload_2022-2-22_23-11-55.png
    Going from major version to major version, there is only a single metric that has improved, and everything else regresses up to ~40% per major version.

    If they included even earlier versions (2019, 2018) I bet the recent numbers would look even worse.
     
    Last edited: Feb 23, 2022
  43. Trindenberg

    Trindenberg

    Joined:
    Dec 3, 2017
    Posts:
    304
    Is there a way to find/disable all the background trackers/loggers? It would be nice to see how fast things run without those. How to make the editor release mode over super debug mode? If a million things happen on reload and most of it is logged, how much time is spent logging? Writing strings is not fast.
     
  44. NWHCoding

    NWHCoding

    Joined:
    Jul 12, 2012
    Posts:
    1,459
    It would help a lot if multi-threading was a norm with Unity. Looking at my 12C/24T CPU it never gets above 20% while doing pretty much anything in Unity. And meanwhile while it is pretty much idling I have to wait 10 seconds for things to finish up.
    The whole refresh thing is a few of the spikes on 3 cores, the rest are just doing nothing:
    upload_2022-2-24_19-4-56.png

    Since the 6+ core CPUs are slowly becoming the norm, maybe it would be a good time to try and start giving some focus to multi-threading. I know it is not straightforward but Unity in general has not been designed with multi-threading in mind and it feels, unfortunately. Still, gets me mad when i have to wait and CPU is at 15%.

    From the images aboe I would conclude that the best hardware for Unity is a quad core CPU with 5+ GHz clock for the most part.
     
    Lahcene likes this.
  45. RobertOne

    RobertOne

    Joined:
    Feb 5, 2014
    Posts:
    205
    yeah, the best compile times ive seen so far where with an i9 12900hk. Strong Singlecore and a lot of threads they say. My surface pro (i7) beats my asus g14 (ryzen 9) in almost nothing, except unity compile times :D

    i think thats due to the roslyn compiler that unity uses and thats a singlecore compiler. But i could be totally wrong tho
     
  46. amiel_ace

    amiel_ace

    Joined:
    Oct 25, 2014
    Posts:
    16
    Unity is getting even more and more broken.:)
     
  47. wheee09

    wheee09

    Joined:
    May 21, 2018
    Posts:
    68
    To quote @Maria_Angelova in a response earlier (on this page), she said:
     
  48. NoTuxNoBux

    NoTuxNoBux

    Joined:
    Oct 2, 2020
    Posts:
    25
    Just to add two cents, but... does it? There are undoubtedly things that absolutely have to run on the main thread, but Unity is currently not particularly friendly towards having multiple threads either, which is why coroutines also exist. Working with TAP in .NET and running asynchronous tasks on separate threads this way, even doing a fetch of a constant such as
    persistentDataPath
    fails hard with Unity complaining that you cannot access it outside the main thread. There seems to be no real reason for this, either, it's just a read of an immutable field (?), but the side effect is that code on other threads now have to wait for a main thread task to be able to read it, or expect it up front somehow - not particularly a great benefactor to concurrency.

    I'm not mentioning this out of grievance, but perhaps there is potential to restructure Unity or its APIs to better support such multithreaded flows and parallelism? Concepts such as immutability and monads that avoid state (.NET's Task is such a beast, similar to promises or futures in C++) might help in making code more friendly to multithreading internally.

    To give a basic example, things such as file reading can happen asynchronously in most languages nowadays using Tasks in .NET, which means Unity could spend its time doing other things whilst waiting for that to happen - of course this is only possible up until a point; processes dependent on the read cannot proceed without the read having finished.

    Hope this helps somehow.
     
  49. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    391
    @NoTuxNoBux Unity is generally good at rewriting systems from scratch but the process of slowly improving a feature over time is an area they seem to struggle with. Developers are very talented at explaining why a system must be rewritten -- we all know the thought process here.

    In Unity's case, I think there is much to be gained from slow evolution. Imagine if you could upgrade to the new scriptable pipeline without having an entire project "break"? This kind of engineering mindset is challenging but also an area of growth for the Unity teams.
     
  50. TheVirtualMunk

    TheVirtualMunk

    Joined:
    Sep 6, 2019
    Posts:
    127