Search Unity

  1. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice

Unity Incremental script compilation

Discussion in '2021.1 Beta' started by jonas-echterhoff, Dec 14, 2020.

  1. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,647
    In Unity 2021.1, we have rewritten the script compilation pipeline in the editor (ie, how we invoke the Roslyn csc compiler to compile all your C# scripts into .NET assemblies) to use an incremental build system. You can use this thread to discuss this change or report any issues with it.

    Incremental script compilation means that we now have a proper build system which drives script compilation, which can track inputs of and dependencies between individual build nodes, and thus can determine what actually needs to be rebuild when making a build - and cache build results for different platforms and configurations so that less rebuilding is needed.

    In practice, this means:
    -If you change a script in an asmdef, only that asmdef or any code depending on that asmdef gets recompiled.
    -If you switch between platforms, or between debug/non-debug mode, only code which has not yet been compiled in that configuration before gets recompiled.
    -If you build a player, scripts only get recompiled if they have changed since the last player build of that configuration.
    -You get better progress reporting of script compilation happening in the background.

    Some small API changes related to this:
    -ScriptCompilerOptions.EmitReferenceAssembly has been removed, since this is now always enabled
    -CompilationPipeline.assemblyCompilationStarted has been deprecated. Its main use was to profile script compilation. Since script compilation is now happening out of process, we cannot invoke these exactly when compiling an individual assembly starts or finished, so you will not get useful results for profiling any more.

    This is the first step in a bigger project to move player builds to an incremental build system as a way to avoid redundant work and make building faster that way.
     
    glenneroo, _met44, Thaina and 20 others like this.
  2. DoctorShinobi

    DoctorShinobi

    Joined:
    Oct 5, 2012
    Posts:
    156
    -If you change a script in an asmdef, only that asmdef or any code depending on that asmdef gets recompiled.

    Wasn't it always like this? I thought that was the point of using asmdefs when they were introduced
     
  3. JoNax97

    JoNax97

    Joined:
    Feb 4, 2016
    Posts:
    459
    I remember something about an assembly triggering compilation of other assembies only if the public API changes. Is that part of this?
     
  4. Ramobo

    Ramobo

    Joined:
    Dec 26, 2018
    Posts:
    199
    From what I've heard, yes. In practice, that horrendous freeze increased for me. Compilation is what, less than 10% of the freeze? It's domain reload that's the issue. {Unity .NET Core rant}

    EDIT: In other words, does this compilation pipeline rewrite really matter right now?
     
    Last edited: Dec 15, 2020
    phobos2077, OndrejP and Kolyasisan like this.
  5. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,647
    You are correct - sorry - I picked an extremely bad example of something which works in the new script build pipeline, as this exact case would have also worked in the old one.

    So: yes - the old script build pipeline had a crude form of dependency tracking, which could handle this exact case of changing a script in an asmdef and then only triggering recompilation of dependent asmdefs. However, here are some things it could not handle (which the new one can):

    -Not recompiling all asmdefs when changing asmdef structure
    In the old pipeline, whenever you changed an asmdef setting or added a new asmdef, etc, all asmdefs would be recompiled. In the new pipeline, we can track what asmdefs might actually be affected by the change and only do those.

    -Not recompiling dependent asmdefs if recompiling an asmdef yielded no relevant change
    If you make a change to an asmdef, which does not change the api surface of the generated dll, dependent asmdefs will not need to be recompiled.
     
    Last edited: Dec 16, 2020
  6. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,647
    Domain reload is still an issue, and is not affected by this. And, no, .NET Core would not magically fix that either.

    Compilation is actually not 10% of the freeze time - it often takes much longer than the domain reload freeze, but it is less painful, since it is asynchronous (and domain reload blocks the editor).

    By itself, this change here will likely not hugely impact your workflows. But it is part of a longer-term initiative to make player builds incremental to give you a much better iteration workflow on builds.
     
  7. Ramobo

    Ramobo

    Joined:
    Dec 26, 2018
    Posts:
    199
    What happened to AssemblyLoadContext? Is there some sort of misunderstanding regarding what it can do?
     
  8. pvloon

    pvloon

    Joined:
    Oct 5, 2011
    Posts:
    590
    This is great news, very much appreciated to see Unity push hard on iterative workflows! I miss Unity 4.3 days where, nostalgia or not, everything felt snappier, a few more of these and Unity will feel snappier than ever :)
     
  9. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,647
    Depends on what your understanding is :)

    AssemblyLoadContext might be a piece of the puzzle for solving this. It allows loading assemblies into separate AssemblyLoadContexts, which can then be unloaded/reloaded individually - which we might be able to use for building a setup where we can reload assemblies at a smaller granularity than everything all at once.

    But: ALC unloading depends on the code being unloaded cooperating - and no other code may hold any references to objects belonging to code being unloaded. Which is not easily enforceable for generic user code (See here for some deeper info if you are interested: https://docs.microsoft.com/en-us/dotnet/standard/assembly/unloadability). We have been considering building some logic around this using a custom CoreCLR runtime, which would help breaking down any such references preventing unloading, and giving warnings which would help users design code with unloadability in mind.

    So, in summary: Yes, .NET Core and AssemblyLoadContext will likely play a part in future improvements to domain reload speeds. But, to manage expectations: No, this will not be a matter of "just switch to CoreCLR" and it will be fast - this will require more R&D (with uncertain outcomes) on our side, and possibly also require projects to be designed with this in mind to take advantage of it - that's what I meant by ".NET Core will not magically fix it" :) .
     
  10. PassivePicasso

    PassivePicasso

    Joined:
    Sep 17, 2012
    Posts:
    92
    You stated that CompilationPipeline.assemblyCompilationStarted, and implied that CompilationPipeline.assemblyCompilationFinished will be deprecated.
    Does this also affect CompilationPipeline.compilationStarted and CompilationPipeline.compilationFinished?

    If so this is a problem for a tool I'm working on, I specifically need to know when compilation completes so I can overwrite certain files.
     
  11. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,647
    No, those will still work.

    What do you want to overwrite and why, though? It is useful to understand such needs for potential future changes in this area.
     
  12. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    724
    Do u have an ETA when this domain reload only the assembly that has code changes? I think I already asked this question quite a few times but until now this feature still haven ship yet.
     
  13. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    595
    What about incremental script compilation for Burst? Couldn't static methods that are completely independent be recompiled safely even in runtime? That'd be similar to Unreal LiveCoding feature.
     
    Jes28 likes this.
  14. jdtec

    jdtec

    Joined:
    Oct 25, 2017
    Posts:
    272
    Will any of these changes affect editor iteration speeds or is it only standalone that benefits at the moment?
     
    The-Oddler likes this.
  15. PassivePicasso

    PassivePicasso

    Joined:
    Sep 17, 2012
    Posts:
    92
    I've been working on a tool to assist in modding unity games that just helps sets up the environment for a given game. In Order to do this, after compilation completes, I need to overwrite files in ProjectRoot/Library/ScriptAssemblies in some cases, such as Assembly-CSharp and Assembly-CSharp-firstpass in order to ensure that the libraries correctly get loaded and are usable in the unity environment.

    Obviously this is a giant hack, and its for type of work that is largely giant hacks.... so take that as you will.

    Edit: Just wanted to add that, this obviously does cause various problems. For Example, If you create a script in Assets which isn't managed by an AssemblyDefinition then it would get compiled by default into Assembly-CSharp, but since that gets overwritten the script isn't loaded correctly and Unity throws various errors.
    This is honestly expected, you also run into issues with various assemblies still not being loaded correctly. This is more or less acceptable since the goal isn't to allow someone to completely boot up a deployed game inside unity, some imperfection is warranted, and possibly even wanted.
     
    Last edited: Jan 12, 2021
  16. amisner2k

    amisner2k

    Joined:
    Jan 9, 2017
    Posts:
    38
    My feedback so far is that this popup modal:
    upload_2021-2-10_20-5-10.png

    needs to go away and the stuff it's reporting on should be relegated to the Background Tasks window as soon as possible. However you guys manage to solve this, it should be a priority because it's a very stifling experience iterating on script changes currently.

    The Flax 1.0 engine editor shows its script compilation progress in the bottom-right status bar (like Unity 2021.1 does) but they don't block their editor. Best I can tell is they just prevent entering play mode until script compilation is complete but their editor remains responsive to input.

    I do notice overall that the time it takes to compile a simple edit to one script file in my current project reduced by a noticeable amount. It wasn't a lot but it was something. So this is promising, just please......get rid of that modal!

    Thank you.
     
    kjyv, m0guz, florianhanke and 4 others like this.
  17. jdtec

    jdtec

    Joined:
    Oct 25, 2017
    Posts:
    272
    I would love it if they could get rid of the stalling on script change completely too. When I first started using Unity years ago I was kind of surprised that it did that, was very jarring.

    On a positive note I have noticed in the recent betas that the 'compiling scripts' is now no longer locking up the editor and appears to show as a progress bar in the bottom right corner. Then the domain reload (I assume) happens after, which locks up the editor for a few seconds. Slightly better experience than before. Maybe I'm only noticing this more now that I manually trigger the refresh...

    I think I remember hearing before that the domain reload UI lock-up is a fundamental part of how Unity / the editor work and the isn't going away anytime soon. I guess it needs to do it if you have made script changes that affect the editor or something? [shrug emoji]
     
    amisner2k likes this.
  18. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,391
    Yeah, it's pretty fundamental. Since the editor UI can be driven by user scripts, they have to reload the UI when user scripts change.

    I'm not sure if it'd be possible to skip reloading the domain the editor lives in. Say your inspector is showing the script you just changed, and that script has a method that's called by a custom inspector's code. When you reload your script's assembly, what does that inspector do?
     
  19. mahdi_jeddi

    mahdi_jeddi

    Joined:
    Jul 18, 2016
    Posts:
    145
    One solution would be not to reload the UI if the serialized properties or editor related scripts are unchanged. I think most people don't change these that often. But I guess that would be complicated to implement.

    They already did something similar for not recompiling other Assemblies if the referenced assembly's public API didn't change. I guess @jonas-echterhoff would have a better idea of how complicated this would be implement, if at all.
     
  20. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    595
    I thought it would be easy to implement using Burst Modding support in 2021 https://docs.unity3d.com/Packages/com.unity.burst@1.5/manual/docs/ModdingSupport.html but it requires domain reload meaning that it can't be done in runtime :/
     
  21. amisner2k

    amisner2k

    Joined:
    Jan 9, 2017
    Posts:
    38
    If it's true that the domain reloading is only necessary to reflect changes to exposed script variables in the inspector or editor windows, then an idea would be to provide a sort of "Refresh" or "Reload Domain" button that's easily visible and pronounced, that when clicked, Reloads the Domain (make it show a slick animating progress popup modal right away and dim the editor behind it).... That way (hopefully) coders can make quick simple script changes, hit Play, and see results without needing to reload the domain.

    If you *do* make a script change that affects an exposed or serialized variable, then you can reload the domain when you want to instead of being always forced to wait.

    Obviously still not ideal, especially for new users learning Unity who won't know that they have to click "Reload Domain" in order to see the new property they want to show up in the inspector, but......if this could be provided as an advanced option (since it's for advanced users) I feel this could go at least some way towards speeding up the iteration process.
     
  22. jdtec

    jdtec

    Joined:
    Oct 25, 2017
    Posts:
    272
    Yep I like the idea @amisner2k suggests.

    ~98% of my DOTs based coding has nothing to do with inspector/editor script values nowadays anyways.
     
  23. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    133
    Does this mean it would be possible to avoid recompiling if the C# doesn't affect the generated IL? ie. The file change is whitespace only?
     
unityunity