Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

2019.3 slow build times

Discussion in 'Editor & General Support' started by CPlusSharp22, Jul 11, 2020.

  1. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    In editor script recompiling is not terrible for me but compiling out a binary is getting horrific.

    I make one line of code change and want to test on Windows standalone and/or Android and/or Linux? We are talking about 5+ minute compile times and the unity editor in complete frozen with no helpful status/progress indicators. Even if I enable the script only debug build, it shows minimal if any improvement.

    In my project, not everything can be tested in editor. You need to run multiple standalone and/or on devices. Besides performance problems in editor, you need multiple instances for proper development.

    What can I do to profile this issue better and address them? If I search around for improving build speeds, I see stuff like the assembly definitions but I'm not convinced the script compiling is the problem.
     
  2. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,193
    Depending on the size of your project, it's easy for build times to be several minutes. Fast iteration on builds really isn't viable except for the most trivial projects.

    However, you should look at the Editor log after performing a build. There should be some timing metrics in it. Typically, one huge expense in creating a build is compiling shader variants. It's possible that Unity is including more shader variants than you strictly need, and perhaps you could cut down on the number of variants to speed up your build.

    Another idea is, if you're iterating on testing something, you could create a build that just builds one scene, or a small number of necessary scenes, so you can quickly build just what you need to run the thing you're trying to test. I have a few build scripts for building smaller portions of my project, which usually just means passing a different list of scenes into the build method.

    Also note how much stuff you've placed into the Resources folder(s). Anything there will always be included in a build, even if it's not used by the scenes you're building. If you have a lot of stuff in Resources, that will increase the size of your build, potentially for assets that aren't even used by that build.

    Using Assembly Definitions probably won't make much of a difference for builds. That's mainly for quickly iterating within the editor when making code changes.
     
  3. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,589
    Is this with or without using the "Scripts Only Build" option?

    Switch from IL2CPP to Mono on platforms where it's supported. The turnaround times with Mono are significantly faster.

    From the documentation:
    Enable this setting to only rebuild the scripts in your application, and reuse data files from the build you previously executed. Before you can use this setting, you must build your whole Project once. The Scripts Only Build option significantly improves iteration times if you only changed the code in your application. This setting is only available if you enable Development Build.

    upload_2020-7-11_19-18-12.png

    https://docs.unity3d.com/Manual/BuildSettings.html
     
    dgoyette likes this.
  4. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,193
    I'm not sure how often I make only code changes between builds (as opposed to changing something within a scene), but I never knew about this feature. Definitely a nice trick I'll keep in mind.
     
    Peter77 likes this.
  5. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    I use Script Only Build option and Mono. The former makes no difference in speed unfortunately. I've timed it and at best it saves 2 seconds, maybe just placebo. Not sure why.

    Alot of time is spent on doing stuff like cubemaps for some reason. It's impossible to tell since the editor is basically broken while compiling, can't be touched and does nothing but flicker text boxes.

    I wish we could profile this stuff better. Using the Editor Profiler, it just stops until you finish or miraculously hit the flickering "cancel" button on something that actually stops the build.
    At the end of a build, for whatever absurd reason, it's doing stuff like "AssemblyHelper.ExtractAssemblyTypeInfo()" which allocates over 4GB in a frame. Pumps the frame to over 400,000ms locking the editor.
     
  6. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,193
    Does the log file tell you anything about the timing? It should at least contain statements about how long it took to compile various shaders. I can't recall whether the log file contains any other timing information that might shed some light.
     
  7. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    Direct comparion
    Options: StripDebugSymbols, CompressTextures, ForceOptimizeScriptCompilation, Il2CPP, Development, AllowDebugging, CompressWithLz4

    The first 4 options seem to be required by unity at all times.

    WITHOUT BuildScriptsOnly:
    -- 1/13 Build player 39s --
    -- 2/2 Compile scripts 54s --
    -- 3/13 Prepare assets for target platform 21s --
    -- 4/13 Building scenes 17s --
    -- 5/13 Building scene Assets/Scenes/bootstrapper.unity 17s --
    -- 6/13 burst 0s --
    -- 7/13 Build GlobalGameManagers file 0s --
    -- 8/13 Writing asset files 17s --
    -- 9/13 Packaging assets - globalgamemanagers.assets 0s --
    -- 10/13 Packaging assets - resources.assets 2s --
    -- 11/13 Packaging assets - sharedassets0.assets 14s --
    -- 12/13 Creating compressed player package 1s --
    -- 13/13 Postprocess built player 29s --


    WITH BuildScriptsOnly:

    -- 1/5 Build player 53s --
    -- 2/5 Compile scripts 48s --
    -- 3/5 Prepare assets for target platform 21s --
    -- 4/5 burst 0s --
    -- 5/5 Postprocess built player 25s --

    Build player, assets, and post processor take *forever*.
    And yes, this is mono.

    Compile scripts at 48s isn't too terrible. But these two are offenders.
    - Finished compile Library/PlayerScriptAssemblies/Unity.Serialization.dll in 8.663926 seconds
    - Finished ILPostProcessor 'Unity.Entities.CodeGen.EntitiesILPostProcessors' on Assembly-CSharp.dll in 30.005564 seconds
    - Starting ILPostProcessor 'Unity.Properties.CodeGen.PostProcessor' on Assembly-CSharp.dll

    If I disable CompressWithLz4, it's an even longer build time so it's not that either.
     
    Last edited: Jul 12, 2020
  8. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    I shaved off 40 seconds by going into Wwise and finding their "AkBuildPreprocessor" script. In the OnPreprocessBuildInternal and OnPostprocessBuildInternal, if I skip the stuff they do entirely by returning early my results are:

    1/5 Build player 19s --
    2/5 Compile scripts 48s --
    3/5 Prepare assets for target platform 17s --
    4/5 burst 0s --
    5/5 Postprocess built player 8s --

    Of course, this stinks for audio programmers and designers, but this is something we can tinker with hopefully.

    Still a bummer about those compile times. I dont know what to do about "EntitiesILPostProcessors"
    There's also at least a minute of time not accounted for in those steps 1 to 5. Just not sure where.
     
  9. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,193
    Those compile times seem really high to me. I just tried doing a build, just for comparison. Hard to say how much code I have, but the whole project is in the 10-15 GB range, with plenty of custom code. I was getting 10 seconds or so for the script compilation, however, I was getting it twice. I don't know why that is. But still, that's a lot less than what you're seeing.

    Some simple things to make sure you're doing:
    • Build to an SSD if possible? That is, the directory where the build ends up should ideally be on an SSD.
    • Make sure your anti-virus isn't scanning that folder. I don't know how much of a difference that makes.
     
    CPlusSharp22 likes this.
  10. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    Thanks for sharing your times.
    SSD and antivirus has the whole drive excluded. It's also a beefy machine, 64GB with 12 threads.

    I'm not liking this codegen thing for Entities. Wherever it shows up is a stall during the compile.
     
  11. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    I've attached what my build output has in case anyone has any ideas by the script compile part takes so long. I know that the idea is you can use asmdefs to break down Assembly-CSharp, but you can see here there are other things like Unity.Entities and Unity.Mathematics.Extensions.Hybrid taking longer...

    EDIT:
    Found this cool tool that kind of organizes it a bit rather than looking at that dump of text.
    https://forum.unity.com/threads/introducing-the-editor-iteration-profiler.908390/

    So the top 3 libs - math, coffee, asmplugins don't need to be recompiled everytime. Any way to cache them?

    Unity.Mathematics.dll
    6,904.000 ms

    Coffee.UIParticle.dll
    6,600.000 ms

    ASMPlugins.dll
    6,554.000 ms

    Assembly-CSharp.dll
    6,066.000 ms

    Unity.TextMeshPro.dll
    4,047.000 ms

    Unity.Timeline.dll
    4,001.000 ms

    Unity.Entities.dll
    3,966.000 ms

    Unity.InputSystem.dll
    3,905.000 ms

    Unity.ResourceManager.dll
    3,804.000 ms

    Unity.Burst.dll
    3,703.000 ms
     

    Attached Files:

    Last edited: Jul 13, 2020
    MartinTilo likes this.
  12. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    There must be a way to avoid recompiling stuff that never changes. Like Unity.Mathematics or this Cofee.UIParticle.dll I have. Code in them never changes. I understand having to recompile if I do a rebuild or switch targets, but doing a scripts only build ontop of a full build should be smart. There must be something I can do, I just cant find the right search terms I guess.

    ASMPlugins is in Assets/Plugins/Wwise folder

    Unity.Mathematics ends up in the Library\PackageCache folder
     
    Last edited: Jul 13, 2020
  13. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,193
    Well, what I wasn't sure about your log was whether each of those compile time values truly the elapsed time compiling just that one DLL, or basically the "time" at which it finished compiling that DLL, even if it started just a few ms before it finished. It seemed like a bunch of the compile times were very similar, and roughly increased in value, making me think it was more like just the time it completed the compilation. But I don't know.
     
    CPlusSharp22 likes this.
  14. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    Yeah it's hard to tell. I also tried moving things like Coffee into the plugins folder which is not supposed to recompile Everytime but definitely does.
     
  15. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
  16. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
  17. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
  18. emrys90

    emrys90

    Joined:
    Oct 14, 2013
    Posts:
    755
    I'm trying to figure out how to improve this to. I want to be able to do quick iteration of script only builds. Right now it takes me 20 seconds, with most of that recompiling assembly definitions that didn't change.
     
  19. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    I unfortunately gave up and left Unity. Their libraries like Unity Mathematics and other packages were killing all of my productivity. I am now much happier in C++ land.
     
    IgnisIncendio and junaid109 like this.
  20. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,589
  21. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111

    Yes, that is what they told me in a support ticket. Unfortunately too experimental, I needed LTS support in 2019 and LTS for 2021 won't be here for a long time sadly. Shame!
     
    CrandellWS likes this.
  22. Minos321

    Minos321

    Joined:
    Jul 7, 2018
    Posts:
    29
    I upgraded from LTS 2020 to 2021 and my build times went from 2 minutes to 15+ minutes. Deleting the "Library" folder, and turning off compress mesh data, brought it down back to 2 minutes after the initial build ;)
     
  23. pistoleta

    pistoleta

    Joined:
    Sep 14, 2017
    Posts:
    539
    We just upgraded to 2022.3.0 and the android I2LCPP builds are taking me more than 30 minutes, maybe 40.
    With the same machine, iOS I2LCPP builds take 4.
    Something is really wrong here.