Search Unity

  1. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice

Unity IL2CPP player build time improvement coming to 2020.2.0b2

Discussion in '2020.2 Beta' started by mike-voorhees, Sep 3, 2020.

  1. mike-voorhees

    mike-voorhees

    Unity Technologies

    Joined:
    Aug 9, 2016
    Posts:
    26
    The process of converting .NET assemblies into C++ files has historically been single threaded. The il2cpp.exe utility performs this task.. You may have noticed the il2cpp.exe process churning along using a single core.


    In the fall of 2019 we began a journey to make il2cpp.exe take advantage of all the cores your machine has available. Over the last year we have steadily shifted il2cpp.exe toward becoming a multi-threaded application.


    Starting in 2020.2.0b2 the process of converting .NET assemblies into C++ files is multi-threaded.


    For 6-8 core machines the conversion process will see roughly a 60-65% decrease in conversion time. On a 64 core machine conversion time decreased by 73%.


    It is hard to generalize exactly what improvement you will see in the overall il2cpp player build times. There are many variables that contribute to the total player build time such as C++ compiler settings, your CPU, and more.


    Broadly speaking here is what you can expect to see


    • Clean player builds will have the least perceived improvement. This is because compiling C++ files dominates the total time.

    • Incremental player builds will see the most perceived benefit because there are fewer C++ files to compile.
     
    lclemens, Arowx, hippocoder and 26 others like this.
  2. charlesb_rm

    charlesb_rm

    Unity Technologies

    Joined:
    Jan 18, 2017
    Posts:
    471
    Please note that 2020.2.0b2 is currently in QA and will only be available next week (around Sept 9th)
     
    transat, dzamani and GliderGuy like this.
  3. LINYAN999

    LINYAN999

    Joined:
    Dec 4, 2018
    Posts:
    1
    2019lts can use?
     
    Joc36 likes this.
  4. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    193
    This is great news. 2020.2 looking even better!
     
  5. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    265
    Unfortunately no, this will only be in 2020.2+.
     
  6. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    389
    Are C++ files regenerated for all assemblies all the time or only the assemblies that changed?
    If it is regenerated for all, do you plan to improve this?
     
  7. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    2,977
    Is this also happening for mac or is this pc only? asking because the talk is about a .exe
     
  8. mike-voorhees

    mike-voorhees

    Unity Technologies

    Joined:
    Aug 9, 2016
    Posts:
    26
    > Are C++ files regenerated for all assemblies all the time or only the assemblies that changed?

    If 1 or more assemblies are changed, all C++ files are regenerated.

    > If it is regenerated for all, do you plan to improve this?

    It's on our radar for the future improvements.

    > Is this also happening for mac or is this pc only?

    This applies to all platforms supported by the Editor. So yes it applies to mac.
     
    bdovaz and Lars-Steenhoff like this.
  9. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    265
    codestage and Lars-Steenhoff like this.
  10. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    2,977
    Great will check it out with my quad core macbook pro
     
  11. Shinyclef

    Shinyclef

    Joined:
    Nov 20, 2013
    Posts:
    421
    Oh looking forward to this. I don't have cloud build or anything so this might move IL2CPP back into the realm of iteration viability. We'll see, but I hope so :D.
     
  12. KamilDA

    KamilDA

    Joined:
    May 21, 2020
    Posts:
    94
    I assume that this affects WebGL Build times as well?
     
  13. andyz

    andyz

    Joined:
    Jan 5, 2010
    Posts:
    1,698
    Sounds great, though why this slowww process was not making use of multi-threading until now I can only guess!
     
  14. Schubkraft

    Schubkraft

    Unity Technologies

    Joined:
    Dec 3, 2012
    Posts:
    977
    Yes.
     
    stonstad and KamilDA like this.
  15. KamilDA

    KamilDA

    Joined:
    May 21, 2020
    Posts:
    94
    My build times went from >3 minutes to 2:40 minutes (or 19 seconds when no code change). Nice!

    Observation... when looking at CPU utilisation during IL2CPP conversion my CPU is barely utilized by 10% (R9 3900x 12c/24t). So I do have build time improvements but I don't think it's from that specific feature. Is it supposed to heavily multi-thread the process or is it light?
     
  16. mike-voorhees

    mike-voorhees

    Unity Technologies

    Joined:
    Aug 9, 2016
    Posts:
    26
     
    KamilDA likes this.
  17. gtk2k

    gtk2k

    Joined:
    Aug 13, 2014
    Posts:
    137
    In my environment WebGL build time.
    >2019.4: 105min
    >2020.2: 25min
    (CPU core: 6)
    And it is about 3 times faster.
     
    tonialatalo and Lars-Steenhoff like this.
  18. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    2,977
    If only the texture compression part could be made faster?
    Only compress textures needed for the build and not all
     
  19. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,584
    Isn't Unity only building assets that end up in the build already? Can't you use the "Compress assets on import" option in the preferences to basically have zero asset compression time in the editor?
     
  20. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    2,977
    Compress on import compresses all the textures in the project, ( making the cache data base very large ) and when switching platforms it needs to compress all the textures for that platform, making switching platforms taking a long time.

    I would like to make a build and then unity checks what textures are needed and only compresses those, saving disk space and time.

    And ideally do it on the GPU to speed up compression time.

    Right now it can take many hours and gigabytes to compress all the textures in my project but I don't even need them all compressed



    Its being talked about in this video, I'm just waiting to hear more about it
     
    Last edited: Sep 20, 2020
  21. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,584
    I meant to turn "compress on import" off, so unity only compresses assets during a build and hopefully only assets that actually end up in the build. The platform switch reimport issue has been fixed in asset database v2 btw.
     
  22. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    2,977
    Yes I have it turned off, I think its still compressing all the textures in the project when making a build, and not just the ones needed. I may be wrong, and something changed recently?
     
  23. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    8,352
    Should only compress the textures that are actually used. Try building just an empty scene to confirm.
     
    Lars-Steenhoff likes this.
  24. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    2,977
    I made an empty scene with just one 8k resolution texture for the skybox.
    The project has 12 skybox textures at 8k each.

    Screenshot 2020-09-20 at 22.18.01.png
    It seems to compress cube-maps for each of the skyboxes while I'm using only one in the the scene.
    In the video you can see the compressing when I press build.
    So it seems to compress at least the cube-maps that are not needed.

    not sure about the textures I need to make another test for that.

     
    Last edited: Sep 20, 2020
    Peter77 and JoNax97 like this.
  25. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    2,977
    Last edited: Sep 20, 2020
  26. ScottyB

    ScottyB

    Joined:
    Apr 4, 2011
    Posts:
    141
    This is great news, thank you very much for working on IL2CPP improvements!

    I have one request for this new feature: would it be possible to provide a way to limit the number of threads that IL2CPP uses instead of using all cores? Something to set in the environment variable
    IL2CPP_ADDITIONAL_ARGS
    or via an editor script calling
    PlayerSettings.SetAdditionalIl2CppArgs
    perhaps?

    We run most of our builds on build servers that often have multiple builds running at the same time and having multiple jobs trying to use all cores available at any one time can have negative implications for all jobs running.
     
    Foriero likes this.
  27. mike-voorhees

    mike-voorhees

    Unity Technologies

    Joined:
    Aug 9, 2016
    Posts:
    26
    A player build involves running a number of processes. Some of them are parallelized, some are not. Some processes support command line arguments to control how many threads they use or processes they spawn, some do not. Sometimes those arguments work for everything that is parallelized, sometimes only parts.

    Truly controlling how many cores are used during a player build and presenting that in a user friendly way would be a moderate undertaking. It's something we could consider in the future.

    Compilation of the generated C++ has always been parallelized. Have you been doing something custom to control the core usage during compilation?


    -Mike
     
  28. ScottyB

    ScottyB

    Joined:
    Apr 4, 2011
    Posts:
    141
    I understand that doing proper control over how many cores are being used and exposing that in an easy to use nature can be difficult. As a first easy step it would be nice to add the functionality per build step and then just expose it as an environment variable so that advanced users who want the feature can opt into it if needed.

    I know that retrofitting this functionality into already exisiting build steps can also be difficult. I just wanted to raise this point as I see the feature being developed for new functionality in the hope that it an be added to new updates going forward.

    Having control over this with existing Unity build steps is hit-or-miss. With our automatic build set up, we try to take as much control away from Unity and do things ourselves where possible, so that we have as much control as possible. So for example, we don't get Unity to build Xcode or Gradle projects, but instead get Unity to export those projects and then we build them ourselves.

    When building Xcode projects ourselves using xcodebuild, we're able to pass in a -jobs 4 command line argument to set how many build jobs Xcode will spin up to reduce core count usage. Or we can set the following global settings to limit build job count for all builds run on the build machine:

    defaults write com.apple.dt.xcodebuild PBXNumberOfParallelBuildSubtasks 4
    defaults write com.apple.dt.xcodebuild IDEBuildOperationMaxNumberOfConcurrentCompileTasks 4
    defaults write com.apple.dt.Xcode PBXNumberOfParallelBuildSubtasks 4
    defaults write com.apple.dt.Xcode IDEBuildOperationMaxNumberOfConcurrentCompileTasks 4


    Going through and finding the right settings for each compiler can be quite annoying but for some users it is worth it. Just need those settings to exist in the first place :)
     
  29. mike-voorhees

    mike-voorhees

    Unity Technologies

    Joined:
    Aug 9, 2016
    Posts:
    26
    Thanks for the feedback on your build setup.

    For advanced users who really want control you could run `il2cpp.exe --help` and find what you are looking for.

    I'm intentionally not listing the argument here for a few reasons.

    1) We view the command line arguments as an internal api. They may change at any time.

    2) There is no way to control the core usage of the compilation stage when ran by il2cpp. The same command line argument should control it, but it doesn't. This is not a great experience for users so I'd rather not draw too much attention to it at this point.
     
  30. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,691
    Yeah, I know. This does not sound related to IL2CPP pipeline at all though. And yes, today switching a platform reimports/recompresses all the textures, which is a bit stupid; there's some work underway to stop that but nothing ready to ship yet. Anyway this is not related to IL2CPP at all.
     
    Lars-Steenhoff likes this.
  31. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,106
    Last edited: Dec 1, 2020
  32. Elmak

    Elmak

    Joined:
    Nov 11, 2017
    Posts:
    10
    Pure truth.
    But the rumor is that it's gonna be fixed in 2020.2.0b15.
     
    ROBYER1 likes this.
  33. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,106
    .f1 actually
     
unityunity