Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  3. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Official IL2CPP Build Time Improvements - Seeking Feedback

Discussion in '2021.2 Beta' started by joncham, Feb 24, 2021.

  1. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    Hello Alpha Users!

    Multiple teams in Unity have been working on improving iteration time of player builds the past few releases. You should have noticed faster IL2CPP builds during the 2020 release cycle and even more improvements are coming in 2021.

    To this end, we have been working on a new option for IL2CPP that generates much less code (up to 50% less). This allows for both faster iteration times and smaller executable files. There *may* be a small runtime performance impact due to the different code generation.

    This graph shows improvements to compile times with this option enabled:



    We are asking that you try out this new option with your IL2CPP builds. We want to know if you see real world performance issues when running with this option. Also, let us know if the builds are faster and smaller.

    Note: This is available in alpha builds only, from 2021.2.0a17 onward.

    How to Enable: Change the "IL2CPP Code Generation" setting drop down in the Build Player window to "Faster (smaller) builds". The default and previous behavior is the "Faster runtime" setting.

    upload_2021-5-20_12-37-26.png


    Please respond with:
    1. Any difference in FPS or startup your applications see when running with this option enabled.
    2. The profile.json file in the generated IL2CPP C++ output folder (next to build output folder) for builds with and without this option. You can also send for cold and warm (incremental) builds. This allows us to see where build time is being spent and saved.
    3. You can also send stats on generated C++ output size and final binary size.
    Respond to this thread with any questions or findings. Additionally, you can DM me if you have information you don’t want to share in public.

    Thanks!
     
    Last edited: May 20, 2021
  2. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    Sounds interesting!
    I'm wondering a bit why you share detailed graphs about the build times but not about the possible performance regression. We're using a lot of generics so it would certainly be interesting which microbenchmarks show what slowdowns, and also how the above examples do in terms of performance. Mind sharing that?
     
  3. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    Hello,

    I shared the build times as the benefits of this option are clear. It is unclear what the negative impacts are for real projects. When running test projects we didn't see negative performance impact beyond ~10% of script time in a frame. That's why we are asking for more feedback, to have a larger sample size for finding regressions.

    Here is an example microbenchmark:

    Code (CSharp):
    1.     public class EqualityComparerEnum
    2.     {
    3.         private MyEnum _enum;
    4.         private EqualityComparer<MyEnum> _comparer;
    5.  
    6.         public EqualityComparerEnum()
    7.         {
    8.             var rand = new Random(12345);
    9.             _enum = (MyEnum)rand.Next(0, 3);
    10.  
    11.             _comparer = EqualityComparer<MyEnum>.Default;
    12.         }
    13.  
    14.         [Benchmark]
    15.         public void CallToEnumEqualityComparerGetHashCode()
    16.         {
    17.             _comparer.GetHashCode(_enum);
    18.         }
    19.  
    20.         enum MyEnum
    21.         {
    22.             A,
    23.             B,
    24.             C,
    25.             D
    26.         }
    27.     }
    This (the call to GetHashCode) runs an order of magnitude slower (~12x) with full generic sharing enabled.
     
  4. Hertzole

    Hertzole

    Joined:
    Jul 27, 2013
    Posts:
    421
    Unfortunately, I don't have any projects to really test the changes on but I just want to add my two cents.
    Personally, I'd rather have longer build times for more performance. The Release setting could be close to final and faster with these new changes while Master could take longer but maximizes performance.
     
  5. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    274
    Presumably this will be a configurable setting. Especially because there is bound to be corner-case code where sharing generics proves to be very slow (like the micro-benchmark mentioned above).

    Glad to see this! I would try this out if we were on the Alpha. Looking forward to seeing what other people observe.
     
  6. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    This is partly why we are trying to gather feedback. In order to know if and how it should be configurable.
     
    Rewaken, phobos2077, Hertzole and 2 others like this.
  7. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    211
    I observed a shorter build time when trying this out, but I immediately get nullreferenceexception when I run the build (happening when loading an addresable scene). Building with mono works though.

    So I have to say that I'd much rather it works than it being fast hehe :)


    Unity 2019.4.21f1
    Clean -
    Build completed with a result of 'Succeeded' in 1431 seconds (1431279 ms)
    Build completed with a result of 'Succeeded' in 468 seconds (467873 ms)
    Build completed with a result of 'Succeeded' in 462 seconds (462387 ms)

    Unity 2020.2.6f1
    Clean - Build completed with a result of 'Succeeded' in 1052 seconds (1051999 ms)
    Build completed with a result of 'Succeeded' in 136 seconds (135982 ms)
    Build completed with a result of 'Succeeded' in 135 seconds (134927 ms)

    Unity 2021.2.0a6.1011
    Clean - Build completed with a result of 'Succeeded' in 681 seconds (681159 ms)
    Build completed with a result of 'Succeeded' in 102 seconds (102462 ms)
    Build completed with a result of 'Succeeded' in 105 seconds (105499 ms)
     

    Attached Files:

    leni8ec likes this.
  8. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    211
    I gotta say I am impressed by the speed upgrade when going from 2019.4 LTS to 2020.2. Cant wait for 2020 LTS version, its going to be very good!
     
    joncham likes this.
  9. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    Do you get a NullReferenceException even without the option enabled? Or only when the option for full generic sharing is enabled?
     
  10. TimHeijden

    TimHeijden

    Joined:
    Mar 5, 2015
    Posts:
    34
    I've just ran some tests and got surprising results, in that building is SLOWER than before!

    (Note: All builds are development builds, with ILC2PP configuration "master")

    Build Time
    Build Size
    I was quite careful in testing, making sure to close unity between each build (& after upgrading before making B.) so this is quite weird. I've also made some profiler snapshots for each build, I'll send you a DM with those & some extra details (like the profile.json) later today.
     
    phobos2077 and joncham like this.
  11. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    Thanks for your testing of this @TimHeijden ! I took a look at your build profiler output and have a few thoughts.

    1. I see you are building in master/"ReleasePlus" build config. This enables LTO (link time optimization). While this may help performance, it's not something you want to iterate with as it is extremely expensive. It looks like 95% of your entire build time is the link event. I'd suggest Release for your daily work and it's something we should note more clearly to improve iteration.
    2. I see you are using the VS 2017 toolchain. We have found the VS 2019 linker much faster. It may be something else to keep in mind when you can upgrade.
    3. It seems like the "C" case didn't pickup the environment variable. You will want to make sure it is set, and kill/restart Unity and Unity Hub.

    If you have time and are interested in providing more data, I would suggest:
    1. Just doing runs for "B" and "C" cases. This is the most interesting data points for this new option.
    2. Use Release rather than ReleasePlus/master config.
    3. Again, for "C" case make sure environment variable is set and all instances of Unity/UnityHub are killed to pick up new value
    4. Do both clean (which is what your data seems to come from) and incremental builds. Tweak one line in a script and build again, as all things should be faster but we are mainly focused on this incremental use case.

    Thanks again for providing feedback!
     
    phobos2077 likes this.
  12. TimHeijden

    TimHeijden

    Joined:
    Mar 5, 2015
    Posts:
    34
    Hey Jon, thanks for clarifying the main use case you're looking for. I think I'll do another test later today and otherwise next week. I had killed unity & the hub, so its strange that "C" didn't pickup the environment variable... will check more closely. (I had assumed it worked because the build time was so much shorter again)

    As for building "ReleasePlus", I was already fully aware that its not really intended for iteration, but I was also very curious about the possible performance drop and thus thought I'd use the "optimum" setting for that. Note that I'm also very much of the opinion that unity devs should be able to choose for the maximum runtime performance, especially when developing for slower platforms than PC and trying to hit a stable FPS, 10% extra performance cost for scripting is quite a lot!

    In addition, I am indeed using VS2017. This is because some platforms require use of 2017 rather than 2019. Good to know that 2019 is faster though, will keep that in mind!
     
  13. TimHeijden

    TimHeijden

    Joined:
    Mar 5, 2015
    Posts:
    34
    Couldn't help myself, did it right away. This time I used the IL2CPP configuration "release" instead, and also did cold/warm tests. All builds are using Unity 2021.1.0a8

    I hope this time the environment variable stuck properly. Stats may be slightly skewed because I restarted my PC between no-env & env builds.

    Build Time
    I again sent a DM with the other details inside.
     
    LeonhardP and mahdi_jeddi like this.
  14. poprev-3d

    poprev-3d

    Joined:
    Apr 12, 2019
    Posts:
    71
    When trying to Il2CPP build an empty project with 2021.2.0.0a9.1376 i get this error during build time:


    Exception: Unity.IL2CPP.Building.BuilderFailedException: Build failed with 190 successful nodes and 1 failed ones
    Annotation: C_Win64_VS2019Pch y2vq/pch-cpp-3692469889848082439.pch (+obj)
    ...
    Files (x86)\Windows Kits\10\Include\10.0.18362.0\ucrt" /I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\cppwinrt" /Fo"y2vq\pch-cpp-3692469889848082439.obj" /D_HAS_AUTO_PTR_ETC=1 /D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS /D_SILENCE_ALL_CXX20_DEPRECATION_WARNINGS /Zc:__cplusplus /std:c++17
    ExitCode: 1
    Stdout:
    pch-cpp.hpp
    c1xx: fatal error C1083: Cannot open source file: 'Files\Unity\Hub\Editor\2021.2.0a9\Editor\Data\il2cpp\libil2cpp\pch\pch-cpp.hpp': No such file or directory
    pch-cpp.hpp
    c1xx: fatal error C1083: Cannot open source file: 'Files\Unity\Hub\Editor\2021.2.0a9\Editor\Data\il2cpp\libil2cpp\pch\pch-cpp.hpp': No such file or directory
    pch-cpp-3692469889848082439.cpp
    C:\Users\User\Desktop\tessddd\New Unity Project\Library\il2cpp_cache\buildstate\y2vq\pch-cpp-3692469889848082439.cpp: fatal error C1083: Cannot open include file: 'C:\Program': No such file or directory


    It seems that pch-cpp.hpp is missing although it is on my computer.

    Any idea ? I really wish i could try the 2021.2 new features.
     
  15. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
  16. poprev-3d

    poprev-3d

    Joined:
    Apr 12, 2019
    Posts:
    71
    @joncham yup it seems due to the space of "Program Files", thank you !
     
  17. KamilCSPS

    KamilCSPS

    Joined:
    May 21, 2020
    Posts:
    444
    @joncham, would these improvement also work when building for WebGL or is it limited to mobile/desktop toolchains?

    Thanks
     
  18. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    It also applies to WebGL. The build time improvement on WebGL may vary due to the build process for that platform being more complex than most.
     
    Haneferd and KamilCSPS like this.
  19. adamgolden

    adamgolden

    Joined:
    Jun 17, 2019
    Posts:
    1,549
    I've noticed faster builds lately (I generally update to the latest alpha when I notice it, or soon thereafter). Actually there have even been times I was like did it really finish already? For WebGL that's a pretty big deal :D So yeah.. whatever you're doing, thanks.
     
    LeonhardP and joncham like this.
  20. gsylvain

    gsylvain

    Joined:
    Aug 6, 2014
    Posts:
    100
    @joncham A very long moment is spent waiting for the Microsoft Incremental Linker. I noticed that this process is single thread. On console, we are wasting a considerable amount of time because of this. Would it be possible to analyze if you could use all available CPU cores? Thank you!
    upload_2021-4-16_13-8-25.png
     
    Ruchir and Lars-Steenhoff like this.
  21. MiFrilke

    MiFrilke

    Joined:
    Dec 14, 2016
    Posts:
    41
    So I was mainly interested in checking out these possible performance implications of this option: "We want to know if you see real world performance issues when running with this option"

    Especially the example provided above regarding equality comparision and GetHashCode being much slower piqued my interest.
    We have a pretty sophisticated save/load system at the heart of our game where we use and abuse Dictionaries a lot, so GetHashCode gets called quite often on various objects all over the place.

    Sadly i couldn't even get as far as testing save performance, since the game just straight out crashes during saving when running with full generic sharing enabled.

    I didn't check build time differences or FPS yet, as I'm mainly interested in the saving and loading impact. Is there anything I could share regarding this issue that would be helpful to you to fix this? Sharing the project sadly is no option since it is way too big.

    For now i've attached the profile.json files for both a regular il2cpp build and one with the new option enabled.

    (Tested with Unity 2021.2.0a12)


    Stack trace of crash:
    Code (CSharp):
    1. ========== OUTPUTTING STACK TRACE ==================
    2.  
    3. 0x00007FF8F1181F20 (GameAssembly) [u:\desperados_unity_2021.2a\_unity_install\editor\data\il2cpp\libil2cpp\codegen\il2cpp-codegen-il2cpp.cpp:890] il2cpp_codegen_runtime_constrained_call
    4. 0x00007FF8F2B03208 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\generics6.cpp:13965] ObjectEqualityComparer_1_GetHashCode_m04F4321254D5C4DB954043E12F6B360683A3FE83_gshared
    5. 0x00007FF8F10C42F1 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\il2cppinvokertable.cpp:40369] RuntimeInvoker_TrueInt32_t680FF22E76F6EFAD4375103CBBFFA0421349384C_Il2CppFullySharedGenericAny
    6. 0x00007FF8F17EB60B (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\assembly-csharp16.cpp:20356] MiCharacterNPCWeaponHolder_GetHashCode_m217FB843B629933D7B6D64C7C67CA7B8EC3FF25C
    7. 0x00007FF8F107445C (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\il2cppinvokertable.cpp:44473] RuntimeInvoker_TrueInt32_t680FF22E76F6EFAD4375103CBBFFA0421349384C
    8. 0x00007FF8F1181F26 (GameAssembly) [u:\desperados_unity_2021.2a\_unity_install\editor\data\il2cpp\libil2cpp\codegen\il2cpp-codegen-il2cpp.cpp:890] il2cpp_codegen_runtime_constrained_call
    9. 0x00007FF8F2B03208 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\generics6.cpp:13965] ObjectEqualityComparer_1_GetHashCode_m04F4321254D5C4DB954043E12F6B360683A3FE83_gshared
    10. 0x00007FF8F10C42F1 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\il2cppinvokertable.cpp:40369] RuntimeInvoker_TrueInt32_t680FF22E76F6EFAD4375103CBBFFA0421349384C_Il2CppFullySharedGenericAny
    11. 0x00007FF8F0FE56A7 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\assembly-csharp-firstpass12.cpp:249] InterfaceFuncInvoker1<int,Il2CppObject *>::Invoke
    12. 0x00007FF8F2A76737 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\generics3.cpp:20746] Dictionary_2_FindEntry_mC9A39BDAA66FDC3D9E96031E0EB4D71E51306E46_gshared
    13. 0x00007FF8F10C42F1 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\il2cppinvokertable.cpp:40369] RuntimeInvoker_TrueInt32_t680FF22E76F6EFAD4375103CBBFFA0421349384C_Il2CppFullySharedGenericAny
    14. 0x00007FF8F2A7ACDE (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\generics3.cpp:21710] Dictionary_2_TryGetValue_mC15984CACD8C4EC727085E5BA025AF6CA0D158DB_gshared
    15. 0x00007FF8F14EC397 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\assembly-csharp-firstpass6.cpp:24401] SaveLoadSceneManager_iRefToID_mEFC75EA9734E2A7C9B9B2349BA1E91674EAD7FE2
    16. 0x00007FF8F14E1690 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\assembly-csharp-firstpass6.cpp:21709] SaveLoadSceneManager_castToYamlSerializable_m842964935ECCCE167A2B89B0D41878866DD0B0BC
    17. 0x00007FF8F14EDFBB (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\assembly-csharp-firstpass6.cpp:20443] SaveLoadSceneManager_reflectMiSerializeInfo_mB54930FF38D3506C72FFCFC27DE816CFA2DF1E4F
    18. 0x00007FF8F14EBB4E (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\assembly-csharp-firstpass6.cpp:17684] SaveLoadSceneManager_handleSerializeInfosForClass_m1BCDFE87EA546F5BAD2FF76355441557E898DF52
    19. 0x00007FF8F14E7E49 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\assembly-csharp-firstpass6.cpp:16509] SaveLoadSceneManager_createValueTypeData_m2AF9AA486E67C808AC63707C7C91E6254F5701BA
    20. 0x00007FF8F1A9F126 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\assembly-csharp42.cpp:26105] U3CcreateSceneSaveDateAndStoreToMemberU3Ed__85_MoveNext_m8FDC2EDE5F0307820A57F531E1CC45367F15B2B0
    21. 0x00007FF8F107E98C (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\il2cppinvokertable.cpp:45097] RuntimeInvoker_TrueSByte_tFEFFEF5D2FEBF5207950AE6FAC150FC53B668DB5
    22. 0x00007FF8F389C243 (GameAssembly) [u:\desperados_unity_2021.2a\library\il2cppbuildcache\windows\x64\il2cppoutput\unityengine.coremodule1.cpp:19961] SetupCoroutine_InvokeMoveNext_m4C8B79D8A35F69EEECAA16C82793525E74D75CE9
    23. 0x00007FF8F11E3126 (GameAssembly) [u:\desperados_unity_2021.2a\_unity_install\editor\data\il2cpp\libil2cpp\vm\runtime.cpp:583] il2cpp::vm::Runtime::InvokeWithThrow
    24. 0x00007FF8F11E2BF1 (GameAssembly) [u:\desperados_unity_2021.2a\_unity_install\editor\data\il2cpp\libil2cpp\vm\runtime.cpp:568] il2cpp::vm::Runtime::Invoke
     

    Attached Files:

  22. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    Hello, just wanted to note that I updated the instructions in the original post as this is now exposed as an option in the Build Player window from 2021.2.0a17 onward.
     
  23. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,042
    Why is not a setting that would go on projectsettings.asset and under version control? The setting on that window is not stored and it's harder to make CI build because you need a custom script.
     
  24. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    The feedback so far indicates people view this functionality as a "temporary" setting like script debugging, something that you want to enable locally for iterating and not commit to source control. We can re-evaluate this, but we made this choice based on feedback to this point.
     
    FernandoMK and bdovaz like this.
  25. adamgolden

    adamgolden

    Joined:
    Jun 17, 2019
    Posts:
    1,549
    For each of the options I did a build, closed the editor, re-opened the editor then did the build again, with the times given below based on being the first thing done after a fresh startup of the editor.

    Faster runtime:
    Build completed with a result of 'Succeeded' in 234 seconds (233790 ms)

    Faster (smaller) builds:
    Build completed with a result of 'Succeeded' in 19 seconds (19431 ms)

    Later after using the editor for a while I also did a 2nd attempt on each one sequentially to confirm the earlier times weren't anomalous, and these came out at 225s and 20s. The WebGL builds both did run and 19s vs. 234s to go from editor to browser is HUGE. However, I couldn't do any kind of performance or framerate comparison. There was an issue with Addressables (1.17.17) for me at runtime which only happened when using the new option (that took 19 s). I switched to development build, full stack trace etc., rebuilt, tried again, here's the error from the javascript console:

    Code (JavaScript):
    1. NullReferenceException: Object reference not set to an instance of an object.
    2.   at UnityEngine.ResourceManagement.ResourceManager.ProvideResourceGroupCached (System.Collections.Generic.IList`1[T] locations, System.Int32 groupHash, System.Type desiredType, System.Action`1[T] callback, System.Boolean releaseDependenciesOnFailure) [0x00000] in <00000000000000000000000000000000>:0
    3.   at UnityEngine.ResourceManagement.ResourceManager.ProvideResource (UnityEngine.ResourceManagement.ResourceLocations.IResourceLocation location, System.Type desiredType, System.Boolean releaseDependenciesOnFailure) [0x00000] in <00000000000000000000000000000000>:0
    4.   at UnityEngine.ResourceManagement.ResourceManager.ProvideResource[TObject] (UnityEngine.ResourceManagement.ResourceLocations.IResourceLocation location) [0x00000] in <00000000000000000000000000000000>:0
    5.   at UnityEngine.AddressableAssets.AddressablesImpl.LoadAssetAsync[TObject] (System.Object key) [0x00000] in <00000000000000000000000000000000>:0
    6.   at UnityEngine.AddressableAssets.Addressables.LoadAssetAsync[TObject] (System.Object key) [0x00000] in <00000000000000000000000000000000>:0
    ..and the above is triggered by a call like .
    asyncHandle = Addressables.LoadAssetAsync<UnityEngine.Object>(the_addressable_name);


    I don't know if it's just something about how I work with Addressables or there's actually an issue. I'll be sticking with the faster runtime setting at least for now and would for builds being released anyway.

    Also, @bdovaz not sure this helps but you can specify the setting via script like this:
    Code (CSharp):
    1. EditorUserBuildSettings.il2CppCodeGeneration = UnityEditor.Build.Il2CppCodeGeneration.OptimizeSpeed;
    2. // or..
    3. EditorUserBuildSettings.il2CppCodeGeneration = UnityEditor.Build.Il2CppCodeGeneration.OptimizeSize;
     
    ImpossibleRobert, fherbst and joncham like this.
  26. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    953
    Great analysis work! Nice to read people getting actual data from the wild, and not just our internal in-house test projects. It always gives a stronger sense of validation if things are working out like they are expected.

    Hmm, this scheme might not be the best test for incremental build times, since it will be repeating the same build with same code and same assets than before, that can cut a lot of the meaningful work from the test.

    I would recommend the following test schemes:

    A. Incremental code build test:

    1. Build the project (not measuring time taken)
    2. Change a single line in one .cs script file
    3. Rebuild the project (measure this time)

    This will give an impression of how long the lightest code iteration build will take. And:

    B. Incremental asset build test:

    1. Build the project (not measuring time taken)
    2. Change a parameter of a single asset in the project, and move a single object position in the scene, don't change any code
    3. Rebuild the project (measure this time)

    This should give an impression of how long the lightest asset rebuild iteration will take.

    In current Unity 2021.2 alphas, I expect that for WebGL the incremental code build test will exceed 19 seconds by a considerable margin, due to the limitations of the Emscripten 1.38.11 compiler that it still employs. By the time Unity 2021.2 hits Beta, this should change and it should be using Emscripten 2.0.19 compiler instead. That should speed up scheme A by some +20%-+30% at minimum, maybe even more.
     
  27. adamgolden

    adamgolden

    Joined:
    Jun 17, 2019
    Posts:
    1,549
    Thanks @jukka_j - I guess with nothing changing, my previous times were meaningless :rolleyes:. I also have editor scripts that might have invalidated something cached. Also, there's the other "Code Optimization" dropdown that went without attention. So another run at this..

    First the baseline - no editor restarting, compiled once and then again for each combination of IL2CPP option and Code Optimization option with no changes to code or assets. Regardless of which combination it took 20-21 seconds.

    For Test A, adding or removing private float variables to a MonoBehaviour and something in Awake that changes the values. Repeated 4 times for each of the 4 combinations:
    Faster (smaller) builds + Code Optimization for Size: 169s, 170s, 169s, 169s
    Faster (smaller) builds + Code Optimization for Speed: 187s, 186s, 199s, 184s
    Faster runtime + Code Optimization for Size: 196s, 195s, 194s, 194s
    Faster runtime + Code Optimization for Speed: 229s, 215s, 279s, 234s

    For Test B, added a 3D Object->Sphere, rebuilt the project. Moved the sphere around and changed properties of the Sphere Collider before each build. Repeated several times and it was always 20-22 seconds, regardless of which setting (sometimes speed took less time, sometimes size). I don't think my Test B results could be considered typical though, because almost no content is actually in my scenes (i.e. I'm loading anything significant into the scene via Addressables and generating a lot of stuff procedurally).

    Anyway it's great to see there's effort going into build times and performance - it is definitely appreciated.
     
  28. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    8,194
    What about sub optimal files sizes especially for aspects of Unity that generate lots of very small and inefficient files* e.g. .umbracache .bin etc.

    Modern HDs SSDs and NVMes hit optimal read and write speeds around the 32-64kb file size.

    *NTFS drives with default sectors of 2k+ lose 2x or higher in space for very small files.
     
    phobos2077 and apkdev like this.
  29. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    1,158
  30. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    I did not specifically test this, but assuming the managed linker didn't strip/remove the code I would expect it to work.
     
    Thaina likes this.
  31. PencilBow

    PencilBow

    Joined:
    Jun 23, 2017
    Posts:
    131
    @joncham Very interesting stuff, code-size has been quite a problem for our project. At one point we considered writing an in-house roslyn rewriter to improve generics sharing, but we decided against it because of the time investment required.

    We'll try update our project to the latest unity beta, to see the results of full generic sharing. In the meantime, would you mind explaining exactly what 'full generic sharing' involves?

    - are ALL valuetypes converted to reference types. eg List<int>, List<MyEnum>, List<MyStruct>
    - I assume that 'full' generics sharing involves boxing each argument?
    - is there a way to customize this behavior for each type/method/all-methods-of-class-or-struct
     
  32. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    We generate a single version of each generic method and produce it in a way that all parameters, locals, etc can handle any type. There is no boxing overhead; we do not involve the GC/boxing more with this approach. There is some additional overhead when passing parameters or loading/storing locals, as we need runtime checks to determine whether something should be copied by value or reference. In addition, there can be more overhead in things like constrained calls, as again, we need a runtime rather than conversion/compile time check.

    Our findings and expectations are that non-generic code performance should have no penalty from this option, and that generic code impact varies but in our testing is much lower performance impact than expected.

    There is currently no way to control this option other than globally.
     
    JesOb, PencilBow and Thaina like this.
  33. Armynator

    Armynator

    Joined:
    Feb 15, 2013
    Posts:
    66
    I quickly tested a small sideproject on Windows 10.
    The build time improvements are nice. They went down by 50-66%.
    The build size improvements are minimal. Only a few megabytes less.
    The runtime performance is worse for me. About 3% slower.

    Personally I think this is a great feature for faster debugging, but that's about it. Sure, add it, improve it, but please, don't force us to use it.
    I don't care if the game is a bit bigger or if final builds take longer. What I care about is runtime performance for my players, and sacrificing even 1% performance for the current advantages is hard to justify in my opionion.

    Honestly, to me it looks a bit like you are seeking confirmation here to consider making this the default/only option in the future. And that's exactly why I'm answering now: don't force us to use it. Make it optional.
    Maybe it's more work for you to maintain both versions, but please, don't make the experience for the players worse just to save some final build time for the developers. I mean what's the ratio, 1 developer waiting for a build for 1000 players running it? Better wait 2 minutes longer then instead of damaging the experience for your whole playerbase.
     
    fherbst and MiFrilke like this.
  34. joncham

    joncham

    Unity Technologies

    Joined:
    Dec 1, 2011
    Posts:
    276
    We are trying to validate with actual projects and workflows the impacts we saw with internal testing, and also ensure correctness when running with this option. We roughly saw like ~50% improvement in build time, smaller binary sizes, and limited/variable degradation to runtime performance. Your results match what I have observed internally.

    There are no current plans to make this the only option, but if build times prove to be 50% faster with minimal performance impact I could envision this option becoming the default. Improving long build times is one of (if not the highest) requests we get.
     
    JesOb, mahdi_jeddi and gsylvain like this.
  35. DanielRiches

    DanielRiches

    Joined:
    Nov 23, 2019
    Posts:
    165
    I'm sure there might be ways to close that performance drop gap anyways, and given that we just hit a new generation of consoles and g-cards making this the default may make sense.

    But TBH I would feel better if Unity could commit to closing that gap as much as possible and not throwing the issue on a "not so important" list to look at later.
     
  36. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    1,085
    My project fails to build, using faster runtime or faster builds, net standard, Unity 2021.2.0b6:

    Code (CSharp):
    1. C:\Program Files\Unity\Hub\Editor\2021.2.0b6\Editor\Data\il2cpp\build\deploy\net5.0\il2cpp.exe @Library\Bee\artifacts\rsp\12307732474854830548.rsp
    2. Error: IL2CPP error for type 'System.ValueTuple`2' in assembly 'C:\projectpath\Library\Bee\artifacts\WinPlayerBuildProgram\ManagedStripped\mscorlib.dll'
    3. System.InvalidOperationException: Attempted to lookup type that hasn't been created
    4.   at Unity.IL2CPP.DataModel.BuildLogic.Repositories.ReadonlyMemberStorageStrategy.GetOrAdd[TArg,TKey,TDataModel](Dictionary`2 mapping, TArg arg, TKey elementType, Func`3 create, Action`2 onCreated)
    5.   at Unity.IL2CPP.DataModel.Creation.FullyConstructedTypeFactory.CreateFieldReference(TypeReference declaringType, FieldReference fieldReference)
    6.   at Unity.IL2CPP.TypeDefinitionWriter.MakeFieldWriteInstructionsForType(SourceWritingContext context, TypeReference type, TypeDefinition typeDefinition, FieldType fieldType)
    7.   at Unity.IL2CPP.TypeDefinitionWriter.WriteFieldsWithAccessors(SourceWritingContext context, IGeneratedCodeWriter writer, TypeReference type, Boolean isUnmanagedType, FieldType fieldType)
    8.   at Unity.IL2CPP.TypeDefinitionWriter.WriteTypeDefinitionFor(SourceWritingContext context, TypeReference type, IGeneratedCodeWriter writer)
    9.   at Unity.IL2CPP.CppDeclarations.CppDeclarationsCollector.BuildCacheData(SourceWritingContext context, TypeReference type, CppDeclarationsCache cache)
    10.   at Unity.IL2CPP.CppDeclarations.CppDeclarationsCollector.PopulateCache(SourceWritingContext context, IEnumerable`1 rootTypes, CppDeclarationsCache cache)
    11.   at Unity.IL2CPP.CppDeclarations.CppDeclarationsWriter.CollectDeclarations(SourceWritingContext context, ICppDeclarations declarationsIn, CppDeclarations& declarations)
    12.   at Unity.IL2CPP.CppDeclarations.CppDeclarationsWriter.Write(SourceWritingContext context, StreamWriter writer, ICppDeclarations declarationsIn)
    13.   at Unity.IL2CPP.CodeWriters.GeneratedCodeSourceCodeWriter.Dispose()
    14.   at Unity.IL2CPP.TableWriters.ScheduledTableWriterBaseChunkedTransform`3.PostProcessWorker(WorkItemData`3 data)
    15.   at Unity.IL2CPP.Contexts.Scheduling.PhaseWorkScheduler`1.WorkerLoop(Object data)
    16. UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)
     
  37. Aoedipus

    Aoedipus

    Joined:
    Jan 31, 2019
    Posts:
    25
    Just a few quick tests:

    WebGL delivery platform


    Build sizes
    2020.3.0f1 (speed)
    build/ = 31.4mb
    webGL/ = 65.2mb (addressables)

    2021.1.16f1 (speed)
    identical to 2020.3.0f1

    2021.2.0b6.3225 (faster (smaller) builds)
    build/ = 31.9mb
    webGL/ = 66.9mb (addressables)

    2021.2.0b6.3225 (faster runtime)
    build/ = 32.5mb
    webGL/ = 66.9mb (addressables)

    Build times
    ~350-400+ secs (lts and tech versions)
    348 secs (faster (smaller) builds)
    168 secs (faster (smaller) builds after minor script change)
    246 secs (faster runtime)

    Performance impact
    'faster (smaller) builds' hangs when trying to load scene, also get MethodAccessException errors on start
    'faster runtime' loads and runs fine

    Preference
    Faster runtime, 0.6mb build size negligible, build time still shows improvement, project runs
     
    lclemens likes this.
  38. mike-voorhees

    mike-voorhees

    Unity Technologies

    Joined:
    Aug 9, 2016
    Posts:
    45
    Hi @Kamyker , it looks like something in your project is getting il2cpp.exe into a situation that we did not think was possible. Would you mind filing a bug report with a repro so that we can locate and fix the issue?
     
  39. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    1,085
    Hi, case 1356746. It was a bit difficult to find the cause but I got il2cpp.exe debugging working using dotPeek.

    After changes I had a different build bug with static field in interface: Case 1356753 (https://issuetracker.unity3d.com/is...g-an-interface-with-static-fields-in-a-script)

    After moving it to static class project finally builds and runs but it has some runtime errors when using one dll that does a lot of marshaling. I'll report more in a month as mono is fine for development currently.
     
    Last edited: Aug 12, 2021
    LeonhardP and ImpossibleRobert like this.
  40. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    1,085
    While making a bug report for another game with il2cpp runtime crash in 2020.3 I've tried to build it on 2021.2.0b7. As you can guess it didn't work. Case 1357806.

    It looks scary:
    Code (CSharp):
    1. Building Library\Bee\artifacts\WinPlayerBuildProgram\ql6z\tjek_ILRuntime2.obj failed with output:
    2. ILRuntime2.cpp
    3. C:\Program Files\Unity\Hub\Editor\2021.2.0b7\Editor\Data\il2cpp\libil2cpp\codegen\il2cpp-codegen-il2cpp.h(411): error C2296: '&': illegal, left operand has type 'T'
    4.         with
    5.         [
    6.             T=Enum_t2A1A94B24E3B776EEF4E5E485E290BB9D4D072E2 *
    7.         ]
    8. \Library\Bee\artifacts\WinPlayerBuildProgram\il2cppOutput\cpp\ILRuntime2.cpp(19938): note: see reference to function template instantiation 'bool il2cpp_codegen_enum_has_flag<Enum_t2A1A94B24E3B776EEF4E5E485E290BB9D4D072E2*>(T,T)' being compiled
    9.         with
    10.         [
    11.             T=Enum_t2A1A94B24E3B776EEF4E5E485E290BB9D4D072E2 *
    12.         ]
    13. C:\Program Files\Unity\Hub\Editor\2021.2.0b7\Editor\Data\il2cpp\libil2cpp\codegen\il2cpp-codegen-il2cpp.h(411): error C2297: '&': illegal, right operand has type 'T'
    14.         with
    15.         [
    16.             T=Enum_t2A1A94B24E3B776EEF4E5E485E290BB9D4D072E2 *
    17.         ]
     
  41. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,132
    The devs are looking into it.

    https://issuetracker.unity3d.com/product/unity/issues/guid/1357806/
    https://issuetracker.unity3d.com/product/unity/issues/guid/1356746/
     
    Last edited: Aug 17, 2021
  42. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,751
    Nintendo Switch is the fastest IL2CPP target to build for (by a huge margin) in my experience. Can't get into details due to NDA stuff, but it seems to use some form of caching I haven't seen in any other IL2CPP target, which greatly speeds up compiling and linking. Not sure why these aren't used on other targets.
     
  43. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,921
    There should not be any special build caching for Nintendo Switch which is not present on any other platforms. The directories used for build caching differ slightly per platform, but the behavior should be the same. I wonder if there is some other factor that makes Switch build faster. Maybe the C++ compiler is faster - I don't know.
     
  44. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,751
    I
    It creates additional files and directories unlike the other targets which also use Clang for compiling, and seems to use that for fast incremental linking.

    Linking is by far the longest part of building with IL2CPP in my experience with other platforms. Linking is also a mostly single threaded task: while I can blaze through the compilation by throwing a 32 core CPU at it, there's no bribing linking into going faster.

    Anyway, it's worth tracking down whatever team was behind the Switch building process and check how they did it, and if it can be leveraged for other platforms (or at the very least other Clang-based platforms like Android.)
     
    OneManArmy3D and FernandoMK like this.
  45. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    1,085
    Found another one:

    (Case 1361232) IL2CPP player crash regression

    Mono works fine and 2020.3 il2cpp also.
     
  46. phobos2077

    phobos2077

    Joined:
    Feb 10, 2018
    Posts:
    350
    Is there any chance this will be backported into 2020.3 LTS? IL2CPP build is the bottleneck in our CI pipeline.
     
  47. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,921
    No, we don't plan to back port these changes. They are too extensive, unfortunately.
     
    phobos2077 likes this.
  48. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    It seems less stable than the older code generation option. I compiled my scene in faster-build, but the player for it ends up crashing whenever started. I compiled it again with faster-runtime and it works as expected.
    Although I was using the HDRP with latest 2022 alpha and SRP github master.

    I ended up sending a bug report but only with the scene as a package(project 2 big), and it's a bit different compared to mine but maybe it'll still lead to a crash.

    Case 1365032
     
    LeonhardP likes this.
  49. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,921
    Thanks for the bug report!
     
  50. PencilBow

    PencilBow

    Joined:
    Jun 23, 2017
    Posts:
    131
    Last edited by a moderator: Sep 20, 2021