Search Unity

  1. Unity 2018.3 is now released.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. Want more efficiency in your development work? Sign up to receive weekly tech and creative know-how from Unity experts.
    Dismiss Notice
  4. Build games and experiences that can load instantly and without install. Explore the Project Tiny Preview today!
    Dismiss Notice
  5. Want to provide direct feedback to the Unity team? Join the Unity Advisory Panel.
    Dismiss Notice
  6. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice

Incremental builds for IL2CPP

Discussion in 'Scripting' started by JoshPeterson, Nov 2, 2015.

  1. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    I’m pleased to announce that we’ve started to ship support for incremental builds in the latest patch releases. I wanted to take a few minutes here to mention how incremental builds will work, what they can handle now, and what is coming in the future. Our plan is to make improvements to incremental builds available in patch releases as we have them ready, so the feature is not complete yet, but should provide some benefit.

    Goal:

    Our goal with incremental builds it to improve the long time required to build most projects with IL2CPP.

    Availability:

    We’re shipping incremental build support in patch releases 5.2.2p3 and 4.6.9p2. Initially, incremental builds will work only with iOS. but we plan to make the feature available soon or other platforms that use IL2CPP, including WebGL, Android, XboxOne, and PS4. We’ve chosen iOS initially, as we can depend on Xcode to only build the changed generated C++ source files.

    How to use it:

    After selecting “Build” from the “Build Settings” dialog, choose the “Append” option for an incremental build. The “Replace” option will perform a clean build.

    How it works:

    Our focus in implementing this feature has been on making the C++ code generated by il2cpp.exe more stable. That is, a small change in the input IL code should correspond to a small change in the generated C++ code. For this initial release, we’ve implemented generated code stability across a few changes.

    • Assembly dependency order - for example, a change to Assembly-CSharp.dll should not impact any code generated for mscorlib.dll (unless stripping is involved, see below).
    • String literal addition, removal, and changes
    • Non-generic Type addition, removal and changes

    On our roadmap, we’re planning to improve code stability with the following upcoming changes:

    • Prevent stripping from impacting type and method names in generated code.
    • Generate stable type and method indices for runtime metadata retrieval.
    • Improve code stability for generic types.

    You should expect to see these improvements in upcoming patch releases. Of course, the C++ compile step is only a part of the build process. We’re also working to improve code conversion times (IL -> C++), but we hope that these changes will improve the IL2CPP build process.

    Reporting bugs:

    We really appreciate bug reports about this feature, as it is can often be difficult to isolate the cause of unnecessary changes to generated code. We can make the most progress on fixing incremental build issues if a bug report can provide the IL assemblies and generated C++ code from before and after an incremental build that failed.
     
  2. ibm777p2

    ibm777p2

    Joined:
    Nov 7, 2015
    Posts:
    5
    When is the 4.6.9p2 patch coming up?
     
  3. ibm777p2

    ibm777p2

    Joined:
    Nov 7, 2015
    Posts:
    5
    Hi Josh in the recent version of unity patch 4.6.9p1 I have an issue loading a level after I loaded a level already, which means the game crash after loading a level twice in iOS. Is anyone experiencing this issue after loading a level twice in iOS?

    Android works fine
     
    Last edited: Nov 8, 2015
  4. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @ibm777p2

    The 4.6.9p2 release was scheduled to be out on Friday, November 6. It seems that it has slipped a few days, but our sustained engineering team is working on getting it out now. I suspect it should be out any day now.
     
  5. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @ibm777p2

    I've not heard of any other problems like this, although you might want to ask on a new thread. You could also submit a bug report, and we will have a look at it.
     
  6. jernau75

    jernau75

    Joined:
    Sep 3, 2013
    Posts:
    2
    Hi Josh,

    Thanks very much for 4.6.9p2. Is there a way of setting the Append or Replace option from script? We have our own build window which ultimately calls BuildPipeline.BuildPlayer directly, so we won't see the option that's built in to the regular build window.

    Thanks again.
     
  7. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @jernau75

    Yes, this is possible, our iOS team asked me to point you to this build option:

    http://docs.unity3d.com/ScriptReference/BuildOptions.AcceptExternalModificationsToPlayer.html
     
  8. frank28

    frank28

    Joined:
    Dec 30, 2013
    Posts:
    12
    Hi Josh,

    So as I read, this setting can only be set manually through UI before building? Can this also be set through code, like other BuildOptions? As we'd like to integrate it into our building process, which is written in C#.

    Thanks.
     
    Last edited: Nov 30, 2015
  9. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @frank28

    Yes, I think that AcceptExternalModificationsToPlayer can be set like other build options.
     
  10. frank28

    frank28

    Joined:
    Dec 30, 2013
    Posts:
    12
    Oh, didn't realise that old setting value gets new functionality. Thanks.
     
  11. Blundig

    Blundig

    Joined:
    Jan 7, 2014
    Posts:
    18
    I'm having problems with incremental builds. I'm not sure if I'm missing something. Here's a simple case (tested with Unity 5.3.4f1 and Xcode 7.2.1):
    1. Create a new iOS project.
      • Create 2 test Monobehaviour files. I just used the default templates created by MonoDevelop.
      • Add the behaviors to a GameObject.
      • Save the scene and add it to the build settings.
    2. Build the project in Unity with IL2CPP.
    3. Open in Xcode and build.
      • This first build will compile 158 c++ files (156 from the Unity core, plus the 2 custom test behaviors).
    4. Go back to Unity, make a change to one of the test files.
    5. Build the Unity project, selecting Append.
    6. Reopen in Xcode and build.
      • This 2nd build will still compile all 158 c++ files. My expectation was that it would only need to recompile the 1 file that was altered.
    I tried the above with and without "Strip Engine Code" enabled in the build settings, with the same results. Even if I do an Append build without making any changes to the Unity project, Xcode will still recompile every file.

    Am I missing a step or a build setting to make this work?
     
  12. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @Blundig

    You're not doing anything wrong here, this is the proper workflow. I'm a bit puzzled by the results. I've tried to follow this workflow, and I don't see the same issue. Note that more than one file will need to recompile - we have one IL2CPP generated file that changes with each code change, and some iOS specific files that also always need to compile. However these files should be pretty fast to compile.

    In step 4, what change do you make to the code? That might have an impact on the behavior difference I'm seeing.
     
  13. Blundig

    Blundig

    Joined:
    Jan 7, 2014
    Posts:
    18
    The only change I made was to add an empty Awake method to the behaviour.

    After looking into this more, it appears that incremental builds are working, but maybe my expectations are too high?

    I did a diff to check what files were being changed in the Xcode project folder after an Append build. They were as follows (note that TestScript was the name of my behaviour file):

    Classes/Native/AssemblyU2DCSharp_TestScript13201784756MethodDeclarations.h
    Classes/Native/Bulk_Assembly-CSharp_0.cpp
    Classes/Native/Il2CppCodeRegistration.cpp
    Classes/Native/Il2CppMethodPointerTable.cpp
    Classes/Native/UnityICallRegistration.cpp
    Data/Managed/UnityICallRegistration.cpp
    Data/Managed/Metadata/global-metadata.dat
    Unity-iPhone.xcodeproj/project.pbxproj

    I could see the addition of the Awake method, and other changes seemed minimal.

    When compiling the incremental build (and paying extra attention), I noticed the compiler skipping over some files faster than before (most notably the first 42 c++ files go by instantly, but most of the others still take about the same amount of time). So, it is definitely a difference from a clean build.

    However, the incremental build is only trimming about ~10% from the total compile time for this test project. That was a small enough savings that I didn't notice and assumed it wasn't working. FYI - I have never noticed a great time savings when compiling our actual game projects (which on average take about 20 minutes), so I'm guessing if I measure them I'll find about the same % of savings.

    Note: if I go and modify some of the c++ files by hand in Xcode, the build is basically instant. This is more what I was expecting, based on when I code natively in Xcode. I know this probably isn't a relevant test case, but I wanted to verify that Xcode wasn't corrupt or causing some trouble on its own.

    Should I be seeing a greater time savings from incremental builds, or does this seem normal?
     
  14. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @Blundig

    You are seeing the normal behavior, although I'm surprised that the improvement is only 10% in compile time.

    The files you listed are the ones I expect to change for an added method. I don't think UnityICallRegistration.cpp should be listed twice - that looks like an error which I'll track down. However, it should only be compiled once.

    I have a few more question to help understand the situation.
    • The 10% improvement in compile time, is that 10% less time spent in the Xcode build step, or is that 10% improvement in the entire Unity build process?
    • Are you building in release or debug in Xcode?
    Your experience with manual modifications of the source files in Xcode is interesting. il2cpp.exe should be doing the same thing, so I'm not sure why Xcode would take longer to process them than it does with a manual modification. I will investigate that as well though.

    Thanks for the feedback - I know it can be time consuming to gather this information, so I appreciate it. I think that we can get to the bottom of this and improve it.
     
  15. Blundig

    Blundig

    Joined:
    Jan 7, 2014
    Posts:
    18
    The 10% was just the compile step in Xcode (from when the build starts until linking starts).

    Release builds (but not Archive builds) with no modifications to the project settings exported from Unity. I would just hit the play button on the Xcode toolbar.

    I ran a few tests related to this with strange results.

    1st Test: Unity Incremental Build
    1. Start with a fresh Unity test project, as described above.
    2. Do a clean build from Unity, into a folder named Test.
    3. In Finder, copy the Test folder and name it Clean (for use later).
    4. Open the Xcode project inside of Test.
    5. Build the project, noting the compile time. This will be our baseline.
    6. Do another build, without changing any files. Notice that it finishes almost instantly. (Keep this in mind for later during the 2nd test)
    7. Quit Xcode.
    8. Open Unity. Add an empty Awake method to one of the behaviours. Do an append build into the Test folder.
    9. In Finder, copy the Test folder and name it Append (for use later).
    10. Open the Xcode project inside of Test.
    11. Build the project, noting the compile time. As an incremental build, the expectation is this should be much faster than baseline, but that isn't the case for me. We'll call this baseline 2.

    2nd Test: Manual Incremental Build
    1. Do a diff of the Clean and Append folders (I used FileMerge). Note which files changed. We’ll refer to these as the Changed Files.
    2. In Finder, copy the Clean folder and name it Test2.
    3. (Optional) Delete Xcode’s Derived Data folder to ensure everything is 100% clean.
    4. Open the Xcode project inside of Test2.
    5. Build the project. No need need to time this one, it should be similar to baseline.
    6. Quit Xcode.
    7. In Finder, copy all Changed Files from the Append folder into Test2, replacing the Test2 versions. This gives us a manual recreation of the 1st append test folder. You can verify this with FileMerge.
    8. Open Test2 in Xcode. Build the project, noting a few things.
      1. The compile time is much faster than baseline 2. On my machine, it only takes 0.1% of the time.
      2. It isn't as fast as the reference build from step 6 in the 1st Test. You can see Xcode's progress pause when processing the changed files.
    Result:
    Seemingly identical build folders take significantly different times to compile.

    Theories:
    • My testing methodology is flawed.
    • My test environment is corrupt in some way.
    • The Unity Append process does something that breaks Xcode’s incremental build system.
      • Maybe it touches files in some way that FileMerge doesn’t consider a change but Xcode does?
      • Maybe there are some hidden files causing trouble that I didn’t take into account?
    I could not narrow down why this happened, but I can repeat the results. Hopefully there is something useful here, not just a wild goose chase.
     
  16. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @Blundig

    I think your testing methodology makes sense. Thanks for the detailed break-down. I'll try this locally to attempt to reproduce the same results and let you know.
     
  17. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @Blundig

    I've been able to reproduce this locally using Unity 5.3.4p1 and Xcode 7.3. I'm not sure yet about the cause of the issue, but this is certainly a bug. Do you want to submit a bug report for this issue? If so, you can be easily informed when it is corrected. If you do, please let me know the bug report number.

    If you would rather not, that is fine as well. I think that we have enough information to correct the issue now.
     
  18. Tommi-Kiviniemi

    Tommi-Kiviniemi

    Joined:
    Aug 21, 2014
    Posts:
    7
    Hi Josh!

    Could you please let me know what the issue number is for this? And is it in any way related to
    "(none) - iOS/IL2CPP: Prevent unnecessary changes to the timestamps of the libil2cpp headers during a build. This allows incremental builds to work correctly in Xcode."
    in 5.4.3p2?

    Also, two questions regarding incremental builds:
    1) We make builds on a Jenkins instance, where the CI builds are optimised for speed and utilise Mono/32bit for iOS. Adding BuildOptions.AcceptExternalModificationsToPlayer to our Unity build script results in the following error:
    "CopyStringsFile My.app/HelpshiftLocalizable.strings /Users/Me/Project/Assets/Lib/Helpshift/Plugins/iOS/HSLocalization/zh-Hant.lproj/HelpshiftLocalizable.strings
    cd /Users/Me/Project/Build
    export PATH=“/path/to/xcode711/7.1.1/Xcode_7.1.1.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:/path/to/xcode711/7.1.1/Xcode_7.1.1.app/Contents/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin"
    builtin-copyStrings --validate --inputencoding utf-8 --outputencoding binary --outdir /Users/Me/Project/Build/My.app -- /Users/Me/Project/Assets/Lib/Helpshift/Plugins/iOS/HSLocalization/zh-Hant.lproj/HelpshiftLocalizable.strings
    error: failed to remove /Users/Me/Project/Build/My.app/HelpshiftLocalizable.strings: “HelpshiftLocalizable.strings” couldn’t be removed."

    It works without BuildOptions.AcceptExternalModificationsToPlayer. Is BuildOptions.AcceptExternalModificationsToPlayer not supported with Mono, or is this a bug? We are using Perforce but with the "allwrite" option, so the files should not have write protection on.

    Of note is that the other CopyStringsFile steps succeed. The following warning is also generated:
    "xcodebuild[22197:5389513] DVTAssertions: Warning in /Library/Caches/com.apple.xbs/Sources/IDEXcode3ProjectSupport/IDEXcode3ProjectSupport-9061/Xcode3Core/LegacyProjects/Frameworks/DevToolsCore/DevToolsCore/BuildSystem/Runtime/PBXTargetBuildContext.mm:739
    Details: unexpected successful exit code from cancelled command <C0002:'CpResource Data':p4>
    Object: <PBXTargetBuildContext: 0x7f9edfd3e260>
    Method: -createCommandInvocationRecordFromInvocation:
    Thread: <NSThread: 0x7f9edfb94f60>{number = 10, name = (null)}
    Please file a bug at http://bugreport.apple.com with this warning message and any useful information you can provide."

    2) 5.4 has incremental build support for WebGL. Do we need to use the BuildOptions.AcceptExternalModificationsToPlayer option to take advantage of it, or is it enough to not clear the cache folder under Library? (I have not tested 5.4 yet but want to prepare).

    Thanks a lot,
    - Tommi
     
  19. Tommi-Kiviniemi

    Tommi-Kiviniemi

    Joined:
    Aug 21, 2014
    Posts:
    7
    Josh, I just found a post by you from August that instructs to use
    PlayerSettings.SetPropertyInt("EnableIncrementalBuildSupportForIl2cpp", 1, BuildTargetGroup.iOS);
    for enabling incremental builds with IL2CPP for iOS.

    I will extend my questions to include this as well. I take that the above is not supported with Mono/iOS, but is it required for incremental WebGL builds in batch mode with Unity 5.4? And is that still a requirement for IL2CPP/iOS?

    There's quite a lot of overlapping concepts here. Even when using mono you can append to the Xcode project - is BuildOptions.AcceptExternalModificationsToPlayer the programmatic equivalent to append (when building in batch mode), and does it even save on build times, or is it just for preserving the Xcode project?

    We are very dependent on batch mode builds so it is absolutely necessary that incremental building can be done from the command line as well. Also, the documentation for these features is very spread out and it's hard to know what is required at the moment.

    Thanks again,
    - Tommi
     
    Last edited: Apr 12, 2016
  20. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @Tommi Kiviniemi

    Actually, we never had a bug report for this issue. We were able to track down the cause and correct it. This entry from the release notes is the correction for this incremental build issue.

    The BuildOptions.AcceptExternalModificationsToPlayer option is support on Mono, so this does sound like a bug, although I'm not sure of the cause initially. If you can submit a bug report, we can investigate it, although I'm not sure how easy it will be to reproduce your CI setup. Still, it is certainly worth a look.

    The BuildOptions.AcceptExternalModificationsToPlayer option is equivalent to Append. For iOS/Mono, it does not save build time. It only preserves the Xcode project. For iOS/IL2CPP, it allows incremental builds of the generated C++ code.

    For WebGL builds, BuildOptions.AcceptExternalModificationsToPlayer should have no impact. As long as the Library/il2cpp_cache directory is not cleared, WebGL builds will use incremental building of the generated C++ code.

    The EnableIncrementalBuildSupportForIl2cpp can be safely ignored for all build targets. It is no longer used. We shipped it only as a way to provide incremental builds during a testing phase, so that users who wanted to try them out could do so. I'll update that forum post so this information is clear.

    So in summary, incremental builds only occur for the IL2CPP scripting backend. On WebGL, incremental builds will happen by default, on iOS, you need to use the "Append" option or set BuildOptions.AcceptExternalModificationsToPlayer to use incremental builds. The Mono scripting backend does not use incremental builds.
     
    Tommi-Kiviniemi likes this.
  21. Tommi-Kiviniemi

    Tommi-Kiviniemi

    Joined:
    Aug 21, 2014
    Posts:
    7
    Thank you so much for the clarifications, Josh! Greatly appreciated.

    (You may want to specify the functionality of BuildOptions.AcceptExternalModificationsToPlayer in regard to IL2CPP builds in the documentation as well, as the relationship is not clear without your clarifications)

    - Tommi
     
  22. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @Tommi Kiviniemi

    Good point, I'll update the documentation for BuildOptions.AcceptExternalModificationsToPlayer
     
  23. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    Note that I've landed a documentation improvement internally to clarify how AcceptExternalModificationsToPlayer interacts with IL2CPP on iOS. You should see it in the 5.3 and 5.4 docs in the next week or so.
     
  24. matmuze

    matmuze

    Joined:
    Jan 14, 2014
    Posts:
    27
    Hello,

    I am writing an iOS app and I am using an OS X virtual machine on Windows to compile the XCode projects files generated from Windows.

    So far so good, my setup works lovely, however I cannot benefit from incremental build when compiling the C++ files with XCode when the Unity project was previously build from Windows.

    As a result, each minor change requires a full recompilation of the XCode proejct which is a bit lengthy.

    When building the project on Windows, I do not get prompted the "append" option like on OS X, instead I only see one popup which says.

    "WARNING: all files and folders located in target folder will be deleted by build process."

    Is there a work-around to get this feature working on Windows too, if not is it in the roadmap ?

    Thanks in advance.
     
  25. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @matmuze

    Supporting "append" on Windows is actually something that we've thought about doing in the past. I've just talked to our iOS team, and it looks like they plan to implement this. I'm not sure yet when it will be ready, but the feature should be coming.
     
  26. matmuze

    matmuze

    Joined:
    Jan 14, 2014
    Posts:
    27
    Wow thanks for asking this, will keep waiting then.

    Also, do you know know if incremental builds would work on Mac OS X when building a project via script with "BuildPipeline.BuildPlayer" too ?

    If not, do you know if this feature will ever be in the pipeline someday ?

    @JoshPeterson

    Cheers and thanks,
    Mat.
     
  27. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,446
    @matmuze

    Yes, incremental builds work with BuildPipeline.BuildPlayer. You will need to pass BuildPlayer BuildOptions.AcceptExternalModificationsToPlayer, which is the equivalent of "Append" in the GUI.
     
  28. matmuze

    matmuze

    Joined:
    Jan 14, 2014
    Posts:
    27
    Thanks a lot !!