Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug "compiling cpp for libil2cpp" takes much longer suddenly.

Discussion in 'Editor & General Support' started by Hyperactive, Aug 24, 2023.

  1. Hyperactive

    Hyperactive

    Joined:
    Aug 13, 2013
    Posts:
    13
    Hey,

    I've been working on my project for a few years, using IL2CPP just fine. Build times for Android have always taken about 30secs on average. Though now suddenly all builds take up to 20mins, and 99% of the time is spend on "compiling cpp for libil2cpp".

    And my CPU goes through the roof when it goes, bogs up my computer!

    What I've tried based on other threads with roughly the same topics:
    - Disabled BitDefender
    - Deleted my project library
    - Reinstalled Unity
    - Reinstalled my IDE (Rider).

    It does seem to be random though, very occasionally the build time will be 30 secs. Then the rest of the time about 15-20mins. And I'm not doing anything different, I've done some experiments to check before building each time, like editing some code in various scripts, editing prefabs, scenes etc... Though it just seems to be quite random as to when it needs to do the "compiling cpp for libil2cpp" for 20mins.

    In my editor logs, all the files being compiled seem to be the same between my 30 sec builds and 20mins builds. The difference is the time spent on certain files. Basically all the files in "C_Android_arm64 Library/Bee/artifacts/Android/d8kzr" (like 50 files or so) take about 0-2secs each in the 30 secs build.
    But in the 20mins build they average from 10secs to 180 secs!

    Some of the threads about IL2CPP are years old, wondering if there's any clear solutions since then to solve builds taking so long? Especially that mine seems to be random, and out of the blue. Like... sometimes it uses the cache and othertimes it doesn't?

    And to clarify again, this has never been an issue over the years. No matter if I make new scenes, make new scripts, import assets etc... The build time would barely take a few minutes the first time maybe, but then 30 secs most of the time.

    Thanks!
     

    Attached Files:

    Last edited: Aug 24, 2023
  2. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    Which version of Unity are you using?
     
  3. Hyperactive

    Hyperactive

    Joined:
    Aug 13, 2013
    Posts:
    13
    2022.3.7f1
     
  4. Hyperactive

    Hyperactive

    Joined:
    Aug 13, 2013
    Posts:
    13
    I had a look at the editor log in realtime for once. It really does the usual for the first 30secs, then just gets stuck on all the "C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/" files for the next 20mins, before getting to the build gradle part which takes about 10secs. Then it's complete.

    It doesn't freeze or anything, it just goes through a huge amount of these .o files while pretty much using 100% of my CPU (i9-10850k), I can barely do anything else on my computer:

    example:
    --------------
    1647/1673 18s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/vm8y7zb3ae6s.o
    [1648/1673 3s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/1jqxnvpgc7em.o
    [1649/1673 17s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/a3bskqtllyxi.o
    [1650/1673 28s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/urefy0vhg78i.o
    [1651/1673 26s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/qv10oc9wnpz5.o
    [1652/1673 15s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/a97orr79qv03.o
    [1653/1673 19s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/nitceoz6wb6z.o
    [1654/1673 22s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/odnwnnfsri39.o
    [1655/1673 15s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/a74asqx57pu9.o
    [1656/1673 18s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/47dwone1gz66.o
    [1657/1673 14s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/8nhx2jxqbfo4.o
    [1658/1673 17s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/y0hwkm1uahhv.o
    [1659/1673 30s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/yh5nbv70lt9v.o
    [1660/1673 20s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/1nsqeziuzatc.o
    [1661/1673 37s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/zgevjw988vkf.o
    [1662/1673 16s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/fslseevlmxb4.o
    [1663/1673 28s] C_Android_arm64 Library/Bee/artifacts/Android/d8kzr/p7zzlbdkr3o1.o
    [1664/1673 1s] Link_Android_arm64 Library/Bee/artifacts/Android/d8kzr/libil2cpp.so
    --------------

    Hope that helps a bit! Thanks for the assist.
     
  5. M_Dhanu

    M_Dhanu

    Joined:
    Aug 25, 2023
    Posts:
    4
    I'm using the same version and it takes a lot of time to do 'compiling cpp for libil2cpp'. Hope you can look into this. Thank you in advance!
     
  6. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    Thanks for the information! In Unity 2022.3.7, the Unity player build system will emit a file named buildreport.json in the Library/Bee directory of the project. This is a Chrome tracing file (see https://aras-p.info/blog/2017/01/23/Chrome-Tracing-as-Profiler-Frontend/ for details).

    If you can find this file, load it up in Chrome - it should break down the timing of all parts of the build process, and help you to narrow down what the specific cause is.

    My guess is that the linker from the Android NDK is the bottleneck now, but I might be wrong. Let me know what you find in this file, and we can go from there.
     
  7. Hyperactive

    Hyperactive

    Joined:
    Aug 13, 2013
    Posts:
    13
    Awesome! That's great, never saw that before.

    So I loaded up a build report that took 10mins to complete. It actually shows exactly how I described it before:
    The usual process happens in less than a minute, then spends the rest of the time of something fishy. Then gradle build quickly before ended.

    I've attached screenshots of the report, can see the majority of time is spent in "UpdateFromStreamAsync", and then inside that there's a huge amount of "ReadAsync". Not entirely sure what any of this means though, so any guidance here would be appreciated.

    If your guess is correct though, is there a more stable version of Android NDK I should be using if not the one that comes with Unity?

    Thanks!
     

    Attached Files:

  8. Hyperactive

    Hyperactive

    Joined:
    Aug 13, 2013
    Posts:
    13
    Sharing this part of the log as well. As this seems to be what we see as well in the build log. The " C_Android_arm64 " which I meantioned previous seems to be hogging up all the power for 10mins.

    Though based onlooking at the log, and the loading bar together I can see there seems to be some kind of repetition somewhere that's causing this? For example, when I make bigger changes to the project, it'll take 3mins or less to build previously.

    Here we can see, it gets through a first phase of compiling at about 3mins, which is normal. And would usually finish then with a build time of 3mins. But instead, even the loading bar returns from almost finished back to the start. And starts all over again. But this time round for the next 7 mins throttles my CPU and then finishes the build after that.

    This is reflected in the loading bar, the build log and the build report.

    It's not like the entire process is just overall longer, it looks like it's trying to do something after it should be complete. Though very occasionally it doesn't. For example, this morning I logged in, pulled changes from git, and the project built in under a minute! Hurrah! But then... I tried to build a second time after a tiny bit of work, and it takes 10mins. And now all builds take 10-20mins, even when I don't do anything.
     

    Attached Files:

  9. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    @Hyperactive: Can you send me the full buildreport.json file from a long build (maybe via direct message)? I'd love to have a look and see if anything jumps out at me.
     
  10. Hyperactive

    Hyperactive

    Joined:
    Aug 13, 2013
    Posts:
    13
    Done! Thanks.
     
  11. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    Thanks, I'll follow up on the private message, then report back here to provide any public details for others.
     
  12. imxaander

    imxaander

    Joined:
    Jan 8, 2023
    Posts:
    1
  13. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    Sorry, I forgot to follow up! It looks like the build cache is unexpectedly being cleared, but we cannot tell why.
     
    imxaander likes this.
  14. Saeed-Barari

    Saeed-Barari

    Joined:
    Jul 12, 2021
    Posts:
    51
    hi, exact issue on my side as well. @JoshPeterson did you and the team find any solutions to this? can this be caused by some build pipeline settings? (maybe 2 options that shouldn't be used together are set to be used together?)
     
  15. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    Unfortunately, we did not track down the cause of this issue. It happens because the build cache was cleared, leading to a full rebuild when we should need only an incremental build. But as to why that build cache was cleared - we are unsure.

    If you can provide any other information or insight about why this is happening, that would be appreciated!
     
    Saeed-Barari likes this.
  16. MSSJ_GAMEDEV_UNITY

    MSSJ_GAMEDEV_UNITY

    Joined:
    Jul 4, 2020
    Posts:
    5
  17. MSSJ_GAMEDEV_UNITY

    MSSJ_GAMEDEV_UNITY

    Joined:
    Jul 4, 2020
    Posts:
    5
    may this be related to UCVS? im not sure excatly but might synchronising work cause the build cache to be reset or modifed in a way forcing unity to full build rather than incremental?
     
  18. MSSJ_GAMEDEV_UNITY

    MSSJ_GAMEDEV_UNITY

    Joined:
    Jul 4, 2020
    Posts:
    5
    if that maybe the problem, Is it a bug? or is there any good practice that can be done in UCVS project to prevent this?
     
  19. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    I'm not familiar with UCVS, so I can't offer specific recommendations. However, we don't want a version control system to be involved with an build cache artifacts. Those can be generated from source code that is under version control, but the Unity build system will assume that build cache artifacts are not managed by some external actor.
     
  20. BkrullCyclomedia

    BkrullCyclomedia

    Joined:
    Jan 3, 2024
    Posts:
    2
    I think we're starting to see this after an upgrade to Unity 2022.3.19f1 from Unity 2021. @JoshPeterson

    Has Unity found any cause for this yet? I have a coworker who had their long build times fixed by using a fresh Windows 11 install partition, but it seems to have reappeared after a day or two in that partition.

    Where are these build caches located?
     
  21. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    We've not been able to determine a cause of this yet, sorry.

    The build caches are in the Library/Bee directory in the Unity project.
     
  22. ChrisAmm

    ChrisAmm

    Joined:
    Feb 23, 2021
    Posts:
    2
    Same problem here. Compiling for HoloLens 2, taking twice as much time.
     
  23. ChrisAmm

    ChrisAmm

    Joined:
    Feb 23, 2021
    Posts:
    2
    Lump_il2cpp_os,cpp Unity 2019.4 VS2019
     
  24. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,894
    Can you install Visual Studio 2022 and attempt to build with it? IL2CPP fully supports it, and we've found that the build is much faster with VS 2022 than VS 2019.
     
  25. soleron

    soleron

    Joined:
    Apr 21, 2013
    Posts:
    546
    Interesting because I have a problem related to that.
    It does not find the folder/file that Bee is supposed to create.

    Only my issue is with UWP not Android and the issue is on 2023.2.5/10
    I use VS22
    I will try building an Android version just to see what happens.

    I am Using a "Dev Drive" so antivirus is not really as "lockdown" as it is on the rest of the system.


    it also does that, a couple of times, and i was wondering what could be making my machine feel like hanging occassionaly while I am building.

     
    Last edited: Feb 20, 2024