Question Concerningly long build times

Hello,

We have a mid sized project that we recently tried Unity Build Automation on. It's a VR project that builds for Android. Our workspace is 9.4 gb, as reported by Unity. Our APK comes out to 700 mb.

On my local machine, incremental builds take 1:58 (one minute, 58 seconds). However, on Unity Build Automation, they take 42 minutes to 44 minutes. One of our goals for build automation is to speed up the process of a developer creating an APK (2 min), uploading it to the cloud (20 minutes), and having another team member download the build (~1 or 2 minutes). Unfortunately, with build times of 40+ minutes, it's a time-loss for us.

I understand cloud builds are expected to be slower than local builds, but the degree of slowness is concerning (especially when we are paying by the minute).

I went to investigate and found a few strange things:

[2023-04-01T05:21:09.283Z] - 7.3.3.2.4.2 - INFO: ++++++++++++++++++++ SECTION ++++++++++++++++++++

[2023-04-01T05:21:09.283Z] - 7.3.3.2.4.2 - INFO: + Checking out Source Code

[2023-04-01T05:21:09.283Z] - 7.3.3.2.4.2 - INFO: ++++++++++++++++++++ SECTION ++++++++++++++++++++

[2023-04-01T05:21:10.384Z] - 7.3.3.2.4.2 - INFO: BVR: SCM Checkout

[2023-04-01T05:21:10.387Z] - 7.3.3.2.4.2 - INFO: GIT: Checkout project with depth of -1...

[2023-04-01T05:21:10.617Z] - 7.3.3.2.4.2 - INFO: GIT: git version (2.31.1)

[2023-04-01T05:21:10.617Z] - 7.3.3.2.4.2 - INFO: GIT: Checking out git@git.assembla.com:xxxxxxxxxxxx.git/master with depth of -1

[2023-04-01T05:21:10.765Z] - 7.3.3.2.4.2 - INFO: Attempt 1 of 5 to clone repository.

[2023-04-01T05:31:46.456Z] - 7.3.3.2.4.2 - INFO: Cloned in 635 seconds.

Incremental pulls of the project seem to take a full 10 minutes. This project was already checked out and built several times. (the first build took 3.5 hours, so I'm fairly confident it's not doing a clean checkout here). The use of the word "clone" and "Checkout" muddies the waters here a bit.

As a point of comparison, a "git pull" on my local computer completes in 1.28 seconds.

It also automatically compresses and decompresses the library between builds.
Extracting and restoring the library from the remote cache takes 3 minutes. Compressing the library takes 14 minutes. The APK is built but inaccessible until this step is completed.

[2023-04-01T05:49:06.063Z] - 7.3.3.2.4.2 - INFO: Zipping cache files from Library

[2023-04-01T06:03:08.834Z] - 7.3.3.2.4.2 - INFO: Created the archive file.

The actual build time recorded on the cloud is 5 minutes, which is pretty close to the local 2 minutes... but there's just an extra 35 minutes of "other stuff" that we get billed for and adds a lot of dead time to our team's dev cycles. More concurrency isn't really helpful for us.

Since this extra library compression and decompression takes so much time:
($0.02 * (14 +3)) = $0.34, per build

We'd rather pay the Unity rate of $0.14/GB/month to not spend the time compressing our 22gb Library folder. It would pay for itself in just 9 builds per month.

I'm also not sure what's going on with the source control step taking so long. We were expecting cloud builds to take 2x or 3x the time of our local builds, not 20x. Is there anything that can be done to improve this process for us?

Thank you!

 

Thanks for the reply! We will experiment with the compression disabled and see how it helps. I appreciate that tip!

I had assumed since it was caching the library folder, it would cache the rest of the assets for the project, that's interesting to know. So that will also download all the files in the repository, including files that have been deleted (but are still in the git history).

We would eventually like to use cloud builds as a form of continuous integration, so that means running after each commit. Most of our commits are very small code changes, so incremental pulls would be a pretty huge time savings for us.

 

I'm seeing the same issue with long Library cache compression times, but apparently only on the Windows builders. We build our game for iOS and Android, using the Mac builder for iOS, and Windows builder for Android.

"Download... (and) Extracting cache files" takes around 30s on the Mac builder, and 3m 30s on the Windows builder.

For a build with no asset changes, "Zipping cache files from Library" takes 2m 30s on the Mac builder, but it takes at least 18m on the Windows builder. Builds with asset changes can take >30m to zip on the Windows builder.

Assuming this is an issue only on the Windows build machines, why would the archive zipping process be so much slower? Like others, we are optimizing our build and repo to reduce build times and build costs, but this doesn't seem to be something we can address. (outside of paying a smaller cost for an uncompressed cache)

Note that I have not tried building Android on the Mac builder recently, as it will be cost ineffective.

 

I set compressionMethod to "none" on the Android build target that uses the Windows builder, then ran the build target twice, both times with no code or asset changes.

"Download... (and) Extracting cache files" was slightly faster by about 30s. The "Zipping cache files from Library" step dropped from 18m in the previous build (with compression enabled) to under 9m in the latest builds with compression disabled.

So yes, for this test, the library caching step on the Windows builder appears to be ~2X faster with compression disabled. It is still quite a bit slower than what I'm seeing on the Mac builder, so I'm guessing that compression isn't the only performance impact.

But if these uncompressed cache times hold true, the total metered cost should be about the same as on the Mac builder.