Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Bug Extreme build performance regression with com.android.tools.build:gradle >= 3.6.x

Discussion in 'Android' started by AlkisFortuneFish, Oct 6, 2021.

  1. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    888
    I have had to upgrade com.android.tools.build:gradle, in order to build AABs (which needs 4.0.x) and to support ARCore (which needs 3.6.x) and noticed that certain gradle build steps are nearly 40 times slower than with 3.4.x.

    I tracked the regression to the jump from 3.4.x to 3.6.x and it is the same on every version after that.

    Here are some Gradle Build Scans for the same build done with different versions of com.android.tools.build:gradle, both done after cleaning the project.

    3.4.3 - Total build time of 14.783s
    3.6.4 - Total build time of 6m 49.483s (!!!!)

    The offending steps are:
    :launcher:mergeDebugJavaResource 6m 41.434s com.android.build.gradle.internal.tasks.MergeJavaResourceTask
    :launcher:desugarDebugFileDependencies 6m 36.926s com.android.build.gradle.internal.tasks.DexFileDependenciesTask
    :launcher:mergeDebugNativeLibs 6m 35.912s com.android.build.gradle.internal.tasks.MergeNativeLibsTask
    :launcher:processDebugResources 6m 34.827s com.android.build.gradle.internal.res.LinkApplicationAndroidResourcesTask

    Any idea what I could possibly do to solve or at least improve this? It massively increases my iteration times.
     
  2. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    888
    I've done a little bit of hunting to see what on earth is being passed as an argument to aapt2.exe to take nearly 7 minutes to complete. To my surprise, I found that it passes the longest regular expression I have ever seen in my life as an parameter to --no-compress-regex.

    The regex appears to be a character by character match of every asset in the build, which in our case is a lot of assets. The regex is 189,189 characters long.

    Heavily truncated example:

    Code (CSharp):
    1.  
    2. --no-compress-regex (((a|A)(a|A)/(a|A)(n|N)(d|D)(r|R)(o|O)(i|I)(d|D)/(b|B)(a|A)(s|S)(e|E)(a|A)(n|N)(i|I)(m|M)(a|A)(t|T)(i|I)(o|O)(n|N)(s|S)_(a|A)(s|S)(s|S)(e|E)(t|T)(s|S)_(a|A)(l|L)(l|L)\.(b|B)(u|U)(n|N)(d|D)(l|L)(e|E)$)|((a|A)(a|A)/(a|A)(n|N)(d|D)(r|R)(o|O)(i|I)(d|D)/(c|C)(u|U)(r|R)(r|R)(e|E)(n|N)(c|C)(y|Y)(i|I)(c|C)(o|O)(n|N)(s|S)_(a|A)(s|S)(s|S)(e|E)(t|T)(s|S)_(a|A)(l|L)(l|L)_(d|D)0(f|F)18(e|E)(e|E)
    3.  
    What on earth is going on!?
     
  3. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,238
    Oh, that's concerning, I assume, it auto converted things assigned to aaptOptions.noCompress in \unityLibrary\build.gradle, do you have a lot of files there?
     
  4. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    888
    In this case it's launcher/build.gradle, but that is indeed the issue, that is precisely what it is doing. It would appear that every asset bundle in streaming assets is ending up in aaptOptions.noCompress and the build tools explode that to that monster regex. Replacing that whole list with '.bundle' brings the build time down to nothing again. This leads to two obvious questions: Why was it fast before 3.6.x and why is Unity putting every file in there individually as opposed to just .bundle.

    Edit: Actually, dumping the aapt2 args that 3.4.x was spitting out, I can see why it used to be fast, it used to send each one to an individual -0 argument. 3.6.x compiles a monster regex out of the whole thing itself.

    Edit 2: Found the offending entry in the changelog:
    • aaptOptions.noCompress is no longer case sensitive on all platforms (for both APK and bundles) and respects paths that use uppercase characters.
    That is an... interesting... way of implementing that.
     
    Last edited: Oct 8, 2021
  5. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,238
    I think -no-compress-regex is a new option in android gradle plugin, it's surprising, that google under hood started using it implicitly. The full paths are specified, because not to collide with other Unity resources.

    Could you report a bug with a repro project - https://unity3d.com/unity/qa/bug-reporting, so we can have this on our radar? Thank you
     
  6. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    888
    Sure, will do on Monday.
     
  7. edvinas-mandravickas

    edvinas-mandravickas

    Unity Technologies

    Joined:
    Aug 8, 2019
    Posts:
    11
    Hi all!

    Recently I did some changes related to how Unity creates no-compress list. If a file has a distinct extension (not shared between compressible and not-compressible assets) then no-compress list will only contain an extension of that asset. This dramatically reduces size of the list in cases where projects contains a lot of streamable assets with the same extension. For those who has the slowdown issue with a gradle plugin I suggest to try it out. It might help a lot. It won't make the plugin faster, but it will reduce a number of operations it has to do.

    The feature is available in:
    2019.4 - planed to be backported
    2020.3.13f1
    2021.1.9f1

    and up...
     
  8. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    888
    Thanks for the heads up. I tried to make a repro against 2021.1.21 and the behaviour has been quite different, so this would explain that. I haven't tested with my actual project (it doesn't build in 2021) but I have copied the directory structure of my StreamingAssets across and it does condense everything to .bundle, which is my current workaround on 2019.4.31 anyway, bringing build times from 7 minutes down to the 15 seconds it was on com.android.tools.build:gradle <= 3.4.x. It is obvious that aapt2 was ending up spending literal minutes evaluating that regex.

    One other thing that is interesting is that if I force 2021.1.21 to generate an asset list as before (by seeding with files that don't have an extension) the issue is slightly different than it was on the project generated by 2019.4.31, instead of the build getting ridiculously slow there is a threshold of files past which aapt2 just immediately crashes.
     
    Last edited: Oct 12, 2021
  9. edvinas-mandravickas

    edvinas-mandravickas

    Unity Technologies

    Joined:
    Aug 8, 2019
    Posts:
    11
    Well there was a file limit with older gradle plugins as well. I believe it was around 620 files or so. If more entries would be added to no-compress-options aapt2 would just refuse to deal with it. That's why I was implementing this workaround I mentioned in the first place.

    Just to be clear: You are saying that newer versions of gradle plugin still has this issue?
     
  10. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    888
    -ish. From 3.6.0 onwards, the Gradle plugin generates a single monstrous regex for everything provided to aaptOptions.noCompress and passes it into -no-compress-regex rather than a series of -0 entries. So, when building from 2019.4.31 (which is missing your fix) it ends up generating an absolutely ridiculous regex for the file list and slows the build down to a crawl, even with fewer elements than 620. When trying to repro, I did get to a number where it was just outright crashing rather than being slow, which sounds similar to the original issue. Either way, your fix should sort both issues out.
     
  11. edvinas-mandravickas

    edvinas-mandravickas

    Unity Technologies

    Joined:
    Aug 8, 2019
    Posts:
    11
    Thank you for all the insight into this. I knew they added regex but did not test how it performs myself. I'll try to push backport of the fix to 2019.4 ASAP :)
     
unityunity