Search Unity

StreamingAssets files are compressed in APK when "Build App Bundle (Google Play)" option is used

Discussion in 'Android' started by Unity_User_2, Sep 5, 2019.

  1. Unity_User_2

    Unity_User_2

    Joined:
    Aug 23, 2017
    Posts:
    2
    1. What happened

    We uploaded Android App Bundle (aab) to Google Play which was build with "Build App Bundle (Google Play)" option enabled in Unity (2019.2).
    Then we noticed the problem that some StreamingAssets files are not loaded during runtime. Our app loads these files directly as byte sequence assuming that they are stored uncompressed inside APK.
    So we get APK sample based on the uploaded aab and found out that these files are stored compressed (using HEX-editor on APK).
    We checked all SteamingAssets files in our app and concluded that this issue takes place only for files that have uppercase letters in their path (relative to StreamingAssets).

    Same behaviour in 2019.2.1, 2019.2.2.

    2. How we can reproduce it using the example you attached

    Sample project is just a new project created in Unity 2019.2 with ILL2CPP and ARM64 enabled for Android. Also there are some test files in StreamingAssets folder.
    2.1. Build Android App Bundle file (aab) from Unity 2019.2.
    2.2. Get APK from this aab using bundletool.
    2.3. Use HEX-editor on APK to check which files from StreamingAssets are stored uncompressed.
    StreamingAssets files in this project are:
    StreamingAssets/file1.txt
    StreamingAssets/File2.txt
    StreamingAssets/folder1/file3.txt
    StreamingAssets/folder1/File4.txt
    StreamingAssets/Folder2/file5.txt
    So only file1 and file3 are stored uncompressed. Their paths do not contain uppercase letters (relative to StreamingAssets).
    2.4. Export Android Studio project with "Build App Bundle (Google Play)" enabled
    2.5. Check build.gradle. noCompress section has all StreamingAssets files in lowercase:
    aaptOptions {
    noCompress = ['.unity3d', '.ress', '.resource', '.obb', 'file2.txt', 'file1.txt', 'folder2/file5.txt', 'folder1/file4.txt', 'folder1/file3.txt']
    ignoreAssetsPattern = "!.svn:!.git:!.ds_store:!*.scc:.*:!CVS:!thumbs.db:!picasa.ini:!*~"
    }
     

    Attached Files:

  2. ShadowAngel

    ShadowAngel

    Joined:
    Aug 7, 2012
    Posts:
    11
    Got similar problem with aab. I can read anything that in StreamingAssets folder root. But anything that is in sub folders unity can't access. Instead i get DirectoryNotFoundException. If i use apk, everything works fine.
     
  3. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    3,060
    There are known problems with app bundle tools with regard to compression. This is not Unity bug, but rather Google tools bug, so you have to work that around until tools are fixed and updated.
    As you've found, uppercase letters cause problems. IIRC the behavior is different depending on which OS you build on. When using a mix of all lowercase and uppercase/mixed case file names, you get some of those files compressed, but not the same one depending on whether you are on Windows or not (could be case-insensitive vs. case-sensitive file system, we haven't digged that deep).
    A workaround is to use all lowercase file names.
     
    hayone1 and Egil-Sandfeld like this.
  4. nik_d

    nik_d

    Joined:
    Apr 27, 2018
    Posts:
    65
    Other workaround is just to fix build.gradle file generated by
    UnityEngine.BuildPipeline.BuildPlayer
    before actual AAB building (
    ./gradlew bundleRelease
    ).

    It's easier than applying lowercases (for example with Addressables package integrated).

    Result is checked via
    bundletool build-apks
    and
    unzip -vl
    on all platfroms (linux, win, osx).

    Code (CSharp):
    1.     [UnityEditor.Callbacks.PostProcessBuild]
    2.     public static void FixAndroidBuildGradleForAab(BuildTarget buildTarget, string buildPath) {
    3.         if (buildTarget != BuildTarget.Android) return;
    4.         var projectDir = Path.Combine(buildPath, PlayerSettings.productName);
    5.  
    6.         var assetsDir = Path.Combine(projectDir, "src/main/assets");
    7.         _log.Log($"collect assets: {assetsDir}");
    8.         var assetsMap = Directory.GetFiles(assetsDir, "*", SearchOption.AllDirectories)
    9.             .Select(x => x.Substring(assetsDir.Length + 1))
    10.             .ToDictionary(x => x.ToLower());
    11.  
    12.         var gradlePath = Path.Combine(projectDir, "build.gradle");
    13.         _log.Log($"fix patterns: {gradlePath}");
    14.         var content = File.ReadAllText(gradlePath);
    15.         var mo = Regex.Match(content, @"noCompress = \[(.*)\]", RegexOptions.Singleline);
    16.         var oldPatterns = mo.Groups[1].Value.Split(new[] {", "}, StringSplitOptions.None).Select(x => x.Trim('\''));
    17.         var newPatterns = oldPatterns.Select(pattern => assetsMap.TryGetValue(pattern, out var realPath) ? realPath : pattern);
    18.         var result = $"noCompress = [{string.Join(", ", newPatterns.Select(x => $"'{x}'"))}]";
    19.         _log.Log($"result replace:\n{mo.Value}\n{result}");
    20.         content = content.Replace(mo.Value, result);
    21.         File.WriteAllText(gradlePath, content);
    22.     }
    23.  
     
  5. josep2

    josep2

    Joined:
    Mar 26, 2014
    Posts:
    13
    On post build the player I does not find the directory:
    DirectoryNotFoundException: Could not find a part of the path 'C:\aab\test\test\src\main\assets'. or
    DirectoryNotFoundException: Could not find a part of the path 'C:\aab\test\test\src/main/assets'.
    Any idea?
    Thanks
     
  6. nik_d

    nik_d

    Joined:
    Apr 27, 2018
    Posts:
    65
    New Unity (>2019) exports the project to subfoler <build-path>/unityLibrary/src/main/assets
    And it's better to re-check noCompress values in new Unity.. possibly it's already fixed.
     
  7. JuliusM

    JuliusM

    Unity Technologies

    Joined:
    Apr 17, 2013
    Posts:
    791
    Unfortunately to properly fix this for both APK and AAB files, Android Gradle Plugin version 3.6 or newer is needed. Our testing has shown that this Android Gradle Plugin version has quite a few behavioural differences compared to currently used version 3.4. These differences cause quite a few projects fail to build and projects have to be modified to be able to use this new version. Because of that we have decided to not upgrade Android Gradle Plugin version to 3.6 in Unity 2019 and 2018 versions for now.
    The suggested workaround for this compression issue is still the same - use lowercase letters.
    There is also another possible solution of using a custom Android Gradle Plugin version yourself. However this is an advanced process and as mentioned before, you might have to modify your project to be able to use this version. If you want to do this, you would have to increase Android Gradle Plugin version in your project's build.gradle file and provide a custom compatible Gradle version in Unity editor preferences. Note that Android Gradle Plugin versions 3.6.0-3.6.2 are compatible with Gradle versions 5.6.4 and newer.
     
  8. nik_d

    nik_d

    Joined:
    Apr 27, 2018
    Posts:
    65
    My proveded workaround that works with any unity and any gradle version is just to generate (or fix already generated) build.gradle to refer *files in it with their real-case* instead of lower-case only.

    And even more - I think this solution is much better than depending on some new lower-case features of gradle.

    And I don't understand - why you cannot generate build.gradle in a such correct way?)
     
  9. JuliusM

    JuliusM

    Unity Technologies

    Joined:
    Apr 17, 2013
    Posts:
    791
    We can't generate it that way because "real-case" as you call it works only for AAB files. If you'd try to build the APK this way, you would get your files compressed there.
     
    nik_d likes this.
  10. nik_d

    nik_d

    Joined:
    Apr 27, 2018
    Posts:
    65
    Funny, it's really gradle bug.)
    I didn't know, because I extract extract APK (for QA purpose) from AAB in the build process.
    Thanks for the explanation.
     
  11. drallcom3

    drallcom3

    Joined:
    Feb 12, 2017
    Posts:
    136
    Is it fixed in 2020.2?
    I can't open any files in steaming assets. Even lowercase and in root.
     
  12. JuliusM

    JuliusM

    Unity Technologies

    Joined:
    Apr 17, 2013
    Posts:
    791
    We are using android gradle plugin version 3.6 since 2020.1.0b14 and 2020.2.0a13, so the streaming assets should not be compressed in AAB files in newer 2020.1 and 2020.2 versions.
    However it seems that you are having a different issue. The issue discussed in this thread was that the loading of those assets are slow, not that they can't be opened. If you are unable to open them, then your issue is different.
     
  13. stgs73

    stgs73

    Joined:
    Mar 23, 2015
    Posts:
    59
    We tried nik_d soln to no avail, it's only AAB and started happening about 2 weeks ago give or take
    Not sure if it was any unity upgrade or addressables update so we're in the processing of eliminating everything
    We are using LZ4 in Addressables.
    This is not an issue whatsover with an apk, only an apk extracted from an AAB..
    Our gradle is still using 3.4.0 but we tried nik_d soln above so it isn't the character casing for us unless addressables is doing something bad.

    We are on 2019.4.13 ( we tried .11 and .14 no difference )
     
  14. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    3,060
    We have an outstanding bug currently, which is related to launcher/Unity library setup. Turns out when creating apk what matters is the launchers setup, not libraries setup, hence even though files are correctly marked as noCompress in library project, they still will be compressed if not specified so in the launcher.
     
  15. stgs73

    stgs73

    Joined:
    Mar 23, 2015
    Posts:
    59
    Are there any workarounds right now until this is fixed? The performance is really bad even on flagship phones. Maybe customizing the launcher gradle? We don't use that at the monent, just a vanilla mainTemplate

    thanks
     
    Last edited: Nov 16, 2020
  16. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    2,844
    in launcher's build.gradle, locate aaptOptions.noCompress, add files which don't need to be compressed there. Then build the apk, you can then open that apk with tool like 7-zip, and check the compression method next to target file, it should be called "Store" instead of "Deflate"
     
    stgs73 likes this.
  17. stgs73

    stgs73

    Joined:
    Mar 23, 2015
    Posts:
    59
    Thnx tomas, we added '.bundle' in there and it seemed to resolve our issues
     
  18. Tomas99912

    Tomas99912

    Joined:
    Aug 2, 2017
    Posts:
    47
    there still have this problem on unity2020.3.x
     
  19. AwesomeDemo

    AwesomeDemo

    Joined:
    Feb 28, 2020
    Posts:
    1
    Same here, did you find any workaround?
     
  20. francismoy

    francismoy

    Joined:
    Jul 5, 2017
    Posts:
    40
    I started experiencing this problem after updating from 2019.4.10f1 to 2019.4.30f1. I tried using the Addressables 1.8.5 (the version used in the former Unity version), but the problem persisted even after downgrading the project back to 2019.4.10f1: APKs would work as usual but the AAB would be very slow during assets loading.

    I've just updated Unity to 2020.3.19f1 (with Addressables 1.18.16) and I don't have the problem anymore (I had to update the launcher and main gradle files, which can be managed automatically by Unity if you uncheck, remove the files and re-check again from the Project settings).

    There was definitely some compression applied in version 2019.4.30f1 since the downloaded APK from the Play Store was around 80 MB, and after building with 2020.3.19f1 the size is around 93 MB.
     
unityunity