Search Unity

Bug (Case 1355053) Addressables keeps downloading data when storage device has no free space left

Discussion in 'Addressables' started by Peter77, Aug 3, 2021.

  1. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,609
    Unity 2019.4.20f1, Addressables 1.18.13, Android Player

    Addressables.DownloadDependenciesAsync
    continues to download data when the storage device has no free space to store the downloaded data, which causes several different errors, such as:
    Code (CSharp):
    1. Failed to write compressed chunk to the archive '/storage/emulated/0/Android/data/com.MyCompany.AddressablesContentUpdate/files/UnityCache/Temp/3c389541627bff2068df80307514d00d/__data'! Error: 14
    2.  
    3. RemoteProviderException : Unable to load asset bundle from : http://192.168.1.2:49632/files_assets_assets/files/file36.bytes_c3728536d68dba52ca2833849f3921ed.bundle
    4. UnityWebRequest result : Unable to write data
    5. ResponseCode : 200, Method : GET
    6. url : http://192.168.1.2:49632/files_assets_assets/files/file36.bytes_c3728536d68dba52ca2833849f3921ed.bundle
    7.  
    8. Failed to create temporary cache directory '/storage/emulated/0/Android/data/com.MyCompany.AddressablesContentUpdate/files/UnityCache/Temp/5f69707ae0cec84e3b075ea5b958dbc2'


    Reproduce
    1. Open attached project
    2. Switch to Android build target
    3. Click from the main menu "BugReport > Create Files"
    4. Open "Window > Asset Management > Addressables > Profiles" and activate "Editor Hosting"
    5. Open "Window > Asset Management > Addressables > Groups" and click "Build > New Build > Default Build Script"
    6. Open "Window > Asset Management > Addressables > Hosting" and enable it
    7. Click "File > Build Settings..."
    8. Enable "Development Build" and "Script Debugging"
    9. Open "Window > Analysis > Android Logcat" and filter for "Unity" in the search-field
    10. Connect a test device with about 200 MB free space left
    11. Click "File > Build and Run"
    12. When the Android Player started open "Window > Analysis > Android Logcat", filter for "Unity" in the search-field and connect to the Player
    13. In the Android Player click the "Check for new content" button

    Actual
    Addressables.DownloadDependenciesAsync keeps downloading data even when there is no free space on the device and fails after it downloaded everything.

    Expected
    1. Addressables provides an API to check if the downloaded and decompressed data can be stored on the device prior downloading the data.
    2. Addressables.DownloadDependenciesAsync fails if the downloaded and decompressed data can't be stored on the storage device due to insufficient free space and does not keep downloading data.
     

    Attached Files:

    Peter77, Aug 3, 2021
    #1
    chrismarch and TreyK-47 like this.

  2. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,609
    Our game went in early-access a few days ago and we see many users run into exactly this issue!

    Here are a few of those errors users experience due to Addressables being unable to detect if sufficient free space is available on the storage device while downloading and decompressing bundles and writing the data to the storage device.

    The problem is that Addressables does not properly report this error, so I'm unable to display a clear error message that indicates the content can't be downloaded, because insufficient space is available. At the moment this behavior causes all sorts of issues and leads to a gazillion of further unwanted side-effects as well. I would really like to catch the "insufficient free space is available" before or during the download.

    Code (CSharp):
    1. Error while downloading Asset Bundle: Failed to decompress data for the AssetBundle 'https://blabla.bundle'.
    2.  
    3. Failed to write compressed chunk to the archive '/var/mobile/Containers/Data/Application/blabla/Library/UnityCache/Temp/41c3d22ed56db43869cbff6d7b8ec5af/__data'! Error: 14
    4.  
    5. Unable to move cache file '/var/mobile/Containers/Data/Application/blabla/Library/UnityCache/Temp/483a9f5c8f6e042cf8297498b26c5fe6' to '/var/mobile/Containers/Data/Application/blabla/Library/UnityCache/Shared/7c25b4cefe53962b3da15afb45cc3d47/02f82fffa81a2ab4dd838b910ca93ea4': error code - 14
    6.  
    7. Failed to write compressed chunk to the archive '/var/mobile/Containers/Data/Application/blabla/Library/UnityCache/Temp/07b6870b5f4614cb399b3d1062d786e6/__data'! Error: 14
    8.  
    9. Failed to create temporary cache directory '/var/mobile/Containers/Data/Application/blabla/Library/UnityCache/Temp/aead231e1527c44739937c243def770b'
    10.  
    11. Failed to initialize storage creator for file: '/var/mobile/Containers/Data/Application/blabla/Library/UnityCache/Temp/86704f653cc7948aca52a4f5465f4d38/__data'
    12.  
    13. Failed to write archive for block based file: '/var/mobile/Containers/Data/Application/blabla/Library/UnityCache/Temp/86704f653cc7948aca52a4f5465f4d38/__data'
    14.  
    15. Unable to reserve header in the archive file: '/var/mobile/Containers/Data/Application/blabla/Library/UnityCache/Temp/28a26c2fd1d344dbeb0a54321aeccd42/__data'
    Please improve the Addressables error handling for this case, so that we can inform players upfront if they don't have enough free space to store the downloaded content.

    In the meantime, can you please provide a workaround or point out how to workaround the issue?

    @davidla_unity gave a few tips here, but these are incomplete to my knowledge. Things that are unclear to me:
    1. How to get the uncompressed size of the content that's about to be downloaded? Addressables.GetDownloadSizeAsync reports the download size, but I need to know the install size. The size that's needed to store the uncompressed bundles.
    2. I guess Unity/Addressables also requires additional space to store temporary files to eg. decompress them?
    3. If the download size 200MB, but older bundles of the updated bundles exist already, I guess 200MB free space are not required, because the old bundles get replaced. How would I handle this case when the CacheClearBehavior is set to "Clear when new version is loaded".
    4. How to get the storage device where Addressables stores the download bundles?
    Thank you for any insights.
     
    Last edited: Nov 6, 2021
    Peter77, Nov 5, 2021
    #2
    chrismarch, Nyankoooo and kjyv like this.

  3. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,820
    Flagging for the team!
     
    TreyK-47, Nov 5, 2021
    #3
    Peter77 likes this.

  4. davidla_unity

    davidla_unity

    Unity Technologies

    Joined:
    Nov 17, 2016
    Posts:
    763
    Hey Peter,

    tl;dr This makes sense and is definitely something we're going to try and address. I've already started some discussions with the team on how to handle this. I don't have anything concrete as of yet, however. I'll try and talk about a few different points below.

    So, I did some asking and a few answers I received may shed some light. If you're using UnityWebRequest, which Addressables does by default, then the AssetBundle is directly compressed into Lz4 from the stream. No additional disk space should be used when loading Lz4 chunks, either. I'm curious, are you using LZMA compression on your bundles? If I remember right, LZMA compresses smaller than Lz4 so I could definitely see a situation where the recompress to Lz4 is causing space issues.

    Let me try and answer a couple of your questions. As far as the CacheClearBehavior goes that does only clear out the old bundles after the new one is downloaded in case there's an issue with the download. In your situation, this doesn't help much. You might try doing an Addressables.ClearDependencyCacheAsync and clearing the cache for a given key before attempting a download.

    You can also use any of the Unity Cache APIs directly as it's the mechanism Addressables uses by default. If you wanted to clean your entire cache you can try Caching.ClearCache but that would require all remote bundles to be downloaded again. I doubt that's a viable option.

    You might could hook into the TransformInternalIdFunc and check if the requested resource was an AssetBundleResource and, if so, use Caching.ClearAllCachedVersions if the download size for a bundle is > 0. That could be difficult to pull off, however. But it might be worth looking into as a possible temporary solution.

    You can use the Caching API in the engine to get the cache path. Caching.currentCacheForWriting.path should get you the path. You can also set the currentCacheForWriting to a cache at any path you want.

    Here's some examples of how it's used with regular AssetBundles https://docs.unity3d.com/ScriptReference/Caching.html.

    If any of what I said contradicts what you're seeing, let me know.
     
    davidla_unity, Nov 12, 2021
    #4
    Peter77 likes this.

  5. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,609
    Thank you for the reply.

    Our game is live since November 2021 and we have thousands of players who already ran in this "no free space" problem according to Cloud Diagnostics. It's actually the number one issue we currently have, because it creates a gazillion side problems as well. For example, Addressables downloads bundles, can't decompress/store them, returns "everything is fine" and the game continues and then runs in all sorts of different problems.

    The sooner you can solve this problem, the better for us.

    I don't know, here is the setup of a remote group:

    upload_2022-1-11_9-53-41.png

    Yes, I use LZMA for all remote bundles and LZ4 for local bundles.

    I use the "Clear when new version loaded" Clear Cache Behavior. Shouldn't this option ensure that any cached data from this bundle is being cleared?
     
    Last edited: Jan 11, 2022
    Peter77, Jan 11, 2022
    #5

  6. chrismarch

    chrismarch

    Joined:
    Jul 24, 2013
    Posts:
    472
    Has any of this been implemented?

    Even looking at the Unity API, including Cache (as stated in the API Docs "does not account for total available disk space"), it seems the only way to test for free disk space in a unity player on mobile is to use a third party plugin, so it would seem Unity (2020.3, etc.) doesn't consider free disk space in the Cache or Addressables systems?
     
    chrismarch, Jan 28, 2022
    #6

  7. davidla_unity

    davidla_unity

    Unity Technologies

    Joined:
    Nov 17, 2016
    Posts:
    763
    Hey all, I don't have much of note to post but I wanted to say we're aware of the problem and are going to look into what we can do to handle the situation better.

    At the very least I think a lot of our problem comes from the file download size vs. the recompressed file size on disk (I posted a longer explanation in a different thread that I can't seem to find at a quick glance). That makes it basically impossible for anyone to perform a disk space check on their own prior to downloading.

    As far as the "if you're using UnityWebRequest" comment, I should have noted then, if you haven't written a custom AssetBundleProvider, you are using UnityWebRequest (UWR). It's what's used by default.

    I apologize that this post isn't more help, but I wanted to make sure you all knew we were aware there's an issue here.
     
    davidla_unity, Feb 3, 2022
    #7
    CineTek and Peter77 like this.

  8. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,609
    Thank you for the reply and additional information.

    I would appreciate when my bug-report (1355053) can be confirmed by QA and turned into an issue that I can track through the public issue tracker. Any chance that could happen?
     
    Peter77, Feb 4, 2022
    #8
    CineTek likes this.

  9. CineTek

    CineTek

    Joined:
    Jul 12, 2013
    Posts:
    98
    This "bug" is another one of those "game breaking" issues that are currently stopping Addressables from being used in a true production environment. I am very afraid of our AppStore submission later this year if those issues persist. I am still highly in favour to mark the addressables package as non stable tbh for those exact reasons.
     
    CineTek, Feb 4, 2022
    #9
    Peter77 and Prodigga like this.

  10. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    1,123
    Can we atleast get an ETA on this being fixed? As I understand it, there are cases like this where no exceptions are raised when there should be.

    We could at the very least present the user with a "something went wrong" dialogue if an exception was thrown
     
    Prodigga, Feb 12, 2022
    #10
    CineTek, SugoiDev and Peter77 like this.

  11. andymilsom

    andymilsom

    Unity Technologies

    Joined:
    Mar 2, 2016
    Posts:
    294
    Any major changes in Addressables will be in 1.20.x version. As any change for this will be a behavioural change.
    Before 1.20 we will be fixing minor cases.

    I would recommend the following for checking against the cache. It does not however check against disk space. If your cache is too small, it will still download and load into memory.

    Code (CSharp):
    1. /// <summary>
    2. /// Only checks against cache space and is not the same as disk space - see https://docs.unity3d.com/2021.2/Documentation/ScriptReference/Cache-spaceFree.html
    3. /// </summary>
    4. /// <param name="key">Addressables key to get download size for</param>
    5. /// <returns>true if there is valid space</returns>
    6. bool PredetermineCanDownload(string key)
    7. {
    8.     var op = Addressables.GetDownloadSizeAsync(key);
    9.     op.WaitForCompletion();
    10.     long downloadSize = op.Result;
    11.     Addressables.Release(op);
    12.     // could use OS methods to get the disk space here if needed
    13.     if (Caching.currentCacheForWriting.spaceFree < downloadSize)
    14.         return false;
    15.     return true;
    16. }
    On a load failure, you can get a download error using:
    Code (CSharp):
    1. string GetDownloadError(AsyncOperationHandle fromHandle)
    2. {
    3.     if (fromHandle.Status != AsyncOperationStatus.Failed)
    4.         return null;
    5.  
    6.     RemoteProviderException remoteException;
    7.     Exception e = fromHandle.OperationException;
    8.     while (e != null)
    9.     {
    10.         remoteException = e as RemoteProviderException;
    11.         if (remoteException != null)
    12.             return remoteException.WebRequestResult.Error;
    13.         e = e.InnerException;
    14.     }
    15.     return null;
    16. }
    If you get "Unable to write data" Then the download failed due to insufficient disk space.
    I would recommend implementing this check for download error where you can, and react to it. Any solution we could do either against the cache, or against disk space has cases where it could fail. Only on error will you be able to fully handle the situation.
     
    andymilsom, Feb 21, 2022
    #11
    Lapsapnow likes this.

  12. better_walk_away

    better_walk_away

    Joined:
    Jul 12, 2016
    Posts:
    291
    Has this issue ever been resolved in the latest version of Addressables? Do we still need to check the available space beforehand by ourselves to handle this?
     
    better_walk_away, Jul 15, 2022
    #12

  13. andymilsom

    andymilsom

    Unity Technologies

    Joined:
    Mar 2, 2016
    Posts:
    294
    Yes, you still have to do this manually.

    After looking into it there has not been any handling of disk space added in Addressables. There is not a very good way to accurately check the disk space for the downloads that we can do at this time. For example.
    The cache size does not represent the correct disk space available.
    If we say download 5 bundles at the same time. We know the download size so could compare that to what is available. But we have no way to know the actual disk space used after recompressing to the cache. We could do it based on the uncompressed size, but again if 4 of those says it's ok, then the 5th says out of disk space, this is actually flawed as the recompression could mean there will be actually enough disk space.
    There is no way to know if other aspects outside of the running app like a browser downloading are also writing to the disk, so can heavily mess with the expected data.

    For now we think the best approach is to try and download it normally, and react to any error you get (this would be done anyway for the inaccuracy of any other method). Checking for available disk space would not be an inexpensive operation, and would have to be done regularly (or every download).
     
    andymilsom, Jul 18, 2022
    #13

  14. rdxtalha103

    rdxtalha103

    Joined:
    Aug 30, 2022
    Posts:
    15
    @Peter77
    @anyone who got the solution for above the problem?
    And if I click on the "disable content catalog" checkbox in Addressable Setting, and then after download the assets in the storage, and next time during initialisation of Addressable, is addressable will hit the server load path link for something or not?
     
    rdxtalha103, Jan 3, 2023
    #14

  15. NaijaNY82

    NaijaNY82

    Joined:
    Mar 7, 2017
    Posts:
    19
    Any update, has this been resolved?

    If the workaround is till to do this manually should I just account for enough storage space for the uncompressed size of the addressable?
     
    NaijaNY82, Aug 13, 2023
    #15