Search Unity

Assetbundle caching and load from file

Discussion in 'Asset Bundles' started by Roryyyyyyyyyy, Sep 14, 2018.

  1. Roryyyyyyyyyy

    Roryyyyyyyyyy

    Joined:
    Jun 8, 2015
    Posts:
    21
    Hi all,
    I am trying to work out a system for downloading when the game is loaded up. I am currently using UnityWebRequest.GetAssetBundle (I am using 2017.4) to cache the assetbundle in the system. I have an asset manager that tracks everything loaded, but I'm wondering which way would be better to handle the bundles. Am I right in thinking I cannot use AssetBundle.LoadFromFile if I have downloaded it from the cache?

    Our current process is that the app gets the bundle crc from the server which replaces the bundle in the cache if so with the UnityWebRequest.GetAssetBundle. It doesn't store the bundle anywhere, it just caches it.

    How is the best way to get the bundles after this? E.g. I want to load a building model, I will want to load the building bundle from the cache, get the asset, store it in the manager then unload the bundle. Would it be best just to use the web request each time?

    Would anyone recommend me doing it a different way or is this a reasonable approach?
     
  2. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,251
    One I thing I suggest you to do is to figure out what you need exactly and see if you aren't reinventing the wheel.
    UnityWebRequest.GetAssetBundle() (the overloads with CRC/version arguments) is loading AssetBundle from cache, if given AssetBundle with given crc/version/hash is in the cache, otherwise it downloads and caches it for future use. As such you just do UnityWebRequest.GetAssetBundle() and then take the bundle via DownloadHandlerAssetBundle.GetContent(), this way you don't need to bother, whether tahe bundle was downloaded or loaded from the cache.

    If you are looking into using AssetBundle.LoadFromFile, the you shouldn't use UnityWebRequest.GetAssetBundle, you need to assign DownloadHandlerFile instead to bypass AssetBundle system at download time. But by doing so you also lose the builtin caching etc. and reinvent them yourself. So it sounds you should first make sure the builtin caching/loading is not enough for you.
     
  3. Roryyyyyyyyyy

    Roryyyyyyyyyy

    Joined:
    Jun 8, 2015
    Posts:
    21
    Ah okay, so the only real difference would be I would need to create a system for caching it manually? I guess the UWR method would be much better for me then. In terms or memory usage is there much in it between UWR and LoadFromFile?
     
  4. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,251
    UWR caching stores AssetBundle on disk decompressed, so the load will be faster. In terms of memory usage there should be no significant difference between the two, only that in case of LoadFromFile you'd have to download and store the bundle first and ensure efficient memory usage during download (such as DownloadHandlerFile).
     
  5. Roryyyyyyyyyy

    Roryyyyyyyyyy

    Joined:
    Jun 8, 2015
    Posts:
    21
    That makes sense, thanks for the information! I already have UWR quite heavily implemented so it seems it makes sense to just stick with that. Cheers!
     
  6. SketchWork

    SketchWork

    Joined:
    Jul 6, 2012
    Posts:
    208
    @Aurimas-Cernius What about if the app is offline with no internet - UnityWebRequest would fail because there is no internet, but will it still download from the local cache if it exists? having a problem with it when offline on Android.
     
  7. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,251
    If AssetBundle is cached, UnityWebRequest should loading without even attempting to download anything. It should only go to internet if the bundle is not cached or if cached bundle is of different version/hash/crc than requested.
     
  8. stgs73

    stgs73

    Joined:
    Mar 23, 2015
    Posts:
    44
    Hi Aurimas,
    We implemented the assetbundle download via a custom DownloadHandlerScript - this works fine aside from being a bit slow.

    All our assetbundles are LZ4 which is fine once on disk but we'd like to take advantage of the extra compression for download and decompress as it is downloading rather than at the end.

    The options we see are:
    1) Gzip the LZ4 and decompress ourselves - works but is clunky, bit slow compression still not as good as LZMA + compression occurs after entire download.
    2) Build assetbundles as LZMA and use the GetAssetBundle to decompress straight to disk

    2) is a problem as there is no way to pipe the result to a particular file/folder

    Is there anyway to get the benefits of Unity AssetBundle decompression and output the file somewhere?

    If not, is 1) our only solution?

    thanks
     
  9. unity_0fmczLs2vNuMTQ

    unity_0fmczLs2vNuMTQ

    Joined:
    Jun 6, 2018
    Posts:
    8
    Hello, I don't understant one thing. How UnityWebRequest understands if bundle is cached or not? If I use version parameter, does it just use Caching.currentCacheForWriting's version and compares it to the one I specified in UnityWebRequestAssetBundle.GetAssetBundle() or what? For example if I have two bundles - bundle1 and bundle2. After first laucng they were downloaded and cached in AssetBundlesCache/bundle1, AssetBundlesCache/bundle2 acordingly. Now I restart my game with no internet connection. Usually I try to load bundle on server response. It sends me uri and version. I don't have connection, so can I just pass empty uri and version of the current cache to UnityWebRequestAssetBundle.GetAssetBundle() and specify currentCacheForWriting accordingly for each bundle? Also I don't know how to get version from the cache. Caching.GetVersionFromCache() is obsolete.
     
    Last edited: Nov 1, 2018
  10. unity_0fmczLs2vNuMTQ

    unity_0fmczLs2vNuMTQ

    Joined:
    Jun 6, 2018
    Posts:
    8
    Also I checked if UnityWebRequestAssetBundle.GetAssetBundle() throws an exception with no Internet. It doesn't. But the problem is DownloadHandlerAssetBundle.GetContent() throws InvalidOperationException: Cannot resolve destination host. So I can't load bundle from cache.
     
  11. unity_0fmczLs2vNuMTQ

    unity_0fmczLs2vNuMTQ

    Joined:
    Jun 6, 2018
    Posts:
    8
    Another thing I tried was to directly assign downloadHandler property of my UWR with DownloadHandlerAssetBundle object. This time there was no error but assetBundle itself was null. Why is it so hard to just load bundle from cache?! Looks like I'd have to use AssetBundle.LoadFromFileAsync() for this.
     
  12. nsmith1024

    nsmith1024

    Joined:
    Mar 18, 2014
    Posts:
    645

    When I load asset bundle with a terrain, the terrain doesnt show up, player falls thru the floor ending the game immediately.

    Please see this post,

    https://forum.unity.com/threads/terrain-doesnt-appear-when-loading-from-asset-bundle.578254/

    Others having same problem

    https://forum.unity.com/threads/bitfieldinsert-and-shader-error.548491/

    https://answers.unity.com/questions...n-fou.html?childToView=1568496#answer-1568496
     
    Last edited: Nov 3, 2018
  13. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,251
    But why do you want to save bundles on disk? When you use GetAssetBundle() with additional parameters, it will use caching system and will avoid downloading an already downloaded bundle.
    As for additional compression, first of all make sure it does reduce the size significantly. If using GZip does benefit you, first thing to do is to check the request headers and configure server to use gzip if cleant accepts it. The gzip compression will work out of the box if device has builtin support for it.
     
  14. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,251
    Cached asset bundles are identified by name, version, crc and hash. For UnityWebRequest name is taken from url and you have to pass at least one of other three for the cache to be used. If you pass only url, it will always download bundle from the internet. Passing the same url and the same crc/hash/version like previously should load asset bundle from cache (if it is cached and you haven't cleared it) and this should work without internet connection.
     
  15. unity_0fmczLs2vNuMTQ

    unity_0fmczLs2vNuMTQ

    Joined:
    Jun 6, 2018
    Posts:
    8
    Oh, OK. So in case I get URL from the server I have to save it somewhere to be able to pass it even without connection. I decided to load it frrom cache folder with the use of LoadFromFile functionality.
     
  16. stgs73

    stgs73

    Joined:
    Mar 23, 2015
    Posts:
    44
    Hi Aurimas, we wrote our own caching system based on prior experiences with earlier versions of Unity caching - our understanding was that there is no guarantee the cache is available or enough space is available to cache, the side result is it loads into memory instead or ejects out other assetbundles in the cache? Is that still the case? We do all our checks up front and ensure we have space and decompress into our persistentDataFolder so we put up disk space requirements etc etc - it all works fine.
    Unless things have changed then we could switch if you feel there are no issues now.

    As for gzip and header re: yes we already did this, we were just wondering if it was possible to just utilize the decompression from LZMA -> LZ4 and stream that into our location that was all.

    best
     
  17. TomNCatz

    TomNCatz

    Joined:
    Jan 6, 2015
    Posts:
    12
    Ok, so that all sounds good and well, but it does not actually play out that way.

    In my case I want the bundles to update when they reach the server if the server file has changed, good so far.
    Then if it doesn't reach the server I want them to use the locally cached files, so I need to use the matching hash from the last downloaded version. Accept the manifest files with those hashes come from the server and don't make it into cache, and the AssetBundleManifest file does not even seem to get a hash.

    I tried taking a step back and tracking an internal version number that I update whenever I get a positive read from the server, but the version number based UnityWebRequest.GetAssetBundle does not seem to pull correctly from the cache. Even checking the cache manually for version numbers seems to have been marked obsolete. There also does not seem to be any methodology for looking up the bundles in cache manually and pulling them out directly.

    I have found a somewhat Frankenstein solution, but while it works, it has a lot of potential for failure cases. Until there is a built in Unity solution, here you go.

    AssetBundle.manifest doesn't contain a hash, but it does contain a CRC value, so if you download a manifest file that has a CRC, but no hash, you know you have a good line on a fresh version of the AssetBundle file. Clear all cached versions of the assetBundle file and then download a new one from the same source.
    Code (CSharp):
    1. string[] lines = manifest.text.Split('\n');
    2.  
    3. if( lines[1].Contains( "CRC:" ) )
    4. {
    5.     Log.LogL( "Bundle {1} CRC: {0}",lines[1].Substring( 5, lines[1].Length - 5 ), config.name );
    6.     uint.TryParse( lines[1].Substring( 5, lines[1].Length - 5 ), out config.crc );
    7. }
    8.  
    9. config.hash = default(Hash128);
    10. if( lines[5].Contains( "Hash:" ) )
    11. {
    12.     Log.LogL( "Bundle {1} Hash: {0}",lines[5].Split (':') [1].Trim (), config.name );
    13.     config.hash = Hash128.Parse(lines[5].Split (':') [1].Trim ());
    14. }
    15. else if(config.crc != 0)
    16. {
    17.     // new manifest bundle found
    18.     Caching.ClearAllCachedVersions( config.name );
    19. }
    Unity won't cache the bundle unless you give it a version number or a version hash, so you have to make one up when asking for it. At the same time, you need to know what version it was next time in case you can't hit the server. This is why I clear the cache when I am about to get a fresh version and use 0 as the version number. This means that the assetbundle file always has to be downloaded, even if it is current, but as far as I can tell there is not a way to find out what the hash should be for it. Oh, did I mention that version numbers do not update in the manifest files either?

    Anyway, after I've gotten a CRC and a hash, just a CRC, or just nothing at all, I now can try to download the actual bundle file. This logic could probably be simplified if I handled the main assetbundle file through a different flow, but they are largely the same logic outside of this point.

    Code (CSharp):
    1. UnityWebRequest download;
    2. if( config.hash == default(Hash128) )
    3. {
    4.     if( config.crc == 0 )
    5.     {
    6.         // file not found
    7.         List<Hash128> listing = new List<Hash128>();
    8.         Caching.GetCachedVersions( config.name,listing );
    9.         if( listing.Count > 0 )
    10.         {
    11.             // No manifest, but previous versions were cached
    12.             download = UnityWebRequest.GetAssetBundle( Path.Combine( config.path, config.name ), listing[listing.Count-1], config.crc );
    13.             Log.LogL( "{0} did not have a hash key, attempting to use most recent version {1}",
    14.                 config.name,
    15.                 listing[listing.Count - 1] );
    16.         }
    17.         else
    18.         {
    19.             // No manifest, no cache, trying to get the bundle anyway just in case
    20.             download = UnityWebRequest.GetAssetBundle( Path.Combine( config.path, config.name ), config.crc );
    21.             Log.LogL( "{0} did not have a hash key and no previous versions found", config.name );
    22.         }
    23.     }
    24.     else
    25.     {
    26.         // was manifest bundle
    27.         download = UnityWebRequest.GetAssetBundle( Path.Combine( config.path, config.name ),
    28.             0,
    29.             config.crc );
    30.     }
    31. }
    32. else
    33. {
    34.     // have everything we need, just grab the file
    35.     download = UnityWebRequest.GetAssetBundle( Path.Combine( config.path, config.name ), config.hash, config.crc );
    36.     Log.LogL( "{0} is cached : {1}", config.name, Caching.IsVersionCached( download.url, config.hash ) );
    37. }
    38.            
    39. download.SendWebRequest();
    Sorry that this bit is such a mess, I just got it working really. I'm going to walk through it backwards because that is actually more the order of case regularity.
    The last else is processing a case where you have a good hash. This means it is a normal asset bundle and we got the manifest file alright. Basically, this is the call just working as intended.
    Next to last case we have no hash, but we do have a CRC. This is a properly loaded manifest file for a AssetBundleManifest bundle. Here we set the version number to zero and check the CRC. Still mostly normal.
    Above that we have no CRC or hash, plus we didn't have a version in cache. This is the worst case scenario and there isn't really anything Unity can do to help you here. I try downloading the file anyway, just for giggles. In truth, I have logic elsewhere that if the download fails tries to pull it from streaming assets instead, but that's not really relevant here.
    Finally we start getting to the workaround, sorry about that. I pull down a list of all cached version hashes for this bundle. We can't get a list of the actual bundles, the version numbers, or the bundles themselves, but we can get this list. If there are items in the list, we have cached versions to work with. This list is in order from oldest downloaded to newest, so I grab the last hash in the list and use it as the download hash, effectively loading that version from cache. As a note, it looks like I pass it a CRC, but it's really zero so I'm basically just ignoring it.

    This all works, but it still has a gaping hole. Lets say you have character sprites in an asset bundle, and you want to have them wear Santa hats for a Christmas event. You upload a new asset bundle for Christmas, and then afterwords you switch back to the old bundle. Here in we have a problem. the list is in order of oldest to newest DOWNLOADED, not used. This means that the Christmas event bundle is the last in the list. So any user who played before and through Christmas will have the Christmas event bundle when they are offline even after the event has ended and they've received a more recent bundle.

    You can work around this by manually clearing items out of the list when you get new ones, forcing the order of cached items, or a few other methods, but honestly? It's a mess.

    The point is, this feature doesn't quite let you do what it says on the box, and the unity API documentation on it is both out of date AND marks it as obsolete. Hopefully this helps someone else to not spend their whole weekend combing through documentation and forums like I did.
     
    FoxsterDev likes this.
  18. TomNCatz

    TomNCatz

    Joined:
    Jan 6, 2015
    Posts:
    12
    Oh, uh...you can replace any of the "Log.LogL" bits with "Debug.LogFormat" if you want to use this code.
     
  19. AustynPember

    AustynPember

    Joined:
    Mar 14, 2018
    Posts:
    17
    Thanks for showing all that you did. This is what I am having to use because Addressable Assets are not mature enough to do this type of thing as I've found and been told.
    Could you help me understand a bit about how you are implementing this? What the code outside of this snippet looks like?
    For example - what the config looks like, and the manifest - and how these things are declared and called?

    It seems like this is set up to just download one asset bundle, but do you have this in a loop to grab many which are declared in the AssetBundle.manifest?