Search Unity

[RELEASED] [Free] Better Streaming Assets

Discussion in 'Assets and Asset Store' started by gwiazdorrr, Dec 12, 2017.

  1. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    Hi guys!

    TL;DR: Read Streaming Assets 10 times faster on Android for free, out on Asset Store (https://www.assetstore.unity3d.com/en/#!/content/103788). Works on other platforms, too.



    Some time ago I came up with a way to read Streaming Assets directly (i.e. without WWW) on Android. APK/OBB are basically ZIP archives and since Streaming Assets, while being archived, remain uncompressed, they can be read - you just need an offset from the beginning of APK/OBB and size.

    The difference is huge. Direct read on Sony D5503 is ~10 times faster than WWW, regardless of file size. Also, because WWW.bytes property effectively clones internal data (leading to memory spikes in case of large files), direct read is infinitely more memory efficient ;) It can be also used in any thread, works with streams etc.

    Finally, to make things nice and pretty I created cross-platform API, that internally falls back to System.IO.File etc. for other platforms. Examples below.

    The plug-in is out on Asset Store free and waiting for feedback!







     
    DhiaSendi and Gekigengar like this.
  2. caletbak2

    caletbak2

    Joined:
    Oct 15, 2014
    Posts:
    13
    Hi,

    It seems a very useful plugin.

    When the file is stored in the OBB this is something transparent for your plugin? So a file stored in StreamingAssets can be easily found when building a full APK without OBB, and can be found when building with OBB by using the same file path?

    Cheers,
    Calet
     
  3. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    Yes. If you look at the source, it uses Application.dataPath to get the path to APK. According to Unity documentation:

    Streaming Assets get put in OBB, so it should be fine. Just make sure OBB exists before you call BetterStreamingAssets.Initialize. If it doesn't, Application.dataPath will point to APK and no Streaming Assets will be discovered.
     
  4. capu087

    capu087

    Joined:
    Jan 17, 2016
    Posts:
    6
    does it work on web gl ?
     
  5. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    No. AFAIK WebGL doesn't allow synchronous file access.
     
  6. Blaze5565

    Blaze5565

    Joined:
    Sep 11, 2018
    Posts:
    11
    Hi,

    I'm having some trouble with your asset? I've got some files in my StreamingAssets folder:
    BetterStreamingAssets1.png
    And I'm using the BetterStreamingAssets.GetFiles() method to try to get the names of these 3 files (all of which are .json files):
    BetterStreamingAssets2.png
    But the end result only seems to have 2 files: BetterStreamingAssets3.png
    I tried this earlier as well, with just 2 files, and only one of them worked (the one with the Japanese name, specifically).
    Could you explain what might be going on or give me a hand?

    Thanks,
    Cory
     
  7. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    Are you by any chance using Android with App Bundle build?

    If so, see this issue: https://github.com/gwiazdorrr/BetterStreamingAssets/issues/10. Basically sometimes some Streaming Assets get compressed - I couldn't find a pattern and am waiting for Unity to fix it.

    Also, what Unity version are you on? Are you able to reproduce it reliably or does it change between builds?

    EDIT: here's a thread about what I'm talking about: https://forum.unity.com/threads/ass...-when-building-the-android-app-bundle.711752/
    A Unity bug report: https://issuetracker.unity3d.com/is...more-time-when-the-project-is-built-as-an-aab
    An interesting take: https://forum.unity.com/threads/str...app-bundle-google-play-option-is-used.739967/

    So, since you already have the files in the root of StreamingAssets, try to change their name to just lowercase and see if it works.
     
    Last edited: Oct 8, 2019
  8. Blaze5565

    Blaze5565

    Joined:
    Sep 11, 2018
    Posts:
    11
    I was using app bundles, yes, and I changed the filenames to lowercase, and now they're showing up! I'm using Unity 2019.2.8f1, and the missing files change only when I change what files are in StreamingAssets (as in even just adding new files without touching the old ones). Maybe helpful (or at least informative), the icon in Unity also changed when I removed the uppercase:
    Annotation 2019-10-10 032019.png
    Although, all four worked correctly this time despite the icon being different in one from the other now-working ones. Originally, the only one that worked was the last one, which has always had this icon (and always worked) (oh, and, I had deleted it when I originally posted the question to see if the icon meant anything). Anyway, yes, everything seems to be working now, so thank you!
     
  9. Jovaan

    Jovaan

    Joined:
    Aug 1, 2014
    Posts:
    8
    Because of our app size (>250MB with videos), we need to still use APK + OBB to launch in Play Store. I am facing a problem now with only builds loaded from Play store, apk + obb. Trying to use BetterStreamingAssets.GetFiles() results in IndexOutOfRangeException:
    upload_2019-12-1_23-18-51.png

    All the files are in root of StreamingAssets, non-capital letters. They are video files with names formatted like "xxx-yy.mp4".

    Works with full APK build, not APK + OBB, no other changes.
     
  10. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    Interesting.
    1) Do you call
    Initialize
    after OBB is mounted? This is needed if say you download OBB from Google Play yourself.
    2) I'm going to need the arguments for your
    GetFiles()
    call and list of files in StreamingAssets to reproduce it. You can PM me.
     
  11. Jovaan

    Jovaan

    Joined:
    Aug 1, 2014
    Posts:
    8
    1) I call Initialize in Awake of the first (and only) scene the app has. The error manifests in first and all later starts of the app, so I am assuming the OBB must be loaded. AFAIK there is no direct way to check in Unity if OBB is actually loaded. I might be wrong.
    2) Arguments are simple: BetterStreamingAssets.GetFiles(Path.DirectorySeparatorChar); so I am reading the names of all files in StreamingAssets root folder. All files are non-capital letters .mp4 files.

    I have tried including Write permissions to External also, but no change in this behaviour. Exactly same crash.
     
  12. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    Ok I finally found the time to fix this. Basically in OBB directories end up as zip entries, in APK - they do not. The fix is on github (https://github.com/gwiazdorrr/BetterStreamingAssets), it will take a while for it to make it into Asset Store.

    EDIT: The fix is already in the Asset Store.
     
    Last edited: Dec 10, 2019
    DhiaSendi likes this.
  13. thefallengamesstudio

    thefallengamesstudio

    Joined:
    Mar 7, 2016
    Posts:
    418
    Thanks man! This is exactly what I was looking for! What a relief!

    I'm wondering, could you make it so that we can list the directories in a directory, not just files?

    -- Lucian
     
  14. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    Simply put I've never needed that :) It's not totally trivial to do it the right and robust way, but if you can live something quick and dirty, this should work:

    Code (CSharp):
    1. var directories = BetterStreamingAssets.GetFiles(path, "*", SearchOption.AllDirectories)
    2.     .Select(x => Path.GetDirectoryName(x))
    3.     .Distinct()
    4.     .ToList();
    Note that this is not going to work with empty directories.
     
    thefallengamesstudio likes this.
  15. NamekGames

    NamekGames

    Joined:
    May 13, 2014
    Posts:
    10
    Hi, why didn't you do the serialization, and the Save part also? Does it still not work on webGL?
     
  16. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    Well, for instance on Android StreamingAssets are read only (embedded in APK). Also AFAIK altering Streaming Assets on platforms that let you (PC?) is a bad practise.

    As for webGL, there's no way to access a file synchronously there, everything needs to be a web request.
     
  17. thefallengamesstudio

    thefallengamesstudio

    Joined:
    Mar 7, 2016
    Posts:
    418
    Didn't think of that. It's hacky, yes. But maybe it'll be my last resort at some point (I figured another way ATM for my particular case).
    Thanks. And maybe in the future, if possible, you'll find a simpler way and post it here.
     
  18. musicdeveloper

    musicdeveloper

    Joined:
    Oct 16, 2019
    Posts:
    22
    Thank you for this asset!

    I'm wondering if there's any way to include multiple extensions in the search pattern? For instance, when reading through a folder of audio assets, I want to include "*.wav" "*.ogg" "*.mp3"

    (I tried "*.wav|*.mp3", which was one suggestion from the internet and that didn't work...)

    Thank you
     
  19. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    It is possible, but that'd mean compatibility with
    System.IO.Directory.GetFiles
    is broken - something I want to avoid, as the plug-in is meant to be an almost drop-in replacement.
     
  20. nyonge

    nyonge

    Joined:
    Jul 11, 2013
    Posts:
    29
    Looking forward to using this asset :) A couple questions. The website seems to be down, I don't know if docs were there or not but the Github docs are pretty limited.

    Can you explain the difference between Initialize and InitializeWithExternalDirectories? I plan on reading files of a bunch of different types (meshes and textures, even unity stuff like materials and physicmaterials) from StreamingAssets and want to make sure I'm using the right initialization.

    The Asset Store page says it's cross-platform, can you clarify which platforms? Clearly Android, likely PC, I suspect others...?

    Also, the "message me" link on the Asset Store page is broken too, it leads to a 404 instead of opening up a conversation with you.

    Thank you!
     
  21. gwiazdorrr

    gwiazdorrr

    Joined:
    Sep 29, 2014
    Posts:
    61
    Thanks for pointing the stuff out, I'll take car of it.

    The docs are not out there. I kept the API clean and as close to System.IO as possible, so you should feel home.

    When it comes to
    InitializeWithExternalDirectories
    (and
    InitializeWithExternalApk
    for that matter) - it is for testing purposes only (notice it is in
    #if UNITY_EDITOR
    block). Just call
    Initialize
    and you'll be fine :)

    Now when I say it is cross platform I mean it works on Android and any platform supporting access to the filesystem with System.IO classes (among them PC & iOS, but not WebGL). Consoles are a mixed bag here, but I don't think I can elaborate here and certainly don't have resources to support the problematic ones.

    Hope you'll enjoy it!
     
    nyonge likes this.
unityunity