[RELEASED] [Free] Better Streaming Assets

Hi guys!

TL;DR: Read Streaming Assets 10 times faster on Android for free, out on Asset Store (https://www.assetstore.unity3d.com/en/#!/content/103788). Works on other platforms, too.



Some time ago I came up with a way to read Streaming Assets directly (i.e. without WWW) on Android. APK/OBB are basically ZIP archives and since Streaming Assets, while being archived, remain uncompressed, they can be read - you just need an offset from the beginning of APK/OBB and size.

The difference is huge. Direct read on Sony D5503 is ~10 times faster than WWW, regardless of file size. Also, because WWW.bytes property effectively clones internal data (leading to memory spikes in case of large files), direct read is infinitely more memory efficient It can be also used in any thread, works with streams etc.

Finally, to make things nice and pretty I created cross-platform API, that internally falls back to System.IO.File etc. for other platforms. Examples below.

The plug-in is out on Asset Store free and waiting for feedback!

 

Hi,

It seems a very useful plugin.

When the file is stored in the OBB this is something transparent for your plugin? So a file stored in StreamingAssets can be easily found when building a full APK without OBB, and can be found when building with OBB by using the same file path?

Cheers,
Calet

 

Yes. If you look at the source, it uses Application.dataPath to get the path to APK. According to Unity documentation:

Streaming Assets get put in OBB, so it should be fine. Just make sure OBB exists before you call BetterStreamingAssets.Initialize. If it doesn't, Application.dataPath will point to APK and no Streaming Assets will be discovered.

 

does it work on web gl ?

 

No. AFAIK WebGL doesn't allow synchronous file access.

 

Hi,

I'm having some trouble with your asset? I've got some files in my StreamingAssets folder:

And I'm using the BetterStreamingAssets.GetFiles() method to try to get the names of these 3 files (all of which are .json files):

But the end result only seems to have 2 files:
I tried this earlier as well, with just 2 files, and only one of them worked (the one with the Japanese name, specifically).
Could you explain what might be going on or give me a hand?

Thanks,
Cory

 

Are you by any chance using Android with App Bundle build?

If so, see this issue: https://github.com/gwiazdorrr/BetterStreamingAssets/issues/10. Basically sometimes some Streaming Assets get compressed - I couldn't find a pattern and am waiting for Unity to fix it.

Also, what Unity version are you on? Are you able to reproduce it reliably or does it change between builds?

EDIT: here's a thread about what I'm talking about: https://forum.unity.com/threads/ass...-when-building-the-android-app-bundle.711752/
A Unity bug report: https://issuetracker.unity3d.com/is...more-time-when-the-project-is-built-as-an-aab
An interesting take: https://forum.unity.com/threads/str...app-bundle-google-play-option-is-used.739967/

So, since you already have the files in the root of StreamingAssets, try to change their name to just lowercase and see if it works.

 

I was using app bundles, yes, and I changed the filenames to lowercase, and now they're showing up! I'm using Unity 2019.2.8f1, and the missing files change only when I change what files are in StreamingAssets (as in even just adding new files without touching the old ones). Maybe helpful (or at least informative), the icon in Unity also changed when I removed the uppercase:

Although, all four worked correctly this time despite the icon being different in one from the other now-working ones. Originally, the only one that worked was the last one, which has always had this icon (and always worked) (oh, and, I had deleted it when I originally posted the question to see if the icon meant anything). Anyway, yes, everything seems to be working now, so thank you!

 

Because of our app size (>250MB with videos), we need to still use APK + OBB to launch in Play Store. I am facing a problem now with only builds loaded from Play store, apk + obb. Trying to use BetterStreamingAssets.GetFiles() results in IndexOutOfRangeException:


All the files are in root of StreamingAssets, non-capital letters. They are video files with names formatted like "xxx-yy.mp4".

Works with full APK build, not APK + OBB, no other changes.

 

Interesting.
1) Do you call

Initialize

after OBB is mounted? This is needed if say you download OBB from Google Play yourself.
2) I'm going to need the arguments for your

GetFiles()

call and list of files in StreamingAssets to reproduce it. You can PM me.

 

1) I call Initialize in Awake of the first (and only) scene the app has. The error manifests in first and all later starts of the app, so I am assuming the OBB must be loaded. AFAIK there is no direct way to check in Unity if OBB is actually loaded. I might be wrong.
2) Arguments are simple: BetterStreamingAssets.GetFiles(Path.DirectorySeparatorChar); so I am reading the names of all files in StreamingAssets root folder. All files are non-capital letters .mp4 files.

I have tried including Write permissions to External also, but no change in this behaviour. Exactly same crash.

 

Ok I finally found the time to fix this. Basically in OBB directories end up as zip entries, in APK - they do not. The fix is on github (https://github.com/gwiazdorrr/BetterStreamingAssets), it will take a while for it to make it into Asset Store.

EDIT: The fix is already in the Asset Store.

 

Thanks man! This is exactly what I was looking for! What a relief!

I'm wondering, could you make it so that we can list the directories in a directory, not just files?

-- Lucian

 

Simply put I've never needed that It's not totally trivial to do it the right and robust way, but if you can live something quick and dirty, this should work:

Code (CSharp):

1. var directories = BetterStreamingAssets.GetFiles(path, "*", SearchOption.AllDirectories)
2.     .Select(x => Path.GetDirectoryName(x))
3.     .Distinct()
4.     .ToList();

Note that this is not going to work with empty directories.

 

Hi, why didn't you do the serialization, and the Save part also? Does it still not work on webGL?

 

Well, for instance on Android StreamingAssets are read only (embedded in APK). Also AFAIK altering Streaming Assets on platforms that let you (PC?) is a bad practise.

As for webGL, there's no way to access a file synchronously there, everything needs to be a web request.

 

Didn't think of that. It's hacky, yes. But maybe it'll be my last resort at some point (I figured another way ATM for my particular case).
Thanks. And maybe in the future, if possible, you'll find a simpler way and post it here.

 

Thank you for this asset!

I'm wondering if there's any way to include multiple extensions in the search pattern? For instance, when reading through a folder of audio assets, I want to include "*.wav" "*.ogg" "*.mp3"

(I tried "*.wav|*.mp3", which was one suggestion from the internet and that didn't work...)

Thank you

 

It is possible, but that'd mean compatibility with

System.IO.Directory.GetFiles

is broken - something I want to avoid, as the plug-in is meant to be an almost drop-in replacement.

 

Looking forward to using this asset A couple questions. The website seems to be down, I don't know if docs were there or not but the Github docs are pretty limited.

Can you explain the difference between Initialize and InitializeWithExternalDirectories? I plan on reading files of a bunch of different types (meshes and textures, even unity stuff like materials and physicmaterials) from StreamingAssets and want to make sure I'm using the right initialization.

The Asset Store page says it's cross-platform, can you clarify which platforms? Clearly Android, likely PC, I suspect others...?

Also, the "message me" link on the Asset Store page is broken too, it leads to a 404 instead of opening up a conversation with you.

Thank you!

 

Thanks for pointing the stuff out, I'll take car of it.

The docs are not out there. I kept the API clean and as close to System.IO as possible, so you should feel home.

When it comes to

InitializeWithExternalDirectories

(and

InitializeWithExternalApk

for that matter) - it is for testing purposes only (notice it is in

#if UNITY_EDITOR

block). Just call

Initialize

and you'll be fine

Now when I say it is cross platform I mean it works on Android and any platform supporting access to the filesystem with System.IO classes (among them PC & iOS, but not WebGL). Consoles are a mixed bag here, but I don't think I can elaborate here and certainly don't have resources to support the problematic ones.

Hope you'll enjoy it!

 

Hi,

this asset works very well on Android, iOS, Oculus Rift... but we've been struggling to make it work on the Quest.

I get a "DirectoryNotFoundException" exception in ApkImpl.GetFiles()

Code (CSharp):

1.             if ( index == s_paths.Length )
2.                 throw new DirectoryNotFoundException();

I've debugged the "PathUtil.NormalizeRelativePath("NameOftheFolder", forceTrailingSlash : true)" return value and it's the same as on Android (which works), "/NameOftheFolder/" which is under the StreamingAssets folder in Unity. That's correct. But still, it fails on the Oculus Quest.

The Unity version is 2018.4.17 and I'm running the latest version of the asset from the store.

Is this asset compatible with the Quest? Would you mind looking into this?

Thanks.

 

Unfortunately I don't own a Oculus Quest device.

Since paths are the same then it is likely some (all?) streaming assets failed to be detected. You can verify this by looking at ApkImpl.s_paths. Make sure you are running development build to have assertions enabled.

 

I checked ApkImpl.s_paths and its length is 6 (it should be 28), which is the same as the index value. Alas, it returns an exception.

Those 6 items are 5 files I don't need under StreamingAssets/ and "PUT_OSIG_HERE" from "Plugins/Android/Assets".

The files I need are under "StreamingAssets/NameOfTheFolder"

Something I've noticed is that the log goes through all the assets I need in ApkImpl.Initialize() -> GetStreamingAssetsInfoFromJar and complains that: "seems to be a Streaming Asset but it's compressed. If this is a App Bundle build, see README...". This log doesn't show up for the 6 files mentioned above.

The files I need are binary but they're not compressed.

Cheers.

 

Thanks for checking that.

StreamingAssets are not supposed to be compressed. If they are, they can't be freely acessed. How about try renaming all files & folders to lowercase? Maybe this is another instance of https://github.com/gwiazdorrr/BetterStreamingAssets/issues/10?

 

I really appreciate you took the time to help me out with this. I fixed it. It had nothing to do with your asset.

The problem was indeed that they get compressed at build time and the way I fixed it was by adding their extension file to the "noCompress" section of the Gradle file.

Thank you!

 

hey!

I get this error:
ArgumentNullException: Value cannot be null.
Parameter name: array

Using this command:

Code (CSharp):

1. bool pathExist = BetterStreamingAssets.FileExists("katalogs.pdf");

Every command i tried running resulted in error.

And this is how my assets folder looks like:


Am I missing something?

EDIT: Looks like I was missing the fact I need to init the plugin it self. Thanks for the plugin

 

Have you called

BetterStreamingAssets.Initialize()

?

 

Hi!
thanks for the plugin !

I chose your asset to manage files on my android project, but I face a problem. In fact I can read my files but when I want to modify them, it simply doesn't work on android, even if it was working on Unity Editor.
Have a look:

Code (CSharp):

1. public class Manager : MonoBehaviour {
2.     public GameObject text;
3.    
4.     void Start () {
5.         BetterStreamingAssets.Initialize();
6.         text = GameObject.Find("Text");
7.         text.GetComponent<Text>().text = BetterStreamingAssets.ReadAllText("test.txt");
8.     }
9.    
10.     public void write()
11.     {
12.         //Debug.Log(BetterStreamingAssets.Root);
13.         File.WriteAllText(BetterStreamingAssets.Root + "/test.txt", "OMG it works");
14.         text.GetComponent<Text>().text = BetterStreamingAssets.ReadAllText("test.txt");
15.     }
16. }

I think I have a problem linked to BetterStreamingAssets.Root.
What do you think ?

 

StreamingAssets should be always treated read-only. In practice, in the editor they are not, on Window they may be, depending where you install your game, on Android (and most other platforms AFAIK) - they always are.

 

Thanks, I finally used PlayerPrefs with json to solve my problem and it works pretty well.

 

I've found the weirdest issue...

This all works when you make a development build, but when that is unchecked in the build settings window it breaks.

The cause of the issue is the weirdest thing, when you are NOT making a development build, Unity is not copying the streaming assets into the APK file, the folder /assets found there has nothing but /bin

Does anyone know what's going on?, this is on Oculus Quest.

 

Hi! I've read that it doesn't work with WebGL...
I need to build an app that display a 360 4K video on web. I used AVProVideo but it lags with 4K videos. Do you know a way to display a 360 4K video with webGL (or with Unity but for a browser)?
Thank you for your time!

 

So far this is an great asset. I have been have been able to deserialize data from XML files, and I was able to display images from StreamingAssets. Can you explain or provide a simple example of how to use video and audio clips from streaming assets. Here is the code for how I used for images. I expect that code for audio and video will be similar:

Code (CSharp):

1. byte[] data = BetterStreamingAssets.ReadAllBytes (imageFilename);
2.  
3.             Texture2D tex = new Texture2D (2, 2);
4.             tex.LoadImage (data);
5.             GetComponent<RawImage> ().texture = tex;

All the best,
Thanks

 

Generally speaking you're better off using Asset Bundles for textures - you get to use compression formats, there's no decompression needed etc. For video and audio clips, if you intend to use them like you'd use regular assets - use Asset Bundles. I don't know if there's a way to create them from raw data.

Now if you are using an external library, that expects you to pass a file stream or byte array, that's where the plug-in can be useful.

 

I have the file text.txt in the StreamingAssets folder. Everything lowercase.

Now if I do

Code (CSharp):

1. BetterStreamingAssets.FileExists("text.txt")

it works in the editor, but it doesn't work on Android. I do use app bundles (and I have read about issue #10).

Unity 2020.2

Is it maybe because 2020 uses a different Android Gradle Plugin (3.6 istead of 3.4)?

 

First thing to do in such case is to peek into APK/OBB and see if the file is compressed or not. This can be verified with 7-Zip (columns Size and Packed Size will have same value) or any other packing tool.

Also, building in Development mode will enable assertions that might provide useful info on what's wrong.

 

As far as I can tell the files exist and are uncompressed in the AAB.

 

And what about testing in development mode?

 

GetFiles does not work correctly on iOS and macOS. Impossible to read all files in streamingassets root folder. "/" results in project root inside editor, "" results in error, and backslash results in error too, it just adds backslash to streamingassets folder. Is there a way to use GetFiles for Streamingassets root folder in iOS and macOS?

 

@Jovaan have you called

BetterStreamingAssets.Initialize

first?

 

Yes, of course. Please tell the exact syntax to read all filenames in streamingassets root folder.

 

BetterStreamingAssets.GetFiles("/", "*", SearchOption.TopDirectoryOnly)

should get you what you want.

EDIT: on iOS/MacOS the plugin does nothing fancy and calls

Directory.GetFiles

behind the scenes. Here's the exact line: https://github.com/gwiazdorrr/Bette...amingAssets/BSA_BetterStreamingAssets.cs#L253

 

@gwiazdorrr
Is it possible to load/use videos with this plugin ?
All examples I found are about text files

Thanks

 

If your Video library accepts a Stream or a file path, offset and size, then yes, absolutely.

 

Does it work with Sqlite?

 

I am not familiar with Sqlite API. If it accepts a stream or file path & offset pair, then yes - you can use this plugin. Note that on most platforms Streaming Assets are read only (Android included), so no updates to the db.

 

Is there any way to get a list of subdirectories in StreamingAssets on Android using this? Alternatively, is there another way to do so?

 

@hurleybird yes:

Code (CSharp):

1. var directories = BetterStreamingAssets.GetFiles(path, "*", SearchOption.AllDirectories)
2.     .Select(x => Path.GetDirectoryName(x))
3.     .Distinct()
4.     .ToList();
5.  

This will omit empty directories, but you generally should not rely on them - empty directories don't end up in a Android build.

 

Hi!

On our webgl build the files seem to be retrieved from [path to streaming assets]//[rest of the path]. It works fine on localhost and our linux server, however retrieving the files fails on other servers because of the double slash. Why is there a double slash in the first place?

Code sample:

Code (CSharp):

1.  
2.         BetterStreamingAssets.GetFilesAsync("ChangeableContentData/Pictures", "*.png", (foundFiles) => files = foundFiles);
3.  
4.         while (files == null)
5.         {
6.             yield return new WaitForEndOfFrame();
7.         }
8.