Copying everyhting under StreamingAssets to persistentDataPath on Android

Hello all. For the last five hours I've been googling a way of copying everything in my StreamingAssets folder to the persistentDataPath folder - in the start up of my Android application. I am doing this because I need to iterate through those files in an easy way; the StreamingAssets are fine in pulling files when you have the precise path to the file (using WWW and: jar:file://" + Application.dataPath + "!/assets" + url) but the problem, as I percieve it, is that I cannot loop through those folders since they are inside the .apk.

I found the following approach that supposedly should make the StreamingAsset folders accessable:

Code (CSharp):

1. string oriPath = System.IO.Path.Combine(Application.streamingAssetsPath, "db.bytes");
2.  
3.   // Android only use WWW to read file
4.   WWW reader = new WWW(oriPath);
5.   while ( ! reader.isDone) {}
6.  
7.   realPath = Application.persistentDataPath + "/db";
8.   System.IO.File.WriteAllBytes(realPath, reader.bytes);
9.  
10. dbPath = realPath;

But I can't get it to work; I checked the persistent data path on my android device and the folder was empty and I also modified the code to loop through that supposedly new correct path but nothing was returned. I've also tried replacing "Application.streamingAssetsPath" with the aforementioned jar:file:// url but that didn't work either.

I just need to be able to loop through folders so that I don't have to hard code file count values etc.

Any ideas? Thanks in advance.

 

It's an old topic but at the moment I'm having the same problem...

So far using .net this is what I've found online to copy from the directories:

Code (CSharp):

1.     public static string[] getFtpFolderItems(string ftpURL) {
2.         FtpWebRequest request = (FtpWebRequest)WebRequest.Create (ftpURL);
3.         request.Method = WebRequestMethods.Ftp.ListDirectory;
4.  
5.         FtpWebResponse response = (FtpWebResponse)request.GetResponse ();
6.  
7.         Stream responseStream = response.GetResponseStream ();
8.         StreamReader reader = new StreamReader (responseStream);
9.  
10.         return reader.ReadToEnd ().Split ("\r\n".ToCharArray (), StringSplitOptions.RemoveEmptyEntries);
11.     }

 

Honestly, if you are able, assetbundle your stuff in streamingassets and then put the bundle in there instead. Then load up the bundle and extract stuff and save it to persistantdatapath that way. Just did this myself. You'll get a smaller build size that way.

 

Hmmm...I'm gonna look more into it.

In the meantime I'm wondering if there's just a way to download the .jar file? What I end up with as a url for it is something like jar:file/// then a url that leads to the /assets/ folder. How can I use this url to download the jar file so I could just extract it from there?

 

https://docs.unity3d.com/ScriptReference/Application-streamingAssetsPath.html

Some platforms, streamingassets can be accessed just like any other file location, but you can use Application.StreamingAssetsPath to get a path without having to worry about what platform you are on. In the case of android, for example, you have to use the www or unitywebrequest to get at a file in streamingassets.

The first post gives an idea of this works on android, however, you can just yield the www call so that it will only continue after it gets something back (error or what you expect to get back). Now, just also note that streamingassets puts the raw files into your build without any compression or etc that Unity might apply (thus the reason I suggest the asset bundle).

 

@paragraph7 ideally how you can do this is:
If you have an idea of all the folders in the main or parent folder which resides in the persistant data path of the android folder, you can create an xml file or json file which represents the folder structure.
You can iterate through the xml file and then do your operations on the file or folder.

I believe there are scripts that will allow you to write your folder and file structure and hierarchy in the xml or the json.

eg;
persistant-data-folder
--theRootFolder(whatever I like to give the name as )
---folder1
---folder2
----folder3
----folder4
---folder5
---folder6
----file1
----file2


Just use an online tool to map this directory structure to the xml or json and in the unity application use the json or xml and iterate or get or use or modify or delete whatever you want.

Cheers!

 

Thanks guys.

The solution I'm currently looking for is to simply take the streamingassets folder which is a jar file on android from what I understand and simply download it to persistentDataPath, then extract it from that directory (if there aren't already files in persistentDataPath i.e this is the first time the game is running)...

What I have is a url to the jar within the apk like this from streamingAssetsPath:

Code (CSharp):

1. jar:file:///mnt/asec/com.AstraCat.TestGame-2/base.apk!/assets/

This is a string, not a url object...

Looking it up online I guess I have to use JarURLConnection to download it using that string? Does this make sense?

 

It's a string for the path.

As far as copying and extracting, never done it that way or seen it done that way. I can't offer any help on that I'm afraid.

 

@Brathnann Alright, thanks anyway though. I think I've found that a best practice for streamingassets is to NOT put your scene files in there. I really only have xml data files and then I placed my scene files as well. I'm going to try placing my scene files outside of it so that it only has the xml files in there, and then using the JarURLConnection or whatever to download the jar file into the persistentDataPath then extract it from there (if I even can, I don't know).

I'll tell you how it goes when I get the chance to work on it. I will keep in mind your tip of compressing assets that are large (or I'll just in that case keep large assets away from the folder if they don't get compressed).

 

I guess you are trying to do is implement an incremental downloadable content ... so you want to download files if they don't exist... I have done that ...

You need xml files and asset bundles

 

I actually got it working compressing the files at build into a tgz file. Then, I downloaded the file using WWW if the game runs on android and extracted it to the persistentdatapath folder. If anyone would like some examples I can post them. It required an external dll to do.

EDIT: Okay, who am I kidding I'm going to post the code here in case anyone ever has this problem.

1) The first thing I needed was a library that works on android and windows universally for zipping up and unzipping tar.gz files (or tgz files, both of those formats the same thing).

The best library I found for this is here:
https://github.com/icsharpcode/SharpZipLib

You need the dll entitled:
ICSharpCode.SharpZipLib

You need to place that in your Plug-ins folder in your Assets directory.

2) So say, like me, you have your main game database files and save files in a directory like "Assets/Data". In this case what's needed is to zip that folder up into a tgz file and place that tgz file within the StreamingAssets folder upon building the game.

One can do that using code like this...I named this file Editor_BuildPreprocess.cs and placed it within my Editor folder:

Code (CSharp):

1. #if UNITY_EDITOR
2.  
3. using UnityEngine;
4. using UnityEditor;
5. using UnityEditor.Build;
6. using System.IO;
7. using ICSharpCode.SharpZipLib.Core;
8. using ICSharpCode.SharpZipLib.Zip;
9.  
10.  
11. class CustomBuildPreProcess : IPreprocessBuild
12. {
13.     public int callbackOrder { get { return 0; } }
14.     public void OnPreprocessBuild(BuildTarget target, string path) {
15.  
16.         string dataDirectory = Application.dataPath + "/Data/";
17.         string fileToCreate = Application.streamingAssetsPath + "/Data.tgz";
18.  
19.         Utility_SharpZipCommands.CreateTarGZ_FromDirectory (fileToCreate, dataDirectory);
20.  
21.     }
22. }
23.  
24. #endif

Now we need to create a new cs file with our sharp zip library methods (we have to create these ourselves if we want to use the dll).

I named this file Utility_SharpZipCommands.cs (I prefix things with something like that typically):

Code (CSharp):

1. using System;
2. using System.IO;
3. using ICSharpCode.SharpZipLib.GZip;
4. using ICSharpCode.SharpZipLib.Tar;
5.  
6. public class Utility_SharpZipCommands {
7.  
8.         //// Calling example
9.         //CreateTarGZ(@"c:\temp\gzip-test.tar.gz", @"c:\data");
10.  
11.         //USE THIS:
12.         public static void CreateTarGZ_FromDirectory(string tgzFilename, string sourceDirectory) {
13.  
14.             Stream outStream = File.Create(tgzFilename);
15.             Stream gzoStream = new GZipOutputStream(outStream);
16.             TarArchive tarArchive = TarArchive.CreateOutputTarArchive(gzoStream);
17.  
18.             // Note that the RootPath is currently case sensitive and must be forward slashes e.g. "c:/temp"
19.             // and must not end with a slash, otherwise cuts off first char of filename
20.             // This is scheduled for fix in next release
21.             tarArchive.RootPath = sourceDirectory.Replace('\\', '/');
22.             if (tarArchive.RootPath.EndsWith("/"))
23.                 tarArchive.RootPath = tarArchive.RootPath.Remove(tarArchive.RootPath.Length - 1);
24.  
25.             AddDirectoryFilesToTar(tarArchive, sourceDirectory, true);
26.  
27.             tarArchive.Close();
28.         }
29.      
30.         public static void AddDirectoryFilesToTar(TarArchive tarArchive, string sourceDirectory, bool recurse) {
31.  
32.             // Optionally, write an entry for the directory itself.
33.             // Specify false for recursion here if we will add the directory's files individually.
34.             //
35.             TarEntry tarEntry = TarEntry.CreateEntryFromFile(sourceDirectory);
36.             tarArchive.WriteEntry(tarEntry, false);
37.  
38.             // Write each file to the tar.
39.             //
40.             string[] filenames = Directory.GetFiles(sourceDirectory);
41.             foreach (string filename in filenames) {
42.                 tarEntry = TarEntry.CreateEntryFromFile(filename);
43.                 tarArchive.WriteEntry(tarEntry, true);
44.             }
45.  
46.             if (recurse) {
47.                 string[] directories = Directory.GetDirectories(sourceDirectory);
48.                 foreach (string directory in directories)
49.                     AddDirectoryFilesToTar(tarArchive, directory, recurse);
50.             }
51.         }
52.  
53.     public static void ExtractTGZ(string gzArchiveName, string destFolder) {
54.         Stream inStream = File.OpenRead (gzArchiveName);
55.         Stream gzipStream = new GZipInputStream (inStream);
56.  
57.         TarArchive tarArchive = TarArchive.CreateInputTarArchive (gzipStream);
58.         tarArchive.ExtractContents (destFolder);
59.         tarArchive.Close ();
60.  
61.         gzipStream.Close ();
62.         inStream.Close ();
63.  
64.     }
65.    
66. }    // Calling example

Now what will happen when you build your game is the tgz file will be created and placed within your StreamingAssets folder. This is accessible then from Android like so when the game starts:

Code (CSharp):

1. #if UNITY_STANDALONE_WIN
2.             Debug.Log("unzipping to persistent data path (windows)");
3.             Utility_SharpZipCommands.ExtractTGZ (Application.streamingAssetsPath + "/" + "Data.tgz",Application.persistentDataPath);
4.         #endif
5.         #if UNITY_ANDROID
6.             //if mg_data doesn't exist, extract default data...
7.             if (File.Exists(Application.persistentDataPath + "/" + "MG_Data.data") == false) {
8.                 ShowMessageOnScreen("\n\nMG_Data.data doesn't exist, creating it for the first time.");
9.                 //copy tgz to directory where we can extract it
10.                 WWW www = new WWW(Application.streamingAssetsPath + "/Data.tgz");
11.                 while ( ! www.isDone) {}
12.                 System.IO.File.WriteAllBytes(Application.persistentDataPath + "/" + "Data.tgz", www.bytes);
13.                 //extract it
14.                 Utility_SharpZipCommands.ExtractTGZ (Application.persistentDataPath + "/" + "Data.tgz",Application.persistentDataPath);
15.                 //delete tgz
16.                 File.Delete(Application.persistentDataPath + "/" + "Data.tgz");
17.             } else  {
18.                 ShowMessageOnScreen("\n\nMG_Data.data does exist, will not extract default data.");
19.             }
20.         #endif

With that done, now you have a file system you can work with on Android that is cross compatible with all other file systems using Application.persistentDataPath...just add in an #if UNITY_IOS or whatever for ios etc etc... above where it says STANDALONE_WIN.

With this it's finally allowing me to start virtual reality games (like heavy data rpg games with xml save files) using samsung vr and also have it compatible with the occulus, for example.

 

Looks cool, but I don't see an advantage of this over an asset bundle honestly. But still interesting to see other options.

 

Maybe you're right. I hadn't tried the asset bundle deal because all I had were xml files in a data folder in my assets folder to work with.

I didn't realize this also but I'm using Unity 4.6 which is .NET 3.5 I believe. If the newer versions of Unity support 4.0 then I think there's a built in method for just compressing it to zip. I'm not sure if that works on android, though.

 

I think if you are using that library you can just open the jar file. The main issue on android the files are in the jar which is compressed. So you can not do a standard file copy. But if you read it as compressed you can get the data out.
Www does this for you.

 

With Unity 2017 on .NET 4.6, the best way I found to solve this problem was to use a plugin : https://assetstore.unity.com/packages/tools/input-management/better-streaming-assets-103788

Then you can easily read from StreamingAssets of any platform, and copy your files into persistentDataPath to easier use.

About the zip part I was unable to make SharpZipLib to work but DotNetZip lib worked with the addition of I18N.dll and I18N.West.dll, which I had to manually copy from Unity into my project Assets.

 

Since it's slow to decompress, I just have SharpZip extract the file only if it hasn't been extracted before on the applications first startup.

@SATAN3 Yep, this is similar to what happened to me. Copy at startup from streamingassets to persistentdatapath.

Due to this making playtesting slow, I had to actually have an enum popup in where you select where the data folder is, in the streamingassets or to be extracted to the persistentdatapath. When you're playtesting in the editor, you don't want to be extracting to persistentdatapath every time you hit the playtest button.

 

I'm curious as to how you coded that with Better Streaming Assets. Do you read the entire file in to memory then dump it to file? I have some pretty huge video files, I think I need to 'chunk' them.

 

Dont u Think Unity should deal with this problem? Just introduce a new StreamingAssetFolderUncompressed and all our Problems are solved. Can this be that difficult?

 

Since this I've found that the best way for me to get it working on all platforms was to use Resources.Load(relativePath); .

What I did is made a .txt file that's created that has the relative path name of the asset every time an asset is imported into Unity. I'm sorry I don't have time to elaborate more on this at the moment, but now my game will work on most any system. Persistent Data Path is still used for save/load files.

 

Still I don't understand how does it works? Accroding to the documentation

https://docs.unity3d.com/ScriptRefe...41.1502410520.1604388002-954575440.1598172327

It is not possible to access the StreamingAssets folder on WebGL and Android platforms.

 

As this seems to be a big problem for many, can't you do a blog post or tutorial on this?

 

I've been able to implement the solution you detailed earlier to get working on some android devices, but I'm still having some issues with full cross-platform support across Android and iOS. Would it be better to download the files into the resources folder, then extract them from there to persistent data path? (In my case I actually need them in the persistent data path for other reasons) thanks!