Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Copying everyhting under StreamingAssets to persistentDataPath on Android

Discussion in 'Scripting' started by paragraph7, Sep 15, 2016.

  1. paragraph7

    paragraph7

    Joined:
    May 9, 2016
    Posts:
    7
    Hello all. For the last five hours I've been googling a way of copying everything in my StreamingAssets folder to the persistentDataPath folder - in the start up of my Android application. I am doing this because I need to iterate through those files in an easy way; the StreamingAssets are fine in pulling files when you have the precise path to the file (using WWW and: jar:file://" + Application.dataPath + "!/assets" + url) but the problem, as I percieve it, is that I cannot loop through those folders since they are inside the .apk.

    I found the following approach that supposedly should make the StreamingAsset folders accessable:

    Code (CSharp):
    1. string oriPath = System.IO.Path.Combine(Application.streamingAssetsPath, "db.bytes");
    2.  
    3.   // Android only use WWW to read file
    4.   WWW reader = new WWW(oriPath);
    5.   while ( ! reader.isDone) {}
    6.  
    7.   realPath = Application.persistentDataPath + "/db";
    8.   System.IO.File.WriteAllBytes(realPath, reader.bytes);
    9.  
    10. dbPath = realPath;
    But I can't get it to work; I checked the persistent data path on my android device and the folder was empty and I also modified the code to loop through that supposedly new correct path but nothing was returned. I've also tried replacing "Application.streamingAssetsPath" with the aforementioned jar:file:// url but that didn't work either.

    I just need to be able to loop through folders so that I don't have to hard code file count values etc.

    Any ideas? Thanks in advance.
     
    astracat111 likes this.
  2. astracat111

    astracat111

    Joined:
    Sep 21, 2016
    Posts:
    626
    It's an old topic but at the moment I'm having the same problem...

    So far using .net this is what I've found online to copy from the directories:

    Code (CSharp):
    1.     public static string[] getFtpFolderItems(string ftpURL) {
    2.         FtpWebRequest request = (FtpWebRequest)WebRequest.Create (ftpURL);
    3.         request.Method = WebRequestMethods.Ftp.ListDirectory;
    4.  
    5.         FtpWebResponse response = (FtpWebResponse)request.GetResponse ();
    6.  
    7.         Stream responseStream = response.GetResponseStream ();
    8.         StreamReader reader = new StreamReader (responseStream);
    9.  
    10.         return reader.ReadToEnd ().Split ("\r\n".ToCharArray (), StringSplitOptions.RemoveEmptyEntries);
    11.     }
     
  3. Brathnann

    Brathnann

    Joined:
    Aug 12, 2014
    Posts:
    5,484
    Honestly, if you are able, assetbundle your stuff in streamingassets and then put the bundle in there instead. Then load up the bundle and extract stuff and save it to persistantdatapath that way. Just did this myself. You'll get a smaller build size that way.
     
    astracat111 likes this.
  4. astracat111

    astracat111

    Joined:
    Sep 21, 2016
    Posts:
    626
    Hmmm...I'm gonna look more into it.

    In the meantime I'm wondering if there's just a way to download the .jar file? What I end up with as a url for it is something like jar:file/// then a url that leads to the /assets/ folder. How can I use this url to download the jar file so I could just extract it from there?
     
  5. Brathnann

    Brathnann

    Joined:
    Aug 12, 2014
    Posts:
    5,484
    https://docs.unity3d.com/ScriptReference/Application-streamingAssetsPath.html

    Some platforms, streamingassets can be accessed just like any other file location, but you can use Application.StreamingAssetsPath to get a path without having to worry about what platform you are on. In the case of android, for example, you have to use the www or unitywebrequest to get at a file in streamingassets.

    The first post gives an idea of this works on android, however, you can just yield the www call so that it will only continue after it gets something back (error or what you expect to get back). Now, just also note that streamingassets puts the raw files into your build without any compression or etc that Unity might apply (thus the reason I suggest the asset bundle).
     
    astracat111 likes this.
  6. iamvideep

    iamvideep

    Joined:
    Oct 27, 2017
    Posts:
    81
    @paragraph7 ideally how you can do this is:
    If you have an idea of all the folders in the main or parent folder which resides in the persistant data path of the android folder, you can create an xml file or json file which represents the folder structure.
    You can iterate through the xml file and then do your operations on the file or folder.

    I believe there are scripts that will allow you to write your folder and file structure and hierarchy in the xml or the json.

    eg;
    persistant-data-folder
    --theRootFolder(whatever I like to give the name as )
    ---folder1
    ---folder2
    ----folder3
    ----folder4
    ---folder5
    ---folder6
    ----file1
    ----file2


    Just use an online tool to map this directory structure to the xml or json and in the unity application use the json or xml and iterate or get or use or modify or delete whatever you want.

    Cheers!
     
    astracat111 likes this.
  7. astracat111

    astracat111

    Joined:
    Sep 21, 2016
    Posts:
    626
    Thanks guys.

    The solution I'm currently looking for is to simply take the streamingassets folder which is a jar file on android from what I understand and simply download it to persistentDataPath, then extract it from that directory (if there aren't already files in persistentDataPath i.e this is the first time the game is running)...

    What I have is a url to the jar within the apk like this from streamingAssetsPath:

    Code (CSharp):
    1. jar:file:///mnt/asec/com.AstraCat.TestGame-2/base.apk!/assets/
    This is a string, not a url object...

    Looking it up online I guess I have to use JarURLConnection to download it using that string? Does this make sense?
     
  8. Brathnann

    Brathnann

    Joined:
    Aug 12, 2014
    Posts:
    5,484
    It's a string for the path.

    As far as copying and extracting, never done it that way or seen it done that way. I can't offer any help on that I'm afraid.
     
  9. astracat111

    astracat111

    Joined:
    Sep 21, 2016
    Posts:
    626
    @Brathnann Alright, thanks anyway though. I think I've found that a best practice for streamingassets is to NOT put your scene files in there. I really only have xml data files and then I placed my scene files as well. I'm going to try placing my scene files outside of it so that it only has the xml files in there, and then using the JarURLConnection or whatever to download the jar file into the persistentDataPath then extract it from there (if I even can, I don't know).

    I'll tell you how it goes when I get the chance to work on it. I will keep in mind your tip of compressing assets that are large (or I'll just in that case keep large assets away from the folder if they don't get compressed).
     
    Last edited: Oct 27, 2017
  10. iamvideep

    iamvideep

    Joined:
    Oct 27, 2017
    Posts:
    81
    I guess you are trying to do is implement an incremental downloadable content ... so you want to download files if they don't exist... I have done that ...

    You need xml files and asset bundles
     
  11. astracat111

    astracat111

    Joined:
    Sep 21, 2016
    Posts:
    626
    I actually got it working compressing the files at build into a tgz file. Then, I downloaded the file using WWW if the game runs on android and extracted it to the persistentdatapath folder. If anyone would like some examples I can post them. It required an external dll to do.

    EDIT: Okay, who am I kidding I'm going to post the code here in case anyone ever has this problem.

    1) The first thing I needed was a library that works on android and windows universally for zipping up and unzipping tar.gz files (or tgz files, both of those formats the same thing).

    The best library I found for this is here:
    https://github.com/icsharpcode/SharpZipLib

    You need the dll entitled:
    ICSharpCode.SharpZipLib

    You need to place that in your Plug-ins folder in your Assets directory.

    2) So say, like me, you have your main game database files and save files in a directory like "Assets/Data". In this case what's needed is to zip that folder up into a tgz file and place that tgz file within the StreamingAssets folder upon building the game.

    One can do that using code like this...I named this file Editor_BuildPreprocess.cs and placed it within my Editor folder:

    Code (CSharp):
    1. #if UNITY_EDITOR
    2.  
    3. using UnityEngine;
    4. using UnityEditor;
    5. using UnityEditor.Build;
    6. using System.IO;
    7. using ICSharpCode.SharpZipLib.Core;
    8. using ICSharpCode.SharpZipLib.Zip;
    9.  
    10.  
    11. class CustomBuildPreProcess : IPreprocessBuild
    12. {
    13.     public int callbackOrder { get { return 0; } }
    14.     public void OnPreprocessBuild(BuildTarget target, string path) {
    15.  
    16.         string dataDirectory = Application.dataPath + "/Data/";
    17.         string fileToCreate = Application.streamingAssetsPath + "/Data.tgz";
    18.  
    19.         Utility_SharpZipCommands.CreateTarGZ_FromDirectory (fileToCreate, dataDirectory);
    20.  
    21.     }
    22. }
    23.  
    24. #endif
    Now we need to create a new cs file with our sharp zip library methods (we have to create these ourselves if we want to use the dll).

    I named this file Utility_SharpZipCommands.cs (I prefix things with something like that typically):

    Code (CSharp):
    1. using System;
    2. using System.IO;
    3. using ICSharpCode.SharpZipLib.GZip;
    4. using ICSharpCode.SharpZipLib.Tar;
    5.  
    6. public class Utility_SharpZipCommands {
    7.  
    8.         //// Calling example
    9.         //CreateTarGZ(@"c:\temp\gzip-test.tar.gz", @"c:\data");
    10.  
    11.         //USE THIS:
    12.         public static void CreateTarGZ_FromDirectory(string tgzFilename, string sourceDirectory) {
    13.  
    14.             Stream outStream = File.Create(tgzFilename);
    15.             Stream gzoStream = new GZipOutputStream(outStream);
    16.             TarArchive tarArchive = TarArchive.CreateOutputTarArchive(gzoStream);
    17.  
    18.             // Note that the RootPath is currently case sensitive and must be forward slashes e.g. "c:/temp"
    19.             // and must not end with a slash, otherwise cuts off first char of filename
    20.             // This is scheduled for fix in next release
    21.             tarArchive.RootPath = sourceDirectory.Replace('\\', '/');
    22.             if (tarArchive.RootPath.EndsWith("/"))
    23.                 tarArchive.RootPath = tarArchive.RootPath.Remove(tarArchive.RootPath.Length - 1);
    24.  
    25.             AddDirectoryFilesToTar(tarArchive, sourceDirectory, true);
    26.  
    27.             tarArchive.Close();
    28.         }
    29.      
    30.         public static void AddDirectoryFilesToTar(TarArchive tarArchive, string sourceDirectory, bool recurse) {
    31.  
    32.             // Optionally, write an entry for the directory itself.
    33.             // Specify false for recursion here if we will add the directory's files individually.
    34.             //
    35.             TarEntry tarEntry = TarEntry.CreateEntryFromFile(sourceDirectory);
    36.             tarArchive.WriteEntry(tarEntry, false);
    37.  
    38.             // Write each file to the tar.
    39.             //
    40.             string[] filenames = Directory.GetFiles(sourceDirectory);
    41.             foreach (string filename in filenames) {
    42.                 tarEntry = TarEntry.CreateEntryFromFile(filename);
    43.                 tarArchive.WriteEntry(tarEntry, true);
    44.             }
    45.  
    46.             if (recurse) {
    47.                 string[] directories = Directory.GetDirectories(sourceDirectory);
    48.                 foreach (string directory in directories)
    49.                     AddDirectoryFilesToTar(tarArchive, directory, recurse);
    50.             }
    51.         }
    52.  
    53.     public static void ExtractTGZ(string gzArchiveName, string destFolder) {
    54.         Stream inStream = File.OpenRead (gzArchiveName);
    55.         Stream gzipStream = new GZipInputStream (inStream);
    56.  
    57.         TarArchive tarArchive = TarArchive.CreateInputTarArchive (gzipStream);
    58.         tarArchive.ExtractContents (destFolder);
    59.         tarArchive.Close ();
    60.  
    61.         gzipStream.Close ();
    62.         inStream.Close ();
    63.  
    64.     }
    65.    
    66. }    // Calling example
    Now what will happen when you build your game is the tgz file will be created and placed within your StreamingAssets folder. This is accessible then from Android like so when the game starts:

    Code (CSharp):
    1. #if UNITY_STANDALONE_WIN
    2.             Debug.Log("unzipping to persistent data path (windows)");
    3.             Utility_SharpZipCommands.ExtractTGZ (Application.streamingAssetsPath + "/" + "Data.tgz",Application.persistentDataPath);
    4.         #endif
    5.         #if UNITY_ANDROID
    6.             //if mg_data doesn't exist, extract default data...
    7.             if (File.Exists(Application.persistentDataPath + "/" + "MG_Data.data") == false) {
    8.                 ShowMessageOnScreen("\n\nMG_Data.data doesn't exist, creating it for the first time.");
    9.                 //copy tgz to directory where we can extract it
    10.                 WWW www = new WWW(Application.streamingAssetsPath + "/Data.tgz");
    11.                 while ( ! www.isDone) {}
    12.                 System.IO.File.WriteAllBytes(Application.persistentDataPath + "/" + "Data.tgz", www.bytes);
    13.                 //extract it
    14.                 Utility_SharpZipCommands.ExtractTGZ (Application.persistentDataPath + "/" + "Data.tgz",Application.persistentDataPath);
    15.                 //delete tgz
    16.                 File.Delete(Application.persistentDataPath + "/" + "Data.tgz");
    17.             } else  {
    18.                 ShowMessageOnScreen("\n\nMG_Data.data does exist, will not extract default data.");
    19.             }
    20.         #endif
    With that done, now you have a file system you can work with on Android that is cross compatible with all other file systems using Application.persistentDataPath...just add in an #if UNITY_IOS or whatever for ios etc etc... above where it says STANDALONE_WIN.

    With this it's finally allowing me to start virtual reality games (like heavy data rpg games with xml save files) using samsung vr and also have it compatible with the occulus, for example.
     
    Last edited: Nov 4, 2017
    SlightField likes this.
  12. Brathnann

    Brathnann

    Joined:
    Aug 12, 2014
    Posts:
    5,484
    Looks cool, but I don't see an advantage of this over an asset bundle honestly. But still interesting to see other options.
     
  13. astracat111

    astracat111

    Joined:
    Sep 21, 2016
    Posts:
    626
    Maybe you're right. I hadn't tried the asset bundle deal because all I had were xml files in a data folder in my assets folder to work with.

    I didn't realize this also but I'm using Unity 4.6 which is .NET 3.5 I believe. If the newer versions of Unity support 4.0 then I think there's a built in method for just compressing it to zip. I'm not sure if that works on android, though.
     
  14. daxiongmao

    daxiongmao

    Joined:
    Feb 2, 2016
    Posts:
    291
    I think if you are using that library you can just open the jar file. The main issue on android the files are in the jar which is compressed. So you can not do a standard file copy. But if you read it as compressed you can get the data out.
    Www does this for you.
     
  15. SATAN3

    SATAN3

    Joined:
    Mar 20, 2019
    Posts:
    2
    With Unity 2017 on .NET 4.6, the best way I found to solve this problem was to use a plugin : https://assetstore.unity.com/packages/tools/input-management/better-streaming-assets-103788

    Then you can easily read from StreamingAssets of any platform, and copy your files into persistentDataPath to easier use.

    About the zip part I was unable to make SharpZipLib to work but DotNetZip lib worked with the addition of I18N.dll and I18N.West.dll, which I had to manually copy from Unity into my project Assets.
     
  16. astracat111

    astracat111

    Joined:
    Sep 21, 2016
    Posts:
    626
    Since it's slow to decompress, I just have SharpZip extract the file only if it hasn't been extracted before on the applications first startup.

    @SATAN3 Yep, this is similar to what happened to me. Copy at startup from streamingassets to persistentdatapath.

    Due to this making playtesting slow, I had to actually have an enum popup in where you select where the data folder is, in the streamingassets or to be extracted to the persistentdatapath. When you're playtesting in the editor, you don't want to be extracting to persistentdatapath every time you hit the playtest button.
     
  17. AccentDave

    AccentDave

    Joined:
    Nov 16, 2015
    Posts:
    43
    I'm curious as to how you coded that with Better Streaming Assets. Do you read the entire file in to memory then dump it to file? I have some pretty huge video files, I think I need to 'chunk' them.
     
  18. schreibtischkrieger2019

    schreibtischkrieger2019

    Joined:
    Jul 12, 2019
    Posts:
    13
    Dont u Think Unity should deal with this problem? :D Just introduce a new StreamingAssetFolderUncompressed and all our Problems are solved. Can this be that difficult?
     
  19. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,643
    StreamingAssets are already uncompressed. They are inside apk, but they are not compressed, reading them means simply reading a particular section of file.
     
  20. astracat111

    astracat111

    Joined:
    Sep 21, 2016
    Posts:
    626
    Since this I've found that the best way for me to get it working on all platforms was to use Resources.Load(relativePath); .

    What I did is made a .txt file that's created that has the relative path name of the asset every time an asset is imported into Unity. I'm sorry I don't have time to elaborate more on this at the moment, but now my game will work on most any system. Persistent Data Path is still used for save/load files.
     
unityunity