UnifiedIO - Cross-platform IO for Unity

I'm downloading Unity 2017.1.0p5 right now and will create a small repro project to send this to the Unity team. Maybe they can tell us what is currently wrong with waiting for async calls.

Non-IL2CPP builds for Windows Store apps use Microsoft's .NET compiler, but it doesn't seem related to that, since my unit tests for UnifiedIO use it and work fine outside of a Unity build. So yes, it sure looks like a Unity bug. I would love to know since which Unity version this became a problem. Might have to test some of the ones I have installed.

Daniel

 

alright, let me know when more is know.
Currently I can't publish due to this issue, so hopefully, it is resolved sooner then later.
Thanks for you help/

 

Will do.

I've already sent them a bug report. I'll see if I can work around the problem or if I'm maybe doing something wrong with the async waiting, though I'm very curious to know what changed in the Unity player that causes this problem now whereas it had always worked fine previously.

You are welcome!

Daniel

 

So I searched around and it seems that hang problems when waiting for async calls are very common if you don't know exactly what you are doing. Because of the way async calls work when they save the thread context for later while returning an incomplete Task immediately, it's possible that forcing a wait for the result (which I have to do for UnifiedIO to be cross platform) can deadlock the thread if the wait and the async call use the same context. This link explains it better if people are interested: http://blog.stephencleary.com/2012/07/dont-block-on-async-code.html. The link also shows how to solve the problem by using ConfigureAwait(false) whenever we await for a Task, which is what I did as a workaround until I'm confident that this will not cause problems.

UPDATE: The above workaround didn't work in every situation, especially using a XAML build type. Another workaround, which seems to work in every case, involves running every internal UnifiedIO call to async methods in their own Task, using Task.Run. That fixed every issue.

The fact that I had never encountered this issue is probably due to the fact that Unity must have recently changed which thread is running the UWP code in the player (I might be wrong, though; they might explain this when they reply to my bug report). It also has always worked perfectly fine in my unit tests, which run outside Unity.

UPDATE 2: Unity was able to reproduce the problem based on my bug report. They haven't added anything else yet.

UPDATE 3: UnifiedIO version 2.2.2 contains this fix and is already available in the asset store.

Daniel

 

Hi Daniel.

I purchased UnifiedIO yesterday, and testing your code now.

But, cannot I use your plugin for IL2CPP backend?

 

Hi @VenisterUT, you should be able to use it in IL2CPP. Is something not working right or were you asking just to confirm? If there's a problem I will need more info.

Cheers!
Daniel

 

Hi.
I'm porting a game in 4.7.2 to Windows store desktop only.
I've issues feeding the serializer because the system.IO is lacking multiple features I was using.

Basically I need to read a data stream and check if the file exits and créate files during serialization process.

Can I use it in a public static class?

Code (CSharp):

1. public static Dictionary<string,TankProperties> Load() {
2.         bool dataFile=false;
3.         Dictionary<string,TankProperties> data;
4.  
5.         #if !NETFX_CORE
6.         //code active when building for platforms other than WSA
7.         Debug.Log (Application.persistentDataPath + "/AI.dat");
8.         FileStream file = null;
9.    
10.         if(File.Exists(Application.persistentDataPath + "/AI.dat")) {
11.             dataFile=true;
12.             file = File.Open(Application.persistentDataPath + "/AI.dat", FileMode.Open);
13.         }
14.  
15.         #else    
16.        
17.         #endif
18.         //DESERIALIZE
19.         if (dataFile) {
20.             var reader = new BsonReader (file);
21.             //reader.ReadRootValueAsArray=true;
22.        
23.             var serializer = new JsonSerializer ();
24.             data = (Dictionary<string,TankProperties>)serializer.Deserialize < Dictionary < string,TankProperties>> (reader);
25.        
26.             #if !NETFX_CORE
27.             file.Close ();
28.             #else
29.             file.Dispose();
30.             #endif
31.         }
32.         else {
33.             data = new Dictionary<string,TankProperties>();
34.         }  
35.         return data;
36.     }
37.  

This code would be placed in a public static class. It does work with binary formatter in editor aswell as with json.net serializer but I was having issues with Windows.threading.tasks because it being public static, among other issues in the returning type of theese clases that I can´t understand what is going on.

Code (CSharp):

1. The methods I tried to use for Windows store are theese:
2. #if NETFX_CORE
3.  
4.     //SAVE-------------------------------------------
5.  
6.     public async Task SaveFileWSA(Dictionary<string,TankProperties> TankDictionary)
7.     {
8.         // create a file with the given filename in the local folder; replace any existing file with the same name
9.         StorageFile file = await Windows.Storage.ApplicationData.Current.LocalFolder.CreateFileAsync("AI.dat", CreationCollisionOption.ReplaceExisting);
10.      
11.         // write  into the file
12.         using (var stream = await file.OpenStreamForWriteAsync())
13.         {
14.             var writer = new BsonWriter (stream);
15.             var serializer = new JsonSerializer ();
16.             serializer.Serialize (writer, TankDictionary);
17.         }
18.     }
19.  
20.  
21.     // CHECKFILEXISTS--------------------------------
22.     static async Task<bool> DoesFileExist()
23.     {
24.         try
25.         {
26.             await ApplicationData.Current.LocalFolder.GetFileAsync("AI.dat");
27.             return true;
28.         }
29.         catch
30.         {
31.             return false;
32.         }
33.     }
34.  
35.  
36.     //    LOAD--------------------------------
37.  
38.     public async Task<Stream> LoadFileWSA()
39.     {
40.         // access the local folder
41.         StorageFolder local = Windows.Storage.ApplicationData.Current.LocalFolder;
42.  
43.         // open the file  for reading
44.         Stream stream = await local.OpenStreamForReadAsync("AI.dat");
45.  
46.         /*
47.         String dic;
48.      
49.         // copy the file contents into the dictionary 'dic'
50.         using (StreamReader reader = new StreamReader(stream))
51.         {
52.             dic = reader.ReadToEnd();
53.         }
54.         */
55.      
56.         return stream;
57.     }
58.     #endif
59.        

 

Hi @TooManySugar, when you ask "Can I use it in a public static class?" do you mean "it" as "UnifiedIO"?

If that is the case, yes, using UnifiedIO in a static class wouldn't be any different from using it in an instance of a class. (Actually, UnifiedIO is all static classes! ) The library waits for all async calls and implements some file/folder handling methods that are missing from Windows Store apps, so you don't need to deal with Tasks at all.

I'm not sure of what problems Windows.Threading.Tasks is giving you. To see if I can be of any help, what exact errors are you seeing? By the way, maybe someone on UnityAnswers could help you as well.

All the best,
Daniel

 

TX Daniel.

Not really whanting to making tasks work but my serialization to work XD.

If I manage to make it work with your asset I'm cool with it. I've stripped the platform specific code so you can see how I do sabe/load data. I do serialize and sabe to a file and if file exists I do read it and deserialize. I'm porting a game to Windows store and found in the process that Binary formater, file.exists, filestream etc... do not exist in .NET at least not what comes packed with unity 4.7.2. I'm using 4.7.2 because the game was developed on 4.x platform and there is no need (and would be a total mess to update) to5.x+. The game is no longer developed I ended up development a month ago or so.

Code (CSharp):

1. using UnityEngine;
2. //to enable serialization
3. using System;
4. //using System.Runtime.Serialization.Formatters.Binary;
5.  
6. using System.Collections.Generic;
7. using Newtonsoft.Json;    //Json serialization
8. using Newtonsoft.Json.Bson;    // binary serialization
9.  
10. using System.IO;
11.  
12. public static class sampleSaveLoad_TankData  {    
13.        
14.     //OFFLINE SAVE--> filestream
15.     public static void Save(Dictionary<string,TankProperties> TankDictionary ) {
16.    
17.         FileStream file = File.Create (Application.persistentDataPath + "/AI.dat"); //you can call it anything you want
18.         var writer = new BsonWriter (file);
19.         var serializer = new JsonSerializer ();
20.         serializer.Serialize (writer, TankDictionary);  
21.        
22.         file.Close();      
23.     }        
24.  
25.     public static Dictionary<string,TankProperties> Load() {
26.        
27.         bool dataFile=false;
28.         Dictionary<string,TankProperties> data;
29.  
30.         FileStream file = null;
31.        
32.         if(File.Exists(Application.persistentDataPath + "/AI.dat")) {
33.             dataFile=true;
34.             file = File.Open(Application.persistentDataPath + "/AI.dat", FileMode.Open);
35.         }
36.        
37.         //DESERIALIZE
38.         if (dataFile) {
39.             var reader = new BsonReader (file);
40.  
41.             var serializer = new JsonSerializer ();
42.             data = (Dictionary<string,TankProperties>)serializer.Deserialize < Dictionary < string,TankProperties>> (reader);
43.            
44.             file.Close ();
45.  
46.             Debug.Log ("dic lenght" + data.Count);
47.         }
48.         else {
49.             data = new Dictionary<string,TankProperties>();
50.         }    
51.  
52.         return data;
53.     }
54.  
55.     /*
56.    
57.     //online
58.     //------------------------------------------------------------------------------------------
59.    
60.     //ONLINE SAVE-->memory stream, llamado x masterclient en la pantalla de player setup (UI).
61.     //En online necesito serializar en vez de a un file a un byte[] para poder enviarlo por photon como master client via RPC.
62.     public static byte[] Save_OnlineData(Dictionary<string,TankProperties> TankDictionary){
63.    
64.         BinaryFormatter bf = new BinaryFormatter();
65.        
66.         using (MemoryStream ms = new MemoryStream())
67.         {
68.             bf.Serialize(ms, TankDictionary );
69.             return ms.ToArray ();
70.         }        
71.  
72.     }
73.    
74.     //En online se recive byte[] via RPC y se envia aqui para deserializar
75.     public static Dictionary<string,TankProperties> Load_OnlineData( byte[] boxedData) {
76.  
77.         TankDic data ;
78.  
79.         BinaryFormatter bf = new BinaryFormatter();
80.  
81.         using (MemoryStream ms = new MemoryStream(boxedData))
82.         {
83.             data = (TankDic) bf.Deserialize(ms);
84.         }
85.  
86.     return data;
87.  
88.     }
89.     */    
90. }

I understand that how it is now, not using binary formatter (on the un commented part) and switching from system.IO namespace to Unified.IO this would work. (Bson serialization is working already in editor).

Converting Filestream to Stream would work out of the box?

I commented the online code cause uses memoryStream witch I've not yet checked if exists or not in WSA.

 

@TooManySugar It's fine if you use Unity 4.7.2 because the build process for Windows Store/UWP builds in Unity 5 and 2017 will still use Microsoft's .NET compiler installed on your system, so those classes won't exist in either case.

If you can still use the BsonReader and JsonSerializer in Windows Store, then you should be able to use UnifiedIO to replace the file-related methods. I'm not sure what you mean by "Converting Filestream to Stream". FileStream extends Stream, so a FileStream is always a Stream as well. BsonWriter (which takes a Stream) can take any of them.

Replacing a serializer like BinaryFormatter for your online section of the code is, however, probably trickier, since Windows Store apps removed a lot of those classes. I assume you can't change the online data format now, right? Solutions I've found in the web seem to rely on changing the serialization itself to something else (like the UniversalSerializer), not interoperability.

Sorry if this isn't of much help to you. Maybe someone on StackOverflow knows how to read a stream created by a BinaryFormatter on UWP apps. I haven't needed to solve this particular problem.

Daniel

 

Thanks a lot Daniel, your answers enlighten the path a Little more.

At least your solution solves the file read write.
Ok to the Stream part.
For file serialization the Json serialization should do fine I believe.
For the online part I'm not so sure, but I'm even ready to strip teh online part for the game make into the store, even at a reduced Price due to lack of some features. I'll keep searching for all the questions and if I manage to see a path for all of them buy a license of your product.
TX for the help.

 

You're welcome! If removing the online portion for now is a possibility then there's at least that.

Alright, if you decide to purchase and have any questions after checking the docs, feel free to post here again or contact me via email!

 

Hello Daniel I got your asset.

Switching namespace from System.IO to Unified IO makes Stream not available. How is that?!

Even in documentation I can find examples like:

Code (CSharp):

1. Stream stream = UnifiedIO.File.GetReadStream("folder/some file.txt");

what I'm doing wrong :\ ?

Need to add some specific namespace?


////////////////
Anyways, I've it working in editor and hopefully in WSA too

Code (CSharp):

1.     public static void Save(Dictionary<string,TankProperties> TankDictionary ) {
2.  
3.         Debug.Log ("save");
4.             using (var writer = new BsonWriter (File.GetWriteStream (Application.persistentDataPath + "/AI.dat") )) {
5.                 var serializer = new JsonSerializer ();
6.                 serializer.Serialize (writer, TankDictionary);  
7.             }
8.     }

Code (CSharp):

1. public static Dictionary<string,TankProperties> Load() {
2.  
3.         bool dataFile=false;
4.         Dictionary<string,TankProperties> data;
5.  
6.         Debug.Log (Application.persistentDataPath + "/AI.dat");
7.  
8.         if(File.Exists(Application.persistentDataPath + "/AI.dat")) {
9.             dataFile=true;
10.             //file = File.Open(Application.persistentDataPath + "/AI.dat", FileMode.Open);
11.         }
12.  
13.         //DESERIALIZE
14.         if (dataFile) {
15.             using(var reader = new BsonReader ( UnifiedIO.File.GetReadStream(Application.persistentDataPath + "/AI.dat"))){
16.            
17.                 var serializer = new JsonSerializer ();
18.                 data = (Dictionary<string,TankProperties>)serializer.Deserialize < Dictionary < string,TankProperties>> (reader);  
19.                
20.                 //Debug.Log ("dic lenght" + data.Count);  
21.             }
22.         }
23.  
24.         else {
25.             data = new Dictionary<string,TankProperties>();
26.         }
27.  
28.         return data;
29.     }
30.  

when saving if file exists does overwrite if does not exist does créate, just the behaviour I had. So for the offline part I'm happy.

In regards to online, the lack of Stream is really an issue. :\ but I'll see what I find.

 

I'm getting an error:

Assets\UnifiedIO\UnifiedIO_WSAAsync_Directory.cs(62,4): error CS0246: No se puede encontrar el tipo o el nombre de espacio de nombres 'SearchOption' (�falta una directiva using o una referencia de ensamblado?)

Assets\UnifiedIO\UnifiedIO_WSAAsync_Directory.cs(84,4): error CS0246: No se puede encontrar el tipo o el nombre de espacio de nombres 'SearchOption' (�falta una directiva using o una referencia de ensamblado?)

Assets\UnifiedIO\UnifiedIO_WSAAsync_Directory.cs(62,32): error CS0103: El nombre 'SearchOption' no existe en el contexto actual


Assets\UnifiedIO\UnifiedIO_WSA_Directory.cs(94,14): error CS0234: El tipo o el nombre del espacio de nombres 'SearchOption' no existe en el espacio de nombres 'System.IO' (�falta una referencia de ensamblado?)

Assets\UnifiedIO\UnifiedIO_WSA_Directory.cs(94,52): error CS0234: El tipo o el nombre del espacio de nombres 'SearchOption' no existe en el espacio de nombres 'System.IO' (�falta una referencia de ensamblado?)

etc...

 

Hello there,

Thank you for getting UnifiedIO. Let's see if I can help...

Stream is a class in the System.IO namespace, so you should still use System.IO. What you can't use are the File and Directory classes (among other things) since they don't work on Windows Store, but System.IO still exists and is required if you use Stream.

Here are a few things I'd like to address:

1. You should probably not import the UnifiedIO namespace directly if you need System.IO as well (you do because of Stream), since it may cause conflicts because both namespaces have classes named File and Directory. You should instead avoid writing "using UnifiedIO" at the top of your files and always prepend UnifiedIO before calling methods in File and Directory, like in the docs. So your "File.GetWriteStream" should be "UnifiedIO.File.GetWriteStream", like you did with UnifiedIO.File.GetReadStream. Same goes for using File.Exists, which should be UnifiedIO.File.Exists, and any other such calls.

2. Please check the "Preliminary notes" section (7.1) in the docs, since you shouldn't use "Application.persistentDataPath" when calling UnifiedIO. Quoting myself:

3. The errors are very strange. When do they happen? When building for Windows Store? I would normally ask if you have a class in your project called SearchOption, but in this case System.IO.SearchOption doesn't work either. Do you have a namespace named System.IO somewhere in the project? Do you have a copy of mscorlib.dll in your project? (not that you should, but if you do it may conflict with the expected one) It is the dll that normally contains these classes, but something seems very wrong.

Daniel

 

Hi there.
1)ok I will
2)ok !sorry
3)
-Yes when doing build for store.
-"Do you have a namespace named System.IO somewhere in the Project?" -->Yes there is at least other class that I've not yet updated with Unified.IO that uses it. Removing the manespace on that script and comenting the code did not solve the issue.
-no mscorlib.dll

could be due to the netcore folder?. I've a faked 4.5.1 versión as netcore folder. I could not find a visual studio versión that did a 4.5.1 so I did copy the files that seemede necesary from the framework to the netcore folder. I did already manage to build during testing with it so I guess that is no the issue but....

 

No problem

It is possible that the issue results from using .NET Core, since it doesn't have the same classes as normal .NET. If it doesn't include the SearchOptions enum for some reason, that's it.

I think that if you install the Windows 10 SDK packages for Visual Studio you should have no problems in that regard (unless your target isn't Windows 10, of course ). Use Visual Studio installer and install the "Universal Windows Platform development". You can also select individual components and install the 4.5.1 target if that doesn't work. Not sure if this will help. What have you tried?

Daniel

 

I'm not targetin W10 since I'm in Unity 4.7.2 8.1 is what I can and it does require 4.5.1.

Make 4.5.1 is not an easy task.. aparently. Here a thread:
https://stackoverflow.com/questions/24697242/missing-netcore-v4-5-1

Tomorrow I'll retry. kinda tired atm

 

Oh, of course! It completely slipped my mind that there was no Windows 10 back them. Yeah, it sounds like your task is harder than this normally is. Let me know if you have updates!

 

Just to make the point:
A WSA Unity3d app cannot access files outside of the persistentdata folder on a user's PC?
I have an app where I want users to identify images and upload them to the cloud. This would appear to be impossible if my understanding is correct.

 

It can, but it usually needs user interaction for that. By default the app uses its own reserved storage folder (Unity makes persistentDataPath point at a folder inside that folder). If you want to read or write a file somewhere outside of that folder, you can call a native WSA file picker and then the user is able to select the file they want, so the app gets "permission" to read/write there. There may be other ways that I'm not aware of, since I've never had to do that for my projects, or in UnifiedIO.

I hope that helps!

 

Looks like there is a WSA native class for Storage file access, where you make a call using the Storage class and it provides an object which can then be used to read the file. The Storage object assures that the only file you have access to is the one that was picked in the file browser. I can't remember the actually class name. Major monkey wrench since the app was fully functional until it is compiled into a WSA build, then testing and figuring out why it's not working becomes a nightmare.

 

Yeah, StorageFile and StorageFolder are the pillars of the API for WSA. The fact that it was so different was what led me to create UnifiedIO.

That said, you should be able to use the System.IO API and possibly not have to use StorageFile at all if you create the build using IL2CPP instead of .NET as the scripting backend (on Unity 2019 IL2CPP is the only option for WSA, actually). I'm not sure if the app can access outside of its storage without a file picker, though, but it lets you use the normal methods.

 

@CanisLupus Hey boss! I just bought this and I'm not sure if I just missed it but I need help with setting this up to read a json file in streaming assets!

thank you!

 

Hi @Egony, have you checked the docs yet? UnifiedIO was built for the persistentDataPath folder, but due to popular request I added support for changing the base folder, so other folders should work too. Section 5 (Advanced usage) in the docs talks about this.

For the specific case of Application.streamingAssetsPath, please be aware that on some platforms this "folder" is not actually a folder, so UnifiedIO won't work there. That is the case on Android, for example.

Let me know if this helps!

- Daniel

 

@CanisLupus Thank you for your reply! Yes I read the documentation however I'm not 100% sure if not in streaming assets how to make a "Persistent Data" folder in the editor if that makes sense.. I've tried looking all over for info on this but i cant seem to find it anywhere in all honesty I really couldn't care less where the files are stored so long as i can get this to work. Sorry if this is a dumb question.. I've been programming for 10 years but I've only been using unity for about 6 months and never once had to deal with IO operations on it, so this is all new territory

Again, thank you for your time!

 

@Egony Hmm... what are you trying to achieve? Or rather, what problem does UnifiedIO solve in your case? In the modern days, if you use Unity 2019 or later, or if you don't care about the UWP platform (Microsoft Store) my asset probably won't help you.

1. I believe you want to read a JSON file, but does that file change over time? If not, you can place it in the Resources folder of your project and simply read it as a TextAsset using Resources.Load.

2. You can also place the file in streaming assets, since they will be included in builds, but the way to read from it changes between platforms.

3. For a file that can change over time, just like savegames, you can use store it in the persistent data folder. The path of this folder is given by Application.persistentDataPath. Unity automatically creates this folder, and its location depends on the platform, but you can always read/write files from/to it using the System.IO.File class (unless on UWP pre-Unity 2019, where you used Microsoft's .NET compiler instead of Unity's IL2CPP).

Did this help in any way?

 

yes thank you lol kinda wish i knew that it was something that could be achieved like that however you have my money so enjoy it

 

And what I'm doing is save games and I have json files with the class properties for all assets so I can pull them on the fly instead of having tons of prefabs.
Example
{
"engineModel": [
{
"ID": "9b5085d2-f369-4b86-b2ae-5c6c2fc21135",
"Type": "Outfit",
"Name": "Afterburner",
"Category": "Engines",
"Cost": "70000",
"Thumbnail": "outfit/afterburner",
"Mass": "15",
"OutfitSpace": "-15",
"EngineCapacity": "-15",
"AfterburnerThrust": "25.0",
"AfterburnerFuel": ".5",
"AfterburnerHeat": "10.0",
"AfterburnerEffect": "afterburner",
"Description": "An afterburner works by dumping hyperspace fuel into your engines and igniting it, producing a large amount of thrust. This can be very useful for dodging missiles or briefly escaping from faster opponents, but you must be careful to avoid using up so much fuel that you do not have enough left for hyperspace travel."
}
}
So for that its a static file where I can pull those files and set them to a weapon class

and the save file will be an encrypted (maybe) json file that will basically have all the game data (systems known relationships with factions money level ships owned etc. ) so I'm not sure based on your answer that your asset will help me much

 

@CanisLupus if that's the case then consider the purchase a donation for your hard work, I appreciate your patients and assistance keep up the good work, mate

 

@Egony Thank you! You're welcome for the assist. I tried warning users at the top of the store description that UnifiedIO is not as useful after Unity 2019, but I haven't deprecated the package because it may still be useful to someone (and I'm unsure if previous buyers would be able to see it in the store).

Looking at this, you could have your "class properties" JSON in your Resources folder, since it doesn't change. As for savegames/encrypted files you should try the System.IO.File class and store the file in Application.persistentDataPath (easy solution) or any other folder that you can guarantee exists on the OS in question.

Cheers!