Question How to add a new data to existing Binary file without erase the previous data?

Hi there, I save my meshes data using Binary format. I can save and load using Binary format. But I saw there a little issue in saving step. I can save then load it back. But when I launch the game and I add new variables into "MeshData" class and I save it then when I load and I see that my previous data erased in the saved file. Because the file is overwriting into existing file.

SHORTLY:

Saving the File Without Deleting What's Already There. I do not want to overwrite the whole file. I just want to add a new line/data into existing file without erase the previous data. How to add a new data into existing Binary file?


Code to SAVE and LOAD Mesh Data:

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using System.Collections.Generic;
4. using System.Runtime.Serialization.Formatters.Binary;
5. using System.IO;
6.  
7. [System.Serializable]
8. public class MeshData
9. {
10.     public List<float> vertices;
11.     public List<float> uvs;
12.     public List<float> normals;
13.     public List<int> triangles;
14. }
15.  
16. [System.Serializable]
17. public class MeshDatabase
18. {
19.     public MeshData[] meshDatas;
20. }
21.  
22. public class GetMeshData : MonoBehaviour
23. {
24.     public MeshData[] meshDatas;
25.     public MeshDatabase meshDatabase;
26.  
27.     public BinaryFormatter binaryFormatter;
28.     public FileStream fileStream;
29.  
30.     void Update()
31.     {
32.         if (Input.GetKeyDown(KeyCode.S))
33.         {
34.             Save();
35.             Debug.Log("SAVED");
36.         }
37.  
38.         if (Input.GetKeyDown(KeyCode.L))
39.         {
40.             Load();
41.             Debug.Log("LOADED");
42.         }
43.     }
44.  
45.     public void Save()
46.     {
47.         binaryFormatter = new BinaryFormatter();
48.         fileStream = File.Create(Application.dataPath + "/StreamingAssets/Mesh.dat");
49.         binaryFormatter.Serialize(fileStream, meshDatabase);
50.         fileStream.Close();
51.     }
52.  
53.     public void Load()
54.     {
55.         if (File.Exists(Application.dataPath + "/StreamingAssets/Data.dat"))
56.         {
57.             binaryFormatter = new BinaryFormatter();
58.             fileStream = File.Open(Application.dataPath + "/StreamingAssets/Mesh.dat", FileMode.Open);
59.             meshDatabase = (MeshDatabase)binaryFormatter.Deserialize(fileStream);
60.             fileStream.Close();
61.         }
62.     }
63. }
64.  

PLEASE EXPLAIN AS CODE!!! THANKS.

 

That kind of binary stream is lacking any kind of formatting, so you need to preserve the previous shape of the datagram if you want to read it.

However, you really shouldn't be using this format:

Don't use the binary formatter/serializer: it is insecure, it cannot be made secure, and it makes debugging very difficult, plus it actually will NOT prevent people from modifying your save data on their computers.

https://docs.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide

If you insist on using it then you need to implement your own versioning system, so that you write (perhaps) a version integer or string out that lets your code read it with the correct shape data.

This also means you must maintain separate structures for every possible previous version of the data structure you wish to support reaching back in time.

Seriously, just use JSON.

 

Thanks for respond Kurt. I did not know Binary stream was a secure. But I saw at this video tutorial

that Binary is a most secure. Nobody can read the binary file because it is converting to unreadable state. And also I took a look at Binary file and I could not read because of encryption. Even I saw that MINECRAFT use Binary format to save data. Still I did not understand. Also I wanted to learn use JSON to save and load data for mySQL database. But my game is not an online for right now. I think to add online later and I think that there a little things to do before to use the JSON. I will pay attention to your advice.

So I have a few question.

1) can I add a new line into JSON file without erasing the previous data/lines? Or I can not do that with this format too?

2) Is there not another easy way to add a new line without lost the previous data? Or there is not, then I need to keep the previous data then, add a new data and save these whole data again into file, Yes, like this ?

 

Why you want to do that?You can make all your save data like one c# object and then,then replace values you need and save it again.

 

Code (CSharp):

1. string filename = Application.dataPath + "/StreamingAssets/Mesh.dat";
2. if (File.Exists(filename))
3.     fileStream = File.OpenWrite(filename);
4. else
5.     fileStream = File.Create....

in your code every time you press save you create a new file...

 

You clearly have not met any of the 12-year-olds who regularly break security by reading binary files. There is nothing different from a binary file and a JSON file. If you doubt me, just use the

xxd

utility (a Posix tool) and hex dump your binary and there will be all your data.

There is. You can write it to separate files. You can write it as separate chunks end to end to end. But if you change any of the underlying structure of the chunks, it won't be trivial to read back in.

This is just how unstructured binary data works. It's just a bag of bytes without any clue as to what is what. The "clue" comes in the shape of your program's structure, and when you change it, well, it no longer matches!

 

Thanks for respond, yeah I forgot to check if exist then use openwrite operation, Thanks

 

there arent many of those 12 years old doing it, especially not for simple savegame editing. those can do it, deserve to do it but binary savegame is the fastest and easiest way as there is no parsing, also easiest to encrypt if you are worried. json xml are simply readable to anyone with a notepad and slow af. this is just a microsoft crap and we are surprised why do we need 16 cores 5ghz pc-s that should run on a 200mhz celeron

 

There is. You can write it to separate files. You can write it as separate chunks end to end to end. But if you change any of the underlying structure of the chunks, it won't be trivial to read back in.

This is just how unstructured binary data works. It's just a bag of bytes without any clue as to what is what. The "clue" comes in the shape of your program's structure, and when you change it, well, it no longer matches![/QUOTE]


Oh, I did not know about 12 years old child hacked binary files and succeed to read binary file. This was a really import information, thanks for gave a such an important info. I don't doubt. Then I need to use JSON later.

You said that JSON is more secure, right? If I am wrong, then you say me, I understood that JSON is more secure than Binary, am I right?

You mean by to save in different files? Create different files for each chunks and load it back and change something or add then save it, is it? And still this means that I need to load all data variables back then save these again?

 

I divide save files. I load variables back with variables and I adding or changing something then I need to save all previous and new variables into a file, right?

Below I load variables back into as variable and (removing or adding new variables) and I saving these whole. Is it optimized or efficient way to do, or not? Did you mean like this?

Code (CSharp):

1.     public MyData[] myDatas;
2.     public void Load()
3.     {
4.         if (File.Exists(Application.dataPath + "/StreamingAssets/Data.dat"))
5.         {
6.             binaryFormatter = new BinaryFormatter();
7.             fileStream = File.Open(Application.dataPath + "/StreamingAssets/Data.dat", FileMode.Open);
8.             myDatabase = (MyDatabase)binaryFormatter.Deserialize(fileStream);
9.             fileStream.Close();
10.      
11.             myDatas = new MyData[myDatas.Length];
12.             for (int i = 0; i < myDatas.Length; i++)
13.             {
14.                 myDatas[i] = myDatas[i];
15.             }
16.         }
17.     }

Did you mean like this?

 

I said nothing about security of any format. There is zero security conferred by the format of the data.

If data is on an untrusted computer (eg, one you don't control), it is not secure.

Therefore, why make life difficult for yourself with binary data? Your costs go up, your security does not change.

 

I got it.

But my last example is right? Can you reply please.

 

I cannot run your code. You can run your code. Devise a quick experiment / test and see if it works!

 

So much misconception is going around in this thread... some of my thoughts around the subject:

I use JSON save file in development but binary in build. The reason is I can check on what I'm doing easily, but I don't need to pay the endless string-processing price in build.
Best of both worlds. The only price is that I can't test my dev saves in builds. (I can switch to binary read in dev though)

The binary formatter contains flaws which makes it unsafe to process data coming from outside sources. If you're on a server where you are processing data from the clients, you need to forget its existence. If you're developing a desktop application and you save and load files on the local computer, you shouldn't care. Nothing outside comes in unless the user deliberately replaces a save file with some arbitrary file. The attack vector here is very weak.

ps: With that said, I myself also switched to BinaryWriter just because I'm maintaining code for multiple uses and if the code ends up on the server side, it shouldn't cause a problem.

In both cases your only right way is to load the data in, make it object format, insert your new data and overwrite the file.
You do not "insert" data into a file. You can append text files though, but data-structure-wise it's not an option.

 

I made like this. I load back my data into variables and I overwrite these with new and previous data into file

CODE to load back data as variables, is it right?

Code (CSharp):

1.     public MyData[] myDatas;
2.     public void Load()
3.     {
4.         if (File.Exists(Application.dataPath + "/StreamingAssets/Data.dat"))
5.         {
6.             binaryFormatter = new BinaryFormatter();
7.             fileStream = File.Open(Application.dataPath + "/StreamingAssets/Data.dat", FileMode.Open);
8.             myDatabase = (MyDatabase)binaryFormatter.Deserialize(fileStream);
9.             fileStream.Close();
10.  
11.             myDatas = new MyData[myDatas.Length];
12.             for (int i = 0; i < myDatas.Length; i++)
13.             {
14.                 myDatas[i] = myDatas[i];
15.             }
16.         }
17.     }

But another way. If there are a lot of data and it takes a lot time to load a huge data. I mean with a "Huge Data" like (mesh vertices, triangles, uvs, position and rotation). My data is like this. And I need to divide to different files for not to take a long time to load and save it. And another question is, if there are trillion saved files in the game folder then it will be take a long time to search from these trillion saved files? I hope to explained well my question.

 

I used to insert data into file using File.Append but then I could not load it, and it gave me an error, even did not load

 

Just follow this tutorial, this way of saving is easy and much more secure.If you create your save file as one object of all data you can then find data you want replace in list and save it all again.

Code (CSharp):

1.  
2.     CreateSaveObjects();
3.             void CreateSaveObjects()
4.             {
5.                 //Get all new data from variables in Save method
6.                 var objectForSave = new PlanetObject(planetNumber, planetSize, planetType, planetGoal, totalMinerals, remainingMaterialsList.Count, saveList, questList,Galaxy.GetPlanetObject(Galaxy.planetToLoad).planetPosition,planetStatus);
7.                 //Get all old data
8.                 var planetList = Galaxy.GetLoadedPlanetObjectList();
9.                 var clusterPosition = Galaxy.GetClusterObject(Galaxy.clusterToLoad).clusterPosition;
10.                 var clusterList = Galaxy.GetLoadedClusterObjectList();
11.                 var galaxyNumber = Galaxy.GetGalaxyObject().galaxyNumber;
12.                 //Find and replace old object with new object in loaded list
13.                 planetList[planetList.FindIndex(ind => ind.planetNumber.Equals(Galaxy.planetToLoad))] = objectForSave;
14.                 cO = new ClusterObject(Galaxy.clusterToLoad, planetList, clusterPosition);
15.                 //Find and replace old object with new object in loaded list
16.                 clusterList[clusterList.FindIndex(ind => ind.clusterNumber.Equals(Galaxy.clusterToLoad))] = cO;
17.                 gO = new GalaxyObject(clusterList, galaxyNumber);
18.                 //Save object with replaced data
19.                 SaveManager.SaveGalaxy(this, galaxyNumber);
20.            
21.             }
22.  

Code (CSharp):

1.  public static void SaveGalaxy(TerrainManager gO, int clusterNumber)
2.         {
3.  
4.             string planetDirectory = StaticRuntime.GetApplicationPath() + "/save/galaxy/";
5.             if (!Directory.Exists(planetDirectory))
6.             {
7.  
8.                 Directory.CreateDirectory(planetDirectory);
9.             }
10.  
11.             string saveGameFileName = clusterNumber.ToString();
12.             string saveGamePath = StaticRuntime.GetApplicationPath() + "/save/galaxy/" + saveGameFileName + ".pgpro";
13.  
14.  
15.             if (File.Exists(saveGamePath))
16.             {
17.                 File.Delete(saveGamePath);
18.             }
19.             BinaryFormatter formatter = new BinaryFormatter();
20.             FileStream stream = new FileStream(saveGamePath, FileMode.Create);
21.             GameData data = new GameData(gO);
22.             formatter.Serialize(stream, data);
23.             stream.Close();    
24.         }

Code (CSharp):

1.  
2.     [System.Serializable]
3.     public class GameData
4.     {
5.  
6.      
7.  
8.  
9.      
10.         public GalaxyObject _gO;
11.  
12.         //SaveManager
13.        
14.  
15.         public GameData(TerrainManager terrainManager)
16.         {    
17.  
18.             _gO = terrainManager.gO;
19.         }        
20.  
21.     }
22.  

 

I will try this code, if I have problems then I will let you know here in this post. But Brackeys save and load system is similar like mine. But your code is not look like in the video. I think this is a scriptable object saving type which you gave in the code. I will try this code.

 

i edited my code first part so make it more readable

 

In my code i am using c# objects Galaxy is constructed from List of Cluster objects and Cluster object is constructed from list of Planet objects and Planet object is constructed from other objects... point is if i save i only save Galaxy object since it is in top of this hierarchy, all what is inside of this object is saved.

 

Can you share as package to download and take a look at whole scene?

 

I attached 3 relevant scripts but it will not work for you because my project have alot more in it but you can find relevant data in DynamicObjects,Terrain manager - save and load method , and save manager.Brackeys code is different because he demonstrate this by saving single variables as gamedata but your game data can be whole object not just one int.Princip is same

 

And I can not give my opinion without testing but I want to say that, There are 98415 objects on one terrain. Yes, I do not spawn these all objects at once. I added 81 chunks on one terrain. Each chunk gets 1215 objects. And these 81 chunks gets 1215 * 81 = 98415 objects on one terrain (TOTAL 98415 OBJECTS on one terrain). I spawn objects on each clicking and after spawned, the object send all mesh data into chunk and it object destroy itself after sending data to chunk and then the chunk draws with given mesh data like (vertices, triangles and uvs). I optimized game using chunks. But the question is, Does your code can store such 1215 objects data such as (vertices, triangles, uv, position and rotation) in a single file and add or remove data from there without laggy? Or is there a size limit in your code? Anyway I will check out your code to understand what is going on then I will tell you if I could not figure out something. THANKS FOR PAY AN ATTENTION.

 

Point is, you never store data like this.You save only default c# variables, this code doesnt care about anythig unity specific so you need to transfer these thingsto for example to int [] like i have in my code and then on load you create your object from this array and if value in array will be 3 grass will be spawned if 4 soil will be spawned etc..


In my code i gennerating about 500 planets in one click and each planet have 300-3000 game objects and and it is working fine.

Last thing it is good to dont use any unity specific code in your "generation" because then you can use threading and make this loading on background

 

can I not save with the huge vertices, triangles and uvs on different files? I do not save 100,000 objects data into a single file. I divide the objects and save these all in different 81 files. I think it will work.

Maybe you did not take a look at my script . I get all data like a variable int[] tringles, float[] uvs, floar[] vertices, float[] position, float[] rotation.

That's great. Even my chunk will not take high then 2000 objects. If you can store 3000 objects, that's great. But there is a main important thing for me is to add, remove or change data inside that existing file, I wanted to know how to do that. I hope that I can add, remove or change data inside existing file without any issue, and then it would be great.

 

For me only way is 1save = 1 file , other options are terrible slow you can pack all these data you talking about to 1 object and then save it it will be much faster then save 81 files

 

I do not save 81 files at once. My game is a different and because of this I can not use 1 file. My game is like A Minecraft, Rust game. I add wall, floor, stairs and etc on the terrain. Simply this is a construction system.

 

You can use this in any kind of game this is not game or unity specific this is just c# object oriented.

 

I will take it a look. It will be a little hard to understand your code. But I will try And I made a little optimization, Now I do not to save vertices, triangles or uvs. I will just save position, rotation, object type, texture id and that's all. Just I will save like this data. Now I don't save vertices triangles and uvs because of memory usage and saving file. Then I will load positions and will draw the mesh without saving vertices triangles and uvs. Now I can store more efficiently data