Question Working with large sizes of dynamic data

I have a need to operate with large data dynamically, this data is often unnecessary and therefore I don’t want to load it into dictionaries and keep it in RAM, the question is how best to organize work with such data?
to save in files to count binary? do databases? or how?

The data must change during the game and the changes are saved, the data can reach several gigabytes...

 

„Data“ and how you use it and what you use it for play a huge role regarding what a good strategy is for your use case.

It could even be entirely unnecessary to create and store this data to begin with just by tackling the problem statement from a different perspective.

But there is no way we can point that out unless you give more concrete info as to what that data is and why you need it and so on.

 

well, I will describe in detail why I need it - for some reason I need to save the vertices of the terrain, it will dynamically change during the game, let's say on one terrain 128 by 128 I have 64 vertices per division, this is already more than 16k records (each store 64 float), also for each cell records data on the possibility of passage, this is for finding a path, another 16k records.
And now let's take into account that there are 100 such terrains, it is clear that there is a lot of data...
but this data is not always needed, it is loaded at the start, the terrain is formed, then a couple of sections are often used, but sometimes they will be required for changes.
something like this...

 

You are looking at a streaming solution in that case, as long as you keep the data around a given position in memory and swap out everything that‘s too far away - essentially what Minecraft does. You can make that work with a flat filesystem, no database needed as long as the filenames encode the world position or terrain index.

Though first I would definitely check whether you can simply work with the existing data and generate everything at runtime. Rather than load a terrain chunk data from a file, you could generate the necessary data from the terrain the moment the terrain is loaded. This may actually be faster than reading data from disk, especially if you can write bursted jobs for that.

 

at first I tried it in the editor and the data was stored in TerrainData after I changed it during the play, but after compiling it in the finished product, the data after closing the application is not saved in the same way, so a data saving mechanism is needed. I was thinking about saving "flat" data, but so far I have not found a scheme according to which all this needs to be done, how to make it work quickly ...