Binaryformatter Serializing/deserializing Too Slow. Need Ideas.

Hi I have a huge procedurally generated 2d voxel game where the block struct contains 4 short variables (tile id, height, etc) and there is a big 1d array of tiles that represents the map grid of size 1024 * 1024.

The issue I'm having is loading the map from file. It takes a good 30 seconds to load my array of tile structs. I'm wondering how other games like Terraria and Starbound do it so quickly. Shrinking my map to 256 * 256 does make loading a lot faster but at a cost of map size.

Should I be fragmenting my map into smaller files? Using a different way to serialize? So lost...

 

Maybe you should rebuild your system with a something else than a array of 1024*1024 items. You could also use images as height maps and another image for values. Depends on your needed values, but you can do a lot with images, colors and alphachannels as well as tileids that are bound to their position instead of saving the name itself.

 

Use Ceras:
https://github.com/rikimaru0345/Ceras

 

If I understand your problem, I'd use BitConverter instead of BinaryForrmatter. BinaryFormatter includes a lot of metadata which you shouldn't need, since you already know the data represents this struct of yours. The metadata wastes space and wastes time processing it.

 

Typically you would use a Flyweight pattern, though it sounds like thats what you are already doing. in either case you'll want to minimize/simplify the data thats getting serialized for storage, and only have it be the proper data type when its in use.

You could try and read/write all that data as a 1024x1024 texture, with each pixel's RGBA values relating to the fields in your block struct. For example, the R channel can neatly fit all possible ids if you have at most 256 unique block types. you can even do bit shifting if one channel needs less while the next channel needs more (example, if you only have 64 block types but need 1024 height values, then blockId could use the 1st 6 bits of R while height uses last 2 bits of R and all 8 bits of G).

Also asynchronously streaming in blocks with priority given to blocks nearest to the player would help them get into the game faster

 

Interesting... Aren't textures still effectively a big array of structs too? Or is it treated differently internally that makes it serialize faster? I do see what you're saying - the color channels save precious bits.. But what makes it better than say... converting my current struct members into chars and whatnot to save bits?

What is your thought on saving the seed instead of the entire map and just having a dictionary of "changes" made to the tiles? Would the overhead be too great?