Save and Load Questions ( serialize and deserialize file )

I have tons of entities which are generated at runtime and to be loaded at runtime, so sub-scenes aren't useful for me afaik. Most entities are grouped into 'game-chunks' -- think of terrain tiles. How do I write to and load from a file per game-chunk's entities? These entities have varying archetypes. Other concerns of mine are: shared components, dynamic buffers, components with references to unsafe collections, and entity references. Additionally, I don't need to save every component per entity and would like to introduce some additional information purely for post load work.

I am aware of the ECS Serialize Utility, but it doesn't appear to have a section in the manual yet. Is it intended for my situation, or do I need to roll my own?

 

I am also interested in knowing more about this.

++interest;

 

Toying around with a concept for my own solution. This may be something I can build with Blobs, but I'll need to research them more.

To reiterate: the goal is to serialize 'anything' that can be associated with an entity: components, shared components, buffers, native collections (using pointers, system components, and unsafe variants). I want to give this system a set of entities and it should work regardless of archetype or content. I don't want to serialize every component directly and will need to handle 'provided' components like physics and rendering. Therefore I'm thinking each data type needs an associated utility class that handles serialization and deserialization.

File Header

Includes number of data streams and number of entities.
​

Data Streams

header information contains data stream id, entry size, entry count, and utility identifier.
Stream payload is right after header info​

Entities

The old identifier is preserved. Used to rebuild entity references just after load.
old entity id --> Entity value--> id (uint), version (uint)
Entity entry header is followed by N number of data references (data stream id, data stream index)​

File Structure:

File Header

[4 bytes] data streams count // uint
[4 bytes] entity stream count // uint​

Data Streams

[4 bytes] id // uint
[4 bytes] entry size // uint
[4 bytes] entry count // uint
[4 bytes] identifier length // uint, probably better than iterating until string terminator
[N bytes] identifier // serialization utility identifier, string?
[size * count bytes] payload​

Entity Stream

[8 bytes] old entity id // uint, uint
[4 bytes] data reference count // uint
[N * 8 bytes] data references // each reference is: stream id (uint), stream index (uint)​

Entity Stream Example [3 entities, 21 uints, 84 bytes]

12, 1, // old entity id, version
3, // reference count
0, 0 // reference to data stream id 0, index 0 (since two entities ref 0,0 it's likely a shared component)
1, 1 // reference to data stream id 1, index 1
2, 7 // reference to data stream id 2, index 7
22, 2, // old entity id, version
1, // reference count
2, 8 // reference to data stream id 0, index 0
281, 4, // old entity id, version
2, // reference count
0, 0 // reference to data stream id 0, index 0 (since two entities ref 0,0 it's likely a shared component)
2, 9 // reference to data stream id 0, index 0​

Serialization and Deserialization Processes

Serialization would involve building data streams and entity stream, merging into one byte stream, and writing to file or sending over the network. Everything that should be serialized needs to be associated with a utility class.

Deserialization would involve unpacking all the data streams, unpacking all the entities, then mapping relative data to ECS entities, components, buffers, etc. Each buffer / component needs to be associated with a utility class.

Each utility system needs a unique identifier (as of now it's a string, so I could use the full class name by default), and has the ability to serialize and deserialize data stream entries.

If these utilities can be used in jobs, I may be able to go wide? Each thread stores it's own data streams and entity stream, and they are stitched together into a single stream. Not 100% sure how the stitching should work, but perhaps each thread creates a block containing it's own data streams and entity stream. These blocks are appended into a single file, with header info to break apart the blocks quickly. Then each block can be fed into a worker thread until all blocks are complete?​

 

(Apart from bugs with BlobAssetOwner that Unity are aware of) what is missing from SerializeUtility?

 

SerializeUtility seems to only work with worlds as a whole. I assume I'm to create a new temp world, access the manager, duplicate the entities I want saved (unsure how this is to be done), utilize the SerializeUtility to write the temp world to a save file, and lastly tear down the temp world? I haven't worked with anything other than the default world yet and don't know how they should exchange data or handle events.

What all does SerializeUtility cover? Is there a chance to override how some data is serialized?