Save/Load and game performance - Automatic or manual serialization?

Hi Uniteers, we are currently investigating the possibility of saving and loading objects with complex structures from and to disc, but we are currently discussing the pros and cons of automatic and manual serialization. By manual serialization, I mean that you manually have to identify and select each field you wish to save, and create save/load methods for each object that refers to each of its fields, or automatically where you simply have to mark classes with [Serializable] and fields with [SerializableField] or [NonSerializable].

From what we know, the manual approach is much more performance friendly, but takes longer time to implement and is more error prone (we assume), while the automatic approach is easy to implement but is not as performance friendly.

We have only little experience in this field, so your experience or knowledge is much welcome. How does your game handle save/load, and why have you selected said method?

 

I would go automatic all the way, for almost all cases. As long as your reading files during 'loading' states then you may only see small performance hits. Also you can reduce the performance hits by saving the data into a different formats, JSON, Binary etc...

I tend to create a data class or struct containing only the data I want to serialise then serialise the entire class automatically so I dont need to tag fields at all. One up side is once serialised I can pass the class/struct straight in as initialisation data for instances.

 

Interesting, I didn't even know that. Is it because the serialization process is different between the two? In what situation should you pick each?

 

Depends on complexity and requirements to end-data size.
If you need to push performance to the edge (dynamically loading serialized data or such) - do it manually.
If you need to minimize size (storing huge amounts of data) - do it manually.
If doing it manually, I suggest reading http://www.codeproject.com/Articles/14164/A-Fast-Serialization-Technique and using serialization anyway. This way you can mix auto in non-crucial parts and manual in crucial parts.
Otherwise, leave it to auto. In most cases it's just 2-10 times speed/size difference and that doesn't decide anything during loading screen. Development speed is usually more important.

Yes, it's different. But both JSON and XML come with huge overhead of working with strings (they are human-readable though) while binary doesn't (and thus size of save data is much smaller and speed of serialization is faster). If you expect user to easily modify save or expect to debug something inside saves, use JSON, otherwise Binary (XML is worse at human readability than JSON, imho, but it's matter of preference). It's very easy to change one into another if using C# serialization - you just use different formatter, and that's it.

P.S. [SerializableField] attribute is for Unity serialization; I'm yet to see a way to use it runtime to save/load data.

 

Perhaps write a custom attribute to specify that a field should not be saved (e.g., "[NonSaved]", similar to [NonSerialized]). Then your automatic serializer can skip these fields. This makes everything automatic by default, but with the ability to optimize manually.

 

BinaryFormatter already uses [NonSerialized] attribute. Or if you create your own formatter/serialization, you can check fields for that exact [NonSerialized] attribute, why create a new one?

 

You might want to serialize a lot of fields normally (for example, in scene files) but only include a small subset in saved games.

 

Yes, and use [NonSerialized] attribute on them. Why create some new mysterious "[NonSaved]" when you can just place [NonSerialized]? It will work with default serializations as-is (BinaryFormatter, Unity serialization, JsonUtility, etc). And you can make your custom serialization check for that attribute (however, if you need to do so, that means you do what BinaryFormatter already does; so why not use it instead?). If using XML serialization (I don't think it's included in mono Unity uses though) there's [XmlIgnore] that does same for it.

 

But what if you want Unity to always serialize these fields except for in saved games? That was the motivation for my suggestion.

 

That would mean you'd need to implement new formatter yourself already; I don't see any other way to check for that attribute. And that means you'd use C# serialization which uses ISerializable while Unity's serialization ignores it. So I'd use manual serialization of that class by implementing ISerializable (if limiting my serialization) or ISerializationCallback (if limiting Unity serialization) to change either how I serialize or how Unity serializes this object.

This use-case is rare enough, so I think this way is more development-time effective than making a new formatter.

 

Good points! I was assuming a custom serializer/formatter. But you're right that it's probably more time-efficient to serialize everything, and only change the approach if it ever becomes an issue.

 

Unless you encounter a problem, it's probably safer for it to be automatic. Even if you do encounter problems, it's probably less time consuming to just wait til the end to implement it. Otherwise you have to maintain your custom serializer every step of the way.

 

What about mixing the approaches? For example, all the serializable objects may have a pair of Load/Save methods that usually just redirect to a "standard" automatic serializer. But if you need, you can inject some custom code there to boost the performance or to support old save formats.

 

I have all my game data in structs, arrays, and primitive types and created deep copy clone constructors for all of them.

Benefits:
- Compact and super fast
- Forces me to keep my data organized
- I like it
- Super easy to save/load games. JSON handles structs of primitive types very well. Starting a new game is just loading a new game memento.
- Portable
- Don't have to worry about reference types vs value types.

Cons:
- Takes more time to code vs throwing it into a serializer.
- Really requires time to organize game data properly.

I'm working on a turn based game so it's nice I can take snapshots after every turn, keep a history of every action made, rewind, etc. I used BinaryFormatter for a while, but it was unexpectedly slower and not portable to the Universal Windows Platform. I also found it nice to be able to edit JSON files by hand during development.

 

I have some articles published on the matter:

http://www.sitepoint.com/saving-data-between-scenes-in-unity/
http://www.sitepoint.com/saving-and-loading-player-game-data-in-unity/
http://www.sitepoint.com/mastering-save-and-load-functionality-in-unity-5/

I have developed a very robust and reliable save/load system in my game.

The core of the system is just "serializables.cs" file which does nothing except specify serializable classes and structures. It's basically a "tree hierarchy" of classes, with the root being:

Code (CSharp):

1. //-----Global save class------
2. [Serializable]
3. public class SavePackage
4. {
5.     public PlayerData Pdata;
6.     public SessionData Sdata;
7. }

This is followed up with PlayerData definition which is basically current position and orientation, accumulated XP, inventory items, saved status effects, yadda yadda yadda, and then Session data is:

Code (CSharp):

1. [Serializable]
2. public class SessionData
3. {
4.     public List<SceneData> sceneData;
5. }

The List is because we'll be having as many list items as there are registered Scenes in Build.

Each SceneData has:

Code (CSharp):

1. /// <summary>
2. /// Modifications to any given scene: Starting object IDs, Dropped objects such as potions, and Weapon drops.
3. /// </summary>
4. [Serializable]
5. public class SceneData
6. {
7.     public int SceneID;
8.     public List<float> PickedUpPlacableIDs;
9.     public List<float> DestroyedContainerIDs;
10.     public List<DroppableObject> AddedObjects;
11.     public List<MeleeWeaponDrop> MeleeWeaponDrops;
12.     public List<MagicWeaponDrop> MagicSpellDrops;
13.     public List<RangedWeaponDrop> RangedWeaponDrops;
14.     public List<ArmorItemDrop> ArmorItemDrops;
15. }

Within the core structure of the game I have a GlobalObject script, which is made Don'tDestroyOnLoad and is a Singleton structure (meaning it has one public static Instance).

At the Initialization of any given scene, the object keeps track of whether the scene is started for a first time (for example by using New Game or actually transitioning into a new level), or is it Loaded (meaning we've been there before and we need to load modified saved data).

It also initializes fresh SessionData structures as needed, serializes them down to disk or reads from them (depending on instructions from the player Pause and Main menus which offer save/load functionality).

Then, there is a LevelMaster object which reads from GlobalControl if the scene is loaded, and then goes through the list of loaded SceneData objects (quoted above) and does specific instantiation for each, for example this snippet:

Code (CSharp):

1. // ....
2. ///Now, weapons part:
3.                 for (int i = 0; i < GlobalControl.Instance.sessionData.sceneData[sceneIDX].MeleeWeaponDrops.Count; i++)
4.                 {
5.                     WeaponLootManager.GenerateFromSavedMeleeWeapon(GlobalControl.Instance.sessionData.sceneData[sceneIDX].MeleeWeaponDrops[i]);
6.                 }
7.                 GlobalControl.Instance.sessionData.sceneData[sceneIDX].MeleeWeaponDrops.Clear();
8.  
9.             for (int i = 0; i < GlobalControl.Instance.sessionData.sceneData[sceneIDX].MagicSpellDrops.Count; i++)
10.             {
11.                 WeaponLootManager.GenerateFromSavedMagicSpell(GlobalControl.Instance.sessionData.sceneData[sceneIDX].MagicSpellDrops[i]);
12.             }
13.             GlobalControl.Instance.sessionData.sceneData[sceneIDX].MagicSpellDrops.Clear();
14. //....

As you can see, after we've instantiated all the needed saved objects, we need to clear the list.

As for the actual Save itself, that's reasonably well explain in the articles posted above, but the core of it is a delegated event system declared in GlobalControl:

Code (CSharp):

1.  
2.     public delegate void SaveHandler(object sender, EventArgs e);
3.     public event SaveHandler OnSave;

And then, any item that wishes to save/load itself, must subscribe to this event, and save it's relevant data.

For example all my droppable objects like weapons and spells (so far) extend from Droppable class, which implements IInteractable interface. Interface mandates several functions such as LookAt and Interact, while Droppable subscribes to event, and un-subscribes from event at Destroy, Disable and Interact.

For example here's a subscribed function to OnSave event from a melee weapon drop object:

Code (CSharp):

1. public override void OnSave(object sender, EventArgs e)
2.     {
3.      
4.             //Make new save item structure:
5.             MeleeWeaponDrop drop = new MeleeWeaponDrop();
6.             drop.Weapon = Weapon;
7.  
8.             //Where is the saved item?
9.             drop.Position = gameObject.transform.position;
10.             drop.Rotation = gameObject.transform.rotation.eulerAngles;
11.  
12.             //Write structure into session data. Global control takes on from here.
13.             GlobalControl.Instance.sessionData.sceneData[SceneManager.GetActiveScene().buildIndex].MeleeWeaponDrops.Add(drop);
14.     }

If you want to see a preview of what we got so far I'd be glad to share with you all. Let me know, we'll prepare a google drive link with our current build. Just be warned it's QUITE early in development (half the swords are missing textures, all spells except Fire Projectile set are basically exact same thing, animations are VERY crude because it's my first time at making game animations etc). But mechanically and structurally, it works quite well and I'm very happy with progress so far.

 

For my case I choose to always use JSON for any time I am going to be manually editing the data at some stage as others have said its easy to edit, more so than XML but thats my opinion. So for all game settings and configuration files they are readable and also automatically serialised via a single generic method. The only time I have used binary files for a terrain system handling well over a million objects positions and rotations contained in thousands of files etc... at runtime I noticed a very large performance increase in the read times. JSON was roughly 8 - 10 seconds for a few thousand JSON files. The same files in binary performed the same task in about 1/3 of a second. So the performance increase was very welcomed considering I'm reading them at runtime.

Also just to clarify when I talked about automatic serialisation I was refering to using a single generic method for saving and loading data, not tagging classes and variables, then using a dll save/load the data. Its all down to personal preference but I prefer being able to make a data class containing only the values/lists/arrays/classes i want to save then having a single generic method save and load them. Its now just become a Util method which I have used in roughly 10 projects over the last 5 years and so far works on all formats.

 

Thanks for the read guys, I have reflected upon your advice and played around a bit with serilization. Using the BinaryStreamer and serializing classes using reflection appears to be quiet expensive so I have instead decided to implement the interface ISerialization into our classes, which is much more performance friendly. Our currenty system is by far fully developed so I am not sure what sort of caveats I will encounter as we progress.

 

Hi, that sounds really interesting! Could you please go a bit deeper in explaining how the code of such a system would look like?
And when you say "it's nice to be able to edit JSON during development" does it mean that there are reliable ways to decrypt and encrypt JSON data later on after publishing the game?