Question Recursive (De)Serialization

I'm working on a project that involves de-serialization, but I've run into yet another issue. The below is an abridged version of the class I am attempting to de-serialize:

Code (CSharp):

1. public class AreaData
2. {
3. public List<ChildDataOrigin> childAreas = new List<ChildDataOrigin>();
4.     private List<AreaChildData> childAreasData = new List<AreaChildData>();
5. }

The issue is that AreaChildData contains a reference to another AreaData:

Code (CSharp):

1. public class AreaChildData
2. {
3.     public string name = "";
4.     public AreaData areaType = null; //This is the issue
5.     public bool optional = false;
6.     public bool unlimited = false;
7.     public int maxNumber = 1;
8.     public int mandatoryNumber = 1;
9.  
10.     public int numberPlaced = 0;
11.     //How to integrate this with the actual Area object
12.  
13.     public AreaChildData(string inputName)
14.     {
15.         name = inputName;
16.     }
17.  
18.     public AreaChildData(string inputName, bool inputOptional, bool inputUnlimited, int inputMaxNumber, int inputMandatoryNumber)
19.     {
20.         name = inputName;
21.         optional = inputOptional;
22.         unlimited = inputUnlimited;
23.         maxNumber = inputMaxNumber;
24.         mandatoryNumber = inputMandatoryNumber;
25.     }
26. }

This gets the serializer in a giant loop, even if the actual depth of the tree is only one layer. I thought using an object (see below) that doesn't include this as the public (i.e. writable) list and making childAreasData private would fix this, but it did not.

Code (CSharp):

1. [System.Serializable]
2. public class ChildDataOrigin
3. {
4.     public string name = "Default Child Data";
5.     public bool optional = false;
6.     public bool unlimited = false;
7.     public int maxNumber = 1; //Does it make sense for this to be a hard cap?
8.     public int mandatoryNumber = 1;
9.  
10.     public AreaChildData CreateChildData()
11.     {
12.         return new AreaChildData(name, optional, unlimited, maxNumber, mandatoryNumber);
13.     }
14. }

I do have some workarounds to this (creating a separate "real" AreaData class that doesn't touch serialization with a ten foot pole, or reworking AreaChildData to not include an AreaData), but is it possible to build this "tree" structure with JSON files?

 

Your class structure is recursive. Unity's default serialisation is by values, and does not support recursive serialisation.

If you want to serialise by reference so that cyclic references are supported, you need to use the SerializeReference attribute. Though I believe that data being deserialised needs to be structured in the same way Unity manages SerializeReference fields. Best to refer to the docs, and play around with it to see how the data is structured.

 

The simplest solution is to not serialize the field itself but instead serialize a identifier as a placeholder for the field. When the code is running it can then do a lookup and cache the right areadata when its requested. You can use ISerializationCallbackReceiver to perform operations right before the class is to be serialized and right after the class is deserialized, either automatically by unity or manually such as via JsonUtility.

Code (CSharp):

1.  
2. public class AreaData : UnityEngine.ISerializationCallbackReceiver
3. {
4.     private static Dictionary<string, AreaData> areas = new Dictionary<string, AreaData>();
5.     public static IReadOnlyDictionary<string, AreaData> Areas => areas;
6.  
7.     [SerializeField] public string areaId;
8.     private List<AreaChildData> childAreasData = new List<AreaChildData>();
9.  
10.  
11.  
12.     public void OnBeforeSerialize() {}
13.     public void OnAfterDeserialize()
14.     {
15.         if (!string.IsNullOrEmpty(areaId))
16.             areas[areaId] = this;
17.     }
18. }
19. [System.Serializable]
20. public class AreaChildData : UnityEngine.ISerializationCallbackReceiver
21. {
22.     [SerializeField] private string areaTypeId;
23.     [NonSerialized] private AreaData areaType = null;
24.  
25.     public AreaData AreaType
26.     {
27.         get
28.         {
29.             if(areaType == null && !string.IsNullOrEmpty(areaTypeId))
30.             {
31.                 AreaData.Areas.TryGetValue(areaTypeId, out areaType);
32.             }
33.             return areaType;
34.         }
35.     }
36.  
37.  
38.     public void OnBeforeSerialize()
39.     {
40.         areaTypeId = areaType != null ? areaType.areaId : null;
41.     }
42.  
43.     public void OnAfterDeserialize()
44.     {
45.         areaType = null;
46.     }
47. }

 

Right. I'd like to emphasise that the "NonSerialized" attribute on the areaType field is important. Just making the field private is not enough. The Unity editor may serialize / deserialize even private variables when it hot-reloaded the domain. This attribute specifically prevents any serialization which is important to avoid those infinite recursion loop.

 

If you can, make the AreaData class ScriptableObject. In this case an object reference will be serialized instead of class instance and this will solve your problem.

 

No serialiser is capable of serialising out references to Unity objects and deserialise them by default, as there is literally no API available from Unity to make this possible. Not even Unity's JSONUtility is capable of doing this; you get the InstanceID instead, which is not reliable between sessions.

Some serialisers like have ways to work around this, such as Odin's external reference resolvers, but you still need to implement some kind of database system to restore these references.

Also, besides, if OP is likely trying to serialise some kind of state, then serialising a reference an object that won't maintain its state between sessions probably doesn't help them.

Honestly OP needs to move away from Unity's basic serialiser and use a proper one, such as Newtonsoft.JSON or the Odin serialiser.

 

... or just Unity's own Serialization package. I've come to love it! Finally, I can fully manage all the data I want to serialize and how.

Nothing is stopping me from serializing a list (int) type as (ushort) if I know the list will never have more than 65k entries. And those two bytes saved in the serialized data can accumulate to significant numbers. Just look at how bloated a Unity scene file becomes when you have 50k Game Objects in the scene (I bet it's getting close to 1 GB). Instead, I can save the data that I need to reconstruct those game objects and get away with maybe 10 MB of serialized data.

Make no mistake though - I love this package ONLY for its binary serialization. If you want to do JSON with that package expect to enter HELL and never return. That Json serialization API is mind-boggingly crappy. After two days trying to serialize a simple NativeList in a struct I gave up. The issues you run into do not make any sense. You're better off constructing the JSON syntax yourself. Which seems pretty close to what that API does, except it won't allow you to do things that seem logical.

 

I haven't had a chance to play with the package yet but it does seem useful if you want serious amounts of control over your serialisation.

I mostly use Odin because I can chuck any blob of data at it and it will serialise it generally without fail, including tons of types that Unity doesn't support natively.

Mind you my save data for my projects is usually all Unity serialisable anyway. But the Odin serialiser, I find, requires the least work to get it running, and then I can just layer anything extra on top easily. It's got plenty of API tools to extend of change its behaviour, such as the aforementioned external reference resolvers, alongside serialisation policies, hell you can even write your own serialisation formatters to be able to serialise types it doesn't support.

And you can get low level if you want, such as this method I wrote with the help with the lead dev Tor, when I needed to, uh... serialise something while serialising something and convert the bytes to JSON (basically I have an interface that, while serialising, would convert Unity objects into surrogate data):

Code (CSharp):

1. public static string LizardSerializeValue<TR>(object valueToSerialize) where TR : class, IExternalStringReferenceResolver, new()
2. {
3.     if (valueToSerialize is Object obj)
4.     {
5.         Debug.LogWarning("Root serialisation object is a Unity Object. Deserialisation of Unity objects is not currently supported.", obj);
6.     }
7.  
8.     string serializedString;
9.  
10.     using (var context = Cache<SerializationContext>.Claim())
11.     using (var stream = new MemoryStream())
12.     using (var writerCache = Cache<JsonDataWriter>.Claim())
13.     using (var resolver = Cache<TR>.Claim())
14.     {
15.         var writer = writerCache.Value;
16.  
17.         writer.Context = context;
18.         writer.Stream = stream;
19.         writer.PrepareNewSerializationSession();
20.         writer.FormatAsReadable = false;
21.         writer.EnableTypeOptimization = true;
22.         writer.Context.StringReferenceResolver = resolver.Value;
23.  
24.         var serializer = Serializer.GetForValue(valueToSerialize);
25.         serializer.WriteValueWeak(valueToSerialize, writer);
26.  
27.         writer.FlushToStream();
28.         byte[] bytes = stream.ToArray();
29.      
30.         serializedString = System.Text.Encoding.UTF8.GetString(bytes, 0, bytes.Length);
31.     }
32.  
33.     return serializedString;
34. }

 

Unity default built in serializer does exactly that. Just create an asset - the instance of SerializedObject denscendant - reference it in your prefab or scene and it will be deserealized automagically along with prefab or scene loading.

Of course you can reference as many assests of the same type as you want and create nestetd hierarchies of objects. Consinder this simple example:

Code (CSharp):

1. [CreateAssetMenu]
2. public class MyBinaryTreeNode : ScriptableObject
3. {
4.     public int value;
5.     public MyBinaryTreeNode left;
6.     public MyBinaryTreeNode right;
7. }

 

ok... now try serializing that to custom save file or text file. and then deserialize back out. Try doing this in a build. It won't work. The point of this thread is about save/loading data to a custom file that sits outside unity's serialization environment, not using in-project assets for serialization

Unity is fine with loading and saving its objects inside its own environment, but the moment you try to directly integrate its objects with a savefile or even a custom serialization surrogate, unity will not play ball. This is why you don't save unity objects themselves to savefiles, you use identifiers in surrogates.

 

But the OP post does not contains a single word about runtime serialization, just loading.
For runtime serialization there's SerializeReference attribute mentioned above.

 

Not in this thread no, but this topic is an extension of his earlier thread. In it you can see that AreaData is already a ScriptableObject. He's trying to perform procedural loading into these assets using the custom JSON data he's writing.

 

I removed the scriptableobject inheritance from AreaData, at your suggestion.

I wasn't able to get this to work. It didn't work on either the AreaData field in ChildAreaData or the list in AreaData.

Thanks for the help everyone. I'm reading over ISerializationCallbackReceiver's docs now.

 

Here's my data importer. Maybe I screwed something up here?

Code (CSharp):

1. public class DataController : MonoBehaviour
2. {
3.     private static DataController _instance;
4.     public static DataController instance
5.     {
6.         get
7.         {
8.             if(!_instance)
9.             {
10.                 _instance = new GameObject().AddComponent<DataController>();
11.                 // name it for easy recognition
12.                 _instance.name = _instance.GetType().ToString();
13.                 // mark root as DontDestroyOnLoad();
14.                 DontDestroyOnLoad(_instance.gameObject);
15.             }
16.             return _instance;
17.         }
18.     }
19.  
20.     public List<AreaData> allAreaData;
21.  
22.     private void Awake()
23.     {
24.         Debug.Log("DataController is awake.");
25.         allAreaData = new List<AreaData>();
26.         //ImportJSON();
27.         ImportJSONTest();
28.         //ImportJSONTestTwo();
29.         foreach(AreaData data in allAreaData)
30.         {
31.             data.SetSubAreas(); //This requires less code changes.
32.         }
33.         Debug.Log(allAreaData.Count);
34.         Logging();
35.     }
36. void ImportJSONTest()
37.     {
38.         Object[] areaObjects = Resources.LoadAll("Test", typeof(TextAsset));
39.         foreach(Object data in areaObjects)
40.         {
41.             AreaData generated = JsonUtility.FromJson<AreaData>(data.ToString());
42.             generated.FinalizePopulate();
43.             allAreaData.Add(generated);
44.         }
45.     }
46. }