Question Importing lists of custom objects from JSON

I'm trying to store my game data in JSON files, but I can't get my loader to properly load it. I've been looking at examples, but I can't see anything wrong with my code.

Code (CSharp):

1. void ImportJSON()
2.     {
3.         Object[] areaObjects = Resources.LoadAll("Areas",typeof(TextAsset)); //This loading works, I get all the objects as files
4.         foreach(Object data in areaObjects)
5.         {
6.             string JSONstring = data.ToString();
7.             AreaListObject newAreaData = JsonUtility.FromJson<AreaListObject>(JSONstring); //This appears to be were the problem is. The AreaListObjects are being created, but the lists are either empty or nonfunctional.
8.             allAreaData.AddRange(newAreaData.areaDataList);
9.         }
10.     }

The object I'm trying to create is as follows:

Code (CSharp):

1. [System.Serializable]
2. public class AreaListObject
3. {
4.     public List<AreaData> areaDataList;
5. }

It's a simple object just meant to be used as a target for JSON reading.
Here's a sample of the JSON as well:

Code (csharp):

1.  
2. {
3.     "AreaDataList":
4.     [
5.         {
6.             "name":"Kitchen",
7.             "layerString":"Room",
8.             "maxLength": 5,
9.             "maxWidth": 3,
10.             "minLength" : 4,
11.             "minWidth" : 2,
12.             "hasWallMaterial" : true,
13.             "hasFloorMaterial" : true,
14.             "floorMaterialTypes":["tile","linoleum"],
15.             "wallMaterialTypes":["plaster","wallpaper"]
16.         },
17.         {
18.             "name":"Hallway",
19.             "layerString" :"Room",
20.             "hasWallMaterial" : true,
21.             "hasFloorMaterial" : true,
22.             "floorMaterialTypes":["wood","carpet"],
23.             "wallMaterialTypes":["wood","plaster"]
24.  
25.         },
26.         {
27.             "name":"DiningRoom",
28.             "layerString":"Room",
29.             "maxLength" : 6,
30.             "maxWidth":5,
31.             "minLength": 4,
32.             "minWidth" : 4,
33.             "hasWallMaterial" : true,
34.             "hasFloorMaterial" : true,
35.             "floorMaterialTypes":["wood","carpet"],
36.             "wallMaterialTypes":["wood","plaster"]
37.  
38.         },
39.         {
40.             "name":"LivingRoom",
41.             "maxLength" : 5,
42.             "maxWidth":5,
43.             "minLength": 4,
44.             "minWidth" : 4,
45.             "hasWallMaterial" : true,
46.             "hasFloorMaterial" : true,
47.             "floorMaterialTypes":["wood","carpet"],
48.             "wallMaterialTypes":["plaster"]
49.  
50.         },
51.         {
52.             "name":"Storage",
53.             "maxLength" : 5,
54.             "maxWidth":5,
55.             "minLength": 3,
56.             "minWidth" : 3,
57.             "hasWallMaterial" : true,
58.             "hasFloorMaterial" : true,
59.             "floorMaterialTypes":["wood"],
60.             "wallMaterialTypes":["plaster"]
61.  
62.         },
63.         {
64.             "name":"Bathroom",
65.             "maxLength" : 3,
66.             "maxWidth": 2,
67.             "minLength": 2,
68.             "minWidth" : 1,
69.             "hasWallMaterial" : true,
70.             "hasFloorMaterial" : true,
71.             "floorMaterialTypes":["tile"],
72.             "wallMaterialTypes":["tile","wallpaper"]
73.         }
74.     ]
75. }
76.  

 

Load/Save steps:

https://forum.unity.com/threads/save-system-questions.930366/#post-6087384

An excellent discussion of loading/saving in Unity3D by Xarbrough:

https://forum.unity.com/threads/save-system.1232301/#post-7872586

Loading/Saving ScriptableObjects by a proxy identifier such as name:

https://forum.unity.com/threads/use...lds-in-editor-and-build.1327059/#post-8394573

When loading, you can never re-create a MonoBehaviour or ScriptableObject instance directly from JSON. The reason is they are hybrid C# and native engine objects, and when the JSON package calls

new

to make one, it cannot make the native engine portion of the object.

Instead you must first create the MonoBehaviour using AddComponent<T>() on a GameObject instance, or use ScriptableObject.CreateInstance<T>() to make your SO, then use the appropriate JSON "populate object" call to fill in its public fields.

If you want to use PlayerPrefs to save your game, it's always better to use a JSON-based wrapper such as this one I forked from a fellow named Brett M Johnson on github:

https://gist.github.com/kurtdekker/7db0500da01c3eb2a7ac8040198ce7f6

Do not use the binary formatter/serializer: it is insecure, it cannot be made secure, and it makes debugging very difficult, plus it actually will NOT prevent people from modifying your save data on their computers.

https://docs.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide

Problems with Unity "tiny lite" built-in JSON:

In general I highly suggest staying away from Unity's JSON "tiny lite" package. It's really not very capable at all and will silently fail on very common data structures, such as bare arrays, tuples, Dictionaries and Hashes and ALL properties.

Instead grab Newtonsoft JSON .NET off the asset store for free, or else install it from the Unity Package Manager (Window -> Package Manager).

https://assetstore.unity.com/packages/tools/input-management/json-net-for-unity-11347

Also, always be sure to leverage sites like:

https://jsonlint.com
https://json2csharp.com
https://csharp2json.io

 

Is there polymorphism in that data? One of the collection elements has different properties. If it does, this won't gel with the way Unity handles polymorphism with SerializeReference. If it doesn't, your data is faulty.

 

Maybe your issue is

AreaDataList

vs

areaDataList

?

 

That was part of it at least. Thanks.

 

Great, but that means you still have "issues"?

This is not a very detailed description of your issue. You have to do the debugging since it's your project, your data and your code. If you still need help, you have to be more specific what issue you actually have. We don't even know what your "AreaData" class looks like.

 

I think it's almost certainly the

Do you know of any example code for this?

 

There's no one-size fits all solution.

Use

ScriptableObject.CreateInstance<T>()

or else you must make a GameObject and then

.AddComponent<T>()

the correct MonoBehaviour to it.

Only then can you pass that correctly-created UnityEngine.Object-derived object onto the JSON lib to populate.

GENERALLY, it is always a less-than-optimal way to do things if your savegame plan involves pulling data back from the Unity Engine. Do games do this? YES. Should you? Probably not.

Instead, YOU should track your truth source of data within your game's data model and serialize / deserialize that.

Ideally Unity is used only for presentation, with things like physics fed back to your data model as necessary.

As I linked above, look at Xarborough's discussion.

 

I'm not trying to save games. This is data for use in a procedural generator. I don't know if that makes a difference, but I'm not interested in encoding anything into JSON at the moment, just using JSON files I've authored.

 

Well as mentioned all these collection elements have different amounts of properties, and Unity's basic serialisation utility does not support this. You need to make sure all the JSON elements all express all the properties in this data type.

 

Someone should tell the people who write the Unity manual then, because it says any unfilled field uses the default.

Edit: I'm pretty sure my issue is that I was trying to go to a scriptable object with FromJSON instead of FromJSONOverwrite.

 

You're pretty sure? Is "AreaData" a scriptable object or not? You still haven't shown that class, so how should we know. If it is a ScriptableObject then of course FromJSON would not work at all. However the question is WHY it actually is a ScriptableObject? If it isn't a ScriptableObject, is it marked as Serializable like your "AreaListObject"?

As I said, you have to share relevant information. Guessing doesn't really help

 

That's actually correct. I believe Spiney was simply pointing out that if you created varying type objects in JSON, Unity is not capable of discerning from an array of a common super type that "Oh this is an ABC object and this is a XYZ object"... so they're all (I believe) gonna come out as the same type, many with

default(T)

values.

Aaaaaaaanyway... I finally ripped out a quick demo of this stuff...



I think you figured it out already, but just in case, see package, with this code:

Code (csharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.UI;
5. using UnityEngine.Networking;
6.  
7. // @kurtdekker
8. // The point of this is to show how to produce real-live ScriptableObject
9. // classes and then populate them using the Unity JSON Utility.
10. //
11. // NOTE: JSONUtility will not handle a lot of common things such as Dictionaries!!
12. //
13. // Use NewtonSoft JSON .NET if you need that.
14.  
15. public class JSONUnityTest : MonoBehaviour
16. {
17.     [Header( "Drag an Asset instance in here.")]
18.     public BagOData PreMade;
19.  
20.     [Header("Leave this blank, a transient one will be made.")]
21.     [Header( "Once made you can doubleclick to browse it while running.")]
22.     [Header( "Object is gone when you press stop.")]
23.     public BagOData Generated;
24.  
25.     [Header("Deglugging output:")]
26.     public Text JSONOutput;
27.  
28.     void Start ()
29.     {
30.         string s = JsonUtility.ToJson( PreMade, prettyPrint: true);
31.  
32.         Debug.Log( s);
33.         JSONOutput.text = s;
34.  
35.         // you must fabricate this scriptable object
36.         Generated = ScriptableObject.CreateInstance<BagOData>();
37.  
38.         Generated.name = "MyNameIsGenerated";
39.  
40.         // then populate its data
41.         JsonUtility.FromJsonOverwrite( s, Generated);
42.     }
43. }

 

Alright, I've gotten to the point where the AreaData objects are being created properly, but they aren't being populated/overwritten.

Here's the new code for importing:

Code (CSharp):

1. void ImportJSON()
2.     {
3.         Object[] areaObjects = Resources.LoadAll("Areas",typeof(TextAsset));
4.         //Debug.Log($"A: {areaObjects.GetLength(0)}");
5.         foreach(Object data in areaObjects)
6.         {
7.             string JSONstring = data.ToString();
8.             AreaListObject newAreaData = JsonUtility.FromJson<AreaListObject>(JSONstring);
9.             foreach(string areaDataJSON in newAreaData.areaDataList)
10.             {
11.                 AreaData generated = ScriptableObject.CreateInstance<AreaData>();
12.                 JsonUtility.FromJsonOverwrite(areaDataJSON, generated);
13.                 generated.FinalizePopulate();
14.                 allAreaData.Add(generated);
15.             }
16.  
17.         }
18.     }

And for AreaListObject:

Code (CSharp):

1. [System.Serializable]
2. public class AreaListObject
3. {
4.     public List<string> areaDataList;
5. }

And AreaData in the flesh (abridged of some non-JSON related functions):

Code (CSharp):

1. public new string name = "defaultArea";
2.     public string layerString = ""; //This turns into the layer through parsing in the constructor
3.     public AreaLayer layer = 0;
4.     public int depth = -1;
5.  
6.     public int maxLength = 2;
7.     public int maxWidth = 2;
8.     public int minLength = 0; //Set to max by default (See constructor)
9.     public int minWidth = 0;
10.  
11.     //Placement Behavior
12.     //TODO: Figure out constraints-Should these reference by string or be coded in directly?
13.     //Heck, should they be here or on children?
14.     public List<string> criteria = new List<string>();
15.     public List<string> criteriaExceptions = new List<string>();
16.     public List<AreaModifier> modifiers = new List<AreaModifier>();
17.  
18.     //Sub-Areas
19.     public TransitBehavior transitType = TransitBehavior.NA; //TODO: Maybe make this variable? Similar to walls
20.     public string transitAreaName = null; //This will need to be a string, to find the target
21.  
22.     public bool fullCoverage = false;
23.     public float coveragePercentage = 0;
24.  
25.     public AreaData transitArea = null;
26.     public List<AreaChildData> childAreas = new List<AreaChildData>();
27.  
28.     //Material Behavior TODO: Convert these to JSON (They don't need to be converted right now though.)
29.     public bool hasWallMaterial = false;
30.     public List<String> wallMaterialTypes = new List<string>();
31.     public List<MaterialType> possibleWalls = new List<MaterialType>();
32.     public List<MaterialModifier> requiredWallTags = new List<MaterialModifier>();
33.     public List<MaterialModifier> forbiddenWallTags = new List<MaterialModifier>();
34.  
35.     public bool hasFloorMaterial = false;
36.     public List<string> floorMaterialTypes = new List<string>();
37.     public List<MaterialType> possibleFloors = new List<MaterialType>();
38.     public List<MaterialModifier> requiredFloorTags = new List<MaterialModifier>();
39.     public List<MaterialModifier> forbiddenFloorTags = new List<MaterialModifier>();
40.  
41.  
42.     public Color visualizationColor = Color.gray; //TODO: Figure out how to turn JSON into Color
43.  
44.  
45.     //Building Parameters
46.     public int aboveFloors = 1; //Variable?
47.     public int bFloors = 0;
48.     public int minArea = -1; //If the design of a building leads to having fewer cells than this, the building will be scaled up.
49.     public int maxArea = -1;
50.  
51.     public List<VerticalChildData> verticalPool = new List<VerticalChildData>();
52.  
53.     public bool hasExteriorMaterial = false;
54.     public List<string> exteriorMaterialTypes = new List<string>();
55.     public List<MaterialType> possibleExteriors;
56.     public List<MaterialModifier> requiredExteriorTags;
57.     public List<MaterialModifier> forbiddenExteriorTags;
58.  
59.     public List<string> footprintNames = new List<string>();
60.     public List<FootprintType> possibleFootprints;
61.  
62.     //Vertical Parameters
63.     public int maxLevels;
64.  
65.     public void FinalizePopulate()
66.     {
67.         //Debug.Log("FinalizePopulate called.");
68.         if(minLength == 0)
69.         {
70.             minLength = maxLength;
71.         }
72.         if(minWidth == 0)
73.         {
74.             minWidth = maxWidth;
75.         }
76.         Enum.TryParse(layerString, true, out layer);
77.         depth = ((int)layer);
78.  
79.         if(footprintNames.Count > 0)
80.         {
81.             foreach(string footprintName in footprintNames)
82.             {
83.                 FootprintType type;
84.                 Enum.TryParse(footprintName, true, out type);
85.                 possibleFootprints.Add(type);
86.             }
87.         }
88.  
89.         //Material Type conversions
90.         if(hasFloorMaterial)
91.         {
92.             foreach(string typeName in floorMaterialTypes)
93.             {
94.  
95.                 MaterialType type;
96.                 bool gotType = Enum.TryParse(typeName, true, out type);
97.                 if(gotType)
98.                 {
99.                     possibleFloors.Add(type);
100.                 }
101.             }
102.         }
103.         if (hasWallMaterial)
104.         {
105.             foreach (string typeName in wallMaterialTypes)
106.             {
107.  
108.                 MaterialType type;
109.                 bool gotType = Enum.TryParse(typeName, true, out type);
110.                 if (gotType)
111.                 {
112.                     possibleWalls.Add(type);
113.                 }
114.             }
115.         }
116.         if (hasExteriorMaterial)
117.         {
118.             foreach (string typeName in exteriorMaterialTypes)
119.             {
120.  
121.                 MaterialType type;
122.                 bool gotType = Enum.TryParse(typeName, true, out type);
123.                 if (gotType)
124.                 {
125.                     possibleExteriors.Add(type);
126.                 }
127.             }
128.         }
129.     }
130.  
131.     public AreaLayer GetLayer()
132.     {
133.         return layer;
134.     }
135.  
136.     public void SetSubAreas()
137.     {
138.         if(transitAreaName != null)
139.         {
140.             transitArea = DataController.instance.SearchAreasByName(transitAreaName);
141.         }
142.         if(childAreas.Count > 0)
143.         {
144.             foreach (AreaChildData child in childAreas)
145.             {
146.                 child.areaType = DataController.instance.SearchAreasByName(child.name);
147.                 if (child.areaType == null)
148.                 {
149.                     childAreas.Remove(child);
150.                     Debug.Log($"Child Areas for {name}, could not find {child.name}.");
151.                 }
152.             }
153.  
154.         }
155.         if(verticalPool.Count > 0)
156.         {
157.             foreach(VerticalChildData child in verticalPool)
158.             {
159.                 child.areaType = DataController.instance.SearchAreasByName(child.name);
160.                 if (child.areaType == null)
161.                 {
162.                     verticalPool.Remove(child);
163.                     Debug.Log($"Vertical Pool for {name}, could not find {child.name}.");
164.                 }
165.                 else
166.                 {
167.                     Enum.TryParse(child.usageString, true, out child.usage);
168.                 }
169.             }
170.         }
171.     }

I'm going to keep working to see if I can figure this out. Once again, thanks for the help everyone.

 

It seems the issue is that while the JSON importer is getting the right number of objects from the lists, it's not getting the content, even as strings. (Line 8 in ImportJSON() above)

Edit:
I was able to get the whole thing working with the following:

Code (CSharp):

1. void ImportJSONTest()
2.     {
3.         Object[] areaObjects = Resources.LoadAll("Test", typeof(TextAsset));
4.         foreach(Object data in areaObjects)
5.         {
6.             AreaData generated = ScriptableObject.CreateInstance<AreaData>();
7.             JsonUtility.FromJsonOverwrite(data.ToString(), generated);
8.             generated.FinalizePopulate();
9.             allAreaData.Add(generated);
10.         }
11.     }

Entire JSON file:

Code (CSharp):

1. {
2.             "name":"Kitchen",
3.             "layerString":"Room",
4.             "maxLength": 5,
5.             "maxWidth": 3,
6.             "minLength" : 4,
7.             "minWidth" : 2,
8.             "hasWallMaterial" : true,
9.             "hasFloorMaterial" : true,
10.             "floorMaterialTypes":["tile","linoleum"],
11.             "wallMaterialTypes":["plaster","wallpaper"]
12. }

So that's good, but it comes with the large inconvenience of needing to have every single AreaData in its own file, so if anyone knows how to make import lists of strings that can then deserialized individually, I'd love to hear it.

 

Have you tried just making AreaData not a ScriptableObject asset? just a C# class. Then AreaListObject becomes the root asset and just stores a bunch of serializable AreaData instances. When serializing/deserializing assets unity only loads the data thats part of that root asset, everything else gets attached in via referencing.

basically the only two ways to get all that area data to fit on a single asset is to:

1. use AssetDatabase.AddObjectToAsset (btw you can only use on .asset files, will not work on scriptableobject, gameobjects, etc.)
2. turn AreaData into a serializable C# class and let AreaListObject be the root object responsible for holding all the areadatas

Option #2 is much more straightforward and less error prone, but will mean you can't drag AreaData references to other objects in a scene (you have to drag the AreaListObject and refer specific areas via that list).

JsonUtility actually supports missing fields. Its lenient in what fields do or don't exist in a JSON file. In the event a field is missing, FromJson/FromJsonOverwrite will just use the default/pre-existing data in the object. In the event that a field exists in the json but not in the target asset (for instance if an old JSON file has a field that was removed in a newer version of the class), JsonUtility will just ignore the field silently and skip to the next field

It may not be to the Json spec but technically its not actually deserializing from json to asset, at least not "directly". Under the hood it first converts the json to another format (possibly yaml or reflection) and just deserializes using the same process it uses to deserialize everything else. This is why features like ISerializationCallbackReciever, ISerializable, and FormerlySerializedAsAttribute work with JsonUtility

 

I've considered it, but it would require other changes in the project. Probably doable though.

I don't mean in the game's data structure (I'm fine with them being multiple files/instances) but just the ability to write JSON files like the one in my first post rather than the one in my last post. I suppose I should clarify I've been writing those by hand, they aren't generated by the program or anything.

 

So I'm just trying to understand your flow here. If AreaData are scriptableObject assets, why do you need to save them as json?

If I were to guess it sounds like you want to use these text assets to "reconfigure" areas on the fly and then have other scripts use said AreaDatas indirectly via reference to object fields on monobehaviours assigned in the inspector.
If that's the case, how is JsonUtility going to know which reference the data its loading its meant to overwrite?

one thing you could do is that AreaListObject holds a list of json strings and then further individually load from there. The problem however is the final json can end up being rather illegible due to it likely having to use escapes all over the place, which would defeat the point of using Json as the human readable format. its gonna look ugly and will not be fun to edit.

However, I think the cleaner implementation is if you move all the json-important fields inside AreaData into an internal class. AreaData simply holds an internal reference to this serializable "AreaDataConfig" class and exposes all the fields via properties for all the scripts to use (thus all the scripts still think they are directly using the data). Then your ImportJSONTest() isn't loading to a List<AreaData>, its loading to a List<AreaDataConfig>. after that you can just have a method on your AreaData that ImportJSONTest can call to load each new config in.

Also! Note that you likely want to make the

[SerializeField] private AreaDataConfig config;

field in AreaData first! before converting the other fields to properties so that you don't lose all your data in your AreaData assets! Then copy all that data over before finally turning the original fields into properties.

 

It's a legacy issue. I had them as scriptableObjects when I using them before, but the JSON system makes that redundant. Just switching away from that is probably what I need to do.

 

Well, the good news is that I got this working, the bad news is well, I have a new problem now: https://forum.unity.com/threads/recursive-de-serialization.1450303/

My sincere thanks to everyone who posted in this thread.