JsonUtility serializes floats with way too many digits

I need to serialize a Vector3 to Json. For this i want to use the build in JsonUtility. But JsonUtility converts a Vector3 to floats with a lot of decimal places. I know that this is the way a float works but it is not practical for serialization to a file. The files will become unnecessarily large because of all the digits.

Here an example:

Code (CSharp):

1. Debug.Log(JsonUtility.ToJson(new Vector3(0.24f, 0.5f, 0.75f)));

will output:

Code (CSharp):

1. {"x":0.23999999463558198,"y":0.5,"z":0.75}

can i do something to make this more efficient? I used Json Net before where this was no problem. However Json Net is a lot slower than JsonUtility (see here) and i would like to use as little 3rd party libraries as possible.

 

I think it was something like (not 100% if is correct). Possibly there is better way.
Vector3 V3 = new Vector3(0.24f, 0.5f, 0.75f) ;
string stringifyVector = V3.ToString ( "F4" ) ; // floating points of 4 digits.But you may loose axis names.
then pass to Json.
Alternatively after getting individual xyz, reconstruct the vector as string, to correct format and then pass to json.

 

I would argue a UTF-8 text file will increase by a mere 15 bytes in your example. How many vectors are you storing that this 15 bytes is that huge an issue?

Lets consider the permutations. Either 0, 1, 2, or 3 digits will have in excess of 15 bytes extra data. That is an average of 23 bytes per vector. That's about 2kbyte of overage for every 100 vectors. Before you even struck 1mbyte of overage, you'd have over 50,000 vectors stored in your file.

I'd personally classify this as an over optimization at this point. You could spend your time on more important things if the file size was actually becoming an issue and you verified that stringified floats were the largest percentage of the issue. At the point where file size was getting to absurd to handle, bloat would not only be coming from stringified floats, but instead all sorts of data types in your file. And at that point compression would probably be a better fix.

...

Note, you're using JsonUtility because you find Json.Net too slow. But if you added in extra code to alternatively stringify floats (which isn't directly supported by unity through their serializer and would have to be processed in post), you'd slow down the serializer. Costing you the efficiency gains you picked JsonUtility for, all to save a minor amount of disk-space in a day and age when disk space is abundant.

 

I'm in Unity 2020.1 Alpha, and now have access to Newtonsoft.Json through the Unity Package Manager. This solves the issue, with some... expertise. This code might not solve every case scenario, as I haven't thoroughly tested it, but this should help you quite a bit.

Newtonsoft.Json's serializer does the same thing with floats/doubles having way too many digits. However, unlike Unity, you can customize the serialization process more intimately by specifying that floats use a certain number of digits. However, just a warning: if you are unfamiliar with Reflection and/or Newtonsoft's Json serialization, this code might look ridiculous for you and take a while to understand.

Code (CSharp):

1. using System;
2. using System.Reflection;
3. using System.Collections.Generic;
4. using UnityEngine;
5. using Newtonsoft.Json;
6. using Newtonsoft.Json.Serialization;
7.  
8. using Object = UnityEngine.Object;
9.  
10. [CreateAssetMenu]
11. public class UnityNewtonsoftJsonSerializer : ScriptableObject {
12.  
13.     #region Static Section
14.     //Not expected to work with UnityEditor types as well, since it won't ignore types like Editor, EditorWindow, PropertyDrawer, etc. -- with their fields.
15.     private class UnityImitatingContractResolver : DefaultContractResolver {
16.         /// <summary>
17.         /// Any data types whose fields we don't want to serialize. When any of these types are encountered during serialization,
18.         /// all of their fields will be skipped.
19.         /// </summary>
20.         private static readonly Type[] IgnoreTypes = new Type[] {
21.             typeof(Object),
22.             typeof(MonoBehaviour),
23.             typeof(ScriptableObject)
24.         };
25.         private static bool IsIgnoredType(Type type) => Array.FindIndex(IgnoreTypes, (Type current) => current == type) >= 0;
26.  
27.         protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization) {
28.             List<FieldInfo> allFields = new List<FieldInfo>();
29.             Type unityObjType = typeof(Object);
30.  
31.             for (Type t = type; t != null && !IsIgnoredType(t); t = t.BaseType) {
32.                 FieldInfo[] currentTypeFields = t.GetFields(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
33.                 for (int i = 0; i < currentTypeFields.Length; i++) {
34.                     FieldInfo field = currentTypeFields[i];
35.                     if (!field.IsPublic && field.GetCustomAttribute<SerializeField>() == null)
36.                         continue;
37.                     if (unityObjType.IsAssignableFrom(field.FieldType)) {
38.                         Debug.LogError("Failed to serialize a Unity object reference -- this is not supported by the " + GetType().Name
39.                             + ". Ignoring the property. (" + type.Name + "'s \"" + field.Name + "\" field)");
40.                         continue;
41.                     }
42.                     allFields.Add(field);
43.                 }
44.             }
45.             //This sorts them based on the order they were actually written in the source code.
46.             //Beats me why Reflection wouldn't list them in that order to begin with, but whatever, this works
47.             allFields.Sort((a, b) => a.MetadataToken - b.MetadataToken);
48.  
49.             List<JsonProperty> properties = new List<JsonProperty>(allFields.Count);
50.             for (int i = 0; i < allFields.Count; i++) {
51.                 int index = properties.FindIndex((JsonProperty current) => current.UnderlyingName == allFields[i].Name);
52.                 if (index >= 0)
53.                     continue;
54.                 JsonProperty property = CreateProperty(allFields[i], memberSerialization);
55.                 property.Writable = true;
56.                 property.Readable = true;
57.                 properties.Add(property);
58.                 //Debug.Log(property.PropertyName + " was added.");
59.             }
60.             return properties;
61.         }
62.     }
63.  
64.     private class FloatConverter : JsonConverter<float> {
65.         private int decimalPlaces;
66.         private string format;
67.  
68.         public int DecimalPlaces {
69.             get { return decimalPlaces; }
70.             set {
71.                 decimalPlaces = Mathf.Clamp(value, 0, 8);
72.                 format = "F" + decimalPlaces;
73.             }
74.         }
75.  
76.         public FloatConverter(int decimalPlaces) {
77.             DecimalPlaces = decimalPlaces;
78.         }
79.  
80.         public override void WriteJson(JsonWriter writer, float value, JsonSerializer serializer) {
81.             writer.WriteValue(float.Parse(value.ToString(format)));
82.         }
83.  
84.         public override float ReadJson(JsonReader reader, Type objectType, float existingValue, bool hasExistingValue, JsonSerializer serializer) {
85.             //For some reason, reader.Value is giving back a double and casting to a float did not go so well, from object to float.
86.             //And I didn't want to hard code 2 consecutive casts, literally, like "(float) (double) reader.Value", so I'm glad this works:
87.             return Convert.ToSingle(reader.Value);
88.         }
89.     }
90.     #endregion
91.  
92.     [SerializeField] private bool prettyPrint = true;
93.  
94.     private JsonSerializerSettings settings = new JsonSerializerSettings() {
95.         ContractResolver = new UnityImitatingContractResolver(),
96.         Converters = new JsonConverter[] {
97.             new FloatConverter(3)
98.         }
99.  
100.     };
101.  
102.     public string Serialize<T>(T obj) {
103.         string text;
104.         Formatting formatting = prettyPrint ? Formatting.Indented : Formatting.None;
105.         settings.Formatting = formatting;
106.         try {
107.             //For now, as I am unsure how I want to move forward with looping references, I'll just have it try first -- if it comes up, show an error
108.             settings.ReferenceLoopHandling = ReferenceLoopHandling.Error;
109.             text = JsonConvert.SerializeObject(obj, settings);
110.         } catch (JsonSerializationException e) {
111.             //and then go to these statements and ignore any looping references. This way, it lets us know if it DOES come across looping references.
112.             //Which aren't supported as this code is written currently.
113.             Debug.LogException(e);
114.             settings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;
115.             text = JsonConvert.SerializeObject(obj, settings);
116.         }
117.         return text;
118.     }
119.  
120.     public T Deserialize<T>(string text) {
121.         return JsonConvert.DeserializeObject<T>(text, settings);
122.     }
123. }

What this does:
1. Changes Newtonsoft.Json to NOT serialize C# properties (just like how Unity doesn't serialize C# properties).

2. Changes Newtonsoft.Json to SERIALIZE any fields (including non-public fields) that are marked with the

[SerializeField]

attribute (just like how Unity does).

3. Changes Newtonsoft.Json to ignore some standard Unity types, since I personally don't need their fields serialized at all (UnityEngine.Object, MonoBehaviour, and ScriptableObject -- you can add to these if you want).

4. Changes Newtonsoft.Json to order the fields by the order you wrote them in source code (strange, this isn't a default for them). This is why I sort by

MetadataToken

.

5. Allows you to specify the number of digits that ALL floats will be serialized with. (This uses the FloatConverter class in the code above)

 

Hello, pardon for my ignorance.. but I am unfamiliar with Reflection and/or Newtonsoft's Json serialization

I need to serialize a List<Vector3> but having this error:

Code (CSharp):

1. JsonSerializationException: Self referencing loop detected for property 'normalized' with type 'UnityEngine.Vector3'. Path '[0].normalized'.

Can the UnityNewtonsoftJsonSerializer be used to overcame that? In case, how to use it?

Many thanks in advance.

 

What is your Newtonsoft.Json? There have errors in your code.

 

Apologies for the delay, but maybe this'll help someone,
I tested the UnityNewtonsoftJsonSerializer code above with the following MonoBehaviour, adding it as a script/component in a scene, and entering playmode:

Code (CSharp):

1. using System;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. public class SerializationTestBehaviour : MonoBehaviour {
6.     [Serializable]
7.     public struct TestData {
8.         public Vector3 position;
9.     }
10.  
11.     [SerializeField] private UnityNewtonsoftJsonSerializer serializer;
12.     [SerializeField] private TestData testData;
13.  
14.     private void Start() {
15.         string text;
16.  
17.         text = serializer.Serialize(testData);
18.         Debug.Log(text);
19.  
20.         text = serializer.Serialize(new List<Vector3>() { new Vector3(3, 3, 3), new Vector3(9, -4.02f, 0) });
21.         Debug.Log(text);
22.     }
23. }
24.  

The 2 info logs were printed as:

Code (CSharp):

1. {
2.   "position": {
3.     "x": 1.988,
4.     "y": 2.0,
5.     "z": 4.0
6.   }
7. }
8. UnityEngine.Debug:Log (object)
9. SerializationTestBehaviour:Start () (at Assets/Scripts/DevTesting/SerializationTestBehaviour.cs:16)

Code (CSharp):

1. [
2.   {
3.     "x": 3.0,
4.     "y": 3.0,
5.     "z": 3.0
6.   },
7.   {
8.     "x": 9.0,
9.     "y": -4.02,
10.     "z": 0.0
11.   }
12. ]
13. UnityEngine.Debug:Log (object)
14. SerializationTestBehaviour:Start () (at Assets/Scripts/DevTesting/SerializationTestBehaviour.cs:18)
15.  

Therefore, it works for me.

The code I provided shouldn't be trying to serialize C# Properties (like the Vector3.normalized property in your error message) because the UnityNewtonsoftJsonSerializer.UnityImitatingContractResolver.CreateProperties(...) method returns a list of JsonProperties telling the serializer what to serialize, and I only iterate through FieldInfo objects, never any PropertyInfo objects.
This basically means only fields are serialized, whereas properties (like Vector3.normalized) are ignored.
So I'm not sure why you got that error message.. I hope you got it working after all this time though!


I tested this in Unity 2020.3.18f1 with the following added to my Packages/manifest.json file: "com.unity.nuget.newtonsoft-json": "2.0.0"

@Yiming075 The code should work and compile, but it does require you add the Newtonsoft.Json library to your project somehow. As an example, you can use Unity's package of Newtonsoft.Json (or bring the DLLs directly into your project).

For example, in my Unity project named "dummy", I added the package manually by editing the manifest.json file in a text editor to contain "com.unity.nuget.newtonsoft-json": "2.0.0":