Feedback Please add System.Text.Json support

Pretty self-describing

 

It lacks a lot of features of proper json serializer/deserializer.

 

You can also point out why exactly you need that namespace, for example, by listing out the features.

 

https://docs.microsoft.com/en-us/dotnet/api/system.text.json?view=netcore-3.1

 

System.Text.Json uses Span<T> and require .NET Standard 2.1, is currently not available for unity see https://forum.unity.com/threads/net-standard-2-1.757007/

 

Note that you can still use System.Text.Json (alongside Span<T> and others) via the NuGet package. You just won't be able to take advantage of specific runtime optimizations associated with Span<T> and such. https://www.nuget.org/packages/System.Text.Json/

 

System.Text.Json won't work in current il2cpp.

There are 2 alternatives:
- features https://github.com/jilleJr/Newtonsoft.Json-for-Unity
- performance https://github.com/neuecc/Utf8Json

 

Given that they want "features for proper json" then the newtonsoft variant is what they're looking for.

 

*May work in Net 6 when reflection will be replaced with source generators

 

They would also like to have lighweight solution. Serializing/Deserializing a very simple object could allocate memory of 10s to 100s of MBs. That's not really viable for mobile game development, or any development to be honest. You should not run out of memory because few kB JSON docs.

 

System.Text.Json support would be greatly appreciated. The Unity serialization system (even the new one) lacks features. Newtonsoft.Json is too random and horrible performance-wise.

System.Text.Json is a good alternative between maintainability, performance, and features. Right now it works on Mono builds but not on IL2CPP/WebGL builds, support for IL2CPP would be greatly appreciated.

 

There is a workaround (until 2021.2 is launched) for value types: https://github.com/dotnet/runtime/issues/49772

 

That's kinda ironic, considering rewrite of .NET built-in JSON parses inspired by Utf2json and the creator was one of the top contributors to new netcode json library

 

Sorry for dragging up an old thread but has anyone managed to get System.Text.Json working at all in Unity?

 

@markinovus yes !

You basically have to follow one to three steps, depending on your situation:

• Gather the required DLL of System.Text.Json with the right version (using NuGet online website, then decompress the nuget package with 7Zip for example). Use the 5.0.0 version of System.Text.Json. You'll need to have all these Dlls:
  • Microsoft.Bcl.AsyncInterfaces.dll
    
  • System.Buffers.dll
    
  • System.Memory.dll
    
  • System.Runtime.CompilerServices.Unsafe.dll
    
  • System.Text.Encodings.Web.dll
    
  • System.Text.Json.dll
    
  • System.Threading.Tasks.Extensions.dll
• For Il2CPP and WebGL: Add the following link.xml in the Asset folder
  • Code (CSharp):

    1. <linker>
    2.     <assembly fullname="System.Text.Json">
    3.         <namespace fullname="System.Text.Json" preserve="all" />
    4.         <namespace fullname="System.Text.Json.Serialization" preserve="all" />
    5.         <namespace fullname="System.Text.Json.Serialization.Converters" preserve="all" />
    6.     </assembly>
    7. </linker>
• For Il2CPP and WebGL: You might have to write custom converters for very specific cases such as serializing generic function instances for value types. See the github thread : https://github.com/dotnet/runtime/issues/49772

​

 

Unity's serializer doesn't run server-side. An important use case is having the same tooling on the client and game server. Newtonsoft.Json works (and it is what I use) but it lacks the minimal allocation and high performance design of System.Text.Json.

 

I've never used Unity's JSON serializer. It just lacks all the features I need. And now we just need all that features of Newtonsoft.JSON plus more performance.

 

As source generators work in 2021.2b you can try making System.Text.Json work https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-source-generator/

 

Quoting my own post

Unity Technologies provide Newtonsoft.JSON through the Package Manager by now. Just add the following line to

Packages/manifest.json

:

"com.unity.nuget.newtonsoft-json": "2.0.0",

 

It's outdated version of https://github.com/jilleJr/Newtonsoft.Json-for-Unity

 

I suppose most of the people here asking for System.Text.Json are well aware of this already. However we are now in need of a better solution: higher perf, less allocation. You seem don't understand this point since the beginning?

 

You too:

 

I know about neuecc/Utf8Json actually, but didn't successfully use it 3 years ago. 2 years later my main concern changed, I worked mainly on offline games without json data. However, it's now changed again, I'm now working in an online game project. I see neuecc/Utf8Json has been inactive for a long time thus I just want to use something that is more actively maintained.

 

I know from previous experience (non games applications) the value of including version numbering into JSON and version migration within the code, so that if the code is upgraded and the JSON structure changes then the app still works and can auto-migrate data saved from an older version to the current codebase.

Unfortunately a limitation of JsonUtility is that it serializes/deserializes the current structure only. If you were to try deserializing data saved from an older version of code, it just wouldn't work.

I want to have full JsonDocument control (as provided by System.Text.Json), the ability to include a version attribute that I can peek at before deserializing and the ability to manipulate the JSON as a structured object.

As a temporary workaround, I am considering this approach:

1. Include the JSON version as a hidden, serializable, private field in the object I want to serialize:

Code (CSharp):

1. [Serializable]
2. public MyClass
3. {
4.     [SerializeField, HideInInspector]
5.     private int _jsonVersion = 1;
6.     ...

2. Before deserializing, peek at the version number:

Code (CSharp):

1.     public void DeserializeJson(string json)
2.     {
3.         private struct JsonVersionStruct
4.         {
5.             public int _jsonVersion;
6.         }
7.  
8.         var jsonVersionStruct = JsonUtility.FromJson<JsonVersionStruct>(json);
9.         if (jsonVersionStruct._jsonVersion == _jsonVersion)
10.         {
11.             ...

But this is a gnarly workaround when there's a perfectly good class in System.Text.Json.

I like JsonUtility, but it's just too basic to trust with a production system.

[EDIT: I originally had _jsonVersion as static in step 1 but this won't serialize.]

 

Has anyone succeeded in getting System.Text.Json to run properly in Unity 2021.2?

 

This exact use case has made me manually inject a $jsonVersion header into my json for my levels, and then an upgrading system where its able to take a file from version n and upgrade it match version n + 1 perfectly. If you chain all of the upgraders then you are able to take it from version 1 to version 24 assuming you implement all of the conversions. This does require you to handle the json structure correctly tho, so using something that is able to unflatten a json string can give you really good control, this is something that unity's jsonutility cant do. If it could unflatten json strings that would be amazing.

 

Seems like System.Text.Json works in 2021.2.7, at least on windows (also il2cpp!) for this class (I've tried int as T):

Code (CSharp):

1. public class Obj<T>
2. {
3.     public string[] TestString {get; set;}
4.     public InnerObj TestObj {get; set;}
5.     public T TestT {get; set;}
6.  
7.     public class InnerObj
8.     {
9.         public string TestString {get; set;}
10.     }
11. }

 

Which IL2CPP Code Generation option did you use?

 

Both work.

 

Is there any official word from Unity on supporting System.Text.Json and when?

This in place of the conflicting forks of newtonsoft (This project vs `com.unity.nuget.newtonsoft-json` · Issue #145 · jilleJr/Newtonsoft.Json-for-Unity (github.com))?

It does sound awkward now to get good il2cpp JSON support (fully featured)

 

Looks like Unity's Newtonsoft package got out of preview. https://docs.unity3d.com/Packages/com.unity.nuget.newtonsoft-json@latest

Older versions have this warning:

 

With il2cpp "faster runtime" value types won't work (not sure about Unity 2022).

@JoshPeterson Would 3rd il2cpp code generation option "faster runtime with full generics" be possible at cost of the biggest size?

 

Just tried, seems to be working with il2cpp "faster runtime" even for value types. For ex.

Code (CSharp):

1. [JsonSerializable(typeof(MyStruct))]
2. [JsonSerializable(typeof(MyStruct2))]
3. public partial class MyJsonContext : JsonSerializerContext{}
4.  
5. public byte[] Method<T>(T obj) =>
6.   JsonSerializer.SerializeToUtf8Bytes<T>(obj, MyJsonContext.Default.Options)

Unfortunately "Options" field was added in later version of the source generator. Unity's Editor\Data\NetCoreRuntime and Editor\Data\DotNetSdkRoslyn have to be updated manually to get it working.

 

Did you manage to get json source generator to work? I couldn't implement JsonSerializerContext at all due to compiler errors.

 

Yes but I ended up symlinking whatever .net sdk I have installed to Unity folder to not relay on their version. For ex. C:\Program Files\dotnet\sdk\6.0.401\Roslyn\bincore > Editor\Data\DotNetSdkRoslyn . This is discouraged by Unity but works for me fine.

This way VS/Rider and Unity uses same sdk and allows me use the most recent versions of NuGet packages.

 

What steps exactly do I need to take? That namespace fails to resolve for me using 2021 LTS with the latest nuget package.

 

I'm not as knowledgeable in this particular realm as some of the folks in this thread. But as I understand it, Newtonsoft doesn't properly support async behavior. And never will. So unless I'm completely misunderstanding something, I struggle to understand why Unity has continued pushing support for this over System.Text.Json.

 

You'll need a lot of dlls. You can try https://github.com/xoofx/UnityNuGet . I build nuget dlls using separate normal csproj but that plugin may be easier to use.

Newtonsoft has 4x more nuget downloads than System.Text.Json, you can easily get async support by running it on separate thread (I like the using the UniTask.SwitchToThreadPool()).

Unity is pushing for net 6 that will have nuget support and thus easy way to use System.Text.Json,

 

Excellent this worked perfectly. Thanks for taking the time to point me in the right direction.

That's a fair point. Performance is the primary consideration for my particular use case, which System.Text.Json seems to have the upper hand in as well.

 

System.Text.Json is part of .Net 6 (since .NET core 3.0) the nuget package is for other versions and .NET standard.
Since the new runtime will probably not come until 2023.2, it could also be that Unity comes with .NET 7, maybe even 8 Since .NET and Unity 202x.2 versions are released around the same time, that would be interesting if each new year version has the latest .NET.

 

Maybe that's just a typo but the player runtime upgrade will be 2023.2 at the earliest, as Josh said they've missed the cutoff for 2023.1. That's also just for the player runtime, the editor runtime upgrade could well be a couple of Unity releases after that. And until the editor upgrade, using new .Net features will not really be practical for development. But Unity should track the latest .Net release pretty closely, so it could be .Net 8 or 9 when we're finally able to use it.

 

Yes, it was a typo.

 

Has anyone managed to get it to work?
I still get:
"Unable to resolve reference 'System.Text.Json'. Is the assembly missing or incompatible with the current platform?
Reference validation can be disabled in the Plugin Inspector."
Even after trying https://github.com/xoofx/UnityNuGet, maybe I've used it wrong but I'd like to ask if anyone managed to get it working after seeing this same error.

 

For example, you can't serialize the List.You need to convert it to a property.

 

there are some hacks to get around this.

The list just needs to be wrapped in an object. If you're receiving a raw list in the JSON, then you can do the following:

Code (CSharp):

1. const string listJson = @"[
2.  1,
3.  2,
4.  3,
5.  4,
6. ]";
7.  
8. var wrappedListJson = $"{{\"value\":{listJson}}}";
9. var wrappedList = JsonUtility.FromJson<Wrapped<List<int>>>(wrappedListJson);
10. var list = wrappedList.value;
11.  
12. [System.Serializable]
13. public struct Wrapped<T>
14. {
15.   public T value;
16. }

If you need to serialize the json to a list format, then you can use a substring.

Code (CSharp):

1. var list = new List<int>();
2. list.Add(1);
3. list.Add(2);
4. list.Add(3);
5. list.Add(4);
6.  
7. var wrappedList = new Wrapped<List<int>> { value = list };
8. var wrappedListJson = JsonUtility.ToJson(wrappedList);
9.  
10. var prefixLength = "{\"value\":".Length;
11. var postfixLength = "}".Length;
12. var listLength = wrappedListJson.Length - prefixLength - postfixLength;
13. var listJson =  wrappedListJson.Substring(prefixLength, listLength);

 

Looks like a big waste of time just use Newtonsoft or system.text.json in Unity 2022.

 

This is slightly related, but I'm curious. Would you prefer a unity-specific JSON library that is Burst-compatible to `System.Text.Json`?

I'm not sure the usefulness of serializing/deserializing JSON in a Burst context, so I'd expect most people would prefer `System.Text.Json` as it's fast, easy enough, and the standard among .NET developers. When using JSON, you're interacting with web services, which aren't generally hot code paths. If you were using JSON in a performant context, you'd probably opt to use a better format anyway.