Search Unity

  1. The 2022.1 beta is now available for testing. To find out what's new, have a look at our 2022.1 beta blog post.
    Dismiss Notice

Systen.Text.Json .Net STandard 2.0

Discussion in 'Scripting' started by CyberAngel, Jan 14, 2022 at 3:18 PM.

  1. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    Has anyone had any luck getting this into Unity 2021.2? Or any Unity version that supports .Net Standard 2.0?

    When I get the package and drop the dll and xml in, all I get is

    Unable to resolve reference 'System.Text.Encodings.Web'. Is the assembly missing or incompatible with the current platform?
     
  2. MaskedMouse

    MaskedMouse

    Joined:
    Jul 8, 2014
    Posts:
    773
    just a little search whether it is implemented or not using Google with the keywords "Unity System.Text.Json" and you'll stumble upon this thread

    As far as I can read out of that thread, there are some complications with supporting System.Text.Json.

    There are alternatives like Newtonsoft Json though. Some of the packages that Unity uses, even rely on newtonsoft json which is why they've made a package of it.
    Package Manager -> + -> Add by name ->
    com.unity.nuget.newtonsoft-json
     
    _geo__ and Kurt-Dekker like this.
  3. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    But I am not interested in the Newtonsoft package, it is a security risk with objects, that I don't want to use. I have read other forum posts in here where people have got the Systen.Text.Json .Net Standard 2.0 version to work in Unity, so now I am asking how they did it.
     
  4. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,266
    Is there a .Net Standard 2.0 version of this library? Because the MSDN says there isn't.
    screenshot.png
     
  5. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
  6. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,266
  7. VolodymyrBS

    VolodymyrBS

    Joined:
    May 15, 2019
    Posts:
    130
    To use System.text.json you need to download all it dependencies into too:
    - Microsoft.Bcl.AsyncInterfaces (NET Standard 2.1 version if you on Unity 2021.2)
    - System.Buffers (not required if you on Unity 2021.2)
    - System.Memory (not required if you on Unity 2021.2)
    - System.Numerics.Vectors (not required if you on Unity 2021.2)
    - System.Runtime.CompilerServices.Unsafe
    - System.Text.Encodings.Web
    - System.Threading.Tasks.Extensions (not required if you on Unity 2021.2)

    with all this libraries you should be able to use System.Text.Json.

    but it still may have issues with il2cpp
     
  8. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    21,292
    You should consider being more selective about where you get your nonsense from.
     
  9. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    I have code that I have serialize as objects, and it is a security risk. Do you mind showing me where it is not?
     
  10. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    21,292
    NewtonSoft is kind of an industry standard for JSON. It's likely no more or less secure than the operating system context it is running within.

    Are you perhaps thinking of binary formatter???

    https://docs.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide

    Do not use the binary formatter/serializer: it is insecure, it cannot be made secure, and it makes debugging very difficult, plus it actually will NOT prevent people from modifying your save data on their computers.
     
  11. VolodymyrBS

    VolodymyrBS

    Joined:
    May 15, 2019
    Posts:
    130
    but how using System.Text.Json will eliminate security risk?
    System.text.json us not more secure than Newtonsoft.Json. it was created for better perfomance and to have Json serialize within System libraries
     
  12. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    No I am not.

    For the ability to serialise this

    Dictionary<string, object>

    You have to have the types stored into the json, which is done as serialising an object, everything I have found on this, says Json.Net has a security flaw if you use the type in the Json. This is defined by OWASP as a security flaw.

    If you have a way to do this safely, I am all ears!
     
  13. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    Yet, I can safely serialise Objects like a Dictionary<string, object> with MS way, and yet I can't with Json.Net safely. If you have a way to do this safely, I am all ears!
     
  14. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,266
    Don't use JSON in production. Problem solved. It's great for debugging, but nothing else. It's insane arguing which crappy JSON is more secure.
     
  15. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    Json, XML, Object

    It has to be done, it is part of my data structure!
     
  16. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,266
    Okay, if we're moving targets: do not use text-based serialization in production. No, it's not 'has to be' done, you chose it.
     
  17. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    It has to be done, there is no way around the data structure.

    Unless you have a workaround that can be done safely, then I need a solution that works!
     
  18. VolodymyrBS

    VolodymyrBS

    Joined:
    May 15, 2019
    Posts:
    130
    one of solutions could be to implement custom converter that will encode type information but not in json string but not so "transparent" as it done by default and limit possible types.
    for example some something like this
    Code (CSharp):
    1. public static class Program
    2.     {
    3.         public static void Main()
    4.         {
    5.            var converter = new CustomConverter();
    6.            var data = new Dictionary<string, object>
    7.            {
    8.                   ["1"] = new DerivedClass() { Data = 42, Data2 = 24 }
    9.            };
    10.          
    11.            var str = JsonConvert.SerializeObject(data, converter);
    12.            Console.WriteLine(str);
    13.            var dictionary = JsonConvert.DeserializeObject<Dictionary<string,object>>(str, converter);
    14.            var obj = (DerivedClass)dictionary["1"];
    15.            Console.WriteLine(obj.Data);
    16.            Console.WriteLine(obj.Data2);
    17.         }
    18.     }
    19.    
    20.     public class CustomConverter : JsonConverter
    21.     {
    22.         public enum TypeCode
    23.         {
    24.             Base = 1,
    25.             Derived = 2,
    26.         }
    27.        
    28.         public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    29.         {
    30.             var jObject = (JObject)JToken.FromObject(value);
    31.             jObject.AddFirst(new JProperty("TypeCode", GetTypeCode(value)));
    32.             jObject.WriteTo(writer);
    33.         }
    34.        
    35.         private int GetTypeCode(object value)
    36.         {
    37.             if (value.GetType() == typeof(BaseClass))
    38.                 return (int)TypeCode.Base;
    39.             if (value.GetType() == typeof(DerivedClass))
    40.             return (int)TypeCode.Derived;
    41.                
    42.             throw new ArgumentException($"Type {value.GetType()} is not supported");
    43.         }
    44.        
    45.         private object GetEmptyInstance(int typeCode)
    46.         {
    47.             return (TypeCode)typeCode switch
    48.             {
    49.                 TypeCode.Base => new BaseClass(),
    50.                 TypeCode.Derived => new DerivedClass()
    51.             };
    52.         }
    53.        
    54.         public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    55.         {
    56.             var jObject = JObject.Load(reader);
    57.             var type = (int?)jObject["TypeCode"];
    58.             if (type == null)
    59.                 throw new ArgumentException("Input data does not contains type code");
    60.                
    61.             var instance = GetEmptyInstance((int)type);
    62.             serializer.Populate(jObject.CreateReader(), instance);
    63.            
    64.             return instance;
    65.         }
    66.        
    67.         public override bool CanConvert(Type objectType)
    68.         {
    69.             return typeof(BaseClass).IsAssignableFrom(objectType) || objectType == typeof(object);
    70.         }
    71.     }
    72.    
    73.     public class BaseClass
    74.     {
    75.         public int Data { get; set; }
    76.     }
    77.    
    78.     public class DerivedClass : BaseClass
    79.     {
    80.         public int Data2 { get; set; }
    81.     }
    Also I believe that almos any System.Text.Json solution could be ported to Json.NET, because as far as I know Json.NET has almost all features System.Text.Json and more
     
  19. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    This is why I am looking for a solution to getting System.Text,Json into Unity 2021.2, the problem with Json.Net, is that the object can still be delivered a payload that is malicious.
     
  20. VolodymyrBS

    VolodymyrBS

    Joined:
    May 15, 2019
    Posts:
    130
    Just curious. how did you implement serialization/deserialization of Dictionary<string, object> with System.Text.Json in more secure way than it possible with Json.NET?
     
  21. CyberAngel

    CyberAngel

    Joined:
    Oct 4, 2014
    Posts:
    77
    Json.Net uses Object serialisation, that means unless you specify the type in that json with a dictionary, it will come back as Dictionary<string,string>() Json.Net can not be made secure in this scenario if you require Dictionary<string, object>. Microsoft's System.Text,Json is supposed to be a replacement for Object serialisation. I don't believe it is perfect, but with custom converters you can over come that. But at least with what I have done and tried, I am not able to exploit payloads into this with MS's approach.

    Json.Net can be exploited with Payloads in the same manner as BinaryFormatter, if you require what I am trying to do.
     
  22. VolodymyrBS

    VolodymyrBS

    Joined:
    May 15, 2019
    Posts:
    130
    I understand you problem. I curious how custom converter implemented. could you share the code of converter?
     
  23. _geo__

    _geo__

    Joined:
    Feb 26, 2014
    Posts:
    423
    Sorry for opening a tangent in this thread.
    I'd be interested what solutions you are refering to. Do you mean using Protobuf etc. instead? I'd be interested in hearing how those faired in production in your experience and what solution you would recommend.

    Newtonsoft has some Docs on how to handle type names in json:
    https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_TypeNameHandling.htm
    Here are some docs by Microsoft on being cautious with type handling in Json.Net:
    https://docs.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2326
     
    VolodymyrBS likes this.
  24. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,266
    No problem from my side. :) I have used many type of serialization over the years, including the Unity ones, popular ones, not so popular ones.
    If you take it seriously, and you don't mind that you need to be mindful what you dump on disk or onto the network, then I really recommend either ProtoBuf or one of the many descendants. I agree, it's a bit convoluted to set it up, but once you have the encoder and all the platform DLLs in place, it's reliable, it can serialize to JSON in debug and binary in production. The big advantage is what the biggest inconvenience according to some people: you need to declare what you serialize. In exchange it cannot be attacked with anything since the schema is in the application, not in the dataset.
    Also it is both backwards and forward(!) compatible. Which makes the usage a fresh breeze. Obviously it can't pull data out from the thin air, but it doesn't throw a belly-flop if not everything is in the dataset. I use it daily in my day job.

    But, in Unity projects, nowadays, because of the unearned hate towards ProtoBuf I recommend Odin Serializer (free and open source). It also has both JSON and binary formatter, so you can write JSON in debug and binary in production. And it is superfast.
    Also, if you have Odin Inspector (separate product, not free and not open source!), then you get a debugger to check the unreadable serializations too. Since it is built in the Odin Inspector, it is battle tested and widely used (Odin is one of the most popular asset nowadays). I also use this when I don't have time to set up ProtoBuf.

    I also want to add: I write my save files manually. Binary in production, handwritten, hand picked data, byte by byte (when it comes to booleans even bit by bit), int by int, string by string. With versioning and all. Obviously in debug mode I dump the save in a simple text file too so it's easy to debug.
     
    Last edited: Jan 16, 2022 at 1:06 PM
    _geo__ likes this.
  25. _geo__

    _geo__

    Joined:
    Feb 26, 2014
    Posts:
    423
    Thank you for your elaborate answer :)

    To me that's an advantage :D

    I have used Protobuf for storage on single platform apps in the past. I shied away from using it for cross platform things out of a vague fear of whether it would work everywhere (WebGL for example). Maybe I should switch. I am still serializing most things with json because it's supported pretty much in any environment. I usually serialize well defined data objects whose sole purpose is to hold the data in a known format. Which brings an alternative to mind. Why not just parse the data on your own into primitive data structures, no types involved (Json.net even supports this iirc). Yes it's a PITA but it will allow pretty much anything to be deserialized.

    I think the OP has the problem of having unstructured data in the form of mixed types, thus the "serialize object" requirement. That sounds like a nightmare to me. But I guess there are scenarios when you need that (like the source is not under your control).
     
    Last edited: Jan 16, 2022 at 5:00 PM
    Lurking-Ninja likes this.
  26. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,266
    Well, if you deploy to WebGL, you might as well use JSON. :D
     
    _geo__ likes this.
  27. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    21,292
    Interesting... I'm always text beginning to end.

    My thinking is that bugs found in development are free, but bugs caught in the wild in production code are actually screwing up my users, causing ill will and making people rate my game badly.

    Therefore no way in hell would I want anything like binary data blobs in the way of finding those bugs.

    Y'all know my position on the silliness of client-side encryption, but I have done client-side checksumming, which involves hashing the save data along with a preset key and saving that hash for later verification before accepting the save data.

    This form of "armoring" on the save data is equivalently secure to actual client-side encryption and yet leaves the data completely human-readable on disk.
     
    _geo__ likes this.
  28. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,266
    The security is nice, but it's secondary for me, I'm more about to generate as small amount of garbage as possible. JSON serialization is not the best in that (obviously this image only should read as relative to each other).

    screenshot.png
    (Source: https://github.com/TeamSirenix/odin-serializer)

    PS: also, there is a giant difference between security like not allowing the player to change game play data and security as in preventing bad actors to run unauthorized code on the player's computer. The first isn't interesting, the second is.
     
    Last edited: Jan 16, 2022 at 7:14 PM
    _geo__ likes this.
  29. _geo__

    _geo__

    Joined:
    Feb 26, 2014
    Posts:
    423
    Ha, I just did the very same in my game :)

    If I look at those graphs then I am glad I actually built all my serialization with Unitys JsonUtility. It's a pain to use but I just love loading my data into SOs to then observe it at runtime. It's so nice for debugging and testing :D
     
    Last edited: Jan 16, 2022 at 10:40 PM
unityunity