Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  3. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Feedback About the com.unity.serialization Package

Discussion in '2023.2 Beta' started by Canijo, Nov 2, 2023.

  1. yu_yang

    yu_yang

    Joined:
    May 3, 2015
    Posts:
    85
    They may want to make the output JSON more concise. This limitation should be known, it's just that the support for generics is currently in a 'Todo' state...
     
    Ice_106 and Canijo like this.
  2. Ice_106

    Ice_106

    Joined:
    Jun 17, 2019
    Posts:
    5
    This is really good, but unfortunately, I am developing a package and relying on it. I will temporarily overwrite the package, and I hope the new version can fix this issue
     
    Canijo likes this.
  3. yu_yang

    yu_yang

    Joined:
    May 3, 2015
    Posts:
    85
    I just finish
    TypeSerialization
    , it is not thread-safe, but I think it's a valuable starting point that might be of interest to someone, so I'm pasting here.

    Code (CSharp):
    1. using System;
    2. using System.Text;
    3. using System.Collections.Generic;
    4. using UnityEngine.Pool;
    5.  
    6. namespace UnityExtensions.Reflection
    7. {
    8.     // Todo: thread safety.
    9.     public static class TypeSerialization
    10.     {
    11.         static Dictionary<Type, string> s_typeToText = new();
    12.         static Dictionary<string, Type> s_textToType = new();
    13.  
    14.         public static string Serialize(Type type)
    15.         {
    16.             if (type == null)
    17.                 return null;
    18.  
    19.             if (!s_typeToText.TryGetValue(type, out var text))
    20.             {
    21.                 using (Build(type, out var builder))
    22.                     text = builder.ToString();
    23.  
    24.                 s_typeToText.Add(type, text);
    25.                 s_textToType.TryAdd(text, type);
    26.             }
    27.             return text;
    28.         }
    29.  
    30.         public static Type Deserialize(string text)
    31.         {
    32.             if (text == null)
    33.                 return null;
    34.  
    35.             if (!s_textToType.TryGetValue(text, out var type))
    36.             {
    37.                 type = Type.GetType(text);
    38.                 s_textToType.Add(text, type);
    39.             }
    40.             return type;
    41.         }
    42.  
    43.         static PooledObject<StringBuilder> Build(Type type, out StringBuilder builder)
    44.         {
    45.             var result = StringBuilderPool.Get(out builder);
    46.             if (type.IsGenericType)
    47.             {
    48.                 if (type.ContainsGenericParameters)
    49.                     throw new ArgumentException();
    50.  
    51.                 string ns = type.Namespace;
    52.                 if (ns != null)
    53.                     builder.Append(ns).Append('.');
    54.  
    55.                 if (type.IsNested)
    56.                 {
    57.                     using var _ = ListPool<Type>.Get(out var list);
    58.                     Type temp = type;
    59.                     do
    60.                         list.Add(temp = temp.DeclaringType);
    61.                     while (temp.IsNested);
    62.  
    63.                     for (int i = list.Count - 1; i >= 0; i--)
    64.                     {
    65.                         builder.Append(list[i].Name).Append('+');
    66.                     }
    67.                 }
    68.                 builder.Append(type.Name);
    69.  
    70.                 var args = type.GenericTypeArguments;
    71.                 for (int i = 0; i < args.Length; i++)
    72.                 {
    73.                     builder.Append(i == 0 ? "[[" : ",[");
    74.                     using (Build(args[i], out var argBuilder))
    75.                         builder.Append(argBuilder).Append(']');
    76.                 }
    77.                 builder.Append(']');
    78.             }
    79.             else
    80.             {
    81.                 builder.Append(type.FullName);
    82.             }
    83.  
    84.             builder.Append(", ").Append(type.Assembly.GetName().Name);
    85.             return result;
    86.         }
    87.     }
    88. }
    89.  

    This is the
    StringBuilderPool
    used above.
    Code (CSharp):
    1. #nullable enable
    2.  
    3. using System.Text;
    4.  
    5. namespace UnityEngine.Pool
    6. {
    7.     public interface IPoolCreator<TObject> where TObject : class
    8.     {
    9.         // Todo: abstract static.
    10.         ObjectPool<TObject> CreatePool();
    11.     }
    12.  
    13.     public class SharedPool<TObject, TPoolCreator>
    14.         where TObject : class
    15.         where TPoolCreator : IPoolCreator<TObject>, new()
    16.     {
    17.         static readonly ObjectPool<TObject> s_pool = new TPoolCreator().CreatePool();
    18.  
    19.         public static int CountAll => s_pool.CountAll;
    20.         public static int CountActive => s_pool.CountActive;
    21.         public static int CountInactive => s_pool.CountInactive;
    22.  
    23.         /// <summary>
    24.         /// Get a new instance from the pool.
    25.         /// </summary>
    26.         /// <returns></returns>
    27.         public static TObject Get() => s_pool.Get();
    28.  
    29.         /// <summary>
    30.         /// Get a new instance and a PooledObject. The PooledObject will automatically return the instance when it is Disposed.
    31.         /// </summary>
    32.         /// <param name="value">Output new instance.</param>
    33.         /// <returns>A new PooledObject.</returns>
    34.         public static PooledObject<TObject> Get(out TObject value) => s_pool.Get(out value);
    35.  
    36.         /// <summary>
    37.         /// Release an object to the pool.
    38.         /// </summary>
    39.         /// <param name="toRelease">instance to release.</param>
    40.         public static void Release(TObject toRelease) => s_pool.Release(toRelease);
    41.  
    42.         public static void Release(ref TObject toRelease)
    43.         {
    44.             s_pool.Release(toRelease);
    45.             toRelease = null!;
    46.         }
    47.  
    48.         /// <summary>
    49.         /// Clear the pool.
    50.         /// </summary>
    51.         public static void Clear() => s_pool.Clear();
    52.     }
    53.  
    54.     public class StringBuilderPool : SharedPool<StringBuilder, StringBuilderPool.Creator>
    55.     {
    56.         public struct Creator : IPoolCreator<StringBuilder>
    57.         {
    58.             public ObjectPool<StringBuilder> CreatePool() => new ObjectPool<StringBuilder>
    59.             (
    60.                 createFunc: () => new(255),
    61.                 actionOnRelease: o => o.Clear()
    62.             );
    63.         }
    64.     }
    65. }

    Test code:
    Code (CSharp):
    1. using System;
    2. using UnityEngine;
    3. using UnityExtensions.Reflection;
    4.  
    5. class A<T> { }
    6.  
    7. namespace Test
    8. {
    9.     struct B
    10.     {
    11.         public class C<T, U>
    12.         {
    13.             public class D
    14.             {
    15.                 public struct E<V, W>
    16.                 {
    17.                 }
    18.             }
    19.         }
    20.     }
    21.  
    22.     class TestComponent : MonoBehaviour
    23.     {
    24.         [UnityEditor.MenuItem("Tools/Type Serialization Test")]
    25.         static void TypeSerializationTest()
    26.         {
    27.             Do(typeof(GameObject));
    28.             Do(typeof(A<bool>));
    29.             Do(typeof(B.C<A<bool>, byte>));
    30.             Do(typeof(B.C<A<bool>, byte>.D));
    31.             Do(typeof(B.C<A<bool>, byte>.D.E<B.C<A<float>, long>, int>));
    32.  
    33.             static void Do(Type type)
    34.             {
    35.                 var text = TypeSerialization.Serialize(type);
    36.                 Debug.Log(text);
    37.                 Debug.Log(type.AssemblyQualifiedName);
    38.                 Debug.Log(TypeSerialization.Deserialize(text) == type);
    39.             }
    40.         }
    41.     }
    42. }
     
    Ice_106 likes this.
  4. yu_yang

    yu_yang

    Joined:
    May 3, 2015
    Posts:
    85
    Then, this is the thread-safe version of
    TypeSerialization
    , with the following main changes:
    1. According to Microsoft's documentation, the name of the core assembly can be omitted, so I omit it. Based on my testing, the core assembly names for Unity and .NET are currently different, so omitting it should provide better compatibility.
    2. Used a
      SpinLock
      to ensure thread safety. If there is no contention, the
      SpinLock
      should be able to skip waiting.
    3. Since Unity's object pool is not thread-safe, I changed the usage of
      StringBuilderPool
      to directly create a new
      StringBuilder
      . Considering that each type is executed only once, I think a little bit of GC is acceptable.
    4. Similarly, the use of
      ListPool
      has also been removed. I attempted to use a lightweight stack-allocated list to avoid creating a
      List
      , but failed in the last step because Unity currently cannot create stack-allocated arrays for managed types. I kept this part of the code, and it should be possible to improve it with inline arrays once Unity migrates to .NET 8.
    Code (CSharp):
    1.  
    2. public static class TypeSerialization
    3. {
    4.     static readonly Assembly s_coreAssembly = typeof(object).Assembly;
    5.     static readonly Dictionary<Type, string> s_typeToText = new();
    6.     static readonly Dictionary<string, Type> s_textToType = new();
    7.     static SpinLock s_spinLock = new(false);
    8.  
    9.     public static string Serialize(Type type)
    10.     {
    11.         if (type == null)
    12.             return null;
    13.  
    14.         using var _ = s_spinLock.Take(false);
    15.         if (!s_typeToText.TryGetValue(type, out var text))
    16.         {
    17.             text = Build(type).ToString();
    18.             s_typeToText.Add(type, text);
    19.             s_textToType.TryAdd(text, type);
    20.         }
    21.         return text;
    22.     }
    23.  
    24.     public static Type Deserialize(string text)
    25.     {
    26.         if (text == null)
    27.             return null;
    28.  
    29.         using var _ = s_spinLock.Take(false);
    30.         if (!s_textToType.TryGetValue(text, out var type))
    31.         {
    32.             type = Type.GetType(text);
    33.             s_textToType.Add(text, type);
    34.         }
    35.         return type;
    36.     }
    37.  
    38.     static StringBuilder Build(Type type)
    39.     {
    40.         var builder = new StringBuilder(64);
    41.         if (type.IsGenericType)
    42.         {
    43.             if (type.ContainsGenericParameters)
    44.                 throw new ArgumentException(type.ToString());
    45.  
    46.             string ns = type.Namespace;
    47.             if (ns != null)
    48.                 builder.Append(ns).Append('.');
    49.  
    50.             if (type.IsNested)
    51.             {
    52.                 SpanList<Type> list = (Span<Type>)new Type[16]; // Todo: use InlineArray
    53.                 Type temp = type;
    54.                 do
    55.                     list.Add(temp = temp.DeclaringType);
    56.                 while (temp.IsNested);
    57.  
    58.                 for (int i = list.Count - 1; i >= 0; i--)
    59.                     builder.Append(list[i].Name).Append('+');
    60.             }
    61.             builder.Append(type.Name);
    62.  
    63.             var args = type.GenericTypeArguments;
    64.             for (int i = 0; i < args.Length; i++)
    65.             {
    66.                 builder.Append(i == 0 ? "[[" : ",[");
    67.                 builder.Append(Build(args[i])).Append(']');
    68.             }
    69.             builder.Append(']');
    70.         }
    71.         else
    72.         {
    73.             builder.Append(type.FullName);
    74.         }
    75.  
    76.         var assembly = type.Assembly;
    77.         if (assembly != s_coreAssembly)
    78.             builder.Append(", ").Append(assembly.GetName().Name);
    79.  
    80.         return builder;
    81.     }
    82. }
    83.  

    Code (CSharp):
    1. public ref struct SpinLockTakenScope
    2. {
    3.     Span<SpinLock> _spinlock; // Todo: ref SpinLock
    4.     readonly bool _lockTaken;
    5.     readonly bool _exitUseMemoryBarrier;
    6.  
    7.     public unsafe SpinLockTakenScope(ref SpinLock spinLock, bool exitUseMemoryBarrier = true)
    8.     {
    9.         // if (Unsafe.IsNullRef(ref spinLock))
    10.         //     throw new NullReferenceException();
    11.  
    12.         fixed (SpinLock* p = &spinLock)
    13.             _spinlock = new(p, 1);
    14.         _lockTaken = false;
    15.         _exitUseMemoryBarrier = exitUseMemoryBarrier;
    16.  
    17.         spinLock.Enter(ref _lockTaken);
    18.     }
    19.  
    20.     public void Dispose()
    21.     {
    22.         // if (!Unsafe.IsNullRef(ref _spinlock))
    23.         {
    24.             if (_lockTaken)
    25.                 _spinlock[0].Exit(_exitUseMemoryBarrier);
    26.  
    27.             // _spinlock = ref Unsafe.NullRef<SpinLock>();
    28.         }
    29.     }
    30. }
    31.  
    32. public static class SpinLockExtensions
    33. {
    34.     public static SpinLockTakenScope Take(this ref SpinLock spinLock, bool exitUseMemoryBarrier = true)
    35.         => new(ref spinLock, exitUseMemoryBarrier);
    36. }

    Code (CSharp):
    1. public ref struct SpanList<T>
    2. {
    3.     Span<T> _span;
    4.     int _count;
    5.  
    6.     public readonly int Count => _count;
    7.  
    8.     public ref T this[int index]
    9.     {
    10.         get
    11.         {
    12.             if (index < 0 || index >= _count)
    13.                 throw new IndexOutOfRangeException();
    14.             return ref _span[index];
    15.         }
    16.     }
    17.  
    18.     public SpanList(Span<T> buffer)
    19.     {
    20.         _span = buffer;
    21.         _count = 0;
    22.     }
    23.  
    24.     public void Add(in T item)
    25.     {
    26.         if (_count == _span.Length)
    27.         {
    28.             Span<T> newSpan = new T[_count > 2 ? Math.Min((uint)_count << 1, 0x4000_0000u) : 4];
    29.             _span.CopyTo(newSpan);
    30.             _span = newSpan;
    31.         }
    32.         _span[_count++] = item;
    33.     }
    34.  
    35.     public static implicit operator SpanList<T>(Span<T> buffer) => new(buffer);
    36. }
     
    Last edited: Jan 19, 2024
    Ice_106 and Canijo like this.
  5. Canijo

    Canijo

    Joined:
    Oct 9, 2018
    Posts:
    50
    @martinpa_unity Does
     [GeneratePropertyBagsForTypesQualifiedWith(Type type,..)]
    include implementations of generic Types?.

    Given the types:
    Code (CSharp):
    1. [assembly: GeneratePropertyBagsForTypesQualifiedWith(typeof(IExample)]
    2.  
    3. public interface IExample
    4. {
    5. }
    6. public class Example<T> : IExample
    7. {
    8.  
    9. }
    10.  
    11. public class SomeBehaviour
    12. {
    13.      private Example<float> _floatExample;
    14.      private Example<bool> _boolExample;
    15. }
    Will a
    PropertyBag
    get generated for
    Example<float>
    and
    Example<bool>
    ?

    ____
    Edit: and what about this?
    Code (CSharp):
    1. public class SomeBehaviour<T>
    2. {
    3.      private Example<T> _genericExample;
    4. }
    5.  
    6. public class SomeProgram
    7. {
    8.      private SomeBehaviour<float> _floatBehaviour;
    9. }
    ___
    Edit2: even worse.. this?
    Code (CSharp):
    1. public class SomeProgram
    2. {
    3.      public Example<T> CreateExample<T>()
    4.      {
    5.          return new Example<T>();
    6.      }
    7.  
    8.  
    9.     public void DoSomething()
    10.     {
    11.             CreateExample<bool>();
    12.             CreateExample<float>();
    13.     }
    14. }
     
    Last edited: Jan 30, 2024
  6. martinpa_unity

    martinpa_unity

    Unity Technologies

    Joined:
    Oct 18, 2017
    Posts:
    484
    Hi @Canijo,

    If I recall, the
    [GeneratePropertyBagsForTypesQualifiedWith(Type type,..)]
    will generate the types that implement that provided interface, but it won't look for usages of that type. In the case of generic types, we don't currently support that concept, so it should only work for derived types that close the generic parameters.

    So, in your example, I believe that none of the closed types of Example will be generated. However, if you tag
    SomeProgram
    , then
    SomeBehaviour<float>
    will be generated.

    You can also generate the property bag for a given generic type using:
    [assembly: GeneratePropertyBagsForType(typeof(Example<float>))]


    This would also work:

    Code (CSharp):
    1. [assembly: GeneratePropertyBagsForTypesQualifiedWith(typeof(IExample))]
    2.  
    3. public interface IExample
    4. {
    5. }
    6.  
    7. // Nothing generated
    8. public class Example<T> : IExample
    9. {
    10. }
    11.  
    12. // Generated
    13. public class StringExample : Example<string>
    14. {
    15.    
    16. }
    Hope this helps!
     
    Canijo likes this.
  7. Canijo

    Canijo

    Joined:
    Oct 9, 2018
    Posts:
    50
    @martinpa_unity thanks for the clarification. I will resort to writing a SourceGenerator that makes a deep scan of my custom marked classes and write a file with all the required [GeneratePropertyBagsForType] for generic constructed types.

    But from what you've said about "SomeProgram", does it mean that if a Type that has a PropertyBag generated, when it gets scanned, a field/property declared in that Type with a constructed generic might trigger the generation of that property/field Type's PropertyBag ?

    will something like this work? :

    Code (CSharp):
    1. [assembly: GeneratePropertyBagsForTypesQualifiedWith(typeof(IExample))]
    2. [assembly: GeneratePropertyBagsForTypesQualifiedWith(typeof(IBehaviour))]
    3.  
    4. // no property bag
    5. public class Example<T> : IExample
    6. {
    7. }
    8.  
    9. // has property bag
    10. public class StringExample : Example<string>
    11. {
    12. }
    13.  
    14. // has property bag
    15. public class Behaviour : IBehaviour
    16. {
    17.      private Example<float> _floatExample;
    18.      private Example<bool> _boolExample;
    19. }
    20. /// Will a bag be generated for Type "Example<float>" and "Example<bool>" ?
    21.  
     
    Last edited: Jan 30, 2024
  8. martinpa_unity

    martinpa_unity

    Unity Technologies

    Joined:
    Oct 18, 2017
    Posts:
    484
    Yes, it should, because the
    Behaviour
    type will be generated.

    I'm not entirely sure if each source generator will see the output of the others, but it's worth a try.
     
    Canijo likes this.
  9. Canijo

    Canijo

    Joined:
    Oct 9, 2018
    Posts:
    50
    I believe that with Incremental Source Generators, when SourceText its added, another Compilation is triggered, calling the SyntaxProviders again. But i will research this =) Thanks!
     
    martinpa_unity likes this.
  10. Canijo

    Canijo

    Joined:
    Oct 9, 2018
    Posts:
    50
    Nope, from the SourceGenerators Docs:

    An implementation for this type of use case is in progress, but with no clear date nor design yet, so no luck with that approach.

    In the Editor i have no problem thanks to the ReflectedPropertyBag, and many of the uses for it are actually just for Editor-only code.

    For types i need at runtime I had a BuildPlayerProcessor that i used to scan some custom serialized types and produce a file with the [GeneratePropertyBagForType] attribute. I will place this same logic there, though i have not yet tested if a script produced from the "PrepareForBuild(BuildPlayerContext)" (and then added directly as AssetDatabase.ImportAsset) will be compiled in the build. But i think this kind of approach is the most comfortable for now.

    I've also tested the examples from the previous post and they sure work, so all Ok =)
     
    martinpa_unity likes this.
  11. Canijo

    Canijo

    Joined:
    Oct 9, 2018
    Posts:
    50
    I noted from previous posts that when a Type is marked
    partial
    , the generator will take advantage and avoid creating IL-emit based accessors. However it breaks when that type is generic.

    I believed that maybe the generated bag from one constructed type is overriding another one (or the "base" generic one) ? But it also happens when it only generates a bag for 1 overriden type (
    Test<float>
    for example, with no other existitng
    Test<T>
    's )

    The errors i get:

    When the Type is not marked partial, the bags are propperly generated and everything OK. Just wanted to report it in case is a problem on my end or a fix could be expected in incoming versions.

    Also, do
    PropertyBag
    's for non-partial types with private members work in AOT (il2cpp) ?
     
  12. martinpa_unity

    martinpa_unity

    Unity Technologies

    Joined:
    Oct 18, 2017
    Posts:
    484
    Oh, can you file a bug for this? I don't think it's on your side, so we'll get that fixed.

    it should work as long as the field/property type is not a nested private type.

    When the generated type doesn't have access to a field/property, it will use reflection.
     
    Canijo likes this.
  13. Canijo

    Canijo

    Joined:
    Oct 9, 2018
    Posts:
    50
    Done! ty
     
    martinpa_unity likes this.
  14. martinpa_unity

    martinpa_unity

    Unity Technologies

    Joined:
    Oct 18, 2017
    Posts:
    484
    Hi @Canijo, what is the bug id for this issue? It hasn't found its way to us just yet and I'd like to fast track it if I can.
    Thanks!
     
  15. Canijo

    Canijo

    Joined:
    Oct 9, 2018
    Posts:
    50
    IN-67433, today at around 4 PM (GMT 0) was updated to "In Review"
     
    martinpa_unity likes this.
  16. martinpa_unity

    martinpa_unity

    Unity Technologies

    Joined:
    Oct 18, 2017
    Posts:
    484
    Thank you, I'll keep an eye on it!
     
    Canijo likes this.
  17. Canijo

    Canijo

    Joined:
    Oct 9, 2018
    Posts:
    50
    Upgraded to bug (link)
     
    martinpa_unity likes this.
  18. martinpa_unity

    martinpa_unity

    Unity Technologies

    Joined:
    Oct 18, 2017
    Posts:
    484
    Thank you, I'll start on this next week!
     
    Willkuerlich and Canijo like this.
  19. JohnPontoco

    JohnPontoco

    Joined:
    Dec 23, 2013
    Posts:
    292
    Hey all! Thanks for the lovely package (love the new properties and serialization packages).

    I stumbled on a bug in the parsing code for floating point values in Unity.Serialization.Json. Very small values will serialize but fail to deserialize (fail to round-trip). You can reproduce via the following code:

    Code (CSharp):
    1. public class Test
    2. {
    3.         [MenuItem("Bugs/Demonstrate Bug")]
    4.         public static void Bug()
    5.         {
    6.             float validFloatValue = 1.200001E-39f;
    7.             Debug.Log(validFloatValue);
    8.            
    9.             // Serialize float:
    10.             string json = JsonSerialization.ToJson(validFloatValue);
    11.             Debug.Log(json);
    12.            
    13.             // Throws parse error: [Underflow]
    14.             Debug.Log(JsonSerialization.FromJson<float>(json));
    15.         }
    16. }
    17.  
    Filed as bug IN-75156. It's possible this may be OSX specific, as I just swapped from Windows.