Discussion State of binary serialization

At some point Unity is going to switch to .NET 5+

I have found serializing byte arrays and writing to files very useful. The .NET BinaryFormatter is going to be removed. DOTS StreamBinaryWriter is already removed from the public API. StreamBinaryWriter relied on System.IO.Stream. The timeline for remove is below.


NET 7 (Nov 2022)

References to BinaryFormatter APIs marked obsolete as warnings in .NET 5 now result in build errors

.NET 8 (Nov 2023)

Entirety of legacy serialization infrastructure marked obsolete as warning

.NET 9 (Nov 2024)

BinaryFormatter infrastructure removed from .NET. Back-compat switches also removed


Will there be a unmanaged binary serializer for DOTS and .NET? It is kind of important for deterministic serialization. Also, what are people using for Binary serialization going forward. Obviously there exist third party or self-rolled options.

 

Currently using

public static void Write(BlobBuilder builder, string path, int version)

in Blob.cs to write out a BlobBuilder to a file. And checking it’s inheritance from StreamBinaryWriter, BinaryWriter, FileStream etc, I cannot see how deprecation of BinaryFormatter is going to affect it…could you clarify?

 

It should not effect BlobBuilder. How are you saving to a file? StreamBinaryReader/StreamBinaryWriter are already deprecated. Those used System.IO.Stream and System.IO.File.

 

I use the Write method mentioned above in Unity.Entities/Blob.cs

 

BlobAssetReference<T>.Write(BlobBuilder builder, string path, int version)

uses StreamBinaryWriter internally.

 

I know, just do not understand how it’s affected by BinaryFormatter deprecation? System.IO.File and FileStream etc ist not going anywhere, is it?

 

I am not 100% sure but it sounds like legacy binary infrastructure is going away in .NET 9. FileStream goes away StreamWriter comes in and everything becomes JSON text based.

 

As far as I understand Microsoft’s information on this matter (and here) only BinaryFormatter is going away and Dots Serialization is not using it. FileStream cannot go away…how should any file be accessed? Also JSON files need to be opened and the stream parsed. In fact Microsoft just proudly talked about having made it faster. But will stay quiet now and let some of the experts on serialization chime in (@tertle?)

 

Deprecation of BinaryFormatter is definitely one of those things that will break a lot of projects outside of DOTS.

Biggest advantage of it was zero memory allocation of what has to be serialized, and decent speed of serialization / deserialization. Plus versioning and pretty much any layout without extra code.

Instead of fixing security issues, MS decided to just dump binary serialization.
And replacement for it is... JSON / XML which is pathetic. With an alternative to hand over all responsibility to third-party packages.

So now there are a few options:
- Use third-party solution - MessagePack, Protobuf, GroBuf (requires manual layout or crappy versioning support);
- BSON (pretty bad performance wise);
- JSON (awful GC wise, any package);

So basically, there's no good solution that:
- Does not require extra code;
- Supports deep / complicated object hierarchies;
- Performant;
- No GC;
- Version / data change tolerant;

Future is grim.

Also, this might be useful:
https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/

Would it make sense to develop custom serialization solution for the DOTS? Maybe.
Would it be usable outside of DOTS? Probably not.

 

I ended up deciding to use a custom serialization format similar to my net code serialization for my pathfinding. Original I was using the BinaryFormatter.

 

To be clear, DOTS does have a custom binary serialization system with the Unity.Serialization package, right? I'm not sure how thorough it is, but it's used for ECS and several other things.

 

Not sure what you are referring to.

 

https://docs.unity3d.com/Packages/com.unity.serialization@1.4/manual/index.html

It's a bit hidden. It's a package used by SerializeUtility under the hood to serialize Subscenes to binary data for ECS. As far as I can tell it's a very general serialization system, and even supports migrations!

 

Is there an example how to use it with binary serialization?
Documentation seems to be work in progress.


Edit: Best I could find is tests, which contain something like this:

Code (CSharp):

1.  
2.             public unsafe void SerializeAndDeserialize_PolymorphicUnityEngineObject_WithUserDefinedAdapter()
3.             {
4.                 var container = ScriptableObject.CreateInstance<TestObject>();
5.                 var adapter = new UnityObjectAdapter();
6.  
7.                 try
8.                 {
9.                     var parameters = new BinarySerializationParameters
10.                     {
11.                         UserDefinedAdapters = new List<IBinaryAdapter> { adapter }
12.                     };
13.  
14.                     using (var stream = new UnsafeAppendBuffer(16, 8, Allocator.Temp))
15.                     {
16.                         BinarySerialization.ToBinary<object>(&stream, container, parameters);
17.                        
18.                         var reader = stream.AsReader();
19.                         BinarySerialization.FromBinary<object>(&reader, parameters);
20.                     }
21.                    
22.                     Assert.AreEqual(adapter.SerializeCount, 1);
23.                     Assert.AreEqual(adapter.DeserializeCount, 1);
24.                 }
25.                 finally
26.                 {
27.                     Object.DestroyImmediate(container);
28.                 }
29.             }

However, I can't find FileStream example or figure out how to convert default C# .Net stream to the native stream.

 

By the way, I've managed to find a decent serializer that covers our needs:
https://github.com/Cysharp/MemoryPack

This is basically MessagePack, but even faster, and it does not require third party tools to generate schema.
It uses Roslyn source generators, and works pretty great.

 

There aren't any official examples. The best you can do is fish around the code with IDE tools and work backwards from the source code. It's used in SerializeUtility, so there are some examples there, but they're all rather advanced. The tests are honestly the best bet, since they should at least be independent.

I have no clue how to convert a C# .NET stream to the stream it expects, but I do know SerializeUtility first reads a file directly into memory, and only then makes a stream out of it. You should also do a 'what implements this interface' query in your IDE for the stream type / interface it's using.

 

Lastly, try Githubs new code search! cs.github.com has been great for digging up arcane repositories using these APIs. Sometimes it's useful (although not always).