Are there any serialization methods that don't have massive amounts of overhead?

I'm currently working on a mobile multiplayer game with custom client/server implementation. I got tired of dealing with dependencies and bugs so I'm writing my own using C# websockets. (It's actually a lot easier than I thought. The sample code for websockets from the MSDN reference only needs to be tweaked a little to get it running with unity.)

Does anyone know of a way to serialize data that doesn't involve a massive amount of overhead?

I have a class made of 2 vector2s, a float, and a bool.

When I use binary formatter and memory stream its over 300 bytes.

When I use binary writer and memory stream, its 21 bytes.

Using binary writer means I have to manually populate a byte array with primitives before sending it which is a bit of a headache. Anyone know a better method to serialize data without incurring overhead?

 

https://github.com/neuecc/MessagePack-CSharp
MsgPack is used by many people and is the fastest around

https://github.com/rikimaru0345/Ceras
Ceras has lower overhead and is easier to use

Json is very slow and produces huge outputs (purely by comparison with binary-serializers, json it's designed for totally different use cases!)
But still very good and very valuable for testing and debugging (as long as you program it right and are thus able to easily swap out your serializer) Most of the time I start my projects with Json because it is so convenient.
https://github.com/JamesNK/Newtonsoft.Json

There's also flatbuffers and others, but those are complicated to use and don't deal with inheritance properly, can't have nested objects, ...

 

I generally use BitConverter and Array.Copy, but it is a manual process, and would result in the same 21 bytes as BinaryWriter + memory stream.

 

https://wobesdev.wordpress.com/2017/12/02/serialization-stuff/

 

Variable length integer encoding is the defacto standard for space optimization for integers both in and out of games. And it's easy to use integers for most everything or at least convert to/from over the wire, so this is pretty much what everyone working on realtime games professionally does.

Protocol buffers is the main format that uses VLE, and https://github.com/mgravell/protobuf-net is the best C# library for it if you want something out of the box. Protobuf-net does allocate a bit, but it's author is aggressive in optimization and you won't find an off the shelf library that allocates less that uses VLE.

 

Flat buffers is allocation free FYI, but it doesn't use VLE, it's much more heavyweight over the wire by far. So while it allocates less and is faster, it's by very small margins. Not enough to make up for the extra size on the wire IMO.

Also, websockets is a very heavy protocol, it's UTF8 for christ's sake. Personally I think it's broken by design. Sure it's easy to use but if you care about performance avoid it.

 

A custom binary format designed for networked games or your game specifically is the best

 

Depends on what the goal is. I have a 'protobuf-light' library but the reason it exists is to remove all allocation. It uses the same variable length encoding everyone else is. Space efficiency I'd love to hear about something better then variable length encoding. Pretty much anything can be reduced to protobuf packed integer arrays, and you really can't do better then that space wise.

 

You can do a lot better than that with float quantization and integer range compression, also having the protocol itself defined in code instead of having to write type information and such into the binary stream. Protobuf is mediocre at best when it comes to the size of the compressed data.

 

Well if you are sending floats you lost before you started in most cases. Range compression won't make up for not converting those to integers, assuming you can get away with two points of precision, which you can in almost all cases.

Range compression takes high density to work out to any real significant gain. When measured against good packing and using a small handful of higher level techniques to reduce big chunks right off the top. Now if you don't utilize higher level approaches well, then range compression will naturally yield bigger returns. I prefer reducing at higher levels when possible because it reduces the overall cost better.

 

What is the average compression rate of using variable length encoding on the whole stream? I assume you just pack the data as is and then run a compressor on top? Or maybe I'm missing something.

In all the tests I have done using a custom bitpacked format is by far the best option with a compression rate of around 75% vs just writing the data into the stream using a generic serializer.

Maybe I'm missing something in what you are saying, care to elaborate on what you specifically do?

 

Protobuf packed/repeated format writes the type once for the entire array, not per value.

How much you gain depends on the size of your ints to start with. So on average VLE takes two bytes to encode most integers you are sending in games. If you delta encode beforehand, where you only send what changed, it goes down to 1 byte per int assuming two points of precision.

Range encoding can't get you much over that. Nothing will fit into one byte (assuming no delta encoding), and most everything fits into two. If you start bit packing you can gain some, but at that point IMO delta encoding is the far simpler solution, and once you do that, range encoding gets you nothing really.

 

At a higher level I don't send whole complex types. It's all arrays of integers. So an int2 is packed into an array of x and an array of y.

So that's one byte overhead for container type, one byte overhead for each array for it's type.

On my own protobuf-light library, I use a VLE encoded Bitvector32 to hold field types. It allows me to not have to use the packed format, I can have a complex type with up to 30 fields with the same space efficiency of packed/repeated. But that's mainly just for ease of use so I don't have to do all the iteration on each end to assemble/disassemble the complex types.

 

Wow. Hell of a lot of stuff in this thread. Some great resources. However I don't quite think my use case qualifies to use anything that was posted on this thread. I'm looking for barebones minimum complexity (I'm not knocking it... there's some good information here and I didn't expect a response like this so I didn't explain my use case.) However after reading through all the links and getting my learn on, I stumbled across this https://en.wikipedia.org/wiki/Variable-length_quantity which gave me the idea that made it very simple to serialize-deserialize using memoryStream and binary writer/reader.

My use case for this is a 2-4 player turn-based / live action game. So I'm essentially streaming a recording of the player state when its their turn.

Each frame state only requires sending 21 bytes of data, so I'm just writing the 5 floats (broke down the two vector2s) and bool in a specific order to a byte[1024]. The 22nd bit says whether the message continues or ends. Then it writes the next frame at the 23-42 and so on... After the byte[] is full, ship it off to the server. I use binary reader on the other end to put the state back together (far less painful than I thought). At 60fps with no other optimization (I can probably take out 1/2 of the data and interpolate without anyone being able to tell) it's 1.3kb / second / player. And since the nature of the game is turn based, I can buffer the 'live' action as long as I need to. Pretty cool setup... Server code is about 100 lines. Tested on EC2 instance and viola, my own unity-streaming relay server.

 

As an alternative to ZigZag, the MessagePack has fixint, fixstr, and so on for efficient data encoding. I'm using bounded floats for position, half precision floats for linear and angular speed, smallest three for rotation, and all these stuff is packed into integers.

 

https://github.com/MidLevel/MLAPI/b...tworkingManagerComponents/Binary/BitWriter.cs

Here is a nice collection we use. Contains Ranged floats, VarInt with ZigZag, and diffed arrays.