UNET Serialization

Good afternoon.

I'd like to ask, is built-in UNET Serialization system of network messages is good for performance and bandwidth or not? I'm talking about HLAPI.

https://docs.unity3d.com/Manual/UNetStateSync.html

Code (CSharp):

1. public virtual bool OnSerialize(NetworkWriter writer, bool initialState);
2. public virtual void OnDeSerialize(NetworkReader reader, bool initialState);

Thanks.

 

Sort of. There isn't any overhead per piece of data (it's just dirty bit + payload). With UNET's network reader/writer an integer consumes 1bit for dirtyMask and 4 bytes for data payload (if dirty). The exception to this is bools consume a dirty bit and 1byte (8bits) when they *should* consume 1 bit (but the reader/writer works in full bytes).

There is some overhead associated with the packet. I am not 100% sure what this contains but I believe it is
2byte: short MessageBase type
1-5byte: NetworkIdentity.id (packed uint I think)
1byte: bool initialState?
+low level packet overhead (unsure how much this is)
And for each NetworkBehaviour script there is a packed dirtybit mask (typically 1 byte, but up to 4bytes at a max of 32 SyncVar's). So there more NetworkBehaviour's you have the more overhead you have.

SyncLists work off a different system I think. They don't use dirty bits and aren't part of OnSerialize/OnDeserialize like syncVars are. Unsure of their overhead (I think they get a MessageBase sent anytime something changes).

 

Such an amazing reply, thank you so much!

 

That's the best explanation out there, I think the current system is very well suited for general usage as we can't assume ranges for values. As for the bool, maybe it's possible to detect all bools that are getting serialized and pack every 8 in a byte with some shifting magic? I'm not sure how the code looks at your end, but it's not big deal when overwriting the serialization logic at my end.

 

What I ended up doing is making a BitWriter/Reader that works in bits instead of bytes. Then as you say it rounds up to the nearest byte before sending the packet. This allows a few things.
1: Bools are written as 1bit instead of 8bits
2: Not limited to 32SyncVars
3: Can read/write float compressed with X # of bits. For 0deg-360deg angles I think I use like 12 bits.
4: Avoids overhead of dirtyBit masks (I have like 30 NetworkBehaviour scripts on the player... so that's 8x30 or 240bits of data overhead anytime it's serialized). Instead if there's 0 syncVars then there's 0bits of dirty overhead. If there's 3 SyncVars then there's 3 bits (instead of the 8-40 bits/script like UNET has).
5: And at a lower level it lets me send data bidirectional on the same object (instead of everything Client --> Server or Server --> Client I can do both.) For example position data is serialized from Client(owner) --> Server --> Clients(other). But other data such as animations are serialized from Server --> Clients(all)

 

Wow, that's pretty awesome, how could I achieve something similar, please, could you share some tips? Thanks.

 

In case someone will be looking for a way to serialize data in the LLAPI, here is my class, you are free to use it.

How to use it:

Code (CSharp):

1. float test = 3.5f;
2. CSerializer serializer = new CSerializer(1500); //call this only once
3.  
4. //now every time you want to serialize use PrepareSerialization() or PrepareDeserialization()
5. serializer.PrepareSerialization();
6. serializer.Serialize(ref test);
7.  
8. //to deserialize use
9. serializer.PrepareDeserialization(data); //data is a byte[] and contains data that you will be deserializing
10.  
11. //your float variable has been serialized to serializer.buffer to take the data size use serializer.dataSize
12. //this class stores 8 bool values in one byte

The code:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. public class CSerializer
6. {
7.     public bool isWriting;
8.     public bool isReading;
9.  
10.     public int dataSize;
11.  
12.     public int bufferSize;
13.     public byte[] buffer;
14.  
15.     public byte[] data;
16.  
17.     private byte[] bytesTmp;
18.     private System.Text.Encoding encoding;
19.  
20.     private int boolIndex;
21.     private int boolBitIndex;
22.  
23.     private bool boolHelper;
24.     private byte byteHelper;
25.     private char charHelper;
26.     private short shortHelper;
27.     private ushort ushortHelper;
28.     private int intHelper;
29.     private uint uintHelper;
30.     private float floatHelper;
31.     private double doubleHelper;
32.     private string stringHelper;
33.  
34.     public CSerializer(int bufferSize)
35.     {
36.         PrepareSerialization();
37.  
38.         this.bufferSize = bufferSize;
39.         buffer = new byte[bufferSize];
40.  
41.         encoding = System.Text.Encoding.GetEncoding("Unicode");
42.     }
43.  
44.     public void PrepareSerialization()
45.     {
46.         dataSize = 0;
47.         isWriting = true;
48.         isReading = false;
49.  
50.         boolIndex = -1;
51.         boolBitIndex = 0;
52.     }
53.  
54.     public void PrepareDeserialization(byte[] data)
55.     {
56.         this.data = data;
57.         dataSize = 0;
58.         isWriting = false;
59.         isReading = true;
60.  
61.         boolIndex = -1;
62.         boolBitIndex = 0;
63.     }
64.  
65.     public void Serialize(ref object variable)
66.     {
67.         if (isWriting)
68.         {
69.             switch (System.Type.GetTypeCode(variable.GetType()))
70.             {
71.                 case System.TypeCode.Boolean:
72.                     boolHelper = (bool)variable;
73.                     Serialize(ref boolHelper);
74.                     break;
75.  
76.                 case System.TypeCode.Byte:
77.                     byteHelper = (byte)variable;
78.                     Serialize(ref byteHelper);
79.                     break;
80.  
81.                 case System.TypeCode.Char:
82.                     charHelper = (char)variable;
83.                     Serialize(ref charHelper);
84.                     break;
85.  
86.                 case System.TypeCode.Int16:
87.                     shortHelper = (short)variable;
88.                     Serialize(ref shortHelper);
89.                     break;
90.  
91.                 case System.TypeCode.UInt16:
92.                     ushortHelper = (ushort)variable;
93.                     Serialize(ref ushortHelper);
94.                     break;
95.  
96.                 case System.TypeCode.Int32:
97.                     intHelper = (int)variable;
98.                     Serialize(ref intHelper);
99.                     break;
100.  
101.                 case System.TypeCode.UInt32:
102.                     uintHelper = (uint)variable;
103.                     Serialize(ref uintHelper);
104.                     break;
105.  
106.                 case System.TypeCode.Single:
107.                     floatHelper = (float)variable;
108.                     Serialize(ref floatHelper);
109.                     break;
110.  
111.                 case System.TypeCode.Double:
112.                     doubleHelper = (double)variable;
113.                     Serialize(ref doubleHelper);
114.                     break;
115.  
116.                 case System.TypeCode.String:
117.                     stringHelper = (string)variable;
118.                     Serialize(ref stringHelper);
119.                     break;
120.             }
121.         }
122.     }
123.  
124.  
125.     public void Serialize(ref bool variable)
126.     {
127.         if (isWriting)
128.         {
129.             if (boolIndex == -1 || boolBitIndex == 7)
130.             {
131.                 boolIndex = dataSize++;
132.                 boolBitIndex = 0;
133.  
134.                 buffer[boolIndex] = 0;
135.             }
136.  
137.             if (variable)
138.                 buffer[boolIndex] |= (byte)(1 << boolBitIndex++);
139.             else
140.                 boolBitIndex++;
141.         }
142.         else
143.         {
144.             if (boolIndex == -1 || boolBitIndex == 7)
145.             {
146.                 boolIndex = dataSize++;
147.                 boolBitIndex = 0;
148.             }
149.  
150.             variable = ((data[boolIndex] & (1 << boolBitIndex++)) == 0) ? false : true;
151.         }
152.     }
153.  
154.     public void Serialize(ref byte variable)
155.     {
156.         if (isWriting)
157.         {
158.             buffer[dataSize++] = variable;
159.         }
160.         else
161.         {
162.             variable = data[dataSize++];
163.         }
164.     }
165.  
166.     public void Serialize(ref char variable)
167.     {
168.         if (isWriting)
169.         {
170.             buffer[dataSize++] = (byte)variable;
171.         }
172.         else
173.         {
174.             variable = (char)data[dataSize++];
175.         }
176.     }
177.  
178.     public void Serialize(ref short variable)
179.     {
180.         if (isWriting)
181.         {
182.             buffer[dataSize++] = (byte)((variable & 0xFF00) >> 8);
183.             buffer[dataSize++] = (byte)(variable & 0x00FF);
184.         }
185.         else
186.         {
187.             variable = (short)(((short)(data[dataSize++])) << 8);
188.             variable |= (short)data[dataSize++];
189.         }
190.     }
191.  
192.     public void Serialize(ref ushort variable)
193.     {
194.         if (isWriting)
195.         {
196.             buffer[dataSize++] = (byte)((variable & 0xFF00) >> 8);
197.             buffer[dataSize++] = (byte)(variable & 0x00FF);
198.         }
199.         else
200.         {
201.             variable = (ushort)(((ushort)(data[dataSize++])) << 8);
202.             variable |= (ushort)data[dataSize++];
203.         }
204.     }
205.  
206.     public void Serialize(ref int variable)
207.     {
208.         if (isWriting)
209.         {
210.             buffer[dataSize++] = (byte)((variable & 0xFF000000) >> 24);
211.             buffer[dataSize++] = (byte)((variable & 0x00FF0000) >> 16);
212.             buffer[dataSize++] = (byte)((variable & 0x0000FF00) >> 8);
213.             buffer[dataSize++] = (byte)(variable & 0x000000FF);
214.         }
215.         else
216.         {
217.             variable = (int)((data[dataSize++]) << 24);
218.             variable |= (int)((data[dataSize++]) << 16);
219.             variable |= (int)((data[dataSize++]) << 8);
220.             variable |= (int)data[dataSize++];
221.         }
222.     }
223.  
224.     public void Serialize(ref uint variable)
225.     {
226.         if (isWriting)
227.         {
228.             buffer[dataSize++] = (byte)((variable & 0xFF000000) >> 24);
229.             buffer[dataSize++] = (byte)((variable & 0x00FF0000) >> 16);
230.             buffer[dataSize++] = (byte)((variable & 0x0000FF00) >> 8);
231.             buffer[dataSize++] = (byte)(variable & 0x000000FF);
232.         }
233.         else
234.         {
235.             variable = (uint)((data[dataSize++]) << 24);
236.             variable |= (uint)((data[dataSize++]) << 16);
237.             variable |= (uint)((data[dataSize++]) << 8);
238.             variable |= (uint)data[dataSize++];
239.         }
240.     }
241.  
242.  
243.     public void Serialize(ref float variable)
244.     {
245.         if (isWriting)
246.         {
247.             bytesTmp = System.BitConverter.GetBytes(variable);
248.  
249.             buffer[dataSize++] = bytesTmp[0];
250.             buffer[dataSize++] = bytesTmp[1];
251.             buffer[dataSize++] = bytesTmp[2];
252.             buffer[dataSize++] = bytesTmp[3];
253.         }
254.         else
255.         {
256.             variable = System.BitConverter.ToSingle(data, dataSize);
257.             dataSize += 4;
258.         }
259.     }
260.  
261.     public void Serialize(ref double variable)
262.     {
263.         if (isWriting)
264.         {
265.             bytesTmp = System.BitConverter.GetBytes(variable);
266.  
267.             buffer[dataSize++] = bytesTmp[0];
268.             buffer[dataSize++] = bytesTmp[1];
269.             buffer[dataSize++] = bytesTmp[2];
270.             buffer[dataSize++] = bytesTmp[3];
271.             buffer[dataSize++] = bytesTmp[4];
272.             buffer[dataSize++] = bytesTmp[5];
273.             buffer[dataSize++] = bytesTmp[6];
274.             buffer[dataSize++] = bytesTmp[7];
275.         }
276.         else
277.         {
278.             variable = System.BitConverter.ToDouble(data, dataSize);
279.             dataSize += 8;
280.         }
281.     }
282.  
283.     public void Serialize(ref string variable)
284.     {
285.         if (isWriting)
286.         {
287.             bytesTmp = System.BitConverter.GetBytes(System.Convert.ToUInt16(variable.Length));
288.  
289.             buffer[dataSize++] = bytesTmp[0];
290.             buffer[dataSize++] = bytesTmp[1];
291.  
292.          
293.             if (variable.Length > 0)
294.             {
295.                 bytesTmp = encoding.GetBytes(variable);
296.  
297.                 for (int i = -1, len = variable.Length * 2; ++i < len;)
298.                 {
299.                     buffer[dataSize++] = bytesTmp[i];
300.                 }
301.             }
302.  
303.         }
304.         else
305.         {
306.             ushort length = System.BitConverter.ToUInt16(data, dataSize);
307.             dataSize += 2;
308.  
309.          
310.             if (length > 0)
311.             {
312.                 variable = encoding.GetString(data, dataSize, length * 2);
313.                 dataSize += length * 2;
314.             }
315.             else
316.                 variable = "";
317.         }
318.     }
319. }

 

It took me a long time to do. I ended up using MessageBase for everything except the initialState data.

But basically every script has a flag... it's either owned by server or client (as opposed to the entire object being owned by server or client). This is the fundamental change that allows for bi-directional communication on the same object . Action commands and position are Client(owner) --> Server --> Client(other) and most other things are Server --> Client(all) such as stats, animations, audio, etc.

I use MessageBase to send everything to commonize code. So I can use the same code sending from Server --> Client as from Client --> Server.

The tricky part is stuff like hero position. The server instantiates a new hero for a newly connected client. The server sets the initial position of the hero. So the initial position data is Server --> Client(all). Then the player object is spawned on the owner client and it is assigned control of the position. Position script then sends data from Client(owner) --> Server --> Client(other). I do this by when the server receives a MessageBase from a client it forwards the data to other clients (cheat proofing could be placed here if you wanted). If a new client connects it gets the latest data from the Server (since the owner client cannot send directly to the new client). For monsters that are always owned by the server it's much simpler because all position data is always Server --> Client(all).

I had to bandaid a lot of UNET bugs to get this to work. For example UNET re-spawns the same object repeatedly each scene change. I had to move away from RPC/Command because they aren't robust and aren't always delivered due to NetworkIdentity.observers dropping off. And another tricky bug is when a new client connects all data is pushed to the new client but existing clients do not receive any dirty data. This bug causes SyncVars to desync between clients. You must push all data to existing clients and clear dirty bits before sending full data to the newly connected client to prevent de-syncing.

There were so many bugs I have since abandoned UNET (it's so unrobust and it seems it's not really supported anymore? It seems like just 1 guy working on it.) and have moved to Steamworks.NET but I'm using a similar network strategy as above.

 

I have updated my Serialization class in the post above, there was a bug in the bool serialization code, now it's working fine.

 

Please give a real example of use with unet.