A anti-segmentation C# networking buffer for TCP connections.

I was in the process of writing a client-api for my server when I noticed that C# didn't really have any 'Anti-TCP-segmentation' buffers, so I decided to write one.

Code (CSharp):

1. /// <summary>
2.     /// A custom buffer implementation that will be read by the ogserver-framework to prevent
3.     /// the TCP stack from causing errors due to segmentation. This buffer stores all data
4.     /// inside of a collection and then parses the length of the collection to the network
5.     /// in the form of a 32-bit integer. The server will read the integer and then wait to
6.     /// process any logic until the amount of bytes sent by the buffer has been received by
7.     /// the server.
8.     /// </summary>
9.     public class ByteBuffer
10.     {
11.         List<Byte> buffer;
12.  
13.         private int packetId { get; set; }
14.  
15.         public ByteBuffer(int packetId)
16.         {
17.             this.packetId = packetId;
18.             this.buffer = new List<Byte>();
19.         }
20.  
21.         public void WriteByte(int value)
22.         {
23.             buffer.Add((byte)value);
24.         }
25.  
26.         public void WriteChar(char value)
27.         {
28.             buffer.Add((byte)(((uint)value >> 8) & 0xFF));
29.             buffer.Add((byte)(((uint)value >> 0) & 0xFF));
30.         }
31.  
32.         public void WriteDouble(double value)
33.         {
34.             WriteLong(BitConverter.DoubleToInt64Bits(value));
35.         }
36.  
37.         public void WriteFloat(float value)
38.         {
39.             WriteInt(floatToIntBits(value));
40.         }
41.  
42.        public void WriteShort(int value)
43.         {
44.             buffer.Add((byte)(((ushort)value >> 8) & 0xFF));
45.             buffer.Add((byte)(((ushort)value >> 0) & 0xFF));
46.         }
47.  
48.         public void WriteInt(int value)
49.         {
50.             buffer.Add((byte)(((uint)value >> 24) & 0xFF));
51.             buffer.Add((byte)(((uint)value >> 16) & 0xFF));
52.             buffer.Add((byte)(((uint)value >> 8)  & 0xFF));
53.             buffer.Add((byte)(((uint)value >> 0)  & 0xFF));
54.         }
55.  
56.         public void WriteBoolean(bool value)
57.         {
58.             buffer.Add(value ? (byte)1 : (byte)0);
59.         }
60.  
61.         public void WriteLong(long value)
62.         {
63.             buffer.Add((byte)(((uint)value >> 56) & 0xFF));
64.             buffer.Add((byte)(((uint)value >> 48) & 0xFF));
65.             buffer.Add((byte)(((uint)value >> 40) & 0xFF));
66.             buffer.Add((byte)(((uint)value >> 32) & 0xFF));
67.             buffer.Add((byte)(((uint)value >> 24) & 0xFF));
68.             buffer.Add((byte)(((uint)value >> 16) & 0xFF));
69.             buffer.Add((byte)(((uint)value >> 8)  & 0xFF));
70.             buffer.Add((byte)(((uint)value >> 0)  & 0xFF));
71.         }
72.  
73.         public void WriteString(string value)
74.         {
75.             WriteShort(value.Length);
76.             foreach(Char c in value.ToCharArray())
77.             {
78.                 WriteChar(c);
79.             }
80.         }
81.  
82.         public void WriteVector2(Vector2 value)
83.         {
84.             WriteFloat(value.x);
85.             WriteFloat(value.y);
86.         }
87.  
88.         public void WriteVector3(Vector3 value)
89.         {
90.             WriteFloat(value.x);
91.             WriteFloat(value.y);
92.             WriteFloat(value.z);
93.         }
94.  
95.         private int floatToIntBits(float value)
96.         {
97.             int result = BitConverter.ToInt32(BitConverter.GetBytes(value), 0);
98.             if (((result & 0x7F800000) == 0x7F800000)
99.                 && (result & 0x80000000) != 0)
100.                 result = 0x7fc00000;
101.             return result;
102.         }
103.  
104.         public void Send(BinaryWriter netOut)
105.         {
106.             netOut.Write((int)buffer.Count);
107.             netOut.Write((int)packetId);
108.             netOut.Write(buffer.ToArray());
109.         }
110.  
111.     }

To use this properly on the servers side you must first read the 32-bit integer from the stream, this is the "Header" value and signifies the length of the packet, if 32bits(4bytes) are not available from the stream, you simply skip that networking loop and process it again, repeat until 4bytes have been received.

Once 4bytes have been received, read the integer from the steam, this will determine how many bytes are in the "Buffer".

Next you should keep doing what you did in step one, skip the networking loop until the total amount of bytes available from the network is equal(or greater to) the amount of bytes in the buffer, once you have the proper amount of bytes available in the stream, read and process your networking logic.

This buffer also sends a packet id, make sure to read it after your initial verification code.

An example on the server-side(Java):

Code (CSharp):

1. // Verify that the packet-header is available.
2.         if(buffer.readableBytes() < 4) {
3.             Log.debug(getClass(), "Not enough data read for header, only read: " + buffer.readableBytes() + " / 4");
4.             return;
5.         }
6.                            
7.         // Set the index to the current position, so we no longer attempt
8.         // to read the same information again.
9.         buffer.markReaderIndex();
10.                            
11.         // Read the packet-header and store it into an integer.
12.         int length = buffer.readInt();
13.         Log.debug(getClass(), "Packet length is: " + length);
14.    
15.         // Verify that we have all of the data that was specified in
16.         // the packet header.
17.         if(buffer.readableBytes() < length) {
18.             // If we do not have enough data, reset the current index that
19.             // We attempts to read from, back to the previous index.
20.             buffer.resetReaderIndex();
21.             Log.debug(getClass(), "Read: ("+buffer.readableBytes()+" / " + length + ")");
22.             return;
23.         }
24.  
25.         // Do logic here(IE Handle the packet)
26.         // int packetId = buffer.readInt();

While the buffer.readableBytes() was not included in here(Because it was on the clients side), I'm sure that anybody that can make use of this can figure out how to get the total amount of bytes available on the stream.


Cheers.

 

That a nice simple implementation. A couple thoughts:

• ByteBuffer as presented looks more like a Message class... the buffering appears to happen on the receiving end.
• We have overloading! Consider using Write instead of WriteVector2.
• If you are going to have a ByteBuffer or Message class that sends data, you should have some way of deserializing it into a similar class.

Here's a helpful tool - an extended version of the built-in BinaryReader and BinaryWriter classes:

Code (csharp):

1. public class MyBinaryReader : BinaryReader
2. {
3.     public MyBinaryReader(Stream stream) : base(stream, UTF8Encoding.UTF8) { }
4.     public ulong ReadVint()
5.     {
6.         ulong result = 0;
7.         byte b;
8.         do
9.         {
10.             b = ReadByte();
11.             result <<= 7;
12.             result += (ulong)(b & 127);
13.         } while ((b & 128) != 0);
14.         return result;
15.     }
16.  
17.     public long ReadZigZagVint()
18.     {
19.         var input = ReadVint();
20.         var result = ((long)(input >> 1)) ^ -((long)(input & 1));
21.         return result;
22.     }
23. }
24.  
25. public class MyBinaryWriter : BinaryWriter
26. {
27.     public MyBinaryWriter(Stream stream) : base(stream, UTF8Encoding.UTF8) { }
28.     byte[] buffer = new byte[10];
29.     public void WriteVint(ulong input)
30.     {
31.         int i = 0;
32.         buffer[0] = (byte)(input & 127);
33.         while ((input = (input >> 7)) > 0) buffer[++i] = (byte)((input & 127) ^ 128);
34.         while (i >= 0) Write(buffer[i--]);
35.     }
36.  
37.     public void WriteZigZagVint(long input)
38.     {
39.         ulong output = (ulong)((input << 1) ^ (input >> 63));
40.         WriteVint(output);
41.     }
42. }

 

I'll be the first to admit that I don't have much experience with C# and that I honestly don't like it very much, but when it's required it's required, My Java Client API is much simpler because it can make use of the built-in functionality of the ByteBuf class from Netty. So I'll go ahead and touch on some of your poitns:

True and false, this was only a very basic implementation of what I'm using, in this case you're correct, it's just sending the data to be buffered; However in my version (for a not-yet-released networking API for multiple languages) it's actually used both ways, obviously this requires some changes to the ByteBuffer class. -- All inbound data is stored inside of the ByteBuffer, and it's removed from the Buffer once it's read, bytes are stored in the order that they arrive, I'm sure this could cause issues in the future, so I've been thinking about a safer implementation. (Grouping the bytes by id) in something such as a HashMap<K,V> (I believe this is a Dictionary in C#?).

I'm sorry, I don't quite understand what you're talking about? The WriteVector2 class is mainly just a 'ease of access' method I implemented for Unity, you could just as easily call WriteFloat twice.

I probably understand serialization, but don't understand the definition of it. I'm going on ten years of completely self-taught and haven't ever really cracked open a book for definitions of things. I've been using practices for concurrency(thread-safe applications) for years and I didn't even find out what concurrency meant until about a year ago. Unity still doesn't like my methods of dealing with this though, so for now I guess an update Queue<delegate> will work.

I'll also be honest and say I have absolutely no idea what those class (extensions) are doing, "ReadZigZagVint"... What? Lost me at Zig. I'll be the first to say I don't completely understand bit manipulation, if anything I hardly understand the basics. I can just get by with what I need to do.

 

Don't worry you'll come around, it's like java but better ;P

I found personally that when you get to the stage where you'd like more safety, a proper serialisation and deserialisation library makes life easier - serialising types rather than calling methods on a message/buffer class.

Code (csharp):

1.  
2. public void WriteFloat(float f) {...};
3. public void WriteVector2(Vector2 v2) {...};
4. //Instead do this:
5. public void Write(float f) {...};
6. public void Write(Vector2 v2) {...};
7. //And down the track with Generics
8. public void Write<T>(T value) {...};
9.  

You can't do it for reading - if you want that to make more sense... then Generics are wonderful.

Google is fun!

BinaryReader and BinaryWriter are classes provided by the .NET framework that handle writing and reading to a Stream... and a Byte Array can be easily considered a Stream (wrap it in a MemoryStream)... so that'll handle all the work you're currently doing for free.

On top of that I've implemented a variable length integer - in short it uses a basic encoding scheme to write only as many bytes as necessary - kinda like how UTF8 can handle 32 bit unicode, but usually needs only one byte.

A ZigZag integer is similar, but ever second value is negative (IIRC) e.g. 1, -1, 2, -2, 3, -3 - so that we can encode positive and negative numbers efficiently.

 

Google is the root of all evil

This is the implementation that's used on my server, and client, and the one that I've been slowly moving over the C# during my free time, I doubt that I'll ever 'come around' because I've been a Java junkie since J2SE5 (Somewhere around '04) and I've never enjoyed writing in any of the other languages.

I guess it depends on how you look at it, I'm not serializing the Vector class, instead I'm just writing the contents of the class (The x/y/z coordinates) as primitive data types, the same happens on the server when readVector2/3() is called, it reads the primitive data types.

I noticed the use of

Code (csharp):

1.  
2. [LIST=1]
3. [*]public void Write(float f) {...};
4. [*]public void Write(Vector2 v2) {...};

[/LIST]
Instead of

Code (csharp):

1.  
2. [LIST=1]
3. [*]public void WriteFloat(float f) {...};
4. [*]public void WriteVector2(Vector2 v2) {...};

[/LIST]
in the binary reader/writer class-- but it doesn't seem to have any change in the overall effect. Similar to naming a method:

Code (csharp):

1.  
2. getBy(Channel channel);
3.  

opposed to

Code (csharp):

1.  
2. getByChannel(Channel channel);
3.  

It really just seems like a preference?; However my goal is to make this as simple as possible for the end-users, the little tads of extra typing don't bother me, I teach a game-development class. It's much simpler for people to keep up when they see exactly what's going on, rather than having to think on it.

 

Largely. It's kinda like using var in C# - not only does it reduce the length of code... but means you can make small implementation changes without having to change as much code.

Code (csharp):

1.  
2. Vector3 vector = new Vector3();
3. buffer.WriteVector3(vector);  
4. //Note how Vector3 has been written three times?  
5.  
6. var vector = new Vector3();
7. buffer.Write(vector);  
8. //Notice how it's only written once?
9.  
10. //Let's say we decided that that third value is kinda redundant:
11. var vector = new Vector2();  //only one change!
12. buffer.Write(vector);  
13.  

Again, this is both cool and up to personal preference - some people like to be a little stricter and explicitly type variables while I prefer the keep intent and like being able to make small implementation changes easily.

 

Makes perfect sense, I'll just implement both methods, considering this is for a API --> http://forum.unity3d.com/threads/a-teaser-of-my-free-game-server-api-unity3d-client-api.283635/