(Netcode For Gameobjects) Getting "Reading past the end of the buffer" when using Custom Messaging

I appear to be misunderstanding how to use Custom Messaging. Allow me to explain fully in case someone can help with the bigger picture.

I am attempting to send a data representation of a tilemap which looks something like this:

Code (CSharp):

1. tilemap {
2.     chunks {
3.         vector2int
4.         int[] // usually 16x16, int is mapped to a true TileBase client-side...
5.     }
6. }

I cannot just send a seed from the server to client because sometimes there are entire player generated, though sometimes procedural.

I have opted to send a state for the tilemap upon joining as a client and keep up-to-date by sending subsequent updates afterwards. I have has some considerable issues getting to this point but have ultimately decided that none of the prepared tools would work so a custom message was in order.

My code looks something like this:

Code (CSharp):

1. Debug.Log("Sending tilemap!");
2. // Note: TileMapChunk class extends INetworkSerializable
3. TileMapChunk[] chunks = tilemapData.TileMapChunks.Values.ToArray();
4. FastBufferWriter writer = new FastBufferWriter(["No idea what to put here"], Allocator.Temp, ["No idea what to put here, either."]);
5. networkManager.CustomMessagingManager.SendNamedMessage("ClientTilemapSync", clientID, writer, NetworkDelivery.Reliable);

This is where I reach the limitations of my current knowledge and the documentation doesn't help much.

This seems to send fine, but I am getting the following message client-side:

Code (CSharp):

1. OverflowException: Reading past the end of the buffer
2. Unity.Netcode.FastBufferReader.ReadNetworkSerializable[T] (T[]& value) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Serialization/FastBufferReader.cs:410)

So my question is, does anyone have a suggestion on how to prevent the overflow exception or a better alternative to sending these tilemaps?

 

I've not looked at custom messages so your post was a good excuse to give them a go.

It might have been helpful to see the code that produced the error, along with where you registered the message handler. In any event I've got something to work that's hopefully similar to what you're after.

Code (CSharp):

1. public class CustomMessageTestService : NetworkBehaviour
2. {
3.     const int OVERHEAD = 12;
4.  
5.     private void Start()
6.     {
7.         Debug.Log("Register message handler");
8.  
9.         NetworkManager.Singleton.CustomMessagingManager.RegisterNamedMessageHandler("myMessageName", MessageReader);
10.     }
11.  
12.     public void SendAMessage()
13.     {
14.         Debug.Log("Send Message...");
15.  
16.         TilemapChunk tilemapChunk = new TilemapChunk(new Vector2Int(3,6), new int[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 });
17.  
18.         FastBufferWriter writer = new FastBufferWriter(Marshal.SizeOf(tilemapChunk) + OVERHEAD, Allocator.Temp);
19.  
20.         Debug.Log("FastBufferWriter length: " + Marshal.SizeOf(tilemapChunk));
21.  
22.         writer.WriteNetworkSerializable<TilemapChunk>(tilemapChunk);
23.  
24.         NetworkManager.Singleton.CustomMessagingManager.SendNamedMessage("myMessageName", 1, writer);
25.     }
26.  
27.  
28.     private void MessageReader(ulong senderClientId, FastBufferReader reader)
29.     {
30.         Debug.Log("FastBufferReader length: " + reader.Length);
31.  
32.         reader.ReadNetworkSerializable<TilemapChunk>(out TilemapChunk tilemapChunk);
33.  
34.         Debug.Log("Received TilemapChunk: " + tilemapChunk);
35.     }
36. }

Code (CSharp):

1. public struct TilemapChunk : INetworkSerializable
2. {
3.     Vector2Int position;
4.  
5.     [MarshalAs(UnmanagedType.ByValArray, SizeConst = 16)]
6.     int[] chunk;
7.  
8.     public TilemapChunk(Vector2Int position, int[] chunk)
9.     {
10.         this.position = position;
11.         this.chunk = chunk;
12.     }
13.  
14.     public void NetworkSerialize<T>(BufferSerializer<T> serializer) where T : IReaderWriter
15.     {
16.         serializer.SerializeValue(ref position);
17.         serializer.SerializeValue(ref chunk);
18.     }
19.  
20.     public override string ToString()
21.     {
22.         StringBuilder stringBuilder = new StringBuilder();
23.         stringBuilder.Append("Position: ").Append(position);
24.      
25.         foreach(int tile in chunk)
26.         {
27.             stringBuilder.Append(" ").Append(tile);
28.         }
29.  
30.         return stringBuilder.ToString();
31.     }
32. }

The way the buffer size is calculated might not be the best option, I just didn't want to hard code a value or use an arbitrarily high size.

 

@cerestorm

Thanks for the response. That's almost identical to one of my attempts, but it yielded the same problem. I think my issue lies deeper. When I get home, I will be attempting a new solution based on this thread I found where luke responded:

https://forum.unity.com/threads/send-big-data-from-host-to-clients.1131025/

 

I don't think it's a data size issue as the writer would complain first. The issue is on the receiving end and looks like the reader is trying read more data out than was actually sent. Without seeing the code it's hard to say exactly what the problem might be, but check the size of the data going out and the array length, and the size of the read buffer.

 

@cerestorm I realized that I was not actually writing to the buffer. Stupid, I know, I had it originally and lost it somewhere along the way. I am getting a different error now.

I am now getting "Writing past the end of the buffer"
I can't seem to get C# to give me a good size for that first argument. When I try to use the Marshal tool, I get:
"TileMapChunk[] cannot be marshaled as an unmanaged structure."

Note: There are usually 64 of these 16x16 chunks and players will typically connect close together. I would like to avoid sending one Chunk at a time (hence the array).

Any advice is welcome.

Perhaps I am going about this all wrong. I've been suggested to check out UTP to use with NGO, but I can't find enough documentation to garner any confidence in swapping to that Transport.

 

I feel like there has to be an easier way. Even if I do something like this:

Code (CSharp):

1. byte[] bytes = TileMapChunksToByteArray(chunks);
2. FastBufferWriter writer = new FastBufferWriter((sizeof(byte) * bytes.Length) + OVERHEAD, Allocator.Temp);
3. writer.WriteValueSafe(bytes);
4.  

I am still getting "Writing past the end of the buffer"

 

I wouldn't get too hung up on the write buffer size, it looks like it's just a temporary memory allocation. Set it to something like 10k or 20k for now and reduce it later.

Saying that, yes network objects could well be a better option, if you post the class you want to send I'll take a look into it when I have some free time later.

 

I've tried a large variety of solutions, from binary formatting this class into a byte[] to just send one at a time as INetworkSerializable (just to test, but ultimately filling the message queue).

Code (CSharp):

1. public class TileMapChunk : INetworkSerializable
2. {
3.     public delegate void OnTilesChangedDel(TileMapChunk chunk, List<TileUpdate> tileUpdates);
4.     public event OnTilesChangedDel OnTilesChanged = delegate { };
5.  
6.     // This is actually a Vector2Int now, but I wanted to try a BinaryFormatter and just sending bytes and BF can't handle Vector2Int
7.     private int chunkPositionX;
8.     private int chunkPositionY;
9.     public Vector2Int ChunkPosition => new Vector2Int(chunkPositionX, chunkPositionY);
10.  
11.     private int size;
12.     public int Size => size;
13.  
14.     [SerializeField]
15.     private int[] tilesInChunk;
16.     public int[] TilesInChunk => tilesInChunk;
17.  
18.     public TileMapChunk()
19.     {
20.         this.size = 0;
21.         this.tilesInChunk = new int[size * size];
22.     }
23.  
24.     public TileMapChunk(int size, Vector2Int chunkPosition)
25.     {
26.         this.size = size;
27.         this.tilesInChunk = new int[size * size];
28.         this.chunkPositionX = chunkPosition.x;
29.         this.chunkPositionY = chunkPosition.y;
30.     }
31.  
32.     public void NetworkSerialize<T>(BufferSerializer<T> serializer) where T : IReaderWriter
33.     {
34.         serializer.SerializeValue(ref size);
35.         serializer.SerializeValue(ref chunkPositionX);
36.         serializer.SerializeValue(ref chunkPositionY);
37.         serializer.SerializeValue(ref tilesInChunk);
38.     }
39. ...

 

Currently, this is my messenger code:

Code (CSharp):

1. private void SendTilesToClient(ulong clientID)
2.     {
3.         Debug.Log("Sending tilemap!");
4.         foreach (TileMapChunk chunk in tilemapData.TileMapChunks.Values)
5.         {
6.             Debug.Log("Size of message: " + chunk.GetSerializableSize());
7.             FastBufferWriter writer = new FastBufferWriter(chunk.GetSerializableSize() + OVERHEAD, Allocator.Temp);
8.             writer.WriteNetworkSerializable(chunk);
9.             networkManager.CustomMessagingManager.SendNamedMessage("ClientTilemapSync", clientID, writer, NetworkDelivery.ReliableFragmentedSequenced);
10.         }
11.     }
12.  
13.     private void ListenForTiles()
14.     {
15.         Debug.Log("Listening for tilemap!");
16.         networkManager.CustomMessagingManager.RegisterNamedMessageHandler("ClientTilemapSync", ReceiveTileMap);
17.     }
18.  
19.     private void ReceiveTileMap(ulong sender, FastBufferReader message)
20.     {
21.         Debug.Log("Got map chunk");
22.  
23.         TileMapChunk chunk;
24.         message.ReadNetworkSerializable(out chunk);
25.  
26.         tilemapData.TileMapChunks[chunk.ChunkPosition] = chunk;
27.     }

This is just for testing, before I move on, I'd want to make this work for an array of chunks because I have to pump up the message queue size to get this to work.

The server actually does send these, it seems, but the clients run into an issue and output a very unhelpful error:

Code (CSharp):

1. KeyNotFoundException: The given key '48' was not present in the dictionary.
2. System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) (at <31c0f51ac5a24a22ba784db24f4ba023>:0)
3. Unity.Netcode.NetworkManager.TransportIdToClientId (System.UInt64 transportId) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkManager.cs:1244)
4. Unity.Netcode.NetworkManager.HandleRawTransportPoll (Unity.Netcode.NetworkEvent networkEvent, System.UInt64 clientId, System.ArraySegment`1[T] payload, System.Single receiveTime) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkManager.cs:1304)
5. Unity.Netcode.NetworkManager.OnNetworkEarlyUpdate () (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkManager.cs:1150)
6. Unity.Netcode.NetworkManager.NetworkUpdate (Unity.Netcode.NetworkUpdateStage updateStage) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkManager.cs:1125)
7. Unity.Netcode.NetworkUpdateLoop.RunNetworkUpdateStage (Unity.Netcode.NetworkUpdateStage updateStage) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkUpdateLoop.cs:149)
8. Unity.Netcode.NetworkUpdateLoop+NetworkEarlyUpdate+<>c.<CreateLoopSystem>b__0_0 () (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkUpdateLoop.cs:172)

Additional info:
The size of each of these chunks is consistently 1036
In order to not fill the connection pool, I increase the queue size to 1280 (not a specific number, I just added a zero)
I plan to group these into fewer calls that pass an array because 200 messages sent rapidly is not great.
Running _pre3

 

I've tried network objects and that is the reason I am at this point. I could not get them to work with a struct that contains an array. The whole "non-nullable" thing.

 

This is a very simplified example, I don't know whether or not it's enough for you to build upon.

Code (CSharp):

1. public class TilemapChunk : NetworkBehaviour
2. {
3.     NetworkVariable<Vector2Int> position = new NetworkVariable<Vector2Int>();
4.     NetworkVariable<int> size = new NetworkVariable<int>();
5.  
6.     NetworkList<int> tileValues;
7.  
8.     private void Awake()
9.     {
10.         tileValues = new NetworkList<int>();
11.     }
12.  
13.     public override void OnNetworkSpawn()
14.     {
15.         base.OnNetworkSpawn();
16.  
17.         Debug.Log("Spawned TilemapChunk: " + this);
18.     }
19.  
20.  
21.     public override string ToString()
22.     {
23.         StringBuilder stringBuilder = new StringBuilder();
24.         stringBuilder.Append("Position: ").Append(Position);
25.         stringBuilder.Append(" Size: ").Append(Size);
26.  
27.         stringBuilder.Append(" TileValues: ").Append(tileValues.Count);
28.  
29.         foreach(int tileValue in tileValues)
30.         {
31.             stringBuilder.Append(" ").Append(tileValue);
32.         }
33.  
34.         return stringBuilder.ToString();
35.     }
36.  
37.     public Vector2Int Position { get => position.Value; set => position.Value = value; }
38.     public int Size { get => size.Value; set => size.Value = value; }
39.     public NetworkList<int> TileValues { get => tileValues; }
40. }

Code (CSharp):

1.     public void OnClickSpawnTilemapChunk()
2.     {
3.         TilemapChunk tilemapChunk = SpawnService.Instance().InstantiatePrefab<TilemapChunk>(true);
4.         tilemapChunk.Position = new Vector2Int(4, 2);
5.         tilemapChunk.Size = 256;
6.  
7.         for(int i = 0; i < 256; i++)
8.         {
9.             tilemapChunk.TileValues.Add(i);
10.         }
11.  
12.         tilemapChunk.GetComponent<NetworkObject>().Spawn();
13.     }

This can send 256 tile values, but for 512 it falls over. There's a bug in NetworkList causing this, it's fixed in the develop branch but that creates two major issues for me so I don't use it.

If you can use network objects you should find them far more convenient than sending custom messages.

 

I've thought about making Chunks networkbehaviors, but I'm concerned about the sheer amount of them. Currently, there are 4 tilemaps, each about about 128 chunks. That seems like too many network behaviors.

 

Hmm not sure, but definitely something worth looking in to. Does the player need to see all tiles at all times, or can you limit it with a subset of chunks in view?

I started off using my own custom messages with almost no game objects but the further I got with it the more it felt like I was re-inventing the wheel with the pain of having to keep everything sync'ed manually. So I decided to embrace the game/network object idea and I'm not going back. Although I seem to spend more time fighting with Netcode than game code at the moment...

 

Yeah, everything else is using network objects. And that's where I started, I spent several days trying. This is my one exception. The existing network objects/network variables/RPCs don't work any better.

Since this will will only fire once during initial joining, I figured it was fine.

I've also explored sending only the ones nearby, but this concept conflicts with the player's ability to dash or teleport significant distances where.


I feel like I am close.
The "KeyNotFoundException" error is really catching me. I can't see why this is happening and will likely need to dive into source code when I get home.

 

I had a quick look and couldn't make much sense of that error, nothing I see in what you're doing would explain it.

Are you able to at least send one chunk successfully? You could stagger your sends with rpc calls which is what I've used to avoid creating too much network traffic all at once. In the end I didn't need it for what I thought but I still use it to break up game initialisation to get around a bug with changed values.

 

What do you mean by stagger with RPCs? Like "okay, ready for next message ServerRPC" and server sends next chunk?

 

Pretty much yes. I lifted what I was using previously for a message service but something simple would be easy to implement. You're essentially using the round trip as a time delay between stages of initialisation.

 

Okay. I got it sending chunks finally. I am going to try the round trip and give it a shot. Thanks.

PS: I figured out the previous issue relating to the dictionary lookup. I am going to keyword here so someone might find a solution in a future time.

keyword: messenger, CustomMessagingManager, SendNamedMessage, RegisterNamedMessageHandler, UnregisterNamedMessageHandler
KeyNotFoundException: The given key
was not present in the dictionary.
UNetTransport

If you are reading this because you suddenly started getting the following message:

Code (CSharp):

1. KeyNotFoundException: The given key '???' was not present in the dictionary.

Try lowering the Max Sent Message Queue Size. It has some arbitrary limit.​

 

The round trip solution worked beautifully.

Thanks for your help!

 

Awesome.