Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on Dec 8, 2022, between 7 am & 7 pm EST, in the DOTS Dev Blitz Day 2022 - Q&A forum, Discord, and Unity3D Subreddit to learn more about DOTS directly from the Unity Developers.
    Dismiss Notice
  3. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

(Netcode For Gameobjects) Getting "Reading past the end of the buffer" when using Custom Messaging

Discussion in 'Multiplayer' started by Kyperr, Dec 21, 2021.

  1. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    I appear to be misunderstanding how to use Custom Messaging. Allow me to explain fully in case someone can help with the bigger picture.

    I am attempting to send a data representation of a tilemap which looks something like this:

    Code (CSharp):
    1. tilemap {
    2.     chunks {
    3.         vector2int
    4.         int[] // usually 16x16, int is mapped to a true TileBase client-side...
    5.     }
    6. }
    I cannot just send a seed from the server to client because sometimes there are entire player generated, though sometimes procedural.

    I have opted to send a state for the tilemap upon joining as a client and keep up-to-date by sending subsequent updates afterwards. I have has some considerable issues getting to this point but have ultimately decided that none of the prepared tools would work so a custom message was in order.

    My code looks something like this:

    Code (CSharp):
    1. Debug.Log("Sending tilemap!");
    2. // Note: TileMapChunk class extends INetworkSerializable
    3. TileMapChunk[] chunks = tilemapData.TileMapChunks.Values.ToArray();
    4. FastBufferWriter writer = new FastBufferWriter(["No idea what to put here"], Allocator.Temp, ["No idea what to put here, either."]);
    5. networkManager.CustomMessagingManager.SendNamedMessage("ClientTilemapSync", clientID, writer, NetworkDelivery.Reliable);
    This is where I reach the limitations of my current knowledge and the documentation doesn't help much.

    This seems to send fine, but I am getting the following message client-side:

    Code (CSharp):
    1. OverflowException: Reading past the end of the buffer
    2. Unity.Netcode.FastBufferReader.ReadNetworkSerializable[T] (T[]& value) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Serialization/FastBufferReader.cs:410)
    So my question is, does anyone have a suggestion on how to prevent the overflow exception or a better alternative to sending these tilemaps?
     
  2. cerestorm

    cerestorm

    Joined:
    Apr 16, 2020
    Posts:
    402
    I've not looked at custom messages so your post was a good excuse to give them a go.

    It might have been helpful to see the code that produced the error, along with where you registered the message handler. In any event I've got something to work that's hopefully similar to what you're after.

    Code (CSharp):
    1. public class CustomMessageTestService : NetworkBehaviour
    2. {
    3.     const int OVERHEAD = 12;
    4.  
    5.     private void Start()
    6.     {
    7.         Debug.Log("Register message handler");
    8.  
    9.         NetworkManager.Singleton.CustomMessagingManager.RegisterNamedMessageHandler("myMessageName", MessageReader);
    10.     }
    11.  
    12.     public void SendAMessage()
    13.     {
    14.         Debug.Log("Send Message...");
    15.  
    16.         TilemapChunk tilemapChunk = new TilemapChunk(new Vector2Int(3,6), new int[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 });
    17.  
    18.         FastBufferWriter writer = new FastBufferWriter(Marshal.SizeOf(tilemapChunk) + OVERHEAD, Allocator.Temp);
    19.  
    20.         Debug.Log("FastBufferWriter length: " + Marshal.SizeOf(tilemapChunk));
    21.  
    22.         writer.WriteNetworkSerializable<TilemapChunk>(tilemapChunk);
    23.  
    24.         NetworkManager.Singleton.CustomMessagingManager.SendNamedMessage("myMessageName", 1, writer);
    25.     }
    26.  
    27.  
    28.     private void MessageReader(ulong senderClientId, FastBufferReader reader)
    29.     {
    30.         Debug.Log("FastBufferReader length: " + reader.Length);
    31.  
    32.         reader.ReadNetworkSerializable<TilemapChunk>(out TilemapChunk tilemapChunk);
    33.  
    34.         Debug.Log("Received TilemapChunk: " + tilemapChunk);
    35.     }
    36. }
    Code (CSharp):
    1. public struct TilemapChunk : INetworkSerializable
    2. {
    3.     Vector2Int position;
    4.  
    5.     [MarshalAs(UnmanagedType.ByValArray, SizeConst = 16)]
    6.     int[] chunk;
    7.  
    8.     public TilemapChunk(Vector2Int position, int[] chunk)
    9.     {
    10.         this.position = position;
    11.         this.chunk = chunk;
    12.     }
    13.  
    14.     public void NetworkSerialize<T>(BufferSerializer<T> serializer) where T : IReaderWriter
    15.     {
    16.         serializer.SerializeValue(ref position);
    17.         serializer.SerializeValue(ref chunk);
    18.     }
    19.  
    20.     public override string ToString()
    21.     {
    22.         StringBuilder stringBuilder = new StringBuilder();
    23.         stringBuilder.Append("Position: ").Append(position);
    24.      
    25.         foreach(int tile in chunk)
    26.         {
    27.             stringBuilder.Append(" ").Append(tile);
    28.         }
    29.  
    30.         return stringBuilder.ToString();
    31.     }
    32. }
    The way the buffer size is calculated might not be the best option, I just didn't want to hard code a value or use an arbitrarily high size. :)
     
  3. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
  4. cerestorm

    cerestorm

    Joined:
    Apr 16, 2020
    Posts:
    402
    I don't think it's a data size issue as the writer would complain first. The issue is on the receiving end and looks like the reader is trying read more data out than was actually sent. Without seeing the code it's hard to say exactly what the problem might be, but check the size of the data going out and the array length, and the size of the read buffer.
     
  5. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    @cerestorm I realized that I was not actually writing to the buffer. Stupid, I know, I had it originally and lost it somewhere along the way. I am getting a different error now.

    I am now getting "Writing past the end of the buffer"
    I can't seem to get C# to give me a good size for that first argument. When I try to use the Marshal tool, I get:
    "TileMapChunk[] cannot be marshaled as an unmanaged structure."

    Note: There are usually 64 of these 16x16 chunks and players will typically connect close together. I would like to avoid sending one Chunk at a time (hence the array).

    Any advice is welcome.

    Perhaps I am going about this all wrong. I've been suggested to check out UTP to use with NGO, but I can't find enough documentation to garner any confidence in swapping to that Transport.
     
  6. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    I feel like there has to be an easier way. Even if I do something like this:

    Code (CSharp):
    1. byte[] bytes = TileMapChunksToByteArray(chunks);
    2. FastBufferWriter writer = new FastBufferWriter((sizeof(byte) * bytes.Length) + OVERHEAD, Allocator.Temp);
    3. writer.WriteValueSafe(bytes);
    4.  
    I am still getting "Writing past the end of the buffer"
     
  7. cerestorm

    cerestorm

    Joined:
    Apr 16, 2020
    Posts:
    402
    I wouldn't get too hung up on the write buffer size, it looks like it's just a temporary memory allocation. Set it to something like 10k or 20k for now and reduce it later.

    Saying that, yes network objects could well be a better option, if you post the class you want to send I'll take a look into it when I have some free time later.
     
  8. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    I've tried a large variety of solutions, from binary formatting this class into a byte[] to just send one at a time as INetworkSerializable (just to test, but ultimately filling the message queue).


    Code (CSharp):
    1. public class TileMapChunk : INetworkSerializable
    2. {
    3.     public delegate void OnTilesChangedDel(TileMapChunk chunk, List<TileUpdate> tileUpdates);
    4.     public event OnTilesChangedDel OnTilesChanged = delegate { };
    5.  
    6.     // This is actually a Vector2Int now, but I wanted to try a BinaryFormatter and just sending bytes and BF can't handle Vector2Int
    7.     private int chunkPositionX;
    8.     private int chunkPositionY;
    9.     public Vector2Int ChunkPosition => new Vector2Int(chunkPositionX, chunkPositionY);
    10.  
    11.     private int size;
    12.     public int Size => size;
    13.  
    14.     [SerializeField]
    15.     private int[] tilesInChunk;
    16.     public int[] TilesInChunk => tilesInChunk;
    17.  
    18.     public TileMapChunk()
    19.     {
    20.         this.size = 0;
    21.         this.tilesInChunk = new int[size * size];
    22.     }
    23.  
    24.     public TileMapChunk(int size, Vector2Int chunkPosition)
    25.     {
    26.         this.size = size;
    27.         this.tilesInChunk = new int[size * size];
    28.         this.chunkPositionX = chunkPosition.x;
    29.         this.chunkPositionY = chunkPosition.y;
    30.     }
    31.  
    32.     public void NetworkSerialize<T>(BufferSerializer<T> serializer) where T : IReaderWriter
    33.     {
    34.         serializer.SerializeValue(ref size);
    35.         serializer.SerializeValue(ref chunkPositionX);
    36.         serializer.SerializeValue(ref chunkPositionY);
    37.         serializer.SerializeValue(ref tilesInChunk);
    38.     }
    39. ...
     
    Last edited: Dec 22, 2021
  9. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    Currently, this is my messenger code:

    Code (CSharp):
    1. private void SendTilesToClient(ulong clientID)
    2.     {
    3.         Debug.Log("Sending tilemap!");
    4.         foreach (TileMapChunk chunk in tilemapData.TileMapChunks.Values)
    5.         {
    6.             Debug.Log("Size of message: " + chunk.GetSerializableSize());
    7.             FastBufferWriter writer = new FastBufferWriter(chunk.GetSerializableSize() + OVERHEAD, Allocator.Temp);
    8.             writer.WriteNetworkSerializable(chunk);
    9.             networkManager.CustomMessagingManager.SendNamedMessage("ClientTilemapSync", clientID, writer, NetworkDelivery.ReliableFragmentedSequenced);
    10.         }
    11.     }
    12.  
    13.     private void ListenForTiles()
    14.     {
    15.         Debug.Log("Listening for tilemap!");
    16.         networkManager.CustomMessagingManager.RegisterNamedMessageHandler("ClientTilemapSync", ReceiveTileMap);
    17.     }
    18.  
    19.     private void ReceiveTileMap(ulong sender, FastBufferReader message)
    20.     {
    21.         Debug.Log("Got map chunk");
    22.  
    23.         TileMapChunk chunk;
    24.         message.ReadNetworkSerializable(out chunk);
    25.  
    26.         tilemapData.TileMapChunks[chunk.ChunkPosition] = chunk;
    27.     }
    This is just for testing, before I move on, I'd want to make this work for an array of chunks because I have to pump up the message queue size to get this to work.

    The server actually does send these, it seems, but the clients run into an issue and output a very unhelpful error:

    Code (CSharp):
    1. KeyNotFoundException: The given key '48' was not present in the dictionary.
    2. System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) (at <31c0f51ac5a24a22ba784db24f4ba023>:0)
    3. Unity.Netcode.NetworkManager.TransportIdToClientId (System.UInt64 transportId) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkManager.cs:1244)
    4. Unity.Netcode.NetworkManager.HandleRawTransportPoll (Unity.Netcode.NetworkEvent networkEvent, System.UInt64 clientId, System.ArraySegment`1[T] payload, System.Single receiveTime) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkManager.cs:1304)
    5. Unity.Netcode.NetworkManager.OnNetworkEarlyUpdate () (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkManager.cs:1150)
    6. Unity.Netcode.NetworkManager.NetworkUpdate (Unity.Netcode.NetworkUpdateStage updateStage) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkManager.cs:1125)
    7. Unity.Netcode.NetworkUpdateLoop.RunNetworkUpdateStage (Unity.Netcode.NetworkUpdateStage updateStage) (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkUpdateLoop.cs:149)
    8. Unity.Netcode.NetworkUpdateLoop+NetworkEarlyUpdate+<>c.<CreateLoopSystem>b__0_0 () (at Library/PackageCache/com.unity.netcode.gameobjects@1.0.0-pre.3/Runtime/Core/NetworkUpdateLoop.cs:172)

    Additional info:
    The size of each of these chunks is consistently 1036
    In order to not fill the connection pool, I increase the queue size to 1280 (not a specific number, I just added a zero)
    I plan to group these into fewer calls that pass an array because 200 messages sent rapidly is not great.
    Running _pre3
     
    Last edited: Dec 22, 2021
  10. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    I've tried network objects and that is the reason I am at this point. I could not get them to work with a struct that contains an array. The whole "non-nullable" thing.
     
  11. cerestorm

    cerestorm

    Joined:
    Apr 16, 2020
    Posts:
    402
    This is a very simplified example, I don't know whether or not it's enough for you to build upon.

    Code (CSharp):
    1. public class TilemapChunk : NetworkBehaviour
    2. {
    3.     NetworkVariable<Vector2Int> position = new NetworkVariable<Vector2Int>();
    4.     NetworkVariable<int> size = new NetworkVariable<int>();
    5.  
    6.     NetworkList<int> tileValues;
    7.  
    8.     private void Awake()
    9.     {
    10.         tileValues = new NetworkList<int>();
    11.     }
    12.  
    13.     public override void OnNetworkSpawn()
    14.     {
    15.         base.OnNetworkSpawn();
    16.  
    17.         Debug.Log("Spawned TilemapChunk: " + this);
    18.     }
    19.  
    20.  
    21.     public override string ToString()
    22.     {
    23.         StringBuilder stringBuilder = new StringBuilder();
    24.         stringBuilder.Append("Position: ").Append(Position);
    25.         stringBuilder.Append(" Size: ").Append(Size);
    26.  
    27.         stringBuilder.Append(" TileValues: ").Append(tileValues.Count);
    28.  
    29.         foreach(int tileValue in tileValues)
    30.         {
    31.             stringBuilder.Append(" ").Append(tileValue);
    32.         }
    33.  
    34.         return stringBuilder.ToString();
    35.     }
    36.  
    37.     public Vector2Int Position { get => position.Value; set => position.Value = value; }
    38.     public int Size { get => size.Value; set => size.Value = value; }
    39.     public NetworkList<int> TileValues { get => tileValues; }
    40. }
    Code (CSharp):
    1.     public void OnClickSpawnTilemapChunk()
    2.     {
    3.         TilemapChunk tilemapChunk = SpawnService.Instance().InstantiatePrefab<TilemapChunk>(true);
    4.         tilemapChunk.Position = new Vector2Int(4, 2);
    5.         tilemapChunk.Size = 256;
    6.  
    7.         for(int i = 0; i < 256; i++)
    8.         {
    9.             tilemapChunk.TileValues.Add(i);
    10.         }
    11.  
    12.         tilemapChunk.GetComponent<NetworkObject>().Spawn();
    13.     }
    This can send 256 tile values, but for 512 it falls over. There's a bug in NetworkList causing this, it's fixed in the develop branch but that creates two major issues for me so I don't use it.

    If you can use network objects you should find them far more convenient than sending custom messages.
     
  12. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    I've thought about making Chunks networkbehaviors, but I'm concerned about the sheer amount of them. Currently, there are 4 tilemaps, each about about 128 chunks. That seems like too many network behaviors.
     
  13. cerestorm

    cerestorm

    Joined:
    Apr 16, 2020
    Posts:
    402
    Hmm not sure, but definitely something worth looking in to. Does the player need to see all tiles at all times, or can you limit it with a subset of chunks in view?

    I started off using my own custom messages with almost no game objects but the further I got with it the more it felt like I was re-inventing the wheel with the pain of having to keep everything sync'ed manually. So I decided to embrace the game/network object idea and I'm not going back. :) Although I seem to spend more time fighting with Netcode than game code at the moment...
     
  14. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    Yeah, everything else is using network objects. And that's where I started, I spent several days trying. This is my one exception. The existing network objects/network variables/RPCs don't work any better.

    Since this will will only fire once during initial joining, I figured it was fine.

    I've also explored sending only the ones nearby, but this concept conflicts with the player's ability to dash or teleport significant distances where.


    I feel like I am close.
    The "KeyNotFoundException" error is really catching me. I can't see why this is happening and will likely need to dive into source code when I get home.
     
  15. cerestorm

    cerestorm

    Joined:
    Apr 16, 2020
    Posts:
    402
    I had a quick look and couldn't make much sense of that error, nothing I see in what you're doing would explain it.

    Are you able to at least send one chunk successfully? You could stagger your sends with rpc calls which is what I've used to avoid creating too much network traffic all at once. In the end I didn't need it for what I thought but I still use it to break up game initialisation to get around a bug with changed values.
     
  16. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    What do you mean by stagger with RPCs? Like "okay, ready for next message ServerRPC" and server sends next chunk?
     
  17. cerestorm

    cerestorm

    Joined:
    Apr 16, 2020
    Posts:
    402
    Pretty much yes. I lifted what I was using previously for a message service but something simple would be easy to implement. You're essentially using the round trip as a time delay between stages of initialisation.
     
  18. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    Okay. I got it sending chunks finally. I am going to try the round trip and give it a shot. Thanks.

    PS: I figured out the previous issue relating to the dictionary lookup. I am going to keyword here so someone might find a solution in a future time.

    keyword: messenger, CustomMessagingManager, SendNamedMessage, RegisterNamedMessageHandler, UnregisterNamedMessageHandler
    KeyNotFoundException: The given key
    was not present in the dictionary.
    UNetTransport

    If you are reading this because you suddenly started getting the following message:
    Code (CSharp):
    1. KeyNotFoundException: The given key '???' was not present in the dictionary.
    Try lowering the Max Sent Message Queue Size. It has some arbitrary limit.​
     
    cerestorm likes this.
  19. Kyperr

    Kyperr

    Joined:
    Jul 15, 2016
    Posts:
    31
    The round trip solution worked beautifully.

    Thanks for your help!
     
  20. cerestorm

    cerestorm

    Joined:
    Apr 16, 2020
    Posts:
    402
    Awesome. :)