Reliably transmit 3-dimensional array?

I'm looking for a way to reliably transmit a 3-dimensional array of the size Block[16,256,16] where Block is a base class and other classes inherit from it.

I tried using SyncLists, but since they get very big (65536 entries; the array is ALWAYS full!), they act like they don't transmit anyhing. They do work with a smaller sample size with the same classes though.

So what other ways could I use to solve this? Channel is reliable fragmented always.

 

I have just a little something, used Cmd/Rpc to send 2D array of bytes, didn't work, after hours searching for my mistake I tried to make it a one dimensional array, *insert blow mind gif* it worked. Anyway I'm not sure with syncing classes, try syncing block ids or something like that..

 

Yeah, I've used one dimensional arrays as cmd/rpcs for another project as well, but unfortunately I can't do that in this scenario.

What I've tried as well was a SyncList containing structs like the following:

Code (csharp):

1.  
2. public struct SyncBlock{
3.     public WorldPos pos; // world pos contains 3 public integers x, y and z
4.     public string typeString; // type string is basically typeof(Block).ToString() so that I can also do something like typeof(BlockAir).ToString()
5. }
6.  

That worked to some extent, for example when I tested it with 128 blocks being synchronized. However, once I tried synchronizing 65536 blocks, or even 16 * 16 * 128, or even 8 * 8 * 128, or EVEN 8 * 8 * 64, it wouldn't go through, despite the channel being reliable fragmented. No error message (neither on client nor on server), the client's SyncList just wouldn't get populated with any entries at all. I imagine that block IDs will yield a similar result.

Perhaps I should also mention that Block, and all of the classes that inherit from it, are serializable (so that they can be saved). Perhaps that attribute can be used somehow to transfer them?

 

Block ids would be the same, except strings are way bigger than bytes, and in my game client requests chunks of 1024 blocks (2D game), so it isn't that of a problem. Maybe "fragment" it a bit yourself, try sending only one chunk at a time.
If I count right, for 5 letters long strings it would be around 1441792 bytes, I don't know what it means really, but I think unet isn't built for sending bigger files. Try sending chunks of like 32*32*32 blocks, with three ints for chunks position and ids of blocks in an array. You only use three int's for an entire chunk and byte array should be WAY smaller, if you are ok with max 256 block types.

 

Right, I didn't even consider the size of a string! I'll try it out with unsigned short IDs then, maybe that'll work out.

Also, I am only sending a single chunk, but its size is 16 * 256 * 16. I only want chunks to spread out horizontally if possible, but if that's not an option I might try resizing them to 16 * 16 * 16 and allowing vertical chunks as well.

 

Well 16*256*16 is only 2 times bigger than 32*32*32, so it's ok. They main problem though I think is three ints in every block, so really try to organize the blocks somehow so you must only send the chunk pos. Good luck!

 

Right, I now transmit block IDs instead of strings (WorldPos is still the same as of right now).

However, instead of an empty SyncList, I now get this and I really have no idea why:

IndexOutOfRangeException: NetworkReader:ReadByte out of range:NetBuf sz:5 pos:5
UnityEngine.Networking.NetBuffer.ReadByte () (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkBuffer.cs:35)
UnityEngine.Networking.NetworkReader.ReadByte () (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkReader.cs:161)
UnityEngine.Networking.NetworkReader.ReadPackedUInt32 () (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkReader.cs:61)
Unity.GeneratedNetworkCode._ReadWorldPos_None (UnityEngine.Networking.NetworkReader reader)
Chunk+SyncListBlock.DeserializeItem (UnityEngine.Networking.NetworkReader reader)
Unity.GeneratedNetworkCode._ReadStructSyncListBlock_Chunk (UnityEngine.Networking.NetworkReader reader)
Chunk.OnDeserialize (UnityEngine.Networking.NetworkReader reader, Boolean initialState)
UnityEngine.Networking.NetworkIdentity.OnUpdateVars (UnityEngine.Networking.NetworkReader reader, Boolean initialState) (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkIdentity.cs:628)
UnityEngine.Networking.ClientScene.ApplySpawnPayload (UnityEngine.Networking.NetworkIdentity uv, Vector3 position, System.Byte[] payload, NetworkInstanceId netId, UnityEngine.GameObject newGameObject) (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/ClientScene.cs:388)
UnityEngine.Networking.ClientScene.OnObjectSpawn (UnityEngine.Networking.NetworkMessage netMsg) (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/ClientScene.cs:443)
UnityEngine.Networking.NetworkConnection.HandleReader (UnityEngine.Networking.NetworkReader reader, Int32 receivedSize, Int32 channelId) (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkConnection.cs:416)
UnityEngine.Networking.NetworkConnection.HandleBytes (System.Byte[] buffer, Int32 receivedSize, Int32 channelId) (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkConnection.cs:372)
UnityEngine.Networking.NetworkConnection.TransportRecieve (System.Byte[] bytes, Int32 numBytes, Int32 channelId) (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkConnection.cs:522)
UnityEngine.Networking.NetworkClient.Update () (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkClient.cs:560)
UnityEngine.Networking.NetworkClient.UpdateClients () (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkClient.cs:774)
UnityEngine.Networking.NetworkIdentity.UNetStaticUpdate () (at /Users/builduser/buildslave/unity/build/Extensions/Networking/Runtime/NetworkIdentity.cs:910)

 

This looks similiar to error i was getting when using 2D array...anyway, just try to find some bad initialization/accessing space that doesn't exist, and if you don't find anything, try something else. I started programming like 2 months ago and unet is still mysterious for everybody, so I don't have an idea what could be causing this.
Oh now I see post from seanr, messages look very nice though I haven't gotten into them yet, I tried but maybe too hard for me. Anyway here is my code if you wanted to use smaller chunks in the end:

Code (CSharp):

1.  
2. [Command]
3. public void CmdAskForChunk(int x, int y) {
4.         byte[] b = new byte[32 * 32];
5.         for (int X = 0; X < 32; X++)
6.         {
7.             for (int Y = 0; Y < 32; Y++)
8.             {
9.                 b[X + Y*32] = dynamicMap[(x * 32 + X) + (y * 32 + Y) * 8192];          
10.              }
11.         }
12.         RpcSendChunk(x, y, b);
13.     }
14.  
15.     [ClientRpc]
16.     public void RpcSendChunk(int x, int y, byte[] b) {
17.         for (int X = 0; X < 32; X++)
18.         {
19.             for (int Y = 0; Y < 32; Y++)
20.             {
21.                 if (b[X + Y * 32] == 0)
22.                     continue;
23.                 else
24.                     World.singleton.blocks[(x * 32 + X) + (y * 32 + Y) * 8192] = b[X + Y*32];
25.             }
26.         }
27.         World.singleton.chunks[x, y].downloaded = true;
28.     }

Heh as I see my code, you should really prefer mesages.

 

@MightySheep:

Looks nice, but as far as I understand [ClientRpc] executes on every machine the object is active on. That wouldn't be a suitable option for me.

@seanr:

I have my code now set up like this:

Code (csharp):

1.  
2. public struct SyncBlock
3. {
4.     public ushort bID;
5. }
6.  
7. // ^ looks nasty, I know, but a SyncListStruct<ushort> generated an error message for me
8. public class SyncListBlock : SyncListStruct<SyncBlock> { }
9.    
10. public SyncListBlock syncBlocks = new SyncListBlock();
11.  

The list is getting populated by adding SyncBlocks (a little more than 37,000 in my test). I'm essentially back to square one, as basically the client doesn't receive anything. No error message either.

I'd really like to go with your option, but I also need it so that if a player walks near a chunk and it displays for him (Network proximity, a little optimized by me) that the chunk already has the data in it (which a SyncVar or SyncList would provide; or even queued RPCs). If you can elaborate on how that would work, or on how I can get my SyncList to work, that'd be great!

 

I have now set my chunks to 16 * 16 * 16 sized objects and implemented the method seanr posted. Thanks for that!

However, when I pre-generate a part of the world and try to send it to the client (Unity actually does this automatically, since the Chunks are network objects), it cannot send it because the message is too long. Is there any way to configure Unity to only send out one network prefab at once instead of the full 1024 that get instantiated when a player joins, or do I also have to script that on my own?

I'd really hate to do the latter, because that would mean I had to lose the convenience of having the Chunk be a network prefab.

Note: NetworkProximityCheck is out of the picture, because the 1024 Chunks that get instantiated are SUPPOSED to be in the player's view.

 

Okay, I now have it set up so that the player requests a chunk from the server and then gets a response. The player can only request one chunk at a time (there is a boolean in the class that is set to false when a chunk is requested, and set to true once a chunk is received) so that there aren't any messages piling up. However, I am still getting the send error message too long on the server when a client connects.

Here is my transmission code:

Code (csharp):

1.  
2.         // Server stuff
3.     [Command]
4.     public void CmdPopulateChunk(int x, int y, int z)
5.     {
6.         Chunk chunk = world.GetChunk(x, y, z, true); // the 'true' makes it so that a chunk is generated if it's not loaded already
7.         ushort[] chunkArray = chunk.GetBlockIDArray();
8.         SendChunk(connectionToClient, chunk.worldPos.x, chunk.worldPos.y, chunk.worldPos.z, chunkArray);
9.     }
10.  
11.     public void SendChunk(NetworkConnection conn, int chunkX, int chunkY, int chunkZ, ushort[] data)
12.     {
13.         var wr = new NetworkWriter();
14.         wr.StartMessage(ChunkMsg);
15.         wr.Write(chunkX);
16.         wr.Write(chunkY);
17.         wr.Write(chunkZ);
18.  
19.         int i = 0;
20.         for(int x = 0; x < Chunk.horizontalChunkSize; x++)
21.         {
22.             for(int y = 0; y < Chunk.verticalChunkSize; y++)
23.             {
24.                 for(int z = 0; z < Chunk.horizontalChunkSize; z++)
25.                 {
26.                     wr.Write(data[i++]);
27.                 }
28.             }
29.         }
30.  
31.         wr.FinishMessage();
32.         conn.SendWriter(wr, 1); // channel 1 is reliable fragmented
33.     }
34.  
35.         // Client stuff
36.     public override void OnStartLocalPlayer()
37.     {
38.         ClientScene.readyConnection.RegisterHandler(ChunkMsg, OnChunk);
39.     }
40.  
41.     void OnChunk(NetworkMessage netMsg)
42.     {
43.         int chunkX = netMsg.reader.ReadInt32();
44.         int chunkY = netMsg.reader.ReadInt32();
45.         int chunkZ = netMsg.reader.ReadInt32();
46.  
47.         canRequest = true; // that's the boolean mentioned above
48.  
49.         ushort[] data = new ushort[Chunk.horizontalChunkSize * Chunk.verticalChunkSize * Chunk.horizontalChunkSize];
50.         int i = 0;
51.         for(int x = 0; x < Chunk.horizontalChunkSize; x++)
52.         {
53.             for(int y = 0; y < Chunk.verticalChunkSize; y++)
54.             {
55.                 for(int z = 0; z < Chunk.horizontalChunkSize; z++)
56.                 {
57.                     data[i++] = (ushort)netMsg.reader.ReadInt16();
58.                 }
59.             }
60.         }
61.  
62.         world.ReceiveRequestedChunk(chunkZ, chunkY, chunkZ, data);
63.         world.GetChunk(chunkX, chunkY, chunkZ).update = true;
64.     }
65.  

EDIT: I solved it myself, the boolean was handled the wrong way. However, now I have the issue that the connection seems to abrupt after a few chunk requests. I have added Debug.Log on request and receive (client side) as well as on recevieRequest and sendResponse (server side) to find out what the issue is. Turns out at some point the client sends a request which the server never gets (or the client never gets the response), because the last server log shows "response sent" and the last client log shows "request sent". On the server, the client's player game object just disappears from the hierarchy. Any ideas as to why that happens? When I run as host, everything works fine.

EDIT2: I now know that it's the client's request that doesn't go through. So after a certain amount of chunks have been transmitted, the client's request for another chunk doesn't go through anymore. What's even weirder is that the number seems to be random, so sometimes it can't request a second chunk, and sometimes it can't request a 300th chunk. Also, sometimes I get "lucky" and it works without a single problem, able to request all the chunks needed. I'm very confused by this. Any ideas why that might happen?

 

WOW! This thread solved my terrain streaming problems! Well, maybe not all of them, but at least now I can reliably send my height and splat maps to the clients.

I tried using SendByChannel before I saw this thread, but ended up with channel buffer growing fast and getting overrun. The problem with SendByChannel is that max size is 1.4k, so, you must send very many packages and doing that by client request would be extremely slow.

Now with this approach and up to 64k packages it's a breeze.

The documentation around these things are so bad it's not even funny. There is old uNet and the new HL and LL APIs in a confusing mix and none of it is well documented, or even close to.

I ended up turning this one around a bit compared to seanr's example though. I'm having the server in command, knowing what the client should receive next and the client can just acknowledge that it is ready for more. I don't want clients to be able to do unlimited amounts of terrain requests. If something goes wrong, the client could need to have terrain resent, but that would still be done using the player position that is owned by the server, not by coordinates sent from the client.

Anyway, thanks for this thread

 

Great to hear at least someone is having success! Unfortunately, I'm still stuck with the client disconnection problem.

UPDATE: I have found that I can reduce the likelihood of the client disconnecting without a notice by removing/disabling the network transform component on the player prefab. That said, the likelihood is still very high (about 9/10). I'd really like to know what could possible cause this, so if anyone has any idea, please let me know.

 

NaXDy, I have not stress tested my code, I could still run into trouble at some point, but now it works for a few players at once at least without errors.

In your case, maybe you are sending too much data on some other channel or the same channel from some other part of your code?

Also, I noticed that using SendWriter doesn't always generate ChannelBuffer limit overrun error messages even though the buffer is overrun and traffic is stopped. Maybe if you keep sending even though the buffer is full, and maybe then you will get disconnects after some timeout period? Just a guess...

I don't use the network transform, I use my own code for that part.. But since removing it helped, it could be an indication that reducing traffic in other parts of your code is helping.

 

This issue is simply impossible for me to debug. If I have the chunks set to 8 * 8 * 8, everything works perfectly, but if I set them to 16 * 16 * 16, the big gamble begins (I mostly lose though). It doesn't matter what I do. I can even implement a time delay in the client, so that only once PER SECOND it requests a new chunk. The client prefab still disappears on the server and the request is never received.

If anyone has even a hunch of what might be going on, please let me know, I'm super desperate at this point.

 

What we did in our game (will only work for some cases) is to generate the world locally and then only send world changes based on position (like an explosion at X,Y,Z) and from there on handle it locally. That way we did not have to send chunk information at all.

We dont have a randomly generated world. But you could just let the server handle seeds and then send the seed to the client in order to generate a certain chunk locally.

Might not help all, but might help some to reconsider their design.