Is it better to have one large message, or lots of small messages?

So this is something that's always been bugging me. What's better:

• one large message containing everything you need to sync for that network update (assuming everything fits in one message)
• lots of small, separate messages?

Right off the bat, I know that one big message saves you a lot of bytes from not having to send many headers. But are there any downsides to it? Will it cause way more problems in the case of packet loss? Or maybe the bandwidth can't be as fully filled when you only have large messages and no small messages to pack the remaining space with?

I am kind of expecting the answer to this question to be "it depends", so my real question would be this: In your experience, what's the best way to understand/troubleshoot/plan out the ideal message size for your game?

 

It's better to gather up as much as possible at once, within reason. Just having multiple bundles of data adds overhead beyond just packet headers (latency from starting and stopping multiple transmissions). At the OS level you have the MTU, which typically is 1500 or 9000 bytes. It's best to cram it with data if the packets are small, and it'll split it into whichever size chunks are most efficient. Each TCP packet has 42 or 66 bytes (IPv4 vs. IPv6) overhead too, which could be quite excessive if every message you send is just a two-word JSON dictionary. Then there's even more overhead at the layers below, and if you're using Unity's own networking there are of course its own headers.

Ideal size is the highest you can manage to send and get a response to before it's too late, basically

Actual numbers change drastically depending on what sort of game it is. A typical mobile game where any player interaction happens via chat or scheduling passively defended fights can have huge updates if needed. An FPS should aim to have small amounts of data from the players, but has perhaps 500k-1MB/s available to send down to players (divided by however many frames the game wants to split it into). An MMO tends more towards very turn-based combat, so could work with only a few updates per second (i.e. not every frame). A fully turn-based strategy/board game only sends updates when a player is ready (and nag requests from the server when they take too much time, ideally).

 

I am no expert at this, but heres my 2cents.

----
tldr; Use multiple messages (within a single packet under size of 400 bytes) so long as it makes your code easier to write/use.
----

What do you mean by messages? For example, I use the LLAPI and every call to NetworkTransport.Send or QueueMessage will add headers for channelID and message size, so ideally I want to make only one call to send with what unity sees as one large message, but behind the scenes I am putting my own headers in those messages so I can sort them as I desire. I think the HLAPI handles that for you.
So even though I am sending all my messages as if its one giant message, its really multiple messages. Currently I am only adding a message ID as the header and relying on the message to be read with no errors in order to know where the next message starts, which saves me some bytes, but does make it so that if one message fails for some reason, then all messages in that packet fail which could have been some reliable messages expected to not fail, so perhaps I should at least put a message size for reliable messages.

However, if you mean multiple messages as in sending a bunch of individual packets, then I think most will say thats a bad idea since that could use a lot of extra bytes.
I aim for a total packet size under 400 bytes. The reason for that is because people say the udp packet size to ensure no fragmentation of your packet is 508 bytes (576 MTU - 60 IP Address - 8 UDP Header), and then I subtract 108 to account for any unity headers and some wiggle room.

I think what you mean is whether you should try to pack as much data as you can into one message ID. As Ive said, I use multiple message IDs to separate my messages since messages might not need to be sent all the time. In some cases I might be using more bytes than what I could have gotten away with if I made the messages more specific, but separating messages out made it more reusable friendly I guess.
For example, each object that has a Network Identity like component can send rpcs or update their variables, but each object would pack its own messages by putting all variables that changed into a message that contains all those changes, and then putting each rpc into its own separate message per rpc. Maybe I could have save bytes by not having such a HLAPI approach and instead make a single specific message that knows about all the objects, but that would seem annoying to work with, and might not even be worthwhile depending on how dynamic your game is (do you spawn network objects in, etc...).
Basically, if you want to use one single giant message, then its going to need to pretty much be a giant static message that uses bits (bools) to let you know if you should skip this or that part of the message (kinda like how unity handles SyncVars), otherwise you are going to need some header to know what message you are at.

So for now I use multiple messages in a single packet and if it becomes an issue then I'll look into it.

 

I know what you mean, but I think the upcoming ECS will really facilitate the "big messages" workflow. I can definitely imagine building a new networking system with this, where networking isn't even the responsibility of "objects" anymore. Instead of every component handling their syncing, networking would be done by "systems" than handle writing/reading sync messages for all entities of given types. What would be really cool about this sort of system is that the conversion from an offline game to an online game would become nearly pain-free. To add online to your game, you just need to write "networking systems" for every type of thing that can be networked. Or at least that's my theory

but thanks @HiddenMonk and @orb for the helpful answers

 

Cant you basically do that by just having a list of INetworkables that have a public serialize and deserialize method, store all spawned objects and network related objects into that list and ensure the list order and items are synced accross the network so that all you need to do is just loop over, call serialize to get one giant message with no headers needed, then on other end just loop and deserialize. Keeping the list synced would be the trouble, and things might get messy if you are frequently creating and destroying. Youd need to buffer messages if you detect you are being sent a list that has been altered (the list would have a header with a number that lets you know that) and wait for the message that tells you what you needed to do to the list. I guess this isnt much of a difference to needing to wait for a reliable message to be received before reading unreliable messages that rely on that reliable message (such as spawning an object and then having unreliable updates for that object). There is still also the downside of if one object has a problem reading the data, then all messages will fail to be read.
I dont see how the new Entity Component System would allow you to do much else other than something like explained above (does the video go over networking? I only watched a little of it).

 

So I first use a messaging structure that is optimized. What you send and how you serialize it should be 90% or more of your optimization. It doesn't do any good to optimize for low level header stuff while your messages themselves are several hundred percent larger then they should be. Currently the best overall approach is use protocol buffers, convert everything to integers.

Protocol buffer varints are the key here, most efficient space optimization you can get using a standardized format without doing really low level bit flag stuff, and it will almost always give you enough optimization where going down to the bit level doesn't give meaningful returns.

There is a limit on size for realtime stuff since most everyone uses UDP. You can't have a message larger then the MTU. 1500 is what I'm finding to be the max I can get away with (it's set at the router level, so really it could be smaller/larger at any specific network hop, but they try to standardize).

And you invariably end up with common messaging patterns for realtime stuff. Movement updates will be 90%+ of all data. I send those all at once in chunks that are just below the MTU threshold. The remaining messages are sporadic and can't really be sent in bulk for the most part.

I strongly prefer an overall pattern of request/response vs push where I can. Which is for most stuff. So a client sends their position/movement update, and in response I send movement updates for all entities in visual range. Pushing data you have no idea of the state of the client, if it is overloaded, if it's lagging behind in processing updates. I can almost always find an existing message to key off of for responses, vs having to waste a message does does nothing but request data.

 

FYI what I mean by convert to ints, is multiply/divide on each end. Convert floats to a set precision, 2 is usually good enough.

Code (csharp):

1.  
2. public static float ToFloat(this int i)
3.         {
4.             return i / 100f;
5.         }
6.  
7.         public static int ToInt(this float i)
8.         {
9.             return (int)(Math.Round(i * 100, 2));
10.         }
11.  

That right there is huge savings over sending a full float. There are very few things you can't convert to integers. This combined with protobuf varints gives you such a huge amount of space savings that you generally don't have to optimize a whole lot more.

Also use creative ways to send stuff like rotations. Like I convert to eulers then to a single float heading, then back again.

 

You shouldn't have any problems as long as you use the appropriate QoS for such messages. If you want to send one large message, then just go for it. Use the dirty mask to send only the data that needs to be updated and compress everything that you can. When your netcode is ready for tests, use something like Clumsy to see how it works in a particular scenario.

 

Optimal packet size - 512 bytes

 

Well it won't do anything we can't do right now, but it'll just make it less of a pain to write systems that are automagically aware of all objects that have a certain set of components. With this data-oriented approach (as opposed to object-oriented), creating one single message to sync all entities at once comes more naturally. But it is definitely possible to do it right now, without the ECS

I'm just worried about the performance implications of sending huge messages on reliable channels, so whenever we have a packet loss, it needs to resend everything. If we sent smaller message and got packet loss, it would only need to resend a small message

 

If you send and process the data efficiently, you don't have to worry about the performance.

Checklist:
The maximum safe UDP payload is <= 508 bytes.
Send the data as less often as you reasonably can.
Send only the data that needs to be updated.
Use compact bit-packing.
Use dynamic area of interest management with caching if required.

 

So if I've calculated that a complete network world state snapshot of a game with 100 characters could take anywhere between about 15 and 5000 bytes per "network tick", depending on who's near, who's visible, who's moving, whether or not I want to compress things to half precision, etc..... (I'd use one big message with bitmasks to trim out the stuff that hasn't changed)

If I'm using the Unet LLAPI, I would have no other choice but to send this via a fragmented QoS channel, correct? (since my regular PacketSize shouldn't be anywhere over ~1470 bytes, according to this and what people have said here)

So, since I'd use a fragmented message QoS for sending this big world state message to clients, my ideal "FragmentSize" to minimize packet loss should be 508 (or a bit less just to be safe). Is this what you are suggesting? @nxrighthere @wobes @HiddenMonk

This would allow messages of up to 64 fragments * 500 bytes = 32000 bytes, which is surely more than enough to contain the entire world state sync message in most games

...or is there some kind of big downside to fragmented QoS that would make it better for me to find a way to manually separate this data into regular (non-fragmented) messages of up to 508 bytes each?

 

Correct. You need to avoid fragmentation at the IP level because when the size of the resulting datagram exceeds the link’s MTU, the IP datagram is split across multiple IP packets, which can lead to performance issues because if any fragment is lost, the entire datagram is lost.

Yep.

I think that only LLAPI developers can give the answer to this question because its source code is closed and we don't know exactly how this QoS handle its job. All that we can do is benchmarking.

 

I dont even think unity would allow you to send over 1500 bytes or so unless if its on a fragmented channel (not tested, but I think I read that somewhere. Maybe thats just for the HLAPI).

For one, I wouldnt use a reliable fragmented qos, but I dont really have any reason for that other than waiting for reliable packets seem like it could be slow. (I think you edited "reliable" out of your post, so you might not be planning on using reliable anyways).
If you are not using a reliable channel, then I dont really know if it makes a difference to if you have unity handle the splitting of the packets, or if you let the router handle the splitting of packets. Staying under 508 bytes is mainly to avoid any fragmentation to begin with, but if you are already fragmenting then there might be no point.

However, if one of those fragments fail to be sent, then all fail to be sent (unless if you were using a reliable channel). Meaning its best to avoid fragmentation to begin with.
Perhaps it would be best to just have multiple world state messages sent out that are under 508 bytes. For example, players 1-50 are in world state message 1, and players 51-100 are in world state message 2. If world state message 1 is lost, but world state message 2 is received, then at least you received an update for players 51-100 instead of getting to update at all. Maybe even have it so one network tick you send an update for players 1-50 and next update tick for players 51-100. This depends on your game though, and your network tick rate.
However, if players states are not updated that often, and it would be frequent to where you are hardly sending many state updates for a player, then separating the message into multiple messages might even be bad, I guess it all depends.

Regardless, I dont even know if sending a giant world state is the way to go for so many players, but I guess the main concern is to avoid any message headers on a per player basis and instead just handle it all as a giant message one after the other.

Have you ever looked at the Quake 3 or Doom 3 network model? I think its called a "State Snapshot Delta Compression" where they send a giant game state (unreliably), and then keep sending the giant current game state until they detect you received one and use that to delta compress the new game states you send. Either they mean delta compress as in they detected you received a game state and so they know they dont need to send that anymore and can use a bit to let you know to skip it, or it means if you for example send a complete position vector of like (1500, 40, 7654), and they detect you got it, they can now send you a new position vector that is relative to that old vector, so if you only moved 5 units in X, theyll just have to send (5, 0, 0) which can be a lot less bytes used.
So long as your previous world states buffer is large enough to handle large latencys, youll be able to delta compress, otherwise youll be sending the large world state all the time.
Doom 3 also does a compression with its bit flags so that if there are many 0 bit flags next to each other, it writes down the number of 0's there are so that it can avoid writing all those 0's. It does this by having it so if it detects there are 3 0's in a row, it then counts how many 0's are after it and writes that number down instead of all the 0's. This way when you are deserializing and you detect 3 0's in a row, you know the next bytes or bits will represent a number of how many more 0's there are. This could end up being 0 if there were no more 0's after the 3 initial 0's, causing an undesired increase in bytes used. They have the number just be represented by 3 bits (so max number of 8), and they say to change these values depending on your game.

Overall, I have yet to finish a game and am only in the middle of working on the networking of mine, so take my advice lightly I guess =). I am just aiming for 16 players, so I am just sending RPC like messages and constantly updating certain variables (like position), whether they changed or not since it just makes the code easier. Some RPCs I do reliably, but I even created an RPC message like system where it sends the rpc twice in separate packets (waits for the next network tick to send it again), so that if there was packet loss for the first one, then maybe the second will make it, and if not then oh well. (I cant rely on this for sequenced channels though).

 

Yeah, I am currently working on having a 100% authoritative server with proper lag compensation and client-side prediction, and I am unsure if the possibility of missing some updates from time to time might cause some problems with the re-simulation of inputs to present time upon receiving a world state from the "past" (therefore causing some hitches in player movement). So I dunno if these state updates need to be reliable or not. I guess I'll wait until I can do some real tests

Edit: I didn't understand this initially, but I just now realized why the thing you explained with "State Snapshot Delta Compression" in Doom 3 / Quake 3 is so important. If my state updates are unreliable and I lose states from time to time, my delta compression won't work. And I also now realize that sending huge reliable messages between 30-60 times per second might result in a network traffic jam that gets a little bit worse every time a packet is lost. Unreliable state updates + a confirmation sent to server definitely seems like the way to go

ah.... that is exactly the sort of problem I had in mind when I made this thread! I think I'll most likely end up trying both approaches and stress test each one of them. But not relying on fragmented messages seems like the "purest" solution to me, so I'm definitely more inclined to try that. Besides, with a maximum of 45 bytes required for syncing a single character, for 100 players that would mean about 10 messages of 508 bytes, which is still not a lot of headers.

That's pretty brilliant! I'll try to remember this for later


Thanks guys for all your input, this is really helpful. And Merry Christmas!

 

Delta compression is just the differences sent between your state and server state, and the other thing is run length encoding (if you're looking this up later).

 

RailgunNet with a full source code created by @ashoulsonrse might help you with this.

 

That's pretty great!

 

I just released BenchmarkNet, so feel free to test it by yourself. I hope it will help you to find the right way.

 

hi, is there any documentation for RailgunNet? Thanks.

 

It's better to ask Alex, but I don't think so.