Question Mega-thread to help me learn to reason about NetCode

I'd say much of network programming is about finding clever tricks to hide network latency & bandwidth problems under the rug. We have this inescapable limitation of network delays and bandwidth capacities, and we must do our best to find ways to make our players not notice that these problems exist

When thinking about the netcode strategy for a specific game, I like to ask myself these 3 questions:

1. Where's the best place to hide the latency in this game?
2. How can I make this work with the limited bandwidth that's available to us?
3. Would it matter if people cheat? And if so; how can I make it as-hard-as-possible for players to cheat even if they figure out my client/server messages protocol and are able to send any data they want

If the answer to question #3 is "no", then this will make things a lot simpler for you. But the answer is rarely "no" if you have any hopes of your game becoming popular, so we'll assume that preventing cheaters does matter here.

It might be necessary to go over why the Prediction loop exists before answering this. You might or might not already know why prediction is a thing, but I figure I'll just go over it anyway:


The cheater problem

Imagine you take LSSS in its current state, and make every spaceship position synchronized over network. Clients are responsible for sending their ship's position to the server, and the server is responsible for sending ALL ship positions to each client (except for the ship that these clients own). With this setup, your ship moves instantly when you press inputs, because you don't have to wait for the server to tell you what your position is; you just handle movement directly and then inform the server about your new position.

The problem with this is that if anyone figures out a "hack" to change their position, then they can start cheating (teleporting, making their ship move faster, etc...). The server can do some case-by-case checks to try and validate the data coming from clients, but this tends to be a flawed solution that doesn't always work and makes it complicated to implement new features to your game.

So what other solution is there? The answers is: make the clients have zero authority over the state of the game. Make sure the only thing clients can send is their inputs, and the only thing servers can interpret is client input; almost as if clients all just had a wireless gamepad connected to the server computer where the game runs, and then the server sends back the state of the game to their machines for display. The worst thing cheaters can do now is to develop a program that controls their gamepad for them (actually they can also do "local" cheats like seeing through walls, etc... but we won't get into this here).

But now we've got a new problem: you press forward input, and your ship only starts to move 80ms later (or whatever your ping is) because you have to wait for your input to do the whole round trip to and from the server before you see the result on screen. The game just feels terrible to control now.


The latency problem

So we just replaced our cheater problem with a latency problem. Can we also solve the latency problem?

What if we did this:

• Clients only send inputs to the server
• The server reads client input, applies it to game simulation, and returns the state to clients.
• At the same time, clients ALSO apply their input to the game simulation, but only on the entities they control.
• When a client receives their spaceship's state from the server, it'll be a state from the past due to latency. So if we simply applied that state directly, we'd have a spaceship that just keeps getting its position overwritten to a past state when it moves
• In order to compensate for latency, when a client receives state from the server, it will:
  • Revert the state of game simulation to what it was at the timestamp of the state data we just received from server
  • Re-simulate every frame that happened between that time and the "present"
  • Hope that this re-simulation ends up exactly with the same result the client ended up with when it simulated its controlled entities locally

This is the "Prediction Loop" in a nutshell. The client tries to "predict" the state of the game even without having received state from the server at the current simulation tick, and it kinda just hopes it arrives at the same result as the server once it does receive state from the server. If there is a difference, then the state is determined by what the server says it should be. Your job is to make sure your client predictions will be accurate most of the time, so you don't get jitter every time the server corrects your predicted game state.

The takeaway here is that much of your "game simulation" needs to happen in an update group that can be triggered potentially multiple times in a frame: in Netcode, this is the PredictedSimulationSystemGroup. This systems group will be updated by netcode as many times as it needs to in order to do the "rollback and re-simulate to present" logic. So all of your game simulation systems that are involved in any way with anything that needs prediction will go in there.

But "what should actually be predicted" is the big question, and there is no general answer to that. Ship movement is almost definitely something that should be predicted, but whether or not it's a good idea to make weapon shooting be part of prediction really depends on your specific game. There are ways to make it work with or without prediction, and the best choice depends on what you are most willing to sacrifice in your specific game: responsiveness? bandwidth? accuracy? CPU time? etc... The best answer I can provide is, unfortunately, "become very familiar with how this whole thing works in order to make the best decisions with all factors involved"


The bandwidth problem

Obviously, you should always just send the data that really needs to be synced over network.

But when prediction rolls back the game state to the past and re-simulates to present, it can create an unexpected problem. Did your game simulation code rely on any kind of stateful "previous frame" data? If so, then that "previous frame" data must also be restored when we roll back the state to the past, which means it must be part of the state data sent from server to clients. This means you're paying an extra bandwidth cost.

For this reason, it is often a good idea to design your game simulation systems in a way where they are as stateless as possible. In other words; they rely on as little previous frame data as possible. This is what differentiates Unity.Physics and Havok for example. The end result is reduced bandwidth consumption, even if it can add more work for the CPU. Bandwidth can often become a bottleneck before your CPU does, but again; it depends on the game.


The netcode game loop

So with prediction, the netcode game loop is essentially a game loop within a game loop. The "game simulation" will be updated as part of the prediction group potentially multiple times per frame, but then things like presentation don't need to update as part of the rollback & re-simulation in any way.

In a frame, you'll get:

• Initialization group
• Simulation group
  • (...)
  • Receive state from server
  • Rollback game state
  • Run prediction group X times to catch up to present (this is the "game loop within a game loop" I mentioned earlier
  • (...)
• Presentation group

I wish I could come up with an easier and more straightforward way to explain all of this, but netcode is a problem that is made quite complex due to latency, bandwidth, and cheating problems. For every networked game feature, there are many valid ways to implement it and they're probably all very specific to your game

 

It may not answer all questions, but I'd give these general rules as a starting point:

• Gathering input must happen in GhostInputSystemGroup
  • ...because this updates at some point before the prediction group and also updates before the CommandSendSystemGroup which sends commands to the server
• Applying input must happen in PredictedSimulationSystemGroup , because we need to use the input we remembered for the predicted tick
  • Note: input that's unrelated to "player commands", like input to toggle a UI thing on/off, can be just gathered/applied anywhere in the update loop
• Any game update logic that affects anything that must be predicted must happen in PredictedSimulationSystemGroup. Predicted logic, by nature, should never care about whether it's on the client or server, because it must be identical for both
  • Ship movement is predicted, so it goes in PredictedSimulationSystemGroup
  • If there's an ability system in your game and some ability can affect your ship's speed, then the ability system update must be in PredictedSimulationSystemGroup as well, because it'll end up affecting the outcome of your ship movement, which is predicted
• Any spawning/destroying of ghost prefabs happens in server-only logic, whether it's in a server-only system, or in a line of code under a "if(isServer)". Anywhere in the frame before the GhostSendSystem should do the trick. When the server spawns/destroys a ghost, that ghost is spawned/destroyed on clients as well
  • You might be under the impression that spawning things during the prediction group on the server would be a mistake, because rollback+resimulation might end up re-spawning a ghost multiple times, but.... in Netcode, you can always trust that the server world never rolls back. The server always simulates the prediction group once per tick, and it's the clients' job to always "stay ahead" of the server so that it doesn't have to roll back. If the client fails to stay ahead, it will see a little server correction happen and try to fast-forward its time in order to catch up
  • But sometimes, not everything that should be visible on all clients should be a ghost. Projectiles are a good example. In some games they might be ghosts, but in other games they might get spawned as non-ghost prefabs by clients based on some "uint shotsCounter" that's synced over network on a weapon ghost. It can be a way to sacrifice accuracy in favor of bandwidth savings. Explosions are another example: you can choose to make the server spawn explosion vfx ghosts when it detects that a ship got destroyed, or you could make clients spawn explosion vfx prefabs locally when it detects that a ship's "bool isDead" is true (or when the ship entity is destroyed).
• I'd say performing lag-compensated collision detection on the server sounds like a good idea for this game, although I agree things are not moving super fast so it's kinda on the fence. You can see an example of how this would work here. I don't think this should be done on the client though. Generally, it's the server's job to reconstruct the interpolated state of things that the clients are seeing, and this is what the lag-compensated server collision detection should accomplish

 

Would a breakdown of the major Netcode steps in a frame (receive data, spawn ghosts, gather input, rollback, prediction, send data, etc...) be the answer you're looking for?

Or are you looking for something that would help you build a general intuition (as opposed to case-by-case examples) for what should be predicted, what should be interpolated, what should be a ghost, what should be synced and how, what should update when, etc...? I think an answer like this would be more useful than my previous replies, but it can be tricky to come up with that answer

 

- Bullet can use predicted spawning (there is documentation about it and sample in Asteroids and HelloNetcodea about how it work).

- For the confirmation of a destroyed ship, it depend on the latency. The client predict the server ahead of time. So at the very least the confirmation for an hit will be received after RTT x SimulationTickRate ticks, plus some ticks in case of packet loss. Usually the effect can be spawned and handled without problem.
The ship can't be destroyed on the client (client MUST never destroy ghosts). At most, you can disable the entity.

 

We have some plans about that. So stay tuned

 

Here's a first attempt at mapping out what netcode is doing in a frame.

Initialization system group
There are no changes to this group itself. It runs before the insulation system group and runs once per frame. For this system delta time and the netcode singletons are not updated yet - so be careful if using them.

The only netcode specific thing in this group is the client only NetworkTimeSystem which calculates the target prediction tick and interpolation tick based on delta time, most recent received snapshot for interpolation tick and server reported command age for prediction tick. The calculations try to keep delta time close to “real time” and applies time scaling in order to keep in sync.

Presentation system group
This also runs once per frame after simulation as usual. The group only exist for client worlds, server worlds and thin client worlds does not present anything so the presentation group is not used. Delta time and elapsed time is not adjusted to network time when this group runs so it is not in sync with the server.

Simulation system group
I left this for last because this is where all the complexity is. This group will run once per frame on clients, but it will run once per simulation tick on servers. This means the simulation group on servers can run 0 times or more than 1 time in a frame. The default behaviour in 1.0 is that it only runs 0 or 1 time, if it needs more iterations it will run with a higher delta time instead of running multiple times (so once with delta time 33ms instead of twice with delta time 16.6ms). This is slightly less accurate, but it avoids the feedback loop where adding more iterations makes it take even longer so it just keeps running more and more iterations in order to keep up.

Before this system group runs Time.DeltaTime and Time.ElapsedTime is updated to the correct time for this simulation step, or the correct time for the prediction tick on the client. This will also update all singletons with target ticks etc.

The first thing that runs in the simulation system group, right after the BeginSimulationEntityCommandBufferSystem, is the client-only GhostSpawnSystemGroup. This is responsible for spawning newly received ghosts. All spawns received in snapshots are put in a queue so they can be spawned at the correct interpolation / prediction tick, and the system spawning them runs right after the ECB system because both of them need to wait for all jobs - and by having them close in time we reduce sync points.

The next thing to run is the NetworkReceiveSystemGroup. This group is responsible for checking for new packets received from the network and storing the data from them as entity data used by later systems.There is a server only CommandReceiveSystemGroup which will decode received input commands and store the command data in the correct entity.

Next up is the GhostSimulationSystemGroup, which is where most of the networking code not related to package send/receive or prediction itself goes. This includes things like Despawn (using a queue similar to spawning and a begin simulation ecb), collection handling which updates the ghost prefab list based on generated code and loaded prefabs, prespawned ghost processing and a lot of other things.

This is also where - on clients - we decode snapshots in the GhostReceiveSystem, then copy the values from the correct snapshot to the target entity - with interpolation/extrapolation for interpolated ghosts - in the GhostUpdateSystem. This copy of values is what results in the rollback before prediction. We also run something called classification systems, these systems will look at the received new spawns from the GhostReceiveSystem and determine if they should be predicted or interpolated so it can place it in the correct spawn queue.

The SimulationSystemGroup is also where - on clients - input sampling happens in the GhostInputSystemGroup. We do some processing of the inputs right after this in order to support IInputComponentData / InputEvent. After that processing we send the commands to the server. We want to do this as early as possible to reduce latency, but we need to decode the snapshots first since the input packet also contains snapshot acks.

Prediction
Everything up until this point is netcode internal stuff intended to run before your code, the first user facing thing that runs is the PredictedSimulationSystemGroup. This is setup to mimic what SimulationSystemGroup would normally do in a single player game, but there are a few differences.

The prediction group will run multiple times on clients, but only once on servers because the full simulation group is fixed step there. The last step on clients can have a lower delta time which we call a partial tick - so it is not quite fixed step on clients. This partial tick is basically reusing the prediction code to do extrapolation so we can render at a higher frequency than the simulation tick rate.

The prediction group will set the enabled state of the Simulate component on all ghosts before it runs. This is important because ghosts can be rolled back to different snapshots which means only a subset are to be simulated in a given prediction step. All systems in this group writing to ghosts must use WithAll<Simulate>. You can always have that check - also in non-netcode single player systems which means you can have systems work in both netcode and non-netcode contexts.

On servers there is also some systems which applies the correct received commands to the IInputComponentData, on the client the prediction loop is the only place you should process inputs since they will not run for all ticks and not handle server corrections outside the prediction loop.

The start of the prediction is the PredictedFixedStepSimulationSystemGroup. This is the same as FixedStepSimulationSystemGroup but for prediction, this is where predicted physics runs.

After that your non-fixed step prediction code runs.

At the very end of the last full prediction step in a frame we will do a backup of the predicted values on the client. This is used for a few things. It is used to restore data in the GhostUpdateSystem if we did not receive new data from the server. It is also used for smoothing and validation. Next time we predict this exact tick after a rollback we compare the values to this backup to detect prediction errors and to smooth out the errors.

After prediction has run you have another game loop which is client or server only, the normal SimulationSystemGroup systems. This has the normal fixed step simulation followed by your systems. So you can really see it as two separate loops, first one for the core game state which can be predicted, then one for client or server specific logic.

Simulation system group end
The last thing that runs in the simulation system group - after EndSimulationCommandBufferSystem - is GhostSendSystem on servers - which serializes the current world state and sends it - and the RpcSystem which serializes all new RPCs created this frame and sends them.

 

RPCs are reliable as far as eventually arriving and arriving in order, but they can be processed out of order. All messages received in one frame are processed in undefined order even if they were send on different frames - use your own sequence number if order is really important and you expect multiple messages in flight. Lower sequence numbers will be guaranteed to have arrived this frame or earlier as long as you send in order.

The two issues you can get is timing - you don't know exactly when the message you just sent will arrive - and flooding, you consistently try to send more messages than the transport layer can handle so the queue keeps growing.

The flooding issue is mostly a matter of keeping the number of RPCs low.

For timing, the model we are using is server authoritative, so I always apply that to RPCs too. I don't care when the client sent something, I only care about when the server received it. In my experience putting the responsibility of dealing with that on the client - which has a single delay to consider - is much more robust than trying to have the server reason about the different delays for each connected client.

There are many different ways to deal with that delay client side depending on context. If you have some countdown for joining you can for example try to adjust the countdown displayed on the client so it takes the expected time to send the message into account. The timer will not be perfectly aligned with the server anyway since time is not synchronized between client and server (neither predicted tick nor interpolated tick is the current server time).

For making sure everyone has loaded a map, that is mostly a matter of the server waiting for all clients to confirm they are done before proceeding. The flow would be something like:
* Server puts all client not in game
* Server broadcasts "load level"
* Server loads the level
* Clients receive the message and start loading level
* Clients when done send "level loaded" to the server
* When server has receive "level loaded" for all clients and has loaded it's own level, start the game again

There is additional complexity if a client joins in the middle of the transition, then you need to send "load level" to that client too, and wait for it - or send "you can't join" and close the connection if joining is not allowed.