Bug [0.51] FPS tightly couple with network condition

mmm. I need more info about what it is slowing down in order to understand where the cause is.
Can you do a profile and see what has changed in your project such to cause a 50% FPS impact?
In term of NetCode, we didn't change that much in between 0.5 and 0.50.1 (major things are bug fixes)

 

Can you please clarify what you mean weird or unstable network condition?
It is about latency changes (it jitter) or high latency? Packet drop?
The fact the prediction group consume a lot of cpu is undeniable. If the latency is high we are predicting a lot of frame ahead of the server and when we receive the snapshot the rollback cost can be quite significant.

I can't guess however how many ticks you are trying to predict ahead (you can get these from the net debug, but you may also just print the average) and which systems are slow (that are your implementation details).
Furthermore, since I presume this is from the editor, how did you run the profile? Thread jobs enabled, Job Debugger enabled, Burst enabled but with with sanity checks (that slow down quite a bit).
Would will be interesting to understand what is exactly taking all this time.

We already worked on batching together multiple prediction frames but will be available for 1.0. That can save quite a lot of performance.

 

Ok, so I was in the player, that mean we can remove some variables.
Trying still to understand more: 70ms ping should imply (given the 2 extra tick of slack) the client should run approx 8 ticks ahead of the server.
So every time you receive a snapshot from server (that send at 60 fps) and assuming (worst/best case scenario) you receive them every frame, you will do every frame on the client at 8 prediction steps.
That means, given you screenshot, every step takes about 4-5ms at least.
Are these number what are you experiencing or are they different? (maybe we are doing some miscalculation).

Other questions are:
How many ghosts?
How many systems are running in the prediction loop?
Are you using physics?

I'm trying to understand if there is something in particular the is dragging your FPS down.

 

Interpolated ghosts are not predicted, as such they don't have a PredictedGhostComponent.

If you are using physics only on the server, and removing all physics component from the client (that imply also PhysicsVelocity) physics prediction on the client in practice does not run for most of the entity.
But..

You said you removed the physics components but you are using physics for collision detection. So now I'm a little confused.

Interpolated ghosts are configured as kinematic physics object (if they have a have a PhysicsVelocity component).
As soon as you a predicted ghost, the prediction loop start running.
When the BuildPhysicsWorld kick in, all the entities with PhysicsVelocity component are considered. That is also true for the StepPhysicsWorld and Export step. As such, event though interpolated ghost are not running "prediction" they are still needed to build and updated the client physics world state used by the few predicted physics ghost.
As such, the interpolated kinematic ghost count (as cost) toward the physics simulation.

Now, there are a lot of optimisation opportunities here: you can just build the PhysicsWorld once, and just update the AABB tree with the only one ghost that it is moving instead or rebuilding every frame.
If the number of physics object ti small, better to disable thread scheduling for physics (see PhysicStepSetting) or just skipping using jobs altogether (I think physics sample show something for that).
It also worth considering to add to the physics systems (Build, Step, Export etc) very little simple checks before executing / scheduling jobs that the query actually has entity to process.
Something like:

Code (csharp):

1.  
2. if(!query.IsEmpty())
3.   call the jobs scheduling
4.  

this alone can easily remove about 5-6ms or even more (given you are doing 8/9 steps per frame, the cost of scheduling the physics jobs is quite large) in case there no entities to process. (but this is probably not your case).

Because of the spiral of death. The server run at fixed time step (60hz lets say). There server it always does only 1 simulation step, until the performance drop below 60 doing more steps per frame
Clients runs at variable frame rate and does multiple prediction steps per frame, that imply multiple physics step per frame as well.
When client and server are on the same process, things can get really bad, and pretty fast.
The client start consuming more CPU because it need to run more physics stuff. That means larger frame time.
The server in turn does more steps, and the packets from the server are now even more ahead (because is doing more step per frame). That will cause in turn client running more prediction, that cause a larger frame time, than more steps for the server and so and so on.

In 0.51 you can configure the server to "batch" multiple step together and limit the number of ticks there server does per frame. See
ClientServerTickRate MaxSimulationStepsPerFrame and MaxSimulationLongStepTimeMultiplier.
This should alleviate a bit the performance issue in the editor.

That being said, if you one 1 single ghost, I think you are also doing something wrong somewhere. Like making tons of structural changes or other things. Also, you many not have burst-compiled some jobs (maybe).
And given the low number of entities, probably scheduling the jobs it is still the larger cost. I suspect is going to be faster to just call Run on the main thread (and burst compile the job of course).