Bug (IN-49838) [1.0.11] Massive performance spike

Hi @optimise,

Thanks for reporting this. I didn't find any information on how to reproduce this issue in the ticket you filed.
Can you provide some information on how to reproduce this or provide a screenshot of the profiler showing the issue here as a first step?

Thanks!

 

Oh yes. That looks indeed quite significant.
Could you please provide us with a description of how to reproduce this?

Our performance regression tests don't show any spikes.

 

Ok thanks. I see that you attached a project for reproduction now. I'll have a look.

 

I was able to produce some spiking on my machine with your test scene.

To confirm my understanding, I consider this here a good frame (no spike):


We can see that the EntitiesGraphicsSystem is waiting on the physics jobs to complete, as expected.
The dashed line above marks the end of the physics jobs and then this graphics system can prepare the rendering of the results of the physics simulation. So far so good.

When we do get a spike, we get this situation:


As before, we have the EntitiesGraphicsSystem waiting on the physics jobs to complete, which is normal.
But at the end of the frame we get an extra time slice in the EditorLoop.
Is that what you are also observing on your end?

Note that this is all run in the editor at this point. I will profile on a player build next.

 

When I run in a player build, the spikes completely disappear btw.
See here:


Regarding the stall on the main thread in EntitiesGraphicsSystem:
There is a data dependency between this system and the physics systems through the transformation components of the Entities which the physics is setting and the graphics system is reading for rendering purposes.

At this point in the framework, these data dependencies are managed on the system level, meaning a system will need to wait for the dependent jobs from other systems to have completed before it can commence its work.

 

I just had another look specifically at the job dependency completion that we see within the EntitiesGraphicsSystem.
The completion is in fact an explicit one within the OnUpdate() call from that system itself.
So I will reach out to the wider team to see if that is something that is absolutely necessary or could potentially be changed, hopefully freeing up some time on the main thread, e.g., by scheduling jobs dependent on the data produced by the physics and letting the job system start them whenever the dependency is resolved, rather than completing the system's input dependencies manually as is the case here.

 

Regarding that sub-optimal parallelization you are pointing out here, we are in fact looking into that at the moment.

The solver is currently parallelized by solving groups of independent constraints in parallel phases. We find these groups with a fast, greedy algorithm that is run sequentially (you will see it in the timeline as "CreateDispatchPairsPhasesJob") which has the limitation that it does not support more than a fixed number of such groups. All remaining constraints are then solved sequentially in a single job at the end of every solver iteration (there are 4 iterations by default), which is the part on the right you are pointing out.
Specifically in these "object heap" cases with many stacked and piled objects, you can run out of parallel phases easily.
Another example of this is the Planet Gravity demo in the PhysicsSamples project.

As for the ParallelCreateContactJobs, if you scroll up you will see that the job is simply run on the main thread in this case, given that there is that "Complete()" call from the EntitiesGraphicsSystem, freeing up the main thread for work.
So, the maximum number of worker threads is in fact used, the timeline just shows one job processing phase on the main thread in this special case. If we were to remove the Complete() call in the EntitiesGraphicsSystem mentioned above, the job processing would again show on the worker thread.
I hope that makes sense.

 

Finally, by using the "Edit Mode" option in the profiler (top left, by default set to "Play Mode") while in play mode I was able to see where the spikes that we observed above came from.
These are caused by the profiler itself which is infrequently updating its window.
See here:


In the frame before the spike (marked on the left), the profiler window is not updated and there is no time spent.
In the frame with the spike (marked on the right), we see the large processing overhead caused by the profiler window update.

 

If you are referring to the

Job.Complete()

call we can see in my comment here, this one is currently necessary because the solver does not know yet how many jobs to spawn while the phasing algorithm is still operating, which decides how many parallel phases there will be.
In a previous version, this

Job.Complete()

call was not present, which caused the solver to spawn as many jobs as there could be maximally required, which were way too many and overwhelmed the job system with unnecessary job schedules and executions (max "solver iteration" times "32 phases" many job executions), which in the end effectively slowed the physics pipeline down significantly. This is no longer the case.

Ideally, we would launch a solver job that then launches the correct amount of solver phase jobs once the information comes in, freeing up the main thread. However, since in the C# job system there is currently no way to spawn jobs from within other jobs, this is not something that we can do at the moment.
We do have an eye on that though as a general area of improvement.

 

@tertle : Many thanks for exploring this avenue.

The approach makes sense. It's not unlike the idea I brought up above of spawning the required solver work from within a first solver job, but rather via a manually managed work queue. Given that we already have all the "work batches" calculated, your approach requires low overhead in the pipeline (all the data is already there and organized as required).

I'll take not of all this and will attach it to our internal ticket on engine performance. There are a few spots in the pipeline which have potential for optimization. Removing the solver sync point without impacting "low load cases" is one of them.

 

@tertle : And btw, regarding the other change you did here ("Uniform scale no longer bakes into colliders."), keep your eyes peeled for upcoming releases . We've had that on our short list for improvements.