Resolved (IN-39547) [1.0.0-pre.65] Significantly slower than older version

Thanks for bringing this to our attention. We will have a look at this.

Did you notice any particular areas in the profiler which show where the change in time consumption originates from?

Note that depending on the amount of content in your scene, there can be a significant overhead that is caused by an integrity check that is only done in the Unity Editor for validation.
See here for more information.

 

Thanks for the added information. Could you share a profiler screenshot that shows this?
I have not observed more than one BuildPhysicsWorld run per frame except if you use Netcode for multiplayer which does predictive simulation (multiple steps per frame). Are you using Netcode here?

This is what I am seeing (one time BuildPhysicsWorld system update). Example profiler run from the scene "Assets\Tests\Pyramids\Pyramids.unity" in the official Unity PhysicsSamples project.


And here is the hierarchy view for some other frame in the same run. As you can see the BuildPhysicsWorld time consumption is low and it is called only once.

 

That's very odd.
Can you try the scene I mentioned above, to make sure we get a baseline to work with? I tested all this with the latest version.

 

@Thygrrr : Great stuff! Thanks for the thorough investigation.

One thing I am wondering is how the 120 Hz display plays into all of that. Could V-Sync and/or the default ECS timestep of 60Hz play a role here?

Btw, the case above where you noticed time lost in the editor loop, wouldn't that just be the loop enforcing real-time framerate with a 60Hz simulation time step (which is the default in ECS; see SystemAPI.Time.DeltaTime)? If there was no "pause" somewhere, the simulation would run faster than real-time.

 

@optimise:

After having had a closer look at your screenshot, I can see that the whole physics system group is run 5 times here per single fixed update.

The reason for this is likely that the FixedRateCatchUpManager is trying to catch up with the real-time simulation rate, but needs to step the physics an excessive number of times since a single step time consumption is likely too high in your scene.

The Entities framework employs simulation sub-stepping for cases in which the system needs to “catch up” due to excessive time consumption in the game loop.

This is done in the FixedRateCatchUpManager, which implements system update semantics similar to UnityEngine.MonoBehaviour.FixedUpdate().
The method can be configured in a way that a single "catch up" step does not exceed the World.MaximumDeltaTime value, which prevents simulation issues such as tunneling and the like.
However, due to this clamping of the variable time step, sometimes the system needs to step multiple times to catch up which then aggravates the issue in a negative feedback loop, as is likely the case here.

So, in summary, have a look at the stepping parameters used by the FixedRateCatchUpManager, and either configure it in an attempt to reduce this effect, or solve the issue at the source by optimizing your scene further.

 

I totally understand that. We have your test project in hand and have filed an internal bug report for investigation and are going to look into this. Stay tuned!

 

Thank you for the feedback! We continue to investigate this issue as part of our overall backlog. We'll share more when we have additional details.

 

There are two major stall in the physics 1.0 that cause the build physics world to wait for other jobs and

Daniel is investigating on it. There are indeed some little more stalls now by "default" on the main thread in the physical loop.
In particular, the internal BuildPhysicsWorldDependencyResolver system is usually the one that add some of these stall, since it introduce a sync point on the main thread: it will wait for all jobs that touches PhysicsWorldSingleton, BuildPhysicsWorldData and / or PhysicSimulation singleton entities.

That being said, the game just entered into the classical death-loop: there was a long frame (maybe because of physics), that cause physics run multiple times, that will produce again an even longer frame, etc etc up to a point where it become like that.
This is never ending, unless the catch-up mechanism decide to say something like: "you know what, let's simulate bigger steps (imprecise) to exit this loop". But this is not the way it is implemented.
I would suggest to implement your own FixedRateManager and do some logic there to determine how to clamp and handle cases where the elapsed time in between frame may cause physics to run to slow and enter into this situation.

Another solution is to not-rebuild the world at all, apart the first time. And that can be implemented by a custom system that selectively:
- Disable the BuildPhysicsWorld
- Ask instead to update the AABB tree (imprecise collision detection though)

That requires some little addition at the moment to the BuildPhysicsWorldData to make this cleaner but it is a viable solution, that led to very good performance improvement.

Because for netcode we had similar problems, if we have high latency the client need to simulate a lot of physics step in one frame to catch up, we experimented on a bunch of thee.
By applying both a sort of "adaptive physics-step rate" (that lead to misprediction, and non-determistic results, as I described) as well as by avoiding to rebuild the physics world, you can limit the problem.

 

@n3b: Thanks for the pointers. Note that we are aware of some data duplication between ECS and core Physics and we are discussing this internally. We'll keep looking into this.

@optimise : quick update on the data and project you provided us with.
Looking at your data specifically for 1.0.0-pre65, I can confirm that in your scene physics is stepped twice per frame.
That's why you see a "sync point" blocking execution within the BuildPhysicsWordSystem.
This is the second BuildPhysicsWorldSystem, which naturally must wait until the first step is completed (marked below with an X).
So it only seems as if this system is wasting time while the jobs are actively working on finishing the previous step as you can see in the screenshot. The sync is obviously necessary since we can start a second physics step before the first one has completed.



I indicated with arrows to the right that the same sequence of jobs will be scheduled again here as part of the second step in this frame.

 

Your suggestions are all very valid. I think these are good general improvements and I took a note for us internally. Thanks again.

 

The performance would not automatically improve since the sync point is happening in the middle of 2 consecutive physics steps. The jobs launched by the physics systems for the first step need to be completed before the now waiting BuildPhysicsWorldSystem can launch step 2.

If we were to remove the sync that now causes the wait in the second execution of BuildPhysicsWorldSystem, and if we could possible schedule all the jobs for step 2 ahead of time without even having any information on what the result of step 1 is (we can not easily do that), the BuildPhysicsWorldSystem would not block on the main thread but the work scheduled by the systems from steps 1 and 2 needs to be completed anyways. So you wouldn't get any difference in the end in regards of job execution time for both sets of jobs (for steps 1 and 2 combined) whether there is a sync in the middle or not, since the sync is waiting on running jobs. So in terms of job processing, there is no gap. The processing is never halted.

If however you want to do some extra work on the main thread after completion of the physics systems from step 1 (at the moment where now you have the BuildPhysicsWorldSystem waiting), that is perfectly fine. You would just need to add your systems to the correct group to benefit from that gap so that your systems are launched after the systems from the first physics step have finished (not necessarily the jobs though!) and before the BuildPhysicsWorldSystem starts execution of the second physics step. Just before of data dependencies here. You will not be able to read any data that the systems and jobs in the first physics step are still manipulating, like the velocities and transforms of rigid bodies, until these jobs have finished (which is exactly what the BuildPhysicsWorldSystem from the second step is waiting for also btw).

That group that you could add your systems to could be
the

AfterPhysicsSystemGroup

.

I wonder if you do indeed want 2 consecutive physics steps happening within one FixedStepSimulationSystemGroup update. This could be the main issue here, which blocks everything up.
The reason why this is happening is due to the activities of the

FixedRateCatchUpManager

which decides that you need an additional physics step. For more details on how to control the behavior of this catch up manager (which is used by default in Entities), please refer to this other post on this topic.

 

@n3b : Would you mind sharing the parallelized implementation of BuildFirstNLevels btw?

 

@optimise : Quick update here. I am getting almost twice the FPS with 20000 spheres spawned in 1.0.0-pre65 compared to 0.5.1 with your stress test. In 0.5.1 it's almost coming to a halt while in 1.0.0 it's going at ~45 fps even after all spheres have dropped to the floor. Before, while the spheres are in the air, it runs at 60 FPS.

I will look into this more now.

Edit: The slow down in the old version might be due to the rendering (more than just the game tab open). I will confirm.

 

Very helpful. Many thanks for the details.

 

Alright, I had a closer look and for 20000 spheres I am getting quite similar results between the two versions.

Here is a common frame at the end of the drop in version 0.5.1:


B roughly marks the beginning of the physics step
E roughly marks the end of the physics step (last ExportPhysicsBodies job is ending there).

So, you can see that the physics is done about at time 10.5 ms since the beginning of the frame.

Here is the same situation in version 1.0.0-pre65:


As you can see, the frame layout on the physics side is very similar and here it also ends at about time 10.5ms since the beginning of the frame.

One big difference here is that these regular spikes that you see on the timeline in 0.5.1 are gone in 1.0.0.

Note that I had to make sure that Burst Safety Checks are disabled in both runs. Otherwise, I was getting worse performance.

So, to conclude, in cases where there is a single step done per frame, the performance doesn't look much different between both versions with potentially a small improvement in 1.0.0 compared to 0.5.1 and more stable frame rates.

Furthermore, as soon as the FixedRateCatchUpManager decides to do a second physics step within the FixedStepSimulationSystemGroup, you naturally get the sync point at the beginning of the second step, and you will get much higher time consumption in a single frame. This is just normal if this happens. If you want to avoid having more than one step per frame, you need to change the RateManager as previously mentioned.

 

Since the timing results with a single physics step per frame and 20000 spheres are very similar, while both being significant in time consumption and they were run on the same machine, we can conclude that there has not been any performance regression.
More spheres might just get you into the "two steps per frame" territory, which might show as a regression, but could just be a different stepping pattern (1 step in old version vs 2 steps in new).

I didn't run this on a particularly performant machine. I ran it on a laptop with i7-10850H, which is good but not the latest and greatest. That said it does compare positively against your CPU overall, with single threaded performance being comparable (rating of 2715 vs 2068). Obviously, specifically with DOTS, parallel performance matters most.

What might be a big factor also is that I made sure that the jobs safety checks were off and I only render one view (the game tab). This made a big difference in performance in both versions.

So, I would urge you to check if you do get into that "2 vs. 1 physics step" situation between the two versions and if you don't want it to ever do 2 physics steps, simply change the RateManager in your 1.0.0 version, as I had suggested above. Details for how to do this can be found in this other post.
In a nutshell, changing the RateManager to the FixedRateSimpleManager might be the solution in your case.

 

After I had some unit issues with the data captured in the profiler in 2020.1 and in 2022.2 (the former used units of seconds while the latter used milliseconds, making it impossible to compare the two data sets in the Profile Analyzer), I have now successfully converted the units (by saving the 2020.1 profiler data as .pdata file in the profile analyzer in 2020.1 and loading it in the profile analyzer in 2022.2) and was able to create this apples to apples comparison:



Highlights:

• 0.5.1 on 2020.1 shows some occasional spiking which can be seen in the analysis (see max in the 0.5.1 range at the bottom right). These spikes are gone in 1.0.0 on 2022.2.
  Note that these spikes are infrequent enough not to skew the comparison, as can be seen by the median and mean data points for 0.5.1 which are almost the same.
  
• 1.0.0 is on average ~8% faster than 0.5.1, as can be seen in the median (and mean) data points (see bottom right).