Huge spikes at random times from the ECS scheduler

So I am getting some big random spikes from job management on the CPU. Seems to happen at random after some period of time.

Here is an example from a deep profile. When the deep profiler is not on, the spikes will be about 5ms which screw up the physics. Running a VR game so I can't have sudden frame drops every few seconds.



Has something to do with Job scheduling I think. I also get this majorly on all the systems I am running.