Question Physics.Simulate Jitter?

So my end-goal is to be able to do clientside reconciliation on rigidbodies.

First, during a single update() frame, I teleported my player back to a position 5 movements ago, and then replayed all physics movements from then until the present using physics.simulate, but that gave me jitter on my character. Obviously your main thought would be, because the non deterministic physics engine, ill end up in a different place 100% of the time right?

But then I did something else, during another single update() frame, I made my rigidbody player kinematic, ran physics.simulate a couple times (shouldn't affect a kinematic body right?) and then after that, made the rigidbody dynamic again, and experienced the same jitter that I was getting before. My goal with this, is to only simulate specific objects and not the entire scene. My movement code happens in fixedupdate(), if that changes anything.

Is this normal? Am I doing something wrong?

Keep in mind, if I removed physics.simulate (still making the character kinematic/dynamic every frame and such), then the stutter disappears.

In the end, am I going to have to do this in a separate scene? Is this how it's intended to be used?

(I tested this on multiple machines with the same result.)

 

Update: Also tried RigidBody.Sleep and RigidBody.WakeUp, still have those weird jitters every time physics.simulate gets called. Super odd.

Just a reminder that if I take physics.simulate() out of the mix, I get no jitter whatsoever. Even with making it kinematic/dynamic or waking and sleeping the rigidbody during a frame.