Using multiple Physics.Simulate instances

I'm developing a VR game and want to be able to move my controllers while the game is paused. The controller models, enemies, and projectiles are moved by the physics engine (thus timescale is unfeasible), and I was planning on disabling Autosimulate and use Physics.Simulate() as below.

Controllers would use Time.fixedDeltaTime in order to run as normal.

Code (CSharp):

1. void FixedUpdate()
2. {
3.    Physics.Simulate(Time.fixedDeltaTime);
4. }

Meanwhile 'pausable' gameobjects would have a scale factor which would allow me to change the timescale without affecting the controllers.

Code (CSharp):

1. void FixedUpdate()
2. {
3.    Physics.Simulate(Time.fixedDeltaTime * ScaleFactor);
4. }

However, I've done a quick test and found that Physics.Simulate() triggers all physics calculations, so I'm wondering is there a way to treat specific rigidbodies as special cases?