Switching authority for rigid bodies?

I'm making a simple soccer game and I'm using NetworkTransform with local player authority for players. Then to have nice physics, I want to assign authority of the ball to the closest player. This way, when they interact with it, it feels good (otherwise they don't properly bounce, etc.). However, when I switch player authority, the ball usually spasms for a few frames. It tends to settle to the correct position...but what's going on here? The server also prints to console:

HandleTransform netId:3 is not for a valid player
UnityEngine.Networking.NetworkIdentity:UNetStaticUpdate()

Not sure if that's related.

 

NOTE: The authority-switching logic is happening on the server-only, inside a FixedUpdate. I've seen other users switch authority via [Command]'s - does it matter when I do it?

 

After some more investigation and thinking, I suspect the snap or spasm is fundamental to authority loss. When you relinquish authority, you're necessarily seeing the object with a delay now, so the object will snap back in time for a little bit. I think I could hide or smooth this out somehow...but probably not within the confines of NetworkTransform.