Question RPC call for AddForce only works when object is visible on remote client

Hi, I hope someone here can help me with this strange networking problem that I have.

I have recently started using mirror to create a multiplayer game and as part of that game players can pickup and throw objects. To get this working across the network the pickup and throw actions happen on all clients via an RPC call, so I have a local call to a method for the local player that does nothing except call into a Command and that does nothing except calls an RPC out to all clients, it’s the RPC that then performs the actions of picking up or throwing.

Pickup works fine and tracks over the network, I am making the picked up object a child of the player which turns kinematic when picked up, via an RPC call to all clients as mentioned above.

The throw command then turns off kinematic on the object, sets parent to null and adds a force to it. This all works ok over the network, via an RPC call, even if a bit jittery sometimes but its generally good enough for what I want.

The strange problem I am getting is, the throw only works ok if the object being thrown is currently visible on the viewport of the client that is running on the server. So if a client that is not on the server throws the object while the client on the server is not looking at them, the thrown object jumps about all over the place. I am assuming this is because the RPC call on the client that is on the server that does not have the object currently visible and does not update it correctly? This then causes the object to jump around on the client performing the throw as it tries to be in 2 places at once.

I can prove the problem is caused by the remote RPC call by checking if the object is visible on the client running on the server and then not running the RPC call when it is not visible which makes everything work for throwing but means that that client (client running on the server) does not know the object has been thrown so this does not work as the object is then in the wrong place on that client.

I have tried performing the throw command locally, and/or on the server or setting the thrown object to have client auth but nothing has worked so far.

Any ideas what is going on here?