Astronaut with arms

I have been struggling with this for a while now and really need some suggestions on how to approach this.

My player will be a FPV astronaut, who is able to push and pull themselves around in zero-g. So far I have a player character torso, which is a rigidbody with mass. I also have the relative position of where the player wants to put their hand (from a VR controller). I also have an in-game hand (the visual model of the hand). When just floating in empty space, the vr controller and hand position are identical.

Now imagine the player is floating a few centimeters on front of a wall, and they stretch out their controller on front of them to arms length, at high speed. In the game world, I want the hand model to move forward until it hits the wall. Then I want the player to start translating backwards, with a force which is probably proportional to the distance between the hand and the controller position. As the player torso moves backwards, the vr controller position moves towards the surface of the wall, eventually to the same point as the hand model, and then the hand lifts from the wall and the player is back out in free space (still drifting backwards).

If the player is not on front of a wall, but on front of some small object like a box, and they repeat this move, i'd like to see the box move away and the player move back in the opposite direction.

There are a few other variations I eventually want to do, involving grabbing and rotating, as well as holding on with one hand while pulling and pushing with the other, but my problem is that no matter how I arrange the rigidbodies or hierarchy I just cant get it fully right. It doesn't seem like a difficult problem, but having the hand rigidbody move the player rigidbody AND the other object rigidbody has eluded me. I'd really appreciate some suggestions on how to tackle this problem!!!

 

Disclaimer, i have zero expience with VR.

What you can do is use a trigger on the hands, when they touch somthing use a bit of code that checks is the object static? If so push the player back the distance the hand reached over the wall, if not static just let the normal physics system handle it?