SteamVR trying to make grabbed objects not freak out with other objects

I'm working on a VR game and I initially created a grabbing code that creates a fixed joint between the controller and the grabbed object but results in the object freaking out when being forced into walls/floors and other objects. Then I tried implementing a mechanism that instead makes the object follow an empty object that is parented to the controller. In the FixedUpdate, the object's rotation matches the target's world rotation and the velocity is set by subtracting the world position of the target and object then multiplied by some value to make the object move faster. The result looked slightly better but the object still freaks out when being forced into other objects and there is also movement lag when the player moves/rotates using the controller buttons.

I saw that SteamVR Home has a very nice physics collision when grabbing objects that I wanted to see if that interaction can be implemented in Unity. I recorded myself grabbing objects and pushing them into static and dynamic objects to show how the objects interact.