SteamVR player keeps going through walls while climbing

I would like to apply a climbing mechanic in my game but the player can grab on the climbable object and force themselves through walls and other objects.The code takes the delta distance vector of the controller and its previous position and add it to the player's transform. I tried applying a velocity to the player with the same vector and it does prevent the player from going through walls but the camera freaks out and the controller moves faster than the player's movement, causing the player to fall when climbing. Overall, it works but I would like to keep the player within the level and not go through walls to escape. Any suggestions?