Getting doors to work properly without an additional toolkit or whatsoever

Hello Unity-Community,
I am new to Unity and have a problem I am not able to solve.

I am using the Oculus SDK and can easily grab objects with adding the component OVR Grabbable to them.

The problem is, that after freezing the rotation and position of some axis and therefore restricting the object's movement won't do anything to the way the object behaves, when I grab it. I want the object to still have the restrictions in movement, specifically for doors or objects with a hinge joint respectively.

As mentioned in the title above I don't want to use any additional toolkit like NewtonVR or something else, because there are way too much objects already, which work, how they should with OVR Grabbable attached to them. And with using something else, I would have to change all of the objects, otherwise grabbing would not work anymore.

Did you understand my problem and can anyone help me with this?