Joints Not Staying In Place When Pushed

So i'm trying to make a "cabinet door" you can pull open.
I have a plane caled "window" that has a rigidbody, box collider and configurable joint.
My controllers also have a rigidbody and box collider (both have isTrigger=false).

The joint is set up to only allow X rotation.
This works fine when touching the door from the front/back, but once i try to "lift" it up the window comes of it's joint and eventually snaps back.

GIF showing the problem

Controller setup


I've been at this for 2 hours now and it's really frustrating, because it is supposed to be such a simple mechanic..

I just want the door to stay in place whatever force hits is and only rotate on the X axis..