Limited Hinge Joint snaps to 0 at the slighest nudge

I'm trying to make a book work in VR by using a Hinge Joint to connect the cover to the back of the book. The Hinge is set to be limited from 0 to 180 degrees, but that's not how it behaves.

For some reason the hinge limit kicks in 3 degrees early at exactly 177.0 degrees, making an open book sit awkwardly on a flat surface.

It also violently snaps shut to 0 degrees if just a light force is applied at this 177 degree point, making the book really annoying to handle.

Any ideas what could be causing this? I've tried all kinds of rigidbody mass and hinge mass scale / connected mass scale configurations at this point, what you see in the screenshots is the least bad configuration I could manage. Turning down mass scale to more reasonable values does not change affect problem and only makes the book even harder to handle since the mass of the hinged cover wobbles the main rigidbody.