Rigidbodies passing through each other with constraints on

Hi all,

I have a weird behavior with two rigidbodies:
One is a drawer of a shelf, I used constraints on rotation and position so that it can only be moved in one direction (out of the casing). It has a collider attached to the bottom.
The other is a simple box with a collider and it is also a rigidbody.
Both are non-kinematic.

I want to be able to place the box into the drawer. For the screenshot, I placed the box just above the drawer and on play, let gravity do the rest. What happens is that the box enters the bottom collider, slows a bit down, the drawer gets pushed a little to the side and then the box passes through. I get my desired behavior if I turn the drawer to kinematic or if I put it on the ground and disable all constraints but of course then I wont be able to move it anymore.

So my theory is the box is hitting the drawer which wants to fall down or rotate but is forced not to do so by the constraints and the box gets confused in all these calculations and passes through. So my question would be how to tackle this example? I can place the drawer in a bigger shelf that sits on the ground etc. but I want to avoid any wiggle by placing an object in it so I would have to place very precise colliders on it. Naturally, I prefer the easy "clean" way of just saying: don't wiggle

Thanks for any pointers!