Angular limits on configurable joints in world space

Hi all!

I am running into the following 'issue' and I am unsure if it is a limitation, incorrect setup or just me
The actual use case is a bit different but for the sake of clarity lets assume the following scenario.

I have an old style lantern with a handle. It is hanging from a static post and is only able to rotate - by a configurable joint - over the x and z axis. Both axis have limitations so it can only move a small bit to the left and right and front to back. This is all working.

Now I want an animated character to handle the lantern. The animated character has multiple animations and its animator component animates physics. Its right hand has an isKinematic rigidbody to be used as the connected body in the configurable joint. The lantern is following the hand properly and even the rotation and limits seem to be working. However, the x and z limits are no longer in world space as it was hanging from a post. If our character rotates its hand upwards the lamp is no longer hanging but point forwards. It doesnt matter if we toggle the Configured in World Space bool.

While there is something to be said that the limitations are relative the connected body I am looking for keeping the angular limits in world space at all times.

How do I achieve this? What am I missing?

Thanks!
Pieter