Bug Weird selection behavior with ray interactor set to Sticky

Here is my setup, it's important for this behavior:

I have a layer for Teleport.

I have a right hand RayInteractor with:
- Interaction Layer = every layer, but Teleport excluded
- Raycast Mask = Everything (Including Teleport)
- SelectionActionTrigger Sticky

A ground plane with the layer set to Teleport which has a Teleportation Area with Interaction Layer Mask set to Teleport and Teleport Trigger OnSelectExited

Now when I aim the controller at the plane the raycast stops at the plane (because the Raycat Mask hits it) and when I press the select button it does not teleport, as expected because I excluded the Teleport layer from the Interaction mask. BUT if I release the select button and then hover an other "valid" grab object it suddenly automatically picks up the object while the select button is not held down anymore.

If I remove the Teleport layer from the Raycast Mask it does not happen anymore, so this behavior must happen because the ray hits the Teleporation Area while it is exlucded from the Interaction layer.