Question 2 different XR Grab Interactable's using one Box Collider

Hi, I'm currently trying to make an interactable door with a handle using the XR Toolkit. I want both the door and the handle to move when interacting with the controller. The handle game object is a child to the door game object. Both of them have XR Grab Interactable and Hinge Joint components. Since they both use XR Grab Interactable with the same Box Collider (placed on the handle), neither of them work. If I disable the XR Grabable on the handle I can move the door with no problem and when I disable the XR Grabable on the door I can move the handle without a problem. Neither of them work when they're both enabled. I think it has something to do with the collider being a sort of trigger for both XR Grabables but I can't seem to figure it out. Looked everywhere but can't find any solution. Would really appreciate it if anyone could help.