Why am I enable to grab object with raycast but the object is not grabbed on hand?

I am using XR Grab Interactable to grab an object. I can grab the object with direct interaction however when using the offset grab, the object is not grabbed in my hand. Whenever I come closer to the object to grab, it moves away from my hand but it can still be grabbed. What do I need to do to make it grabbed in my hand?

Link to the demo:
https://drive.google.com/file/d/1mPBxzisqkspoCq0mXx6m58UQj2xPutT9/view?usp=share_link