Can only pick up objects while not wearing headset

I got my rift s a couple days ago and decided to get it working in unity. I enabled virtual reality support and installed the Oculus integration. I made a grabbable cube and got the hand tracking working, but I notice that i was not able to pick up the cube while wearing the headset. as soon as i take off the headset and hold it in my hand (not blocking the sensor) I can pick up the cube with no problem. I have never messed with vr in unity before so I really don't know where to start.

Thanks for any help!