XR Interaction Tool Kit - Disable player collision with selected objects

Hello,
I am developing a game for Quest where at some point I can hold an object with my right hand and scale it up and down by holding the trigger on the left controller. Whenever the object gets big enough and touches the camera I can no longer scale it up. I couldn't figure out how fix it, disabling the collision between the object and the player layers in the collision matrix didn't do the job.
I feel like I'm missing something, and would glad to get some help.