Question Change UI click binding but not in XRI Default Input Actions

I am trying to make it so that the UI in my game can only be pressed using the trigger button, but for some reason, it also allows for the right primary button to click on it as well, but only when I'm not pointing at it. It also seems to be in a random spot every time I click the right primary button. I have tried to fix it using the XRI Default Input Actions, but it still happens. I don't want the right primary button to be what interacts with the UI, because that is what will make my player jump. The only thing I can think of that would be the issue would the XRI Default Input Actions, but I have already tried to remove the RightHand primaryButton from the actions, and it still does it. I'm kinda new to this, and I don't see another way on fixing this issue. How else would I fix this? If it makes a difference, I am using Unity 2021.3.16f1 Personal, and I am trying to build to Android, as I am testing on my Quest 2. I have also tested using Link cable onto my computer and switching the platform to PC, but nothing has worked so far.