Question Help, XR is confusing

So back in 2017 I was dong AR and VR dev with unity. I was somehow able to get movement for the character from my smartphone and what not. Used AR to make virtual buttons to make magic circles appear and shoot fire balls.

Fast forward today. Honestly nothing I've tried or google has helped me so far with XR. Basically I'm trying to recreate a fun little escape room me and my friend played over the weekend, which was in VR with the HTC vibe. What I am trying to do is use iPhone(bc we all have one (() as my VR headset. I did this before with an android in 2017. Yet with the new updates and the XR SDK I cant get the movement of looking around and what not from my phone onto the game. I can click my screen to activate buttons though. How do I get this to work? I am using the Unity remote app to stream from my windows PC to my iPhone, would this be causing the issue. I will try out the mock HMD today to double check if it even works at all. Any help would be great.

Ultimate goal is to create a basic level to an realistic size. Use our phones as the headset like google carboard and track our positions in the game like a multiplayer game. Was going to hack the google dream controller to work with Unity (already found a way to do this). Something along the lines of this, is any of this possible???