AR+VR, how to match positions in two different scenes?

I am making a multiplayer game where both ARCore, SteamVR and Photon are used. The players consists of two VR players and two AR players, all in the same IRL room, so all positions has to match. The VR players can see each other in the same position in the game as they are in real life, but the mobile phones cannot.
When the phones join the Photon room, all the networked objects, VR avatars included, are instantiated, but in different locations each time. So the movements of the VR avatar can be seen, but the avatar is not placed on the person which the camera sees.

Does anyone have a clue on how this can be done?