Mixed Reality Multiplayer calibration issues

I was developing a multiplayer experience with Mixed Reality utilizing tools like Steam VR and Photon, and the project works fine, but there is a weird issue where the player heights are not picked up correctly. Some players see the others on the correct height level, while some see others as giants (very high).

Example observed: The first player connects and becomes the host, everything is mapped correctly with all the props on the correct height level.But then the second player connects and the first player sees the second player 2-3 inches higher than himself (FYI: both players have almost identical height irl). On the other hand the second player sees the first player on the same height as he is himself 0_o.

I was wondering if this is an issue with Mixed reality space calibration and are there any "hard coding" solutions to have all the players have the same height? (or not hard coded methods, whatever works)