Model positioning app ARCore with Google VR (Cardboard)

Hello,

We are developing a AR/VR(?) project where a structure appears in the scene, and the user can see it and walk around it. We are developing in Unity 2019.2, using Unity AR Foundations with Google Cardboard (Google VR).

The structures is a digital mesh placed in Unity, while the rest of the screen seen by the user is the real world displayed by the phone camera, using AR Core (through AR Foundations) for getting the camera image and tracking. However, we are using Google VR, since we want to display the image with stereoscopic rendering (the user will wear a headset, not plugged to anything, to hold the phone). Basically it is an AR application, being displayed in a VR Headset.

However, we are having issues with the positioning of the structure, where they seem to "follow" the camera. When we build for VR, when the user turns the phone around the structure seems to follow him (see attached file), making it impossible for the user to walk around the structure (since the structure always follows his line of sight). This problem doesn't happen when we build exclusively for AR, no VR and no stereoscopic rendering, since in that case the user can pretty much walk around the structure without it "following" him.

Also, not sure if this is related, but there seems to be an issue with AR Image tracking using VR.

All help regarding this issue will be appreciated,
Thanks.

Note: I previously posted this issue in another thread, but I believe it was the wrong one. I am posting it again, since I don't know how to move the thread (if a moderator could help with this issue it would be appreciated).