Question Whole scene suddenly translates far away?

I've created a very basic scene using the Mixed Reality Toolkit, Hololens 2, and primitive shapes like capsules. I placed them all a few meters apart and started the Unity level before starting to wander around my home without looking at them. When I came back to that position, I noticed all of the objects had disappeared. After looking around for a bit, I noticed the whole level shifted about 50 meters away and up into the sky.

The exact same thing happened but in a different direction my next try.

During my third test, I went behind some walls and kept looking back at the objects in space. The objects flashed for a moment, then the whole scene suddenly translated about 2 meters away at a slightly lower height.

I'm convinced this is an issue with the Hololens getting surprised while mapping spaces. I know very little about the device, so I'm unsure if it's inadvisable to use the Hololens across multiple rooms like this.

Is this a known problem, where the HL suddenly horribly loses track of its space and everything shifts some distance? Or is there something I may have done in Unity that causes this whole translation? Or it's a problem with older Unity/MRTK?

If it matters, I'm using MRTK v2.2.0, and Unity v.2019.2.5.f1.