HoloLens Anchors Jumping

I have a use case where I am mapping an area around black cabinets with glass doors. This is probably one of the worst case scenarios (for a static object at least). My app works well until I get close to the cabinet and then it loses track of everything. Sometimes it can recover but other times it cannot. I have a bunch of world anchors around the scene and when it starts tracking again, all the models will be in a different location.

If I stop the Spatial Mapping Observer using the StopObserving() command, could this potentially help prevent the anchors from moving around the scene? Is there any common method to deal with this issue?

Also, I am using Axis Aligned Box for my volume type. Would any of the other options be better for a larger area?