This is pretty much working as intended. Occluders have a normal and if you are on the wrong side then it doesn't have any effect. I usually recommend people have an occluder that only orient around Y inside the building. That creates a cylinder rather than a box, but it's cheap and often good enough. Alternatively you can have occluders that don't rotate on each side of the building.