Make navmesh ignore obstable that's on the same gameobject?

I have a prefab with a navmesh on it. There will be multiple of these prefabs in the scene, so I would like them to avoid eachother. What's the best way to do this?

I've come up with adding a obstacle component to the prefab, but this causes a weird behaviour. I figured out that I need to ignore x's obstacle if it's on x's gameobject. How can I do this?