HDRP: Reflection probe bake breaks scene

I'm making a fan made map for the Skater XL game.
I baked the lighting using Bakery and everything looks fine, except my materials reflect the sky too much even where they shouldn't. In the first screenshot you can see that the smooth part of the tiles is too bright. I thought that the reason could be that I don't have reflection probes in the scene so I wrapped the whole thing in a reflection probe and it seems to fix it but now objects that are far away are completely dark. The darkness position depends from the camera position (I mean it's like the camera has a 100m radius where it works and anything after that becomes dark). This happens on every far object, no matter if inside or outside the reflection probe. Any idea what is going on? If I remove the reflection probe and remove the baked lighting everything goes back to normal