Light leak from a directional light only at a distance

Strange light leak on a directional light on the far wall, only from a distance. I've tried setting the wall to two-sided shadows, I've tried using thicker and additional geometry. Messed with the shadow bias on the light. Nothing works but rotating the light to another direction, but then it happens on another wall.