Question Best way to stop a raymarched shadow by the depth buffer?

I'm trying to make a shader that just raymarches a soft shadow using SDF with a simple primitive like spheres, i start my ray from the depth buffer's world position, my problem is this:


(Green good, Red bad) (Green sphere is the shadow caster)

My shadow gets cast through every normal object, since there's nothing there to stop it, in the SDF map those walls don't exist, so it ends up like a tube that cuts through everything, appearing on the other side of objects.
Maybe there's some way to have the depth buffer be present in the SDF map(), each pixel acting as an extruded plane? Or maybe attenuate shadows that face away from the light normal too much? Or maybe there's something obvious i'm missing?