Collider.Raycast doesn't find what OverlapSphereNonAlloc just found

I'm working on avoidance for flying NPCs. Forces are not used, only kinematic rigidbodies and colliders. To avoid crashing into other similar flying NPCs and into static geometry, I do a Physics.OverlapSphereNonAlloc to find all nearby colliders in the layers involved. This works as expected.

However, as I need normals, I then do a Collider.Raycast (not a Physics. Raycast) on each collider thus detected to get collision data from a RaycastInfo struct. Collider.Raycast detects only the given collider – it ignores everything else – and doesn't even take a layer since the layer is given by the target collider. It's clearly designed for situations like this.

However, while Physics.OverlapSphereNonAlloc always works as expected, Collider.Raycast almost never finds the colliders. It succeeds sometimes, but in the vast majority of cases it fails.

Am I missing something fundamental here? Here's the code, reduced to its bare essentials:

Vector3 staticAvoidanceRange = 3f;

Vector3 pos = transform.position;

private Collider[] hitColliders;

int hits = Physics.OverlapSphereNonAlloc(pos, staticAvoidanceRange, hitColliders, staticAvoidanceLayers, QueryTriggerInteraction.Ignore);

if (hits > 0) {

    int i = 0;

    while (i < hits) {

       Vector3 otherPos = hitColliders[i].transform.position;

       Ray ray = new Ray(pos, (otherPos - pos));

       RaycastHit hit

       if (hitColliders[i].Raycast(ray, out hit, staticAvoidanceRange + 1f)) {

           Vector3 modificationDir = hit.normal;

           float d = hit.distance;

           /* Avoidance vector updated here (omitted) */

       } else {

           Debug.LogError("Failed to hit collider picked up by OverlapSphereNonAlloc " + hitColliders[i]);

       }

       i++;

    }

}

Changing the range to a larger value, or doubling the range in the raycast call doesn't change the outcome. Neither does offsetting the ray's starting point along the ray to avoid self-collision (which shouldn't be necessary since Collider.Raycast should skip all colliders except the target one). Everything is done in the same frame, in Update (not FixedUpdate).

Grateful for any input.