Physics.Raycast misses randomly

I had similar issue. It turned out my raycast origin was in a small cube with a box collider on it. The cube rotated and eventually the ray hit the box collider of the cube whoose transform I was using as an origin point. After deactivating the collider, which I didn't need anyway, the problem disappeared.

 

thanks, the flat collider mentioned really helped me