I'd probably need to investigate the options here. The problem occurs because the physics engine postpones the delete operations for a little while until the moment it can be sure everything related to the particular deleted collider was notified properly. Currently this may take 1 frame. Apparently, it's about the code that makes the collisions react to the new contacts 1 frame later than they occur. Adds and removes are the same level operations from that perspective. Probably there is a way to hack something inside our cast filter callback. I'll explore more here, but won't promise to deliver anything I'm afraid.