OnMouseDown and 2DBoxColliders on top of each other

I'm programmatically adding a bunch of 2DBoxColliders to an otherwise empty Gameobject. This is so when the user clicks I can identify which of the colliders they clicked inside.
This works, and I can pick it up with a simple:

void OnMouseDown() {

, then a Ray and a loop to figure out which box was clicked. Great.

However... that GameObject is the child of another GameObject which itself has a 2DBoxCollider. The child and all of its 2DBoxColliders is entirely contained within the parent.

The problem is, when this parent's 2DBoxCollider is active, OnMouseDown no longer works reliably on the child's colliders. Some of the child's Colliders register the click, others do not, no matter where I click in them. Then if I change the rotation of the child GameObject (even by 0.001), the child's 2DBoxColliders which register the click change and a different set work/don't.

So my question - how do I handle this scenario? I guess I can refactor into an Update that detects mouseDown then raycasts, but why doesn't OnMouseDown work reliably in the first place?

There's no Physics and no Rigidbodies (but probably will be on the parent at some point). All I want is the mouse-clicks.

 

A canvas with an event trigger component isn't an option?

 

Alas no, it's not a UI thing.