EventSystem click-through issues

I'm using the EventSystem's IDragHandler interface to handle dragging my world (non-UI) objects. This works well for the most part, but if I have one object contained within another object that doesn't have an IDragHandler, the contained object doesn't recieve the event.

To clarify, I have chair, in a room, on a world. The chair and the world subscribe to IDragHandler, the room (which does have a collider) does not. If the chair is outside of the room, I can click and drag on it no problem. If the chair is inside of the room, the click goes through the room (which is correct) and the chair (which is incorrect) and hits the world.

How can I get the click to go through the room and hit the chair properly?

Thank you.