EventSystems PointerEvents don't seem to Fire Correctly

I've had so many problems making the EventSystems work as expected for ages and it doesn't seem any better now.

If I hook up the IPointerDown handlers for example, the click events don't seem to trigger every frame. Sometimes an object is hit, sometimes it isn't.

The only way to get accurate hits seems to be polling mouse Input on an Update loop.

Why do the eventSystems methods never seem to fire on time?