Disallow children of Slider Handle from initiating Drag

I have a Slider with its Handle outside the Slider area. Above and below this Handle I have a two Buttons that increase and decrease the Slider value. These Buttons are children of the Handle object so they follow it up and down.

When I press the buttons, everything is fine, but they also initiate dragging the Handle which they should not. I could remove their child status, just making them children of the Slider, and add a script to each of them to follow the Handle up and down, but I would prefer a more elegant solution.

Any ideas?

Here's a picture of my scenario: