Scrollview: Disable scroll on PointerDown [only scroll when dragging the scrollbar]

Is there a way I could only scroll whenever I am dragging the scrollbar?
It's really annoying when you click any part of the scrollbar and the scroll view just auto scrolls based on where your cursor's location on the scrollbar is.

 

I think this should be easily solvable by disable "Raycast Target" on the Image of the Scrollbar background. This way, clicks are not detected on that part anymore, so, it shouldn't trigger the jump logic.

 

Thanks for the reply. Sadly there is no raycast in either the scrollbar background, scroll bar, or even the viewport. I however managed to "ease" it a bit by changing the movement type to clamped and disabled the Inertia feature.