Easiest way to simulate ball flicking

I'm using mouse down, mouse drag and up to calculate distance/time then in fixed update add velocity to a ball. I'm storing the start time on mouse down. and in mouse up I'm calculating the difference in time. then dividing distance / time . then multiplying it with a force figure to apply the force.

however, I'm trying to solve a problem if a user keeps finger on top of a ball (almost motionless) for awhile or cancels a move to go back to origin without moving. How to go about to reset the start time. while allowing a small motion move on the ball itself.

I'm thinking about resetting start time in onmousedrag if mouseposition.x y stops changing above XX amount of distance in for YY fraction of second. but not sure if that's the best way.

Any ideas?