Jumping Straight up and Down

I have a player that I use Addforce on to jump straight up. When I repeat this several times the player gradually moves along the X axis and Z axis.
I am using Rigidbody (not 2d) and even constrained the axis yet they still move.
The player is animated and has colliders and is not kinematic.
Is this from the colliders and if so how should I set it up to avoid this.

 

Using Addforce when moving a player is a really bad idea, even if they tell you to. When moving a player, you might want to change the velocity, so it won't look like, the player is on ice