Seamless Axis Control?

My game currently only support controllers, as it is being developed for Xbox, so the camera's axis controls are set to work with the controller's right analog axis.

There's only a minor issue though, and that is that the camera likes to hang for a second when trying to change direction or combine X and Y axis control with the analog stick.

Is there a way to make the direction change more smooth? All the controls so far are very tight except for this one little detail