Need help with rigidbody connected via joint jiggles on rotation but not on translation

I have a problem where two objects connected by a joint are stable during translation but jitter during rotation. I would like it to be stable during rotation also.
My character is controlled using the Unity Standard assets FPS character controller.
The character has a gun that is a child object of FirstPersonCharacter.
The gun can pick up objects that it touches. I pick them up by adding a joint to the collision object and make the gun the connected object. Code below:

Code (CSharp):

1.                 Debug.Log("Grabbing Object.");
2.                 //CollisionGameObject.gameObject.transform.position = rb.transform.position;
3.                 CollisionGameObject.gameObject.AddComponent<FixedJoint>();
4.                 CollisionGameObject.gameObject.GetComponent<FixedJoint>().connectedBody = rb;
5.                 HeldGameObject = CollisionGameObject;
6.                 hasJoint = true;

Any idea why the object jitters or is rubbery during rotation but is completely stable during translation?

Thank you for any help provided!