Question How To Move Camera Forward When Velocity Is Applied To HMD

When horizontal velocity is applied to the physical HMD I want the camera object to move in the direction perpendicular to the HMD's horizontal velocity vector.

Originally I used the cross product of the velocity's vector to get the perpendicular direction, but the problem is, assume the player turns around and then moves the HMD horizontally (applying a horizontal velocity). The cross product (consisting of the horizontal (and up) vectors) will still point towards the original direction. So, the camera will move backward (ie., in the original direction).

This problem is driving me crazy! What is the solution here?