Constrained camera rotation with touch

Hello everyone!

I'm trying to make a first-person camera rotate using screen swipes, but I'm running into some problems.
What I want is a rotation that, regardless of resolution and screen dimensions, follows the movement of your finger accurately. An example of what I mean is how Google Streetview functions on mobile: your "focal point" remains fixed on your finger during the gesture at all times.

My initial approach is to take the finger positions of the previous and current frames, create rays from them using Camera.ScreenPointToRay, and then use Quaternion.FromToRotation(ray1.direction, ray2.direction) to get a rotation delta which I then apply to my camera's rotation (camera.transform.rotation *= rotationDelta).

However, this results in rotations eventually accumulating a roll (rotation around the camera's forward axis), which I absolutely do not want. Does anyone have some advice how to eliminate this roll rotation?
Thanks in advance!