Rotating view without rotating the camera using projection or view matrices?

Is there a way to rotate the view in a VR camera without rotating the camera? I suspect this can be done with world/view/projection matrices but I cant quite wrap my head around it. I've tried playing around a little bit with projection matrices but the view gets quickly distorted the more I try to rotate. My motivation for doing this is to change the alignment of the eyes for people who have misaligned eyes. The problem with rotating the cameras is that it messes up the head tracking. If for example the "eyes" or L/R camera are rotated to be cross-eyed then leaning forward moves the eyes closer together as each camera moves along a path it is looking at. To take an extreme example(not one I would implement), if the right camera is rotated 90deg to the right and the left camera is rotated 90degrees to the left when the person with the HMD leans forward in his chair 1ft the left and right eyes/cameras will move apart 1ft in opposite directions. I obviously dont want the "eyes" to be 2ft apart or wander away from the "head" or collide into each other so I figure the solution is to keep the cameras pointing forward but rotate the views presented to the user in the HMD.
Any help would be much appreciated.
Thanks
View attachment 442250

 

An image to help explain. The top half I know how to do but is not suitable because tracking follows camera direction. Id like to do the bottom which I think is possible with view or projection matrices but I cant quite get my head around it.