Some weird issues w/ SetStereoViewMatrix

I'm using a variety of builds of Unity, ranging from 2018 - 2019. For this discussion, I'm using 2019.2.7f2.

I need to know the exact different between calling camera.worldToCameraMatrix = xxx VS camera.SetStereoViewMatrix( eye, xxx ).

Q1: does setting camera.worldToCameraMatrix mean "for the current eye", and I would want to move it once per L/R eye pair, or does it mean, "set the center of the camera there, and Unity will move the camera slightly L or R to account for stereo" ??

The example in Unity code shows calling SetStereoViewMatrix with the result of camera.worldToCameraMatrix. Actually it shows this for both the L and R cameras, plus or minus some offsets. This is sort of misleading (at least to me).

Q2: does calling camera.SetStereoViewMatrix modify camera.worldToCameraMatrix? If that was the case, then the above example in the docs becomes pretty weird. recursive, sort of. If calling it doesn't modify camera.worldToCameraMatrix, then how does camera's worldToCameraMatrix affect the camera when you're currently calling camera.SetStereoViewMatrix? (or, does it not affect the camera). Are the two modifications additive, or exclusive?

Q3: when I call camera.SetStereoViewMatrix(), in my VR app (Windows Mixed Reality), the camera's position in the Inspector stops moving around. Prior, it's position every frame was seemingly set by the XR subsystem to the headset's current location. After calling SetStereoViewMatrix(), the updating stops. WHY? As far as I understand it, calling SetStereoViewMatrix is simply setting the camera's VIEW on the scene, not the camera's actual position. Is this not true? It's certainly true for setting camera.worldToCameraMatrix. When I use that method (only), the camera's position and rotation is updated from the current headset's tracking.

Q4: Why does calling camera.SetStereoViewMatrix( ) goof up in Unity 2019 when in "single instance" 3D render mode? MultiPass works fine, but single instance breaks it quick. (the camera's output looks very funny). (this question is not as important).

Q5: exactly what is the difference between camera.worldToCameraMatrix and camera.SetStereoViewMatrix( )?

 

I've done some thorough investigation tonight, and came up with some observations to save y'all time. Keep in mind this is a Windows Desktop project, using a Mixed Reality Headset, using OpenVR. The same results are seen on Unity 2018.1.9F2, 2018.4.9F1, 2019.2.3F1, 2019.2.7F2 (I wanted to be thorough!)

* If you call camera.worldToCameraMatrix = xxx in Update(), or LateUpdate(), it won't have diddley effect. it does nothing.

* If you call camera.worldToCameraMatrix = xxx in OnPreRender (by putting a script on the camera and overriding OnPreRender), for some reason THIS does work.

* There is no noticeable difference between using the method of setting camera.worldToCameraMatrix = xxx, or camera.SetStereoViewMatrix(), they end up doing the same thing. However, when you use camera.worldToCameraMatrix, and move the camera around (by moving the headset), the CULLING that is done is based on the camera's CURRENT position, not by the position you set in the worldToCameraMatrix. If you use camera.SetStereoViewMatrix(), the culling is done based off the view matrix you send in. which is nice.

* When you use camera.worldToCameraMatrix, and move the headset around, the headset's current position is stuffed into camera's position each frame. But the worldToCameraMatrix is still honored with each draw. Keep this in mind for culling and getting the camera's position. Getting the camera's position will return the headset's current position.

* When you use camera.SetStereoViewMatrix(), the camera transform's position and rotation are "thumped" and stuck at whatever their values were when you first called it. It renders the way you would expect it to with the matrix you set in the call, but subsequent frames do not result in the camera's position being set by the VR headset's current location. I have no idea why, but it seems like it should be.

* Whichever version you use, you need to take into account the stereo separation. You need to position the matrix manually, for both the L and R passes. This is obvious for camera.SetStereoViewMatrix(), but NOT so obvious when calling camera.worldToCameraMatrix. You need to pass in different matrices based on if the pass is the L or the R eye!

All of the above observations are based on the rendering mode being "MULTI-PASS". In single-pass mode, it's plain busted, for either set method. For camera.SetStereoViewMatrix( ), the left camera's view looks fine, and the right camera's view is "stuck" and doesn't update. For camera.worldToCameraMatrix, the matrix has no effect, the camera is just set to the headset position.

In single-pass instanced mode, for camera.SetStereoViewMatrix(), it's just utter Crazy Town. the left and right eyes show a split screen, left and right, and each eye renders a weird width sliced of both eyes. something here is clearly busted. This is in all versions of Unity I tried. (all that is in my scene is a sky box background and a single cube with default texture on it!). For camera.worldToCameraMatrix, the matrix has no effect, the camera is set to the headset position.

Summary: Use camera.SetStereoViewMatrix( ) in MultiPass mode. Anything else apparently doesn't work.

 

I keep running into this thread, very useful information.

The only thing that changed so far is that in multi pass instanced mode attempting to touch view/projection matrices results in unity spewing ton of console errors (2020 and 2021 version).