Feedback Cinemachine + XR design issues resulting problems

So we reported a bug where if you use Cinemachine (CM) and XR you get issues with CM trying to set FOV as does XR and XR thus spams a warning to your log every frame tanking performance.

The responce "By design" ... ouch!

Multiple issues here

1) Spaming a warning every frame is not helpful all it does is trash performance. You already have cases where you show warnings in inspectors or print the warning once on first occurance and never again why are you spaming this warning (and the warning about overlay cameras)

2) Your assumption that a game is XR or Desktop (monitor) is wrong ... many games and all of ourse apps do both the player can at any time pop the headset off and resume normal monitor + keyboard + mouse play. So little workarounds to the bad design such as put the CM brain on a GO parent of the camera so it doesn't set the FOV ... that is a problem as linse control from CM is a major reason to use CM in the first place ... positioning a camera really isn't that hard but CM does so much more for you when you couple transform and lense control

How to fix this?

There are several ways this issue could be fixed very simply

Options:
1) simply dont spam, having the camera ignore the attempt by CM to set the FOV while in XR mode is compeltly fine you dont need to SCREAM it at the log every frame tanking performance. Its a warning not an error and even if it was an error you shoulnd't be screaming down the log every frame .. if you hit a critical error turn off the offender or otherwise ensure its not going to vomit log entries ... doing the log vomit just makes reading the log near impossible ... this little tip effects other aspects of your logging ... for example XR multipass complaining about overlay cameras ... yes you told me already you will skip rendering of overlay cameras ... why are you screaming it at me every frame

2) have CM and XR I dont know work together?
Why is it CM can tie into post process to set depth of field but we cant get CM to say ... Ah your using and XR presentiaton I shoudl F*&^ off with setting the FOV for now

Or XR say Ah your using CM I should tell you ONCE that I will ignore CM FOV operations and drive on with life

3) give us control
That is let me toggle on and off the CM Brain's desire to set FOV ... actually it would be nice to enable/disable various aspects of CM brain transform and transitions temporarly during run time for various reasons but at the minimal let me say ... Hay CM Brain I am turning XR on now so F&^* off with the lense stuff K-tnx-bye

 

We have a sample project uploaded with the bug report 1320356 that demonstrates

in short though

This starts XR up

XRGeneralSettings.Instance.Manager.InitializeLoaderSync();

XRGeneralSettings.Instance.Manager.StartSubsystems();

This stops it resuming normal presentation

XRGeneralSettings.Instance.Manager.StopSubsystems();

XRGeneralSettings.Instance.Manager.DeinitializeLoader();

So in our apps we always start up in Desktop mode ... the user indicates they want to go into VR so we start up VR look for the devices and there we go ... we can also detect the loss of user presence and stop VR when they simply remove the headset or we can make it more user deliberate and let them turn it off with hot key bit like WMR does.

Point is we and many of our clients build apps that start in Desktop and transition in and out of a VR presentaiton on demand so these assumptions made about an app is VR "or" non-VR is outright wrong. Ran into a lot of that sort of assumption when sorting out what should be simple tasks ... like starting and stoping XR presentaiton on demand

Anyhow now assuming that yes the app can go in and out of XR presentaiton and CM is the greatist thing to happen to camera's since ... well ever. We will want to use CM with apps that do XR ... this means either XR needs to work with CM better or CM needs to work with respect to needs of XR or CM needs to give us more control so we can help it work with XR something.

But yes I am aware its XR that is vomiting warnings to the log I complain about log vomit alot as a bad bad practice that needs to end. XR also vomits when you have overlay cameras and multipass mode ... its just saying that it will ignore the overlay cam rendering ... whcih is completly fine ... why o why must it say that every frame! .. . I assume the assumption was "Ah they are a VR app and wrongly used overlay cams with multipass" .... in reality the app does both Desktop (where overlay is used a lot) and XR where its used but has limitations.

 

I have never looked into extending our own vCams ... whats in the box has always enabled everything we wanted

So I will look into that but would still love to see teh out of the box play nice together be that a new feature to CM or an adjustment to XR or little of both

As to turning CM off
Ya that is what we will unfortunetly need to do ... we sitll have the tracking move a vCam so hate to turn the brain off ... really dont want to have a second camera rig ... we use a multi-scene set up where our main camera, brain, etc. is in our "bootstrap" scene not in the "game" scene and we liked the game not needing to know about those bits ... now we need to create a system I suppose in bootstrap that monitors presenation method whcih we arleady do e.g. we have an enum that tells us PC, VR, AR and do the nessisary disabling brain and so on ... this however also means that we have to make a fake vCam that is we have to go get the Transform of the Main Camera and drive it around as CM already does and just not set the FOV.

That or make our own vCam if that is an option that doesn't set FOV. That same system will probably also have to purge the cameras from the stack of the Main Camera since XR feels the need to vomit the skip overlay cam rendering every frame.


Please never spam an info or warning on every frame ... just in case I haven't mentioned that before.

You might wonder why we dont put head tracking on the camera?
We put it on a transform within the avatars head and use it to drive IK same for hand tracking it drives hand movements so all the tracking bits are in the player controller rig which links up with visual representation (avatar) and works on transforms that sync data via our network kit (its a multiplayer expernace)

all this assumes that camera behaviour is a system
XR tracking is a system
The two dont really need to know about eachother

 

Having 2 camera rigs can be done its just a shame that its nessisary when all that is really needed is for CM to be configured to toggle setting FOV on and off on demand.

doing this feels a lot less sledge hammery

Here are some pain points when you have 2 camera rigs
Your assuming that no other system needs to access a given camera or that no other system makes assumptions about a given structure as being the camera rig ... actually our world editor and a few other IO systems do need to assume what the camera rig is so changing it in flight could have issues ... of course we could sort that out its just a lot of work to get around the FOV fight.

Right now I am looking into creating our own vCam that doesn't set FOV if thats possible ... if this works then ya simple fix. No messing with the brain, no duplicating camera rigs simply toggle on and off the vCam setting the the FOV based on presentation model.

And ya what we have now does give a nice smooth transition out of VR into Desktop when the headset is removed the user presses teh hot key or we detect the lose of presence and shut down the XR system since CM is there the vCam for the Desktop is now takign over giving a nice smooth transiton and adjusting the FOV back to where it should be with a prety little transiton.

Switch over to a system with 2 rigs its a hard switch ... not the end of the world but not as smooth and fluid as we had.

Again can work around it but it feels very much a hack for a problem that should be simple to solve in a few different ways ... if I can sort out a custom vCam or custom CM Brain that gives me a simple public toggle (control FOV yes or no) then that would be ideal .. I will share if I manage such

 

very cool
I'll give that a go and see if I can expose a bool or toggle method we can call from our events to turn it on and off ... or I assume I can disable it ... anyway this is the sort of solution that feels nice wiht the systms as opposed to 2 rigs and all that

 

Here is what I ended up using and it works perfectly for us

   public class CinemachineNoLensExtension : CinemachineExtension

    {

        public bool IgnoreFov = false;

        /// <summary>Standard CinemachineExtension callback</summary>

        protected override void PostPipelineStageCallback(

            CinemachineVirtualCameraBase vcam,

            CinemachineCore.Stage stage, ref CameraState state, float deltaTime)

        {

            if (IgnoreFov)

            {

                if (stage == CinemachineCore.Stage.Finalize)

                    state.BlendHint |= CameraState.BlendHintValue.NoLens;

            }

        }

    }

hats off to you ...

 

Ya we dont put the device tracking for XR on the camera
we put it on a GO under which we have a vCam

So XR drives the transform of a vCam ... vCam drives the camera

Again its all about seperating the systems

1) Pose that is driven by XR tracking and has nothing to do camera ... yes the pose of the head is where the camera needs to be (sorta) its where the head is we actually adjust the vCam within that to suit the offset as we are rendering an avatar here as well and that head position is used to drive head look
actually little note here when not in XR we simulate hand and head movements so while a Desktop user wont animate exsactly like a proper XR user they arn't static lumps we are simulating some human like movement bsaed on input from the user like mouse movments and all to move the head make it feel like they are looking around.

2) Camera that is driven by CM only CM sets the camera position not XR at all
XR does set FOV when its active but not transform ... as noted above we have pose tracking from XR driving a GO representing the players head and within that there is a vCam so it is a daisy chain but its what gave us the smoothest results and easiest management

3) Rendering or "presentaiton" as we usually call it
This is URP and XR ... so if XR loader is initalized its pumped through HMD if its not then URP is rendering to main display

None of these 3 systems really need to know much about the other ... that I think is part of the difference in how we use it and how its imagined to be used
The asusmption seems to be that a dev puts the tracked pose driver on the main camera and that an app either is or is not VR all the time and that when XR is active it is the lord and master of all things camera ... that is just not the case for us

 

Exsactly and Desktop can actually use 1 of a few vCams depending on what the user is doing like looking through a "camera" in the world to zoom in on something and take a picture and so forth. These are just difference vCams with different lense settings and or behaiours we transiton to based on Desktop state.

In VR we have an analoug for our users ... we give them a little digital tablet they can grab and move about to express the GUI and give them a view port for those special view cases like a in game camera for example. Works out well for us.

 

I think it was so the XR vCam can be the driver right after XR has went down with the XR FOV (109 I think it is)
then once we have cleaned that up (half a second or so) we transition to whatever Desktop cam is appropreate for the situation. So I think its desired that XR vCam resume driving the FOV right when XR goes down so that when we turn on Desktop cam its a transtion to that FOV not a snap ... or would the transition of FOV still occure even though XR vCam isn't adjusting FOV?

The order of operations is something like this

1) detect request end XR presentation so start clean up
2) Transition avatar into Desktop pose ... while in XR its head and hands might be all over hte place so lerp this in nice and smooth ... this takes a second or so
3) Transition the FOV from whatever it was in VR to whatever it should be for hte current Desktop state ... this takes a second or so ... hapens at the same time as the avatar blending back to its Desktop pose
4) bring the Desktop UI back to front (alpha transition) again ... takes a second or so


whole point is that it should feel like you slide into and out of VR both from the player's point of view and the rest of the world so other players seeing your avatar take its helm off then smoothly transtion into the default Desktop pose.
For the player its self it sees the UI come back smoothly, the FOV change smoothly, etc.

Have to remimber the user may have turned XR on or off with the HMD already off or already on its head ... so we want it to be smooth and avoid as much mangling of the avatar's body like lifting its head off its sholders or inverting elbows as possible