Cardboard VR/Non-VR jittering

I have been experiencing an issue using the Google Cardboard plugin with Unity XR to develop a VR/non-VR app where once I've turned off the VR mode I want to keep the motion controls of when in VR but I am now encountering a jittering effect that even though the device is held still (or placed in a phone holder) there is still some movement going on.

At first I thought it was my phone (Samsung s8) but after sending the APK out to a couple of other people who also encounted the same issue.

I used the google documentation about using the "Magic Window"
https://developers.google.com/vr/develop/unity/guides/magic-window.

I even tried to filter the gyro.attitude on each axis by adding up the last 16 readings and taking the average and while it did have a noticeable improvement I encounted a new problem where the camera would start to move in a diagonal direction or spin 180 degrees for no reason.

I've seen a couple of posts about using the Slerp method but I'm not sure how to implement that, let alone implement it into the "magic window" code from Google.

Any help or ideas of how to stop the jittering of non-VR motion would be very much appreciated.