FPS drops to 8 when I turn my head 90+ degrees

There's nothing (special) behind me but the same stuff that's all around.

When I look forward(ish) I'm running 90 FPS in all my scenes. But as soon as I turn my head 90-180 deg, FPS drops like a rock, but recovers when I look back forward.

However if I do that one more time - in most cases it won't go back up and stays 8-10 FPS.

That only happens when I'm done building in VS and running the game (Windows Mixed Reality). No problems when running from Unity editor.


ETA: Eventually it crashes here:

Code (csharp):

1.  
2. global::WinRTBridge.MethodTools_InvokeMethodDelegate__WinRTBridge __safeTargetMethod = global::System.Runtime.InteropServices.McgModuleManager.GetCurrentCalleeDelegate<global::WinRTBridge.MethodTools_InvokeMethodDelegate__WinRTBridge>();
3.            __value = __safeTargetMethod(
4.                                unsafe_instance,
5.                                ((long*)unsafe_args),
6.                                unsafe_methodPtr
7.                            );
8.            global::System.Runtime.InteropServices.DebugAnnotations.PreviousCallContainsUserCode();
9.            // Return
10.            return __value;  //<<<==== EXCEPTION THROWN
11.