Vive VR: Spikes + How to interpret "VR.WaitForGPU"

Hi everyone,

I'm developing a Vive game with Unity using the standard settings of SteamVR, without the lab renderer. On the whole, I get good frame rates, but every now and then (10-20 seconds), there is a drop down below 60fps.

This even happens if the headset is lying still on the floor and the controllers are off. There is no indication in the profiler that anything else except graphics is to blame, as it looks like this:

What could be the cause of this?



Also, all the time is reported to be spent in VR.WaitForGPU (and, given the yellow color, in VSync). Am I correct in assuming that says that the CPU is waiting for the GPU to finish operations, and if the GPU doesn't manage within its allocated time for the frame (11ms for 90fps) a drop will occur?

Thanks!!

 

I am having this exact same issue.

 

Same here, same issue. No idea what's causing it...

 

Hi,

some more info on this (which I've also crossposted here: http://steamcommunity.com/app/358720/discussions/0/355043117522766433/):

• The spikes can be seen in the SteamVR performance frame timing view, too.
• Further investigation seems to indicate that, contrary to what the Unity profiler suggests, this seems to be a CPU issue (the frame-by-frame analysis of the SteamVR performance chart shows a CPU spike before the GPU spike)
• It is still unclear what is causing this, exactly.

However: For me, the issue goes away entirely when creating a standalone build and running that - i.e., not starting the game from within Unity. The game stays at ~6ms per frame then (measured by SteamVR performance frame timing).

This is still very annoying since testing the game is tedious. An interesting question now is whether this is a "pure" Unity problem or a SteamVR/OpenVR-In-Unity problem. If Unity is entirely to blame I would expect this to happen in non-VR projects as well. If any Non-VR Unity developers are listening and have this same issue, I'd appreciate a heads-up. Otherwise, this may be an issue with the VR integration.

-plmx

 

FYI I've seen similar spiking behavior on GearVR - still looking for a solution. Will post if I find one although it may not cross-apply since mine is not vsync related.

 

We also have this issue and is a major roadblock in our development right now (using Vive). We thought it was initially our terrain shaders as this only happens when you look down at a 60 degree angle. For some reason, no matter what terrain shader or texture combo we use, we can maintain 89 fps until we look at the ground at a 60 degree angle. At this point FPS drops to a locked 44 and in profiler we get VR.WaitForGPU. My framerate will not go back up until I look away from the angle, and then at that point the VR.WaitForGPU goes away and FPS is butter smooth at 89 again. :/ No combo of shaders or textures changes this, I can have my terrain with a blank material on it, butter smooth until I look down at that one angle, and then BOOM 44 fps.

 

Does it happen in the build? From the sound of things so far this goes away in the build. The source seems to be that the CPU takes too long, then has to wait for the GPU to finish the frame late. But... There really isn't enough hard information to confirm that, it also really depends on your own profiling.

Technically you will always see the CPU and GPU waiting for each other at some point, the goal is to just make them both do their work fast enough and have the load distributed evenly so that neither is the bottleneck. Seems like VR.WaitForGPU is a little too ambiguous to apply to every project in some generic way as it could be caused by both CPU or GPU which means more contextual profiling is required to find the source of the problem.

In your case, it could be a slew of other things in your project. Do more specific profiling, then maybe restart with a blank scene and add features until it presents itself more clearly.

 

We have it set to Deep profile already and no additional details. It happens in the build. I've also been trouble shooting this for 2 1/2 weeks already (on and off of course). Its not shaders, its not scene objects. I'm down to a blank hill. Walk around and cant get under 89fps.... until I look at the ground at an about 60 degree angle. Of course the area that does this is sloped... its almost like opposite converging angles. I'm looking down at 45-60 degree at a 40-60 degree opposite slope.

 

Any resolution or solution?

 

This is killing our development process on a daily process. I cannot get away from this blasted awful weirdness. Standing still on a terrain with nothing in the environment, not moving. I look at one hill 89fps, I turn around and look at a different one and I'm stuck at 44FPS, VR.WaitForGPU/VR.DeviceSDK and it might as well be called VR.PermanentlyStuckFucked because there is no wait about it, it will never stop doing this, its just indefinitely waiting for something that will never happen. My FPS will never go back up. The other thing that grinds my gears is the whole, I can only have 89 or 44, there is no in-between besides when I'm walking and the occasional drop to 71 or 75. However, whenever there is a problem, its always immediately locked into, 44FPS with that awful message in profiler that obviously is confusing since I keep finding threads with the same problem and rarely a solution. This one right here has it at 72.8% and 14ms .

 

I'm sorry Tethys, but that made me giggle.

I'm also having odd FPS issues. Before I added VR to my level, I was having shotty FPS despite having an Asus 1080 ROG STRIX and Intel i7-7700K... What I read somewhere is that if I added the following line of code, it would help.

Code (CSharp):

1. QualitySettings.vSyncCount = 0;

So I created an empty script and on Start(), I added that line of code. Attached the script to my terrain. POOF! FPS went through the roof.

However, SteamVR looks to have their own way of doing things and that script does not have any impact with my Vive gear.

What I'm wondering is, does SteamVR throttle their FPS to 90? I don't think I've ever gotten a FPS above that.

But, I'm here also looking to see what's going on with WaitForGPU. I will admit, I have kind of an ambitious level for VR. Lots of vertexes. I have not really noticed the 60 degree issue. But, have not looked for it either.

 

Hahaha, well I'm glad someone got a good laugh. I meant to post this 2 nights ago - I sorted this out to a degree.

There is still an angle issue. I cannot tilt the headset on its side or my FPS instantly drops to 44. However, I realized that my terrain LOD system could be reduced in the distance. Come to find out, it was the hill BEHIND the hill I was looking directly at that was causing the instant lock up. Reduced my LOD for stuff in the distance and boom, back to 89. I still occasionally have the weird angle thing but its much more rare now. However, maybe this speaks again to the issue about this sort of blanket catch all message in the profiler when there is an issue. I get the same "VR.WaitForGPU" message if I have too much grass in the scene, if there are too many trees with shadows, if I have distance replace on any terrain shader and who knows what else. I know its still early in VR but man, this needs to be tightened up.

Thanks for the tip about hard coding some sort of vsync setting. I have not seen above 90 fps myself.

 

Well, the Vsync has no impact on VR. This was all before I added in my SteamVR hooks.

 

After having worked on multiple projects, I'm now having this issue. My scene is nothing but 8 cubes and a plane, and even with VSync disabled (through script or the editor), it takes up 90% of the CPU usage in the Profiler. Does anyone have ideas about what could be causing this??

 

I'm seeing the same issue I think? Mine seems to come from a sudden drop of WaitGetPoses causing the 44fps pegged value drop and then suddenly back up to 89fps. Over and over it does this. I see mine in the build which really sucks.

Unity 5.5.3, vSyncCount = 0 (essentially off in code and in editor), renderScale = 1 ...

Oddly enough I don't see the issue if I use quality setting of (Unity stock "Fastest" or "Fast")

Here is a trace of what I'm seeing if it helps (Unity stock quality setting "Simple")



Update:
I have been using Aquas-Lite for an ocean with islands. If I remove the Water then I never see this issue up until quality level Unity stock "Good". Quality levels "Fastest", "Fast", "Simple" all show no issue. "Good" shows the issue but very infrequently.

So this is interesting and frustrating at the same time.

 

Update:
In general I found so far that developing on Vive is as tedious and picky as mobile.

I used an asset called terrain to mesh to convert 6 texture unity terrain and that helped but terrain looks crap It's only a 500x500 terrain!!

Profiling led to some script optimization but not a huge change.

I updated my Vive hardware. No change.

Using most recent SteamVR in 5.5.3 and no change.

That damn WaitGetPose keeps dropping badly. Most of the time I am at 8ms or less but then boom WaitGetPose hits and up to 14ms and lag in play.

Rebuilt the project step by step and no discoveries.

Maybe the Vive simply can't do a flight sim style game?!

Anyone have an suggestion that worked?

I'm almost ready to ditch Unity for Unreal

 

Was banging my head on this for a while too.. for me it turned out to be steam VR settings. I had both 'Allow asynchronous reprojection' & 'allow interleaved reprojection' checked on. I unchecked interleaved and my fps doubled.

 

But who knows.. I didn't change anything and now my fps is dropping again 20 minutes later. Before the above post, I was getting 48fps due to mysterious vsync operations. I made the change and it went up to 90+ fps. Now it's back down to low 60-70s at best. No changes made. Super frustrating and obviously no rhyme or reason for this S***.
GFX 1080 TI
Unity 2017.2

 

Does anyone know how to fix it?
Having the same problem.

 

I've seen this kind of thing happen a lot in the editor, when the overhead of the editor slows things down to the point where the Vive drops frames and the WaitForGPU thing starts happening. But not I'm having the same problem with my game even when I build it. It happens even right after I start my game when I'm simply displaying a simple menu. So there are no heavy scripts, large # of objects, or physics happening, so there shouldn't be anything in my game causing this to happen. See below how XR.WaitForGPU is pegged out taking 29ms. What I've found is that this problem turns on somewhat randomly (it doesn't always happen when I start my game), and it also disappears randomly. I've noticed that sometimes when I move our maximize my game windows, that this can sometimes cause the problem to go away. This is a built Vive game (Unity 2017.3.1p1). I'm banging my head trying to figure out a way to prevent this from happening, but as far as I can tell this is a Unity bug. There's at least 1 other thread where a few people are discussing this, but I don't have the link handy to post. Super frustrating though.

 

I think I am having the same issue and it is in a build - not just the editor.

 

So that large spike was me covering the sensor light on the inside of the rift. The regular XR.WaitforGPU avg about 11-15ms. Is that normal?

 

Check this out - I am doing something very bad

 

Had this issue on Oculus and it was especially visible when entering the Dashboard menu - after playing around on a large scene, the nightmarish yellow spikes started to appear in the profiler and stayed there forever. Entering the dash was turning off all the scene objects in my case so nothing was rendering underneath, yet the XR.WaitForGPU made the Oculus system menu nauseably clunky.

What helped me was halving the textures size in the Unity's Quality settings which apparently boosted the overall performance enough so the dreaded yellow spikes were no more.

What's interesting is that I couldn't reproduce this problem just by intentionally lowering the performance on other scenes by spawning thousands objects or hundreds realtime lights. Even when the scene was performing at stunning 0.8FPS, the XR.WaitForGPU didn't bother to appear and the dash menu was super-smooth.

 

Disabling Async Reprojection helped a ton for me.

 

I always hide my scene view when running VR in editor, so i'll usually only have game view, and either the asset store, or animator view in the forefront, to try and reduce the issue from happening. I still have times where XR.WaitForGPU spikes every couple seconds though.

 

Brandon, you are missing the point here - many of us are getting WaitForGPU spikes even when rendering just the skybox. I have a 1080ti. While Unity is running, my whole desktop is lagging, even when it's tabbed down. Something is seriously wrong. Just updated my video card drives and rebooted, to no avail. Task manager reports 9% CPU usage, so it's likely GPU related. This issue came suddenly, everything was working fine yesterday.

 

Brandon, thanks for chiming in; original poster here. I am not sure I understand your answer correctly, so I have the following add-on questions:

a) If the CPU waits for the GPU, shouldn't the reason for the wait not be also apparent somewhere? I.e. shouldn't there be a clear indication in the profiler (presumable in the GPU tab) what exactly the GPU is doing at that point which causes the wait? Because in my original case, there was nothing in the profiler regarding that - no specific spikes in GPU - or I don't know how to interpret the data (is there a proper documentation with examples for XR.WaitForGPU somewhere? It would be helpful if you could show us a mini-project which shows in which cases XR.WaitForGPU occurs).

b) I would assume that this can also happen for non-XR projects; why is the method called XR.WaitForGPU then? Or is it another method in non-XR projects?

c) You say you "synchronize" GPU and CPU at the beginning of each frame. This would suggest to me that either CPU or GPU have more to do that the allotted time for the frame (say, 11ms), i.e. can it also happen that GPU is faster and must wait for CPU? However, that is at odds with the method name "WaitForGPU", so I think its only the CPU with ever waits for the GPU, right?

Philip

 

@BrandonFogerty Any more advice or recommendations on this?

I'm getting the same XR.WaitForGPU spike on a 1080Ti. Unity version is 2017.4.8, not Vive but Rift; latest Oculus SDK.

Editor is closed - build only running. Not moving, standing still looking in front of me, nothing changing in the scene. All starts fine at a solid 90fps, then intermittently a spike comes and the framerate tanks (4-15fps), then holds 5 seconds or so at 44fps, before getting back to 90fps. Repeats every 30-60 seconds.

Here are screenshots with a debug build and profiler connected (scene is not running in the Editor)








Anyone with any idea on how to interpret this, or debug this further, it would be very much appreciated.

 

I'll answer my own question since I've managed to work it out after spending the afternoon on this...

I've learned a whole lot about optimisation and the profiler. Here are some of the things I tried:
...
Try
Application.targetFrameRate = -1
QualitySettings.vSyncCount = 0

Neither made any difference (I'd already turned off vsync in quality) My uneducated guess is that the 90fps is capped by the Oculus SDK, like its own kind of vsync.
...
**Check frame before spike**
https://forum.unity.com/threads/help-me-understand-these-profiler-results.542463/

For Oculus Go, "It could also mean that your CPU took more than 16ms and you're vsync locked. In this case, WaitForPresent is waiting for the next vsync and the GPU might not be doing a lot. That might be your problem - the frame right before the hitch looks like it took 17.75ms. When it's running at 60fps what is the average frame time? I would also check the breakdown of CPU usage in the hierarchy view on that previous frame.

There are some tools [listed here](https://developer.oculus.com/documentation/unity/latest/concepts/unity-integration-perf/) along with some suggestions. I don't know if the Remote Monitor can be used but it looks pretty handy. It looks like you can't do GPU profiling in Unity on Android sadly. Here's [a blog post](https://developer.oculus.com/blog/squeezing-performance-out-of-your-unity-gear-vr-game-continued/) also that has some suggestions."

For Rift, frames times should be:
To maintain 90fps, you need to be < or about equal to 11.1 ms per frame (1000ms/s / 90 frames/s = 11.11111).
https://forums.oculusvr.com/develop...fficult-to-profile-other-issues-video-example
...
So as soon as the CPU takes more than that magic 11.1ms you're sync locked and the GPU has to wait until the next vsync.
To hit below that magic number, try reducing texture sizes and improving batching. Use the frame debugger to figure out why stuff might not be getting batched.
...

So all of the above failed in my case - bit by bit I deleted everything in my scene until I got down to nothing except a skybox and a plain camera (not even an OVRPlayerController) and still I was getting those unexplained spikes :/

... the asnwer?
Pretty sure that I was CPU bound I then decided to look at Task Manager, and lo and behold, I was getting 70% CPU spikes from explorer.exe

**The reason**? It appears to have been caused by having too many folders opened at the same time (about 15) I tend to get lazy closing them and had never realised this would cause a problem. My PC's an i7-7700K CPU @ 4.20Ghz with 64GB of RAM so never considered that would be an issue.
I closed the folders and bingo - smooth as silk 90fps. Go figure!

Hope this helps someone with the same problem. (does make me wonder how as a developer you can control your app's playback performance when you have no control over how many windows a player might have open on their own machine...)

 

I am also experiencing this problem. When I am on an EMPTY Scene, frame rate stays at 90 for few seconds then drops to 86. When that happens I see FrameEvents.XRBeginFrame -> XR.WaitForGPU kicking in. I won't lose sleep over 4 frames but the real problem is when do have things to render in the scene, how little (less than 200 batches) it may be, frame rate drops to 45 (from 88-90).


I have Vsync off. This happens in Single pass and Multi-pass.And it happens on Editor and Build
Empty Scene= ONLY Camera, no lights, not even directional light, no vr stuff nothing!

I tried on an empty project with nothing imported with only "Virtual Reality supported" box checked.

Unfortunately what @jeromeWork (closing folders) said did not help. Has any one found the any promising information on this?

 

We had this issue plague us for a long time back in the day and after submitting bug reports it turns out that it was caused by GeForce Experience's overlay feature. Try uninstalling GeForce Experience to see if it helps your situation.

 

That makes more sense... I will check it out. Thanks for sharing!

 

We're back to facing this issue when looking down. I believe it has something to do with position tracking. When ever distance swap on a shader is enabled or LODS on trees are on, looking down breaks the solid 89 fps and I get stuck on 44.5. My fps wont budge until I look back up again. It's like it's waiting for something and gets locked in that phase.

 

Dude this is like 100% the problem we are having down to the fps has anyone found a concrete solution?

 

I have found a solution, by changing properties of my OVR camera manager

 

Since this is an ongoing issue, here are some thoughts:
As it has already been mentioned, the WaitForGPU is not the reason for bad performance, it just means that there is a wait which must happen so that the GPU and CPU can be synced. This can also happen if you have 200 fps, it might be that 80 percent of your GPU time is Wait time, because you are rendering too fast, and have to wait for the 90 fps 11ms mark.
As for the 45 and 90 fps, you cant change those locks, and you dont really want to. It would lead to really bad motion sickness if your fps varies between something and 90.
The WaitForGPU is really vague, it doesn't tell you anything beside you are taking longer than 11 ms on a frame. You can still be GPU or CPU bound. Here are some stuff you can do to figure out what's going on:
1. Add up the rendering time you need for your Opaque, transparent, shadow pre passes, post processs and other rendering time. Do the same thing for your scripts and physics operation. if you end up at something around 8-9, then you have classic performance issues which you need to fix. You should leave a few mili seconds overhead for things like MSAA (which cost around 0.3 ms itself), other application running in the background, difference in density of vertices based on which direction you are looking at ...
2. If you should be finished with the frame in 6 ms but you are still apparently over 11ms, something is stalling your process. This is very probably an issue of the CPU, lots of probable reasons for this, for example, a classic problem is Windows, drivers, new programs installed, other application running, anti virus and ... I had horrible fps in an empty VR scene, turned off Windows Defender and then everything was fine. Turned out the defender stalled the unity process which led to the GPU missing the mark, then having to wait for the next frame. Stuff like these are really common, happens in Unreal too btw, once had to change the slot of my RAM to fix my fps problem. Once you hit one of these issues, I am truly sorry, you need to just try things out and investigate things systematically until you figure out bit by bit what is stalling your CPU. On the GPU another classic case is the GPU profiler. If you are to close to 11 ms and turn on the GPU profiler, it wont manage to collect its samples on time and has to wait for the next frame.

 

I also did have trees when I tested it at some point. My own LODs didn't cause any spikes, so unless the tree LODs are different(I think LODs are LODs?) so it might be the shader. In any case, have you found a solution?

 

Care to share what those properties are?

 

I am still having this problem. I am certain it has to do with the tree shaders. Happens as soon as I turn on trees or foliage

Were you able to resolve your issue? I am still struggling with this.

 

For the next person who finds this old thread via google search at 2am...

I too was seeing stuttering issues. Manifested as the WaitForGPU in the profiler. In my case I had URP with anti-aliasing left on at 8x from when I was doing non-VR. Disabling anti-aliasing VR now works smoothly in editor.