Time.deltaTime Not Constant: VSync CameraFollow and Jitter

It's a bit of a bummer for the VR as it would have made a big difference to user experience (but definitely appreciate all the effort on these fixes!). At the same time I can totally understand that we are getting really close to 2020 cycle feature cut off now so this wouldn't be totally unexpected.

While it doesn't make VR fix land any earlier, I'd love to hear what makes it more challenging when rendering VR for example on Windows DX11?

 

Oh.. in that case I do hope OpenVR will still get support for this despite going 3rd party with Valve providing the new XR Management system plugin.. :/

This feels something that should get natively implemented by the new XR API by the XR providers but I don't know if there's even any API there that lets you query things like this.

 

Will 2020.1 be released with this fix?

 

I think you should make this issue high priority and fix it on all backends. Time is one of the most important aspects of any game. Every system relies on it. Do I need to say anymore? You have many AAA games running on your engine. Just my two cents.

 

This Is A Major-Bug in All Of Unity! This should have been taken care of many years ago and you're just noticing it now?? So, Unity will You Be Fixing This Bug For Unity 2019, 2018 versions? This is a big show stopper for Unity and should be addressed for ALL VERSIONS!.

 

You should still be fixing this bug, and release a patch, so devs can apply it if they wish.This will not break existing games?

 

Big changes like this can always break x amount of other things and more crucially, it changes the default behavior of tons of systems as deltatime affects a lot of things beyond your own code. Do note that this fix isn't something you can toggle on and off, it's replacement for the old setup.

 

Do I understand this issue correctly, that every single Unity game ever released has jittering problems? Meaning that all those years of people complaining that Unity games are stuttering was because there's actually a bug preventing the games from running smoothly by default?

How is this issue not more known? I was in the 'it's the programmer's fault if the game doesn't run well' camp. But if this issue is affecting every Unity game ever, then those complaints were actually valid? I feel like this is such a major bug that it should be near the top of every 'disadvantages of using Unity' list ever. I second the notion that this needs to be ported to 2019 LTS.

 

I'm afraid it's not all that simple. First of, most game engines available to you do deltatime calculation like Unity used to do before 2020.2 so it's kinda hard to just fault Unity specifically for this. In fact, Unreals DT's fluctuate even more than Unity's did before this change. There's also been ways to filter the DT yourself on all these engines if you really needed to but granted it's an extra step that most devs would not make + it would only apply to your own code, not to internal systems like animation etc.

While I can see CLEAR difference with the DT fix and without (I can pretty much ditch my own DT filtering code now), there's like dozen reasons why Unity games can jitter and most visible jittering is not from the thing fixed here. I can list few common reasons:

Garbage collection spikes: This is easily number 1 reason. GC in Unity is ancient, and all around patched up solution which can result in huge and long lasting GC spikes. Unity did come up with incremental GC few years ago which should ease the pain a lot but the underlying GC is still the same (but at least they now split GC across frames so the spikes will not be noticeable). In past people often even stalled GC while gameplay was active and then triggered GC manually upon pause or level completion to avoid hitches when it was more noticeable. Some games also hitched a lot since devs didn't pool resources or simply didn't care of garbage generation at all. There's been a ton of common things that generated garbage in past and not every dev even realized all these. Many built-in Unity systems/packages also have in past generated or still do generate some garbage.

Failure to understand FixedUpdate and interpolation: This is probably 2nd biggest reason. Physics in Unity run async to rendering by default, also at somewhat conservative 50Hz update rate. Since physics run by fixed invervals in relation to deltatime, this can cause a lot of visible jitter if the devs don't enable rigidbody interpolation or extrapolation. Unity docs imply there's a perf overheard from interpolation so many just opt to not do it. Many devs also simply don't understand what fixed update is and place things like camera movement there which would obviously fail big time if the aim is to get any smooth movement.

Using built-in (old) input system: Especially mouse input data can be super noisy here. I've seen HUGE hitches by simply using built-in mouse input to drive character turning in Unity. The results you see probably depend a lot by your own hardware and mouse polling rate but for example I've seen pretty horrific noise on some common branded wireless mice. For comparison, gamepad input is butter smooth in comparison and you'll not see similar hitching on panning motion with it.

 

@Tautvydas-Zilys just wanted to say thank you for keeping us in the loop here and coordinating all this. Awesome stuff!

 

As this issue is being fixed in most platforms, does it mean that the methods in FrametimingManager will be finally working in those platforms? Currently they work in XBox One AFAIK.

https://docs.unity3d.com/ScriptReference/FrameTimingManager.html

Discussion thread:
https://forum.unity.com/threads/fra...esttimings-always-returning-0-timings.544325/

 

I think VR is gonna profit from this the most looking forward to the fix

 

Hey Unity. I've fixed this problem on Android in a custom engine, and think I can explain it well, if at least to the community.

It looks to me like Unity is doing Buffer Stuffing.

This page has a good write up:
https://developer.android.com/games/sdk/frame-pacing/

Buffer stuffing breaks down and causes this deviation from 16ms. You fill the buffer, the GPU driver blocks your render loop, and your dt exceeds 16ms. The buffer frees a frame, you render it in 2ms, and start the cycle again.

If ya'll actually hardcoded your DT to 1/60 in that scenario, it would look smooth. But then if you were late on a frame, you would never catch up to "real time", your game would just slowly fall behind from real time.

Basically the solution is this:

If the buffer is full, congratulations, render a frame at the target refresh rate and stuff it.

If the buffer isn't full, you were slow, but you can _still_ render at the target refresh rate, assuming that you will catch up before it empties.

If the buffer is empty, your stuttering and have fallen behind the real time! You must change dt from the target refresh rate to catch up!

The hard problem here is that _you dont know how long the next frame will take to render_. So your running along, at 16ms dt, hitting every vsync refresh, when suddenly a frame that we requested _with a dt of 16ms_ takes longer to render. This frame is 'behind' real time! But, did we stutter?

We didn't stutter if the buffer didn't empty! The buffer dropped down to 2 or 1 frame in it, and we can still push another frame with 1/60 delta time.

But if the buffer emptied, we need to "catch up". The only way to do that is to alter dt.

**Note, that my experience with Android's Choreographer showed that it is still not perfect. We could submit a frame "on time" (before the vsync), and the vsync would still render the previous frame (on a google pixel 1 we reproduced this).

We ended up tracking our own representation of how many frames are in the buffer, and preferring to error on the side of "we probably missed it".

This was kind of a hack, but it worked. Delta Time would be locked to the vsync interval, and the system would render at 60fps, and then would properly drop to 30fps if we couldnt keep up. Synchronization with real time was achieved.

Proper implementation would use these extensions to track when the GPU driver _actually displayed the frame_

Check these OpenGL extensions for querying how many frames are in the buffer, and to query the display timings for the previous frames.

https://www.khronos.org/registry/EGL/extensions/ANDROID/EGL_ANDROID_presentation_time.txt

https://www.khronos.org/registry/vulkan/specs/1.2-extensions/man/html/VK_GOOGLE_display_timing.html


CroTeam has a good writeup of this problem:

https://medium.com/@alen.ladavac/the-elusive-frame-timing-168f899aec92

And here's CTO of CroTeam talking about the problem:

Feel free to email me if you'd like more information regarding this technique.

 

Very awesome discussion, and very awesome work. Thanks to @Tautvydas-Zilys for communicating so transparently about this, and thanks to everyone working on this - including the community! This thread really kind of feels like the "good old days" of Unity and I appreciate that more than words could capture.

Just chiming in express my gratitude for the work on this, and also to add one more user to the list of "needs this for VR", and in fact, for pretty much all VR platforms (OpenVR, PSVR, Oculus native desktop, Oculus Quest, and potentially Pico Neo 2, which I believe have their own proprietary SDK).

I'm still on 2019.4 because I haven't made the move to the new XR plugin framework (which I have *very* mixed feelings about) but having this issue fixed for VR will probably be a strong enough reason for us to move to whichever version has this for enough platforms for VR (OpenVR and PSVR being the most important for us, with Quest a close second ;-) ).

 

It's pretty amazing how we gamedevs had this bug in our update loops for so so long but we didn't need to fix nor did we fully understand the problem until only recently.

I think we got away with it by some games _actually slowing down_. Modern games are more heavily networked so actually synchronizing to time is more important.

I'm just shocked that I didn't see this problem more often back in the raw OpenGL days.

To be honest I think the problem is simply excaberated on Android by the aggressive CPU Scheduler, which causes the aforementioned "we randomly missed a single frame" scenario far more often. Turns out the best way to react to that is to _do nothing_. Who would have thought?

We need Glenn Fiedler to write "Fix Your Timestep 2".

 

Thanks for all the hard work on this issue and keeping us informed.

 

I hope you are able to address OpenVR/SteamVR too despite it not being developed in-house anymore. It's one of the primary targets on PC VR.

 

The only problem I've got - for some reason Time.deltaTime cannot go larger than 0.1f (on Windows 10 DirectX12 HDRP 10.1 with ray tracing). So, if fps drops below 10 it's not reliable anymore (those low fps are common case for low-end hardware in my benchmark project)...
(Case 1282420)

Upd: oh, my bad - there is a Time.maximumDeltaTime parameter that limits it to 0.1 by default... didn't expect that there is a good reason to have this limit.

 

This is great! Can't wait to try it in VR

 

Great blog post! It clears up some misconceptions I had about unity’s pipelining. It would be awesome to have the profiler timeline convey more of that pipelining. Sometimes it’s not very clear to me why frames get dropped in my game (most likely a GPU bound issue but it would be great to see it clearly).

Anyway, thanks for all the hard work, and please don’t worry too much about all the negative replies starting with “it took ten years”, we all have this tendency to complain about things that could be improved (I have quite a lost myself ;-) and take for granted parts that are just great.

 

Hello. Please introduce an EarlyUpdate callback and put it in the loop before PumpOSMessages. That way, I could put all input independent code into EarlyUpdate, thus moving input polling to later in the frame, thus reducing input latency. All AI driven characters and their animations don't depend on input and so they can be processed before updating it.

 

Instead of introducing yet another callback, couldn't you just poll the input on LateUpdate instead? It's meant for things that process as late as possible. For example you can process input events manually on new Unity Input System so afaik you could just poll input on LateUpdate.

 

Assuming I understood your question, yes. I mean, from the blog post:

Code (CSharp):

1. while (!ShouldQuit())
2. {
3.     WaitForLastPresentationAndGetTimestamp();
4.     // EarlyUpdate would go here. All but player character's animators can update here, etc.
5.     PumpOSMessages();
6.     UpdateInput();
7.     Update();
8.     WaitForRenderThread();
9.     IssueRenderingCommands();
10. }

 

Can't you already do this yourself using the PlayerLoop APIs? There is already an EarlyUpdate system for which you can specifically configure your own SubSystem and insert it wherever you like.

Unfortunately, Unity's documentation for specific systems leaves much to be desired. Most SubSystems' documentation (including UpdateInputManager) is very unhelpful:

As of Unity 2019.4, it looks like what you'd want to do is target adding your own SubSystem to the Initialization System. It appears that time updates (PlayerUpdateTime) and input synchronization (SynchronizeInputs) occur within that "phase". Clearly, this may look very different in Unity 2020.2.

For what it's worth, I cannot recommend enough the Player Loop Visualizer. It will allow you to investigate your PlayerLoop and help you understand the order of operations of Unity's internals. It will also help you quickly identify a target for inserting your own custom logic. And to be clear, you can insert your own custom logic with C#.

 

Hmm... if that's the case, then according to your blog entry's explanation of frame timing:

Code (CSharp):

1. ——————— // Input event arrives from the OS!
2. WaitForLastPresentationAndGetTimestamp(); // Wait for frame -2 to appear on the screen
3. PumpOSMessages(); // Pump input OS messages for frame 0
4. UpdateInput(); // Process input for frame 0
5. Update(); // Update game state for frame 0 with the input event that we are measuring
6. WaitForRenderThread(); // Wait until all commands from frame -1 are submitted to the GPU
7. IssueRenderingCommands(); // Send rendering commands for frame 0 to the rendering thread
8. WaitForLastPresentationAndGetTimestamp(); // Wait for frame 0 to appear on the screen. This is where the changes from our input event appear.

would put the "WaitForLastPresentationAndGetTimestamp()" also outside of the PlayerLoop system, no? If that's the case, then what is the PlayerUpdateTime subsystem doing? (And what does the AsyncUploadTimeSlicedUpdate subsystem do for that matter?)

Glad to hear this. It is surprising that it's not already within the overall PlayerLoop system's purview.

On a related note, is there documentation anywhere that describes what portions of the engine's loop are not included within the PlayerLoop system?

 

Can you explain what the expected fix would look like? I'm having trouble understanding how a fix for the PlayerUpdateTime subsystem would address @AnsisMalinsUbisoft's issue.

That is a good hint! Thanks!

So there are still operations that are "sibling" to the PlayerLoop function. Got it.

This is a little confusing to me. It sounds like you're describing PumpOSMessages() here. However, the list of things the Windows Player seems to do as part of that operation is explained to include "updates gamepad state in InputManager". How is that different from the UpdateInputManager and SynchronizeInputs subsystems? (Also, what exactly do those systems do?)

Perhaps an appropriate question here is how does the PlayerLoop function "mix" with the frame timing stack you listed in the blog entry?

I would guess something like this:

Code (CSharp):

1. ——————— // Input event arrives from the OS!
2. WaitForLastPresentationAndGetTimestamp(); // Wait for frame -2 to appear on the screen
3. PumpOSMessages(); // Pump input OS messages for frame 0
4. {
5.     // PlayerLoop!
6.     UpdateInput(); // Process input for frame 0
7.     Update(); // Update game state for frame 0 with the input event that we are measuring
8.     WaitForRenderThread(); // Wait until all commands from frame -1 are submitted to the GPU
9. }
10. IssueRenderingCommands(); // Send rendering commands for frame 0 to the rendering thread
11. WaitForLastPresentationAndGetTimestamp(); // Wait for frame 0 to appear on the screen. This is where the changes from our input event appear.

but your explanation has me wondering if things like "UpdateInput()" are outside the purview of the PlayerLoop (even though there are input-related subsystems within it...).


---------------
Side Note: Someone should double your salary for a week and let you write some documentation about the core Player Loop. There's more helpful information in that one paragraph than, like, the entire PlayerLoop section of the script documentation...

 

Hmm... based on what I've seen of the System/SubSystem relationship of the PlayerLoop, it sounds like you may need to take the PlayerUpdateTime subsystem out of its current parent system (PlayerLoop_Initialization()) and insert it into a new system that can be handled earlier in the core loop, alongside WaitForLastPresentationAndGetTimestamp(), yeah?

This is extremely helpful. I seriously wish the PlayerLoop documentation covered this stuff. It would make working with it all the better.

Regardless, the explanation clears up the ambiguities about what is going on internally and the role that each of those subsystems play. That each platform features platform-specific "input state wrangling" code makes sense. That this is the abstraction layer that unifies everything for Unity managed code consumption makes sense. Thanks for clearing it up!

 

Thanks for posting the details of your fix. Had no idea Time.deltaTime was so complicated when I started the thread. It explains some of the things I noticed like...

1: Moving the mouse caused a larger Time.deltaTime variance
2: Background apps caused larger Time.deltaTime variance (even if they did not generate high enough load and cause dropped frames).
3: The same app has drastically different Time.deltaTime variation on different hardware.

 

I noticed this showed up on issue tracker: https://issuetracker.unity3d.com/is...-issues-on-secondary-monitor-in-windowed-mode (not reported by me but apparently on my earlier findings)

 

Ah nice, I'll give it a go again too as I'm now on latest platform update, just to verify if it's changed on my end.

 

2004 is acting differently but it is bit weird too. So far what I've seen is:

- If I keep 144Hz monitor as default, I get dt ~ 0.00694 (144Hz) on all monitors when running windowed despite the app's window is sitting on is 60Hz monitor. If I toggle to fullscreen, it corrects to that monitors refresh rate.
- With 144Hz monitor as default still, if I drag the built Unity app window by holding mouse left down on the app's top bar (or just hold the mouse left down there), it gets that mixed DT thing again where it can't decide which monitors DT to use. This happens on all monitors, not just when hovering over the 60Hz ones.

- If I set 60Hz monitor as default and keep 144Hz as secondary, it will always get 0.01666 deltatime on all monitors no matter what I do as long as the app is running in windowed mode.

So it does feel like windows compositor runs at primary monitor's refresh rate but it gets confused if you drag the window on desktop with mixed monitor refresh rates IF some of the monitors are running under the primary monitors refresh rate. This all really seems like a windows issue.

 

How would EarlyUpdate would help here with input latency though? May be I'm missing something. As Tautvydas-Zilys, "Input is actually snapshot during "PumpOSMessages"". it is still gonna take this snapshot every frame.

Looks like the link leads nowhere