How far are we away from framerate/renderloop independent input?

Speaking for many*, I'd love to get an update on this.
And this seems like the place to get it I hope it is!

Purpose: To be able to make an app or game with minimal battery drain for non-game apps and UI-only apps. Ability to set framerate to 1 and have input still poll at 60hz. Or pause the render loop until input is received.

If there's nothing happening on the screen, like a UI just being showed, the app should not be hammering away at 60 FPS.

It's currently easy to set it to run at 1 FPS, but then touch input also runs at 1 FPS (at least according to my understanding and tests).

My personal reasoning is that I may be able to ditch making apps in frameworks like React Native, Titanium, Xamarin, etc. and use Unity exclusively. That would truly be a dream, since I really like the visual workflow of the "new" UI system.

* This suggestions has almost 1000 votes on it:
https://feedback.unity3d.com/sugges...er-loop-for-lower-power-consumption-on-mobile
And there's many more threads with votes that are of similar wording.

 

There's plans to tackle this post 1.0 but that's as far as it goes ATM.

One gotcha with the thing is that applicability is somewhat limited. Like, you can't do anything visually. Even if all you do is just show a UI, that UI has to be completely static. No animation effects, nothing.

But yeah, we do realize that for some applications, this is just fine. And indeed a great way to preserve lots of battery power and not require gloves to handle to hold your device.

We discussed it at various points internally and the idea things have so far arrived at is having an API like

Code (CSharp):

1. InputSystem.WaitForInput(timeout: 1);

But there's probably a number of details to hammer out once work is actually happening.

 

The usage is not limited to non-game apps, most word games and many puzzle games would benefit a lot from this feature. We create word games and currently we do like most other word game developers, which is to develop 100% native or use some kind of native framework, because screens are almost always static and it would be a huge waste of battery to use a game framework like Unity.

The downsides of this approach are that we have to create and maintain 2 code bases (Android/iOS), and that making animations natively and moving stuff on the screen is much harder. Unity would be a great choice, if only the input system was not tied to framerate.

It is becoming especially important now that Google has said that battery usage can impact your ranking in the search results.

 

Couldn't it be so you can turn down graphics frame rate to 0, but keep input at 60hz? If something like this were to be made, I'd make everything in Unity and go for the pro sub in a heartbeat. I make tons of apps, but every now and then I get a unity assignment, and I love it, cause I get to work in Unity, but it's so sad to go back to using my other app tools like react native etc.

 

The how is still quite up in the air. My ideal is something that is both an elegant solution and at the same time doesn't require a lot of cross-platform refactoring in Unity. The player loop is hosted by platform-specific code so making changes there can be a pretty arduous undertaking.

The thread has sparked some renewed internal discussions and we've had some ideas. No concrete news yet but it's good you keep pushing

 

That really means a lot to me! I've been hoping for something like this since probably 2014.
And I'm not the only one, so I've searched a bit in the feedback, and tallied up votes below.
The people in those threads are also a bit more eloquant than me I think you'd be surprised to see how many people would use this.

https://feedback.unity3d.com/suggestions/realtime-input-framerate-indepe - year 2011 - 64 votes
https://feedback.unity3d.com/sugges...er-loop-for-lower-power-consumption-on-mobile - year 2015 - 959 votes
https://feedback.unity3d.com/suggestions/better-support-for-guionly-appl - year 2013 - 135 votes (incorrectly marked as completed)

And some forums threads where people have wanted it too, trying various hacks, talking about wanting to use Unity over other regular app frameworks:
https://forum.unity.com/threads/using-unity-for-a-gui-only-application.200300/#post-1359432
https://forum.unity.com/threads/suggestion-low-battery-mode-for-non-game-apps.343632/

runevision has a thread which discusses how to implemented it, but no activity since may:
https://forum.unity.com/threads/bat...t-with-immediate-new-frame-upon-input.476263/

 

has anybody at Unity been tasked with simply thinking, and articulating the answer to: "What is the best possible manner for input events to behave and be?"

 

I don't feel like this problem rests entirely on the Input System, but certainly involves it.

How do native apps solve this problem?

I had thought they actually kept running at ~60 FPS but simply left the previous frame untouched until dirty flags indicated a redraw was needed.

 

At least in Android I think it is like that, the main thread/UI thread issue a draw event every 16.66ms like Unity (yt) but Android does not redraw the entire screen like a game engine, but only the part that are invalidated (1, 2).

All Android app starts with that main thread/UI thread that draws, and it must also dispatch input events.

https://developer.android.com/guide/components/processes-and-threads#Threads
(yt)

Currently Unity is running on an other thread that is not a main thread, but receive all touches from the main thread callback via nativeInjectEvent method seen in the decompiled UnityPlayer class. I think the drawing of Unity (OpenGL) is entirely offloaded to this other thread and leave the main thread job to just receiving input and few others. So the input in Android is in parallel to Unity, and will be synched in some way to Unity's game loop that I have no idea.

And so theoretically even if Unity is running at 1 FPS it should always be aware of all the incoming touches since the main thread/UI thread of Android is always feeding touches to nativeInjectEvent.

We want a 1 FPS Unity game with 60 FPS touch handling, and updates smoothly (e.g. went to 60 FPS temporarily) only when the touches came in.

The core problem is that, suppose if nativeInjectEvent is able to keep all the touches that occur when the app is running at power saving frame rate (I am assuming that currently it cannot do this?), to actually react to it (that is changing to 60 FPS on input so we can smoothly update UI again) we need to wait for a frame ultimately to process a touch. Because currently the New Input System's callback comes in a frame.

The way to fix this might be we have to add a special callback portal that maps directly to the native ones so we can react immediately independent of Unity.

And then when that callback fires, hope that setting FPS number there works so that Unity's drawing thread understand and adjust the drawing. Now that Unity is fast it can draw smoothly and finally setting the FPS to low number again in Unity's thread. All of this just to achieve "partial drawing" imho is going a bit much against a design of game engine which aims to run logic every frame rate time and redraw the entire screen.

Question : Can the new scriptable rendering pipeline prevents a drawing in some frame? Or redraw only part of the screen in some frame? If that is possible then the whole problem can be solved by using that while running the game at 60 FPS. (and think that this FPS is not related to the drawing but just the input) We can make something like "Unity App RP" in addition to the current LWRP/HDRP?

 

When considering performance and ideals, Android is the last place to look.

 

Indeed,

I was doing a ugui only app like project a while ago, and no matter how much I tried to get some solution where application fps was lowered down to conserve battery, where the moment touch input was detected the application fps would be lifted, the lowest I could really go was 10-15fps any lower and just interacting with ugui, was abysmal, touch scroll was the worst as the most the input is missed, same with buttons, though they worked better than anything that involved scrolling. Infact I even enountered Unity bugs with settings application fps not even working properly on htc adnroid device.

In the end I shelved the project, aside from the battery life issue ugui is pretty awful in terms of tradition components like treeviews, listviews etc.. sure there is addons and even some github stuff but compared to rich well designed well documented components found in WinForms (like this beautiful project http://objectlistview.sourceforge.net/cs/index.html ) and even JavaFX (which is what I ended up using for cross platform app with the intention of porting to android later) ...yeah it's really sad Unity can't be used in this area.. it would be bloody amazing if it were possible.. maybe with reboot and major enhancement to ugui and unity editor improvements for it ie seperate ugui editor window etc). As I cannot stand most the other alternatives like react-native (or literately anything even slightly touches on javascrap usage)

So yeah I hope something is done about it, seems like a no brainer to me to really push Unity usage into other areas.

 

This is an interesting solution, however even if it works it has the problem of preventing only the rendering, it will still update everything else on each frame, and on mobile, Update() calls and other overhead can be a main source of CPU usage.

It would be nice to benchmark it and see how much battery does it save, and if it is enough or not to consider it a viable solution, because by definition it would be inferior to simply stopping the game loop completely until the user interacts with the app. I don't think that it goes against the design of a game engine, it goes against the (bad) design of the current input system.

Once the new system is here, the only thing we need is a way to stop and resume the game loop (e.g. set fps to 0/60 which may already work) and have a way to be notified of new input in real time. Online multiplayer games would also benefit from this, by using such events to process input immediately and reduce input lag as much as possible, without having to wait for the next frame. In FPS games 5-15ms matters, at least it matters to many players who buy special keyboards just so that input is processed a few ms faster.

 

Maybe the current PlayerLoop experimental API can help with this? At 0 FPS you set a new empty player loop and restore the default ones on touch. If it is possible use that API off-main thread then the solution might be possible today. (Because Android's native touch is off-main thread when looking from what thread Unity is running on AFAIK)

But I doubt the cross-thread player loop modification is possible because what will happen if we are currently in mid-phase (like mid-physics update phase) and we tear the player loop out from other thread.

 

in the Prefabs beta forum I've asked a question about the possibility of utilising Scenes as standalone entities with the rights to manage touch and their own transforms.

It is my opinion that this use of Scenes MIGHT solve a lot of the problems of rendering efficiency tasking, and independence of input management and the ability to change/modify all. In particular, for UIs, this might be a good idea.

But I may be completely misguided and utterly wrong.

It occurred to me because the new and Improved Prefabs are really just scenes used to hold an arbitrary amount of stuff of arbitrary stuff, with arbitrary amounts of user/developer control, largely limited by the notions of the Prefabs being prefabs.

If that obligation (of scenes pretending to be prefabs) was removed, and a transform and uniquely manageable input given to each scene, then this might make for a much more creative and problem free approach to many creative problems...

https://forum.unity.com/threads/if-scenes-had-a-transform-and-input-handling.549700/

The root of this idea, if you'll excuse the pun, comes from node based and graph node based game engines, wherein a master scene has a root, as do all scenes added to it, and they can either choose to be the boss, or not, for other scenes and input.

And this works, because it permits a completely freeform approach to abstraction design, from game manager to levels to "prefabs", to grouped and nested objects and on down to individual objects.

 

There's been a notion for a while of removing the difference between scenes and prefabs (since for the longest time internally they were almost the same thing), and before at least the decision was made to keep them separate since the tooling was different and they served different needs.

Now that prefabs have nesting, and are actually assets with sub-assets instead of blobs of scene data in another wrapper, making them the same may make even less sense than before. AFAIK scenes don't contain sub-assets, but they can reference any asset. They also have special handling for things like lighting, etc.

You could make a similar argument for MonoBehaviour and ScriptableObject, as they're fundamentally the same C++ type under the hood, but they're conceptually (and have actual implementation differences in C#-land) useful enough to keep them different.

Also they'd have to rewrite a lot of docs, tutorials, and major parts of the engine surrounding prefabs and scenes, and they just finished doing that.

 

It would seem more consistent with the Unity API to have an InputUpdate() on MonoBehaviour, same as Update() and FixedUpdate()

That said, I'll take any API for this feature!

 

Just adding to the list of people who want this feature badly.
Currently lowering the FPS to 10 when no input has been registered for some seconds, then back up to 30/60 when input is registered.
Saves some battery, but there's a lot more to be saved if the input was truly frame rate independent.

 

Quick update on this one. We're still pushing and it's on the radar, but it won't be something available for 1.0 of the new input system.

 

well goood to hear

 

Good to hear that is on the radar.
Are there any updates on this?

 

Does anyone know if rewired supports frame rate independent input?.
I also noticed, on Linux at least, that FixedUpdate calls see the new input state before the Update that reflects them rather than after them. One frame of input lag isn't the worst but it's not ideal. The SFV community made a huge deal out of the 2f reduction in input lag so there is definitely a lot of passion about this issue.

The idea my input handling will also become less responsive if frame rate dips makes me sad. I suspect that some of my deaths while playing Hollow Knight were a consequence of this.

 

https://docs.unity3d.com/ScriptReference/MonoBehaviour.FixedUpdate.html
This is actually a very big misconception. By default the FixedUpdate calls are at 50FPS, independently from the Update cycles (which can be at 30FPS or 60FPS or any value you set in the target FPS, or any value if you leave it to roam free). Which means you can have any number of FixedUpdates before/after any Updates. You may have none of them, you may have two or even three if your Update cycle is painfully slow (rendering issues), although during heavy load the FixedUpdate calls will be affected as well.
You cannot count on these comes after/before each other. They are virtually independent by design.

https://guavaman.com/projects/rewired/docs/FAQ.html#framerate-independent-input

 

This is useful information for those thinking Fixed Update is a possible solution to this issue. I get poked about this so often I added it to the FAQ:

http://guavaman.com/projects/rewired/docs/FAQ.html#fixed-update-framerate-independent-input

 

UnityEngine.Input updates its input values once per cycle on the main thread before FixedUpdate or Update ever run. All FixedUpdate frames that may run during one cycle and the Update frame share the same input values which were sampled at the beginning of the main thread loop. So input is always slightly more fresh in FixedUpdate than in Update.

In the case of a 20 fps render and 60 fps physics refresh:

Frame 0:
(UnityEngine.Input Sampled)
(FixedUpdate)
(FixedUpdate)
(FixedUpdate)
(Update)

Frame 1:
(UnityEngine.Input Sampled)
(FixedUpdate)
(FixedUpdate)
(FixedUpdate)
(Update)

Frame 2:
(UnityEngine.Input Sampled)
(FixedUpdate)
(FixedUpdate)
(FixedUpdate)
(Update)

 

@Rene-Damm Any Updates?

 

@Rene-Damm Any Updates?

This is a must have for my latest project and I love to use Unity for it.

 

It seems it landed in Unity 2019.3 alpha and is called "On Demand Rendering":
https://forum.unity.com/threads/on-demand-rendering-new-in-a7.701369/#post-4706984

 

Just tested it. Finally! It's amazing. Has anyone done some tests or comparisons on mobile devices? For example energy consumption in menu only apps?

 

As an additional question to this, is there a way to know exactly when an input happened? The
CallbackContext objects have a time field, and it seems to have finer resolution than update/fixedupdate intervals. The problem is it does not match the unity Time.time value, and is generally off by several seconds or more (and different every startup). You are able to tell how much time has elapsed between events, but there is no way to figure out when it happened relative to the current frame.

I'd like to be able to "rewind" my game and appropriately compensate the physics and skip through animation frames to give the illusion of a gameworld updating at a constant interval even if the framerate is low.

For example, this is what things look like in the callback for me. It is over 1 second off (and yes I'm running over 1fps lol)

Code (CSharp):

1.  
2. Time.time: 6.31, context.time: 5.367893399990862, realtimeSinceStartup: 8.65886
3.  

There is a thread that sort of deals with it, but I don't think anything came of it and I can't see the code from the linked commits in the stable or develop branch, was it removed at some point?

https://forum.unity.com/threads/new-input-system-and-framerate-independance.543666/

Timeslicing would help and I am also having an issue with that, but that's probably a problem for a different thread.

I am using 0.9.4