Welcome! New input system resources and info (please read)

Hey Runevision,

Over the years using Unity i have created multiple custom input managers with corresponding inspectors and am thoroughly interested in creating them. Apart from helping test the new input manager, is their anyway that we can proactively contribute to the project in anyway? If so who would be the best person to talk to?

 

Also interested in helping. Though the issue with closed source products like Unity is that this would probably involve getting a job with them... I truly hope Unity follows Unreal by going open source, and only has the bare minimum proprietary (just enough to enforce the paid subscription exclusives... or simply go to a royalty model)

 

Hey @fabianzz

Thanks for your reply, I am not apposed to the model that Unity has implemented, in fact i support it. It allows them to have a lot more control over the direction of the company. I know from experience and colleagues of mine that if Unity finds something cool in the community, they will offer to work with them This just makes us strive and work towards making cool third party content that will help the community grow and in doing so brings us closer together.

 

The managed (C#) part of the new input system will be open source just like the prototype code was. That means all the input device classes, the ActionMap system and more.

We don't have an established plan for how we're going to take contributions, but it'll likely be similar to how it's manged for the UI System.

 

I'm fairly new to unity. I managed to get a action map created and use that to produce player movement using the quick start guide. However there is nothing on the documentation (https://sites.google.com/a/unity3d.com/unity-input-advisory-board/home) about how to do rebinds on runtime and the runtimeRebind script in the folder doesn't make any sense to me. How would I be able to rebind on runtime by attaching a function to a button on a canvas where if they click it they will rebind that key to the next key they press (similar to the demo scene)?

I attached a script for how I am currently doing something like this in a game I'm producing.

Would it be something similar to this maybe?

 

It's based on OnGUI script based GUI as opposed to the UI system. In the single player example scene, you can see the rebinding settings by hitting 'R'.

But apart from that, we're not actively supporting the prototype anymore since we want to focus our resources on getting the proper system (not prototype) ready for an experimental release.

 

This looks fantastic, nice work @runevision

 

We're still working hard towards an experimental release.

Here's a video showing a feature we recently got implemented: Edit Mode Input support.

It may seem at first like not a very important feature, but for editor tooling and smooth authoring workflows, it actually makes a big difference and opens up some nice possibilities.

 

@runevision keep up the great work Unity team Looking good

 

@runevision
Is this new System will be frame-independat?

For example i do an App like UI and set fps to 5 (to save battery drain and device getting warm) and input still works?
Thanks

 

We haven't encountered this use case in the past two years we've collected feedback and input for the new system, so we're not explicitly testing for this.

What problems do you run into when trying this with the current input system? I imagine for certain input methods, clicks could go unnoticed, due to the mouse down and mouse up happening in the same frame?

If you program you app to listen to events directly (in the new input system) rather than using our input-querying APIs then it should definitely work, but of course you loose some of the more high level functionality that way.

 

If we set FPS = 5 in current situation, then Inputs dont get always trigger (due to low FPS)
So we have to keep it to 24 to always register users input even though our UI deoesnt change at all as we are building applications rather than games.

If new Input System will be frame-independant, then it will automatically solve our problem. So my question should be is New System going to be frame-independant or not?

 

All scripting in Unity is tied to the frame loop, so in that sense you can't get anything truly frame-independent out of the engine. The question is in what cases this cause issues and in what cases it doesn't.

If you use the event-based API of the input system, that will work more or less as if it was frame-independent. Alternatively you can place your input logic in FixedUpdate (a new possibility in the new input system) rather than Update and have FixedUpdate run at, say, 24 fps while Update runs at your low 5 fps. Then only minimal code will run more frequently while e.g. rendering won't.

 

@Alternatively you can place your input logic in FixedUpdate (a new possibility in the new input system) rather than Update and have FixedUpdate run at, say, 24 fps while Update runs at your low 5 fps. Then only minimal code will run more frequently while e.g. rendering won't.


Sounds like a good plan. I'll wait for Input System to become public in summer.
Keep up the great work

 

@runevision First of all, thank you very much for your hard work into making this possible, the new input system will be of most value.

In my case, I'm mostly interested in the gamepad control standardization and hot-plugging support, so I have a small input manager wrapper on my game (currently in development phase) which will be implemented by the new input system - rather than my custom code + current APIs.

I would like to use your summer experimental release as soon as it is available. Where will you make the release announcement? Only here in the forum, or in other mediums (such as twitter)?

 

Certainly here in this forum; probably a blog post and tweet too.

 

Any idea when this will be released ?

 

The latest update we posted on that - that we don't have any official timelines to share, but I expect that by around summer time we'll have a public experimental preview - is still on track.

 

So... Since friday, beta 7 is available. And because I haven't seen any announcement regarding the new input system, I'll ask:
In the release notes, under features, it says:

• Input: Added new Player Settings property 'Input Manager' for enabling backend of new input system preview feature. The front-end is supplied separately and is not included with the release.

I didn't had time to download the new beta, so, is this THE backend for the input system? And is it compatible with the one available here?

 

I tried looking for the new system in the usings and intellisense but I couldn't find anything

 

ILSpy is your friend! ;-)

UnityEngineInternal.Input

 

I just found out it was also in Unity 5.6. I wonder how long it's been there, and if it's really what we think it is

 

It wasn't in 5.5, so it is probably indeed what we think!

 

And what about this?
https://drive.google.com/file/d/0B7cJ0DQLnM-CTkZqMjdQSGxZMFk/view
Edit/ Project Settings/ Player

 

Yes, it is THE backend for the new input system. It is NOT compatible with the input system prototype released a year ago however.

We will release the new front-end in due time. The initial versions of this will be preview releases. We're still working on it, making some backwards-compatibility-breaking changes here and there, and writing documentation. Please have patience with us.

The good news is that once it's released, we're not tied to Unity releases and it'll be technically possible to release new versions of the front-end independently (and more frequently) than Unity releases. Only updates that are tied to changes in the backend will be tied to Unity releases.

 

A bit more details:

The current plan is that the new Input System will be included as a preview release in Unity 2017.2. Additionally we're aiming to release the preview by other means a bit earlier - likely through the Asset Store in the Unity 2017.1 release cycle. The preview releases will support a limited set of platforms and devices while we roll out broader support.

 

@runevision

Could you mention the limited set of platforms/devices which will be supported for the preview release?

 

This is great! I just wanted to be sure, as you said that was going to be an announcement.
Now that I think about it, makes more sence to announce the front-end, right? Oh well...
Don't worry, I'll be very patient

 

One of the biggest requests when you were asking for feedback on the input system THREE years ago was to make it have a frame rate independent input event for situations like this on mobile. (see https://forum.unity3d.com/threads/official-input-system-improvements.248098/page-3#post-1674052 )

I am disappointed that this was ignored. This is primarily for situations that the user described. You have a mobile game, you want to limit the frame rate while the user is just looking at a static screen (help screen, book, etc. or whatnot). When the user interacts, the frame rate will jump up until a point where the interactions animations conclude and the application can go into a lower rendering frequency again.

Granted, I do understand that all scripting is currently tied to the Update frequency, but we need a way to get an input update frequency that can trigger and start increasing the application's frame rate so that we can smoothly animate again immediately after input. If the mentioned event structure will trigger outside of the Update cycle, but on the main thread so that we can set this value that is fine, but this was a major request that users had for the input system. Running a FixedUpdate is not ideal either since the physics will also run on FixedUpdate causing computations for a system I'm not interested in. Also FixedUpdate is not actually running on a separate thread, it is just executed with a fixed deltaTime meaning you're aren't actually polling input at a different rate than Update.

So if you have the example with a default fixedUpdate time of 0.02 and an Application.targetFramerate of 5 (0.2 frame time). You will see when accessing the Time.realtimeSinceStartup.

Fixed Update: 9.1389
Update: 9.1402
Fixed Update: 9.3277
Fixed Update: 9.3292

You can see that fixed update is delayed 0.2 seconds as it is still tied to the frame rate. I don't want input events missed while the application's frame rate is delayed for battery reasons. I want a way to immediately trigger the application's render time back to a higher framerate on the main thread, outside of the update loop.

 

@Ferazel It's not been ignored It's actually been quite persistently on our minds including the very case you describe where you want to run at low-frame rate yet sample something like touch at much higher frequency and not lose input.

To clarify, we've split the input *gathering* part completely from the input *processing* part in the new system. The processing parts we've built indeed do operate on time slices and given their state-based nature that makes it inherently susceptible to collapsing multiple state changes that happened within a single time slice.

BUT... the data that is coming in is *not* tied to any time slices -- neither is the gathering of that data. And the information coming in as events is directly available to you. And where we need to poll for that information instead of getting events from the OS, we're doing so on background threads that can sample at user-defined speeds.

So, say we're talking touch on Android. You will see all touch data as reported to us by the Android APIs surface 1:1 at the event level in Unity.

So if we take an application drawing curves on screen as an example and it wants high-precision sampling of touch but wants low-frequency screen updates, then yes, input at the state level will not have the desired granularity. BUT input at the event level has. You can go through all touch events and simply ignore the aggregated state in the Touchscreen.

Whether there's better ways to deal with this in the state architecture as well is a good question I think. However, at least to me the most important part was that no matter what we do at the state level, the source data is there. Regardless of framerate.

Does that sound more towards something that would give you the tools you need?

 

For 2017.1 it will be only Windows and OSX in terms of platforms. Device support is pretty wide including pen and touch input on Windows. Additionally, the various VR platforms are supported.

For 2017.2 the plan is to add Android (we have a partial implementation; didn't get ready in time for 2017.1) and Linux.

Overall, we're at a point where we want to be going wide on platforms and get as many online as quickly as possible. But it's a ball that only recently started rolling so there's still uncertainty around whether when exactly what we'll have beyond 2017.2.

 

Then again, standardized support of various gamepads and joysticks likely won't be nearly as wide as in some of the popular Asset Store input packages in the first few releases. But some things are not yet set in stone, so it's hard to be more specific.

 

If I'm understanding correctly, if the raw event list is available, I'm not sure why the input system can't process the events into the correct processed events by putting it into an input queue. While reading the raw input events works for your drawing application example you gave, it doesn't work very well for interacting with UnityUI (if the EventSystem doesn't do it's own raw input processing). Which means the inputs could be lost without rewriting our own input processor and sending the events into the UI manually. Which is a good chunk of work IMO.

 

Everything that works event-based (like the UI) we can totally convert 1:1 and not collapse any inputs. The only issue is with systems that essentially aggregate sequences of events into state that represents time slices (like the old input system does).

But TBH I'm not sure how exactly it'll work with the Unity UI system. ATM that code doesn't exist yet. Given it's all event-based, I don't see a reason why it wouldn't just see every single input event coming in and thus not lose anything -- regardless of what the framerate does. But I may be missing something that complicates that picture.

Requiring people to go all the way back to the input event queue definitely isn't the ideal scenario and we definitely do not want to force people into that. The good thing IMO is the data is there, though (unlike the old system where at the C# level, all that was already lost). From there we can figure out how we can route the data through the system such that it best satisfies what people need out of the system.

 

A new input module would be all that would need to be created (that spits out all the events based on the lower level data). Obviously this doesn't involve rendering since you've set your framerate lower. But to make a crazy example, if you set your framerate to 1 fps and then clicked 3 checkboxes in that amount of time, with a new input module all 3 would be checked when your next frame came around.

 

Thank you so much!

 

Thanks @Rene-Damm for the reply, appreciating it!

 

While being able to write an input module from the low-level input is better than nothing, I feel this is a poor solution. If I'm understanding correctly, the events handlers are still going to be affected by the target framerate because they will not trigger until they can sync on the main script thread context (which I'm guessing it isn't until the framerate is released). The reason I think this is that Unity already stated that the scripting API is only meant to be able to affect change at certain segments of the engine (mainly the Update). So our new input handlers would still be delayed until the next update frame thus creating a delay on input. Which makes it so that the user doesn't see the checkbox's pressed state, they would just immediately animate on. This isn't an ideal user experience. Also needing to listen to a lot of low level events just so you can repackage them and feed them again through the UI isn't ideal either.

Another option (which is what I'm trying to fight for) is to have an API that can modify the Application.targetFramerate independently of the scripting hooks. Something like InputManager.applicationFramerateOnInput (or something like that). That's all that we'd need to solve this problem for my use case adequately. It wouldn't expose the entire Unity scripting API to a multi-threaded hook (like what a C# event handler would need to do) and it wouldn't require me to write a redundant input manager that is still delayed by the framerate. The goal would be to immediately release the hold on the update framerate when input is detected. Then I can tell the Application.targetFramerate to go down again if there is no user input myself using the necessary input hooks in an existing script.

Framerate is the easiest way we can reduce battery drain on Unity applications. I feel it is important to many types of applications that don't always have something animating on the screen, yet I don't want to create a delay or miss inputs on the user experience. I shouldn't need to choose between them in a proper multi-threaded input system.

 

I'm totally with you there, I was just trying to reiterate what it sounded like @Rene-Damm and @runevision were saying. I also agree with you that simply allowing the "input gathering" portion to force a frame even if it isn't time yet from the previously set target framerate if input is detected would be ideal (for only one frame so Unity can then go back to the previously low framerate). This would save a ton on battery without anything else needing to change other than probably a settings checkbox for allowing this behavior or disabling it. It didn't sound like this was built though but perhaps they can chime in.

I think anyone on mobile games with large portions of static UI's (trading card games, solitaire, non games, etc.) would really want this. Obviously, this was brought up a long time ago, I'm hoping it's in but hopefully they can clarify.

 

If the input gathering is event based, wouldn't it be possible to listen for any inputs and change your update-rate on-demand?

 

I hope so, maybe @Rene-Damm and @runevision can clarify. It's clear there is the input gathering piece which will be running faster than the set framerate which is good (or totally event driven but that is going to depend on the operating system and controller types, I'm sure there are some places where they'll have to poll). It's unclear if they've built in the machinery for that input gathering piece to be able to coax the normal giant Unity loop to do or change things before it wants to. In the case above we'd want even if Unity wasn't due to have another frame for half a second to force a frame early because some input was detected. That way you could essentially save a lot of battery when nothing is going on with your game or app but still "wake up" right when input is detected rather than needing to wait for the preset amount of time which would be unresponsive.

 

Input is gathered at a faster rate than the frame rate, either with polling or based on events from the OS.

However, the queued events are still only sent through the system at specific points in the player loop: Just before fixed update and just before update, since that's as close as possible to the point when user scripts can actually make use of the events.

I think what @Rene-Damm talked about is that you can use the events to reconstruct what happened throughout the last frame, so that the precision of your input dependent logic doesn't have to be tied to the frame rate.

However, if I understand correctly, what you want is basically that the game can be set to have a normal high target frame rate when there’s input but a very low frame rate (to save battery) when there’s no input. The critical thing about this that you can’t do today is having the frame loop “break out” of waiting for the next long frame as soon as there’s input available. Is that correctly understood?

 

That's correct, hopefully @Ferazel and can confirm on his end as well. In games or apps that don't always have things moving (Trading card games, strategy games, non game applications, games that get paused, etc.) the biggest way to save battery is by drastically lowering the framerate when it's not needed. In the old system this had side effects of course, the biggest either missed input or the user noticing delayed input because of the low framerate.

Ideally, with the new input system this side effect would not need to occur anymore. When nothing is going on in my game I can set the framerate to 1 fps to save battery, but know that Unity will "break out" of it's waiting early if the user actually touches the touchscreen, or other input occurs (so the framerate can be set higher again). This would probably be paired with an option in player settings on whether to enable this behavior or not.

 

Yeah as @greggtwep16 mentioned, his use cases are exactly the problem. As a mobile developer, I want to be able to be able to set a low target framerate in static image situations for battery reasons and I don't want the low framerate to have an affect on user input feedback responses. Currently, Unity will only notify me of input events on the next delayed frame meaning the input and user input is delayed by the artificial framerate and there is nothing we can do to reset that until the next frame.

I want to be able to immediately resume higher framerates as soon as a registered input type receives input so that I can process that input without delay. This should only happen for registered input types so that if my application doesn't use the gyroscope or accelerometer, those inputs are not going to kick me out of my desired framerate. There is currently no way to reset the Application.targetFramerate outside of the scripting loop. If the Input system was able to do that, this creates a much more seamless experience for the end-user and the developer. As a user, my input is immediately displayed and reacted to. As a developer, I don't need to create a listener to raw events as try to reconstitute raw inputs into events and manually feed it through the necessary systems.

I don't feel that many other systems (if any) would require this framerate altering functionality. Exactly how you want to achieve this I'll leave it up to you. If you want to make it so that the Input system has a flag that drops the artificial limiter for one frame, that's fine I can set the Application.targetFramerate back in a script call. If you want us to give you a framerate that the application will return to on Input, that works too. We just don't have anyway to do this currently until the next artificially delayed frame which causes input delay problems.

I hope that these use cases makes sense and you can achieve it relatively easily with the new input system. If you need more clarification please let me know.

 

All right, I understand the use case now. The problems we're facing with this is that

1) We were not aware of this use case until it was brought up in this thread.
Input independent of frame rate was brought up years ago, but this can mean many things as the discussion here has shown. Being able to poll input in FixedUpdate now is one way it's now independent of frame rate; being able to get all the queued events and reconstruct what happened is another way. The particular use case of having a long frame which is part of a low frame rate be prematurely interrupted upon input is quite different and specific, and not something we understood from the discussions years ago.

2) It won't be trivial to implement.
Having a check in the player loop for whether there's pending input is by itself a significant alteration to how the Unity player loop works. Having to somehow inform the low-level back-end of which inputs it should treat as triggering this condition and which should be ignored is further something that requires non-trivial changes to the current design.

3) It's not clear to us how widely requested this feature is.
While we've covered a lot of things in the input system already, it's no secret the initial releases won't cover everything that people have requested, and we'll be busy for a while implementing more features. Inevitably, there will be some things we never get to too, since there will always be more requests. It's not clear to us yet where this feature fits in the roadmap, priority-wise.

I can talk with our product management and try to get help with determining how wide the demand for this feature is compared to other input features so we can make a decision about this.

First though, I want to hear if an alternative approach could work, from your perspective. If you could set your frame rate to be normal high all the time, but each frame where nothing happened you could tell Unity to entirely skip rendering that frame, would that work as well? This depends on how much battery is consumed by rendering versus other parts of the player loop, but I have a feeling most other things would require very little computation if nothing changes. This approach has the advantage that you can exactly determine which input should trigger or not trigger rendering based on your own logic instead of the back-end having to have to keep track of which inputs specifically should short-circuit a frame or not in a way that can't be specifically tailored to your game.

 

I would prefer than you release a base implementation and add some optionals features in the future than wait for 2 years before release.

 

Thanks for the reply @runevision I had a feeling this was the case. As far as how widely this feature is I know there are tons of threads on this topic going all the way back to 2010, but that's obviously not a great meter so perhaps something can be put up that people can vote on. I went back an looked at the official input thread from years ago and you are correct that what was requested was a vague "framerate independent input" or a vague "save battery". There were actually a few other threads from around that time that were a bit more specific.

As far as alternative approaches goes, there are probably a couple of ways to skin this cat. My impression informally is that the only reason that framerate has the most effect on battery is that it makes the amount of time the cpu is idle a high percentage. Lets say a frame takes 16ms to complete (60fps) but you set the target framerate to 30 then the cpu should be idle around 50% (ignoring apps in the background and other things). If you set it to 15 it should be idle 75% of the time and so on. Basically, Unity would do it's business and then spend a certain amount of time waiting. Informal guessing on your alternative:

Your suggestion of flagging "nothing happened" could work as well. Most likely, you'd want to stop more than just rendering to save as much battery as the other approach (physics, etc.). This would change the paradigm from frame with a long wait to more frames with a shorter wait. I have a feeling as long as the % idle is the same it would be fine. My gut tells me that there is probably a decent amount of "setup" unity does every frame though so having a faster framerate won't be quite as battery friendly. I don't know if per frame Unity has some average statistics on which categories take the most cpu time, but I have a feeling lowering the framerate stops them all and the whole engine waits, whereas doing the opposite will only save the "categories" that will listen to you and skip.

 

As I mentioned earlier, this is not a solution. FixedUpdate is NOT framerate independent. It just gets called the number of times it is behind and increments Time.time by the fixedUpdateStep. So if you have an Application.targetFramerate of 5 each player loop will still be up to 0.2 seconds delayed. Only that fixed update is called multiple times to "step to" the current simulation time. So even though I will receive the input as part of the fixedUpdate (I'm assuming) the delay that the user see is still present. If the user pressed down 3 times on a controller between updates. The UI will still jump down 3 without any intermediate animation possible.

I understand that altercations to the player loop is risky, but I find this difficult to imagine the implementation being difficult to implement. The player loop should not be reading into input, input should be telling the player loop via a thread synchronized bool. Or depending on how the player loop works, reducing the time to wait to 0. Now to test this solution on all platforms I can see being a concern. However, this is what I expect when an input rewrite is on the docket.

I will make sure that we get in touch with our business contact as well so that I can push this feature request. This is very disappointing to hear that when you hear "framerate independent input" you assumed that delayed framerate input would still be ok.

I'm guessing you mean by disabling all of the render cameras? As you mentioned, this has other trade-offs that make life harder on the developer. There are a lot of systems in Unity that would need to be disabled in order to reach the CPU usage that simply turning down the frame rate would. Animations, Physics, and Scripting would all need to be keyed to turn off all of their components during this low-power mode. While this could improve cpu-usage and render time, this is a more difficult solution than I feel it needs to be.

 

It's a feature I didnt know I need until it was brought up in this thread, anything that save abttery is a win and a huge marketing point!

 

I feel like there is a lot of confusion about this input gathering stuff, I had to explain the logic behind it to an artist (and even some designer don't get it before you explain it), let me post back what I said so that both party understand what's at stakes:

That's one reason I think unity should make a deep action game to REAL LIFE understanding of input and many other decision they are making (animation blending and co), else we got a dialogue between deaf people.

Decoupling the gathering of input from the framerate isn't as literal as it might sound, it mean we should make decision about what's happening between two signal send to the player in regard of reactivity. Saying that is still losing a lot of nuances.