Recording and replaying input

Hi! I've been playing with the new input system for some time, and I found this system very useful for prototyping.

Right now I'm working on a way to record all the input (or at least input from some devices), save it to some asset and then later run the game with all the recorded input. This is not a gameplay feature, but rather a prototyping tool. Saving it somewhere is crucial, since one of the requirements is to have an ability to replay the input at any time

Since I want to fake all the input, I want to have clean state of InputSystem without any physical devices (to later add virtual ones). For that I use InputTestFixture, since it enables me to do that and I'm fine with running the game as a test (although it doesn't seem like a proper way)

In my understanding these are my steps:

1. [Recording] Listen to input events
2. [Recording] Serialize and save them to some asset
3. [Replaying] Deserialize them and create InputEvent and\or InputEventPtr structs
4. [Replaying] Queue these events into the input system

Step 1 seems pretty easy: there are couple of ways to listen to events: either subscribe to InputSystem.onEvent, use InputEventTrace or use InputStateHistory. Step 4 is also straightforward given a valid InputEventPtr (using InputSystem.QueueEvent).

Now (de)serialization (steps 2 & 3) is what I can't seem to solve. I believe plainly serializing InputEvent struct into byte[] is not the brightest idea (given that it contains NativeInputEvent and I don't completely understand the contents of this struct). I believe you have some workflow for this case. As I see it now I need following data to recreate input events from scratch: InputControl, state and timestamp. What's my best strategy to retrieve this info for given InputEventPtr? Or maybe I should use some other flow?

 

Okay, after studying source code, documentation and forum posts I finally managed to make a working (kinda) setup. Key information is stored in the state of the event. So I used GetStatePtrFromStateEvent to obtain pointer to the state buffer and then ReadValueFromStateAsObject to read raw state bytes (or one can use ReadValueFromStateIntoBuffer to avoid creating garbage). These bytes (along with device id and event time) are what I save.
To replay this input I use InputSystem.QueueStateEvent (QueueDeltaStateEvent also works) with a bit of a pointer juggling to convert state bytes to IInputStateTypeInfo (btw, QueueStateEvent internally seems to need only state format and a pointer to state bytes, so it would be nice to have a method overload which takes state format and a pointer instead of a struct. This will make it possible to pass format-agnostic data without pointer casts)
All this sounds straightforwards, but I spent couple of evenings trying to figure out how to feed my data into the virtual device (it turns out I forgot about the InputSystem.Update, since I supposed that testing environment would take care of updates).

Anyway, this can't be the right way to accomplish my goal, right? Moreover, it seems that InputTestFixture shouldn't be used in this case, at least since it uses default InputSettings asset, and I want to recreate game events exactly as they were during the recording (I believe Update type would affect that). Actually, the only reason I use InputTestFixture is because it can reset real devices and then restore them (since I create virtual devices with matching device ids). Maybe I can do it another way? Something like a public static method in InputSystem

One more thing (a bit off-topic): from the first glance it seems that parameters time and timeOffset in InputTestFixture.Set<TValue> are used to change time of event (e.g. time offset of 5 would mean that event would be processed 5 seconds later). But as it turns out that's not the case, since all non-outdated events will be processed in the next Update method (at least it appears to me this way)

 

Thanks for the solution, it seems to suit well for me
Although I have a problem with 2 exceptions, which are thrown during application exit after I replayed all the input (both related to InputUser):

The first one happens after I'm removing virtual devices in OnApplicationQuit:

Code (text):

1. IndexOutOfRangeException: Index was outside the bounds of the array.
2. UnityEngine.InputSystem.Utilities.ArrayHelpers.SwapSlice[TValue] (TValue[] array, System.Int32 sourceIndex, System.Int32 destinationIndex, System.Int32 count) (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Utilities/ArrayHelpers.cs:679)
3. UnityEngine.InputSystem.Utilities.ArrayHelpers.MoveSlice[TValue] (TValue[] array, System.Int32 sourceIndex, System.Int32 destinationIndex, System.Int32 count) (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Utilities/ArrayHelpers.cs:717)
4. UnityEngine.InputSystem.Users.InputUser.AddDeviceToUser (System.Int32 userIndex, UnityEngine.InputSystem.InputDevice device, System.Boolean asLostDevice, System.Boolean dontUpdateControlScheme) (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Plugins/Users/InputUser.cs:1121)
5. UnityEngine.InputSystem.Users.InputUser.OnDeviceChange (UnityEngine.InputSystem.InputDevice device, UnityEngine.InputSystem.InputDeviceChange change) (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Plugins/Users/InputUser.cs:1512)
6. UnityEngine.InputSystem.InputManager.RemoveDevice (UnityEngine.InputSystem.InputDevice device, System.Boolean keepOnListOfAvailableDevices) (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/InputManager.cs:1194)
7. UnityEngine.InputSystem.InputSystem.RemoveDevice (UnityEngine.InputSystem.InputDevice device) (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/InputSystem.cs:908)
8. UnityPrototype.InputInjecter.RevertEnvironment (UnityPrototype.InputInjecter+SavedEnvironment env) (at Assets/InputRecorder/InputInjecter.cs:156)
9. UnityEngine.Debug:LogException(Exception)
10. UnityPrototype.InputInjecter:RevertEnvironment(SavedEnvironment) (at Assets/InputRecorder/InputInjecter.cs:160)
11. UnityPrototype.InputInjecter:OnApplicationQuit() (at Assets/InputRecorder/InputInjecter.cs:57)

The second one happens in PlayerInput.OnDisable:

Code (text):

1. IndexOutOfRangeException: Index was outside the bounds of the array.
2. (wrapper stelemref) System.Object.virt_stelemref_class_small_idepth(intptr,object)
3. UnityEngine.InputSystem.Utilities.ArrayHelpers.EraseSliceWithCapacity[TValue] (TValue[]& array, System.Int32& length, System.Int32 index, System.Int32 count) (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Utilities/ArrayHelpers.cs:742)
4. UnityEngine.InputSystem.Users.InputUser.RemoveUser (System.Int32 userIndex) (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Plugins/Users/InputUser.cs:1011)
5. UnityEngine.InputSystem.Users.InputUser.UnpairDevicesAndRemoveUser () (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Plugins/Users/InputUser.cs:659)
6. UnityEngine.InputSystem.PlayerInput.UnassignUserAndDevices () (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Plugins/PlayerInput/PlayerInput.cs:974)
7. UnityEngine.InputSystem.PlayerInput.OnDisable () (at Library/PackageCache/com.unity.inputsystem@0.9.5-preview/InputSystem/Plugins/PlayerInput/PlayerInput.cs:1137)

All preparations (creating virtual layouts, updating supported devices and then creating virtual devices) are done in OnEnable, whereas reversion (removing devices, reverting supported devices settings and then removing layouts) happen in OnApplicationQuit. I now test only keyboard input. I'll try to create minimal working example if my description is obscure
Notable thing is that these exceptions are being thrown only if one or more events are queued (it seems that in this case InputUser pairs the virtual keyboard, which causes device unpairing to happen during game termination)

 

Hiya!
I think I'm trying to do the exact same thing. I haven't touched the new input system but maybe it's time to give it a try. I'd like to record my input in the editor to be able to create tutorial animations in the editor that I can play back in the game. I realise there will be some slight difference with physics etc but I think it doesn't have to be super accurate.
Just wondering how did this work out for you??
Any tips? I think I'll switch over to the new system for step one.

 

We're thinking of doing a similar thing, mostly for testing. That is, record actual user interaction and save it as a test, then the ability to play back the interaction and see if we get (approximately) the same result.

But why would we need to use the new input system for that?

 

I'm actually very surprised to see it's not natively possible with the editor. I mainly need this kind of tests since I'm working on a puzzle game and don't want to have to replay all levels over and over to make sure they're not broken by some tiny adjustment I made on some collider's size.

Croteam did exactly that for The Talos Principle actually. They recorded a bot that would do all levels in the game and they would be able to replay it to test the whole game. They could even run the game faster (x4) with the bot, saving even more time for feedback on the game. They said during a conference that it would take only 20 minutes to test all puzzles could be completed in the game! Link to GDC conference (start at 26:30):

That shouldn't be too difficult to implement with Unity's timeScale. Maybe I'll give it a shot with mentioned method of input recording. I don't want to spend hours playing my levels repeatedly.

 

Hey, I am working on a similar thing. Can you explain how you write the data into QueueStateEvent? I understand that it takes a struct a copies the data into its internal state. But I can't, for the life of me, figure out how the byte array that you get from GetStatePtrFromStateEvent must be formatted so that it's read correctly. :/

 

Thanks for the reply! In the meantime I found a solution that is similar to the first one, which kind of works. I am doing the following.

Code (CSharp):

1. // record the data
2. using (StateEvent.From(device, out InputEventPtr eventPointer))
3. {  
4.     void* statePtr = device.GetStatePtrFromStateEvent(eventPointer);
5.     object state = device.ReadValueFromStateAsObject(statePtr);
6.     InputAction action = new InputAction // custom class to serialize the input data
7.     {
8.          deviceId = device.deviceId,
9.          eventTime = Time.time,
10.          data = (byte[])state,
11.          device = device.GetType().FullName
12.     };
13. }

To replay I figure out the correct device and the corresponding state struct, then deserialize the byte array directly into the struct like this: (using the virtual keyboard and mouse code you posted above)

Code (CSharp):

1. byte[] data = // deserialize data and figure out the device; let's pretend we found the mouse here
2. MouseState state = ReadStruct<MouseState>(data);
3. InputSystem.QueueStateEvent(inputDevice, state);
4.  
5. private T ReadStruct<T>(byte[] buffer) where T : struct
6. {
7.     using (MemoryReader mw = new MemoryReader(buffer))
8.     {
9.         T t = mw.ReadStruct<T>();
10.         return t;
11.     }
12. }
13.  
14. internal class MemoryReader : BinaryReader
15. {
16.     public MemoryReader(byte[] buffer) : base(new MemoryStream(buffer)) {}
17.  
18.     public T ReadStruct<T>()
19.     {
20.         int byteLength = Marshal.SizeOf(typeof(T));
21.         byte[] bytes = ReadBytes(byteLength);
22.         GCHandle pinned = GCHandle.Alloc(bytes, GCHandleType.Pinned);
23.         T instance = (T)Marshal.PtrToStructure(pinned.AddrOfPinnedObject(), typeof(T));
24.         pinned.Free();
25.         return instance;
26.     }
27. }
28.  

This seems to work perfectly fine with the keyboard. But mouse input is always "too weak". Like it is doing the thing, just not as much as in the original. I.e. when I do a full rotation with my character, the replay only does 3/4 or so. Maybe it's because of the edge of the screen? Do you have any idea why this might happen?

 

Hey,

So I'm trying to do the same except I don't want to make this myself if possible. I need to record and playback input for a scene that's rather low FPS, to record a video of this using the recorder from the preview packages. This video recorder makes the scene super slow but it's perfect because the resulting video plays in real time without dropped frames and with sound. The problem is when recording the video things get so slow that, if you're using the kb/mouse input directly, the apparent input plays back unnaturally fast in the resulting video, as slow as you try to go with it, specially the mouse. And this is why I want to record the input first, then play it back when recording the video so that everything feels smooth.

I found this: https://letsbuild.gg/rcarlson/record-save-playback-user-input-in-unity-2phd
But I found several problems with it:

1. I spent the last 24h trying to solve a tedious "namespace or type cannot be found" kind of error, because the guys said in the instructions to just drop the LetsBuild folder in the assets directory. I couldn't figure out why VSCode nor Unity can not "see" the LetsBuild namespace/s from the Standard Assets First Person Character Controller. At last and after trying a million things I realized it needs to be within the same folder hierarchy branch in order to find the namespace. So I moved the LetsBuild folder inside the Standard Assets folder and I finally got it to work. I also tried changing the execution order of scripts and put the LetsBuild stuff before the FPS controller, but I didn't get it to work that way.

2. Not really a problem but the fact that its recording directory defaults to "C:\" and that all the recorded files are prepended by "\" (on Linux, anyways), makes it a little non-multi-platform.

3. It assumes all input happens on either Update or FixedUpdate, but the FPS controller from Standard Assets uses both. It uses mainly FixedUpdate but also Update for jumps; not sure to what extent this isn't the best thing to do, tho.

4. Plain and simple it just doesn't work well for me. I tried recording and playing back in both FixedUpdate and Update modes and the results are always very inaccurate. Even with the objects tracking ("Objects to Sync") enabled and setting in there the FPS controller and its child camera it just doesn't work right. The playback is like at one time the recorded input data is taking inaccurate and offset control of the input, and at the next time it's setting the objects transforms like they should, but then immediately after, they snap back to the badly offset input and on, and on... Really bad.

I think part of the problem with the inaccuracy is that they're serializing and deserializing, comparing strings and creating objects at every darned frame. I'm not very sure how expensive this is, but I'd think it isn't the greatest idea to do this at every frame. I would do this: Record everything as it comes like a stream, then after you're done recording, see if you want to remove duplicated stuff, serialize it for storing or whatnot. Same with playback: Deserialize if needed and recreate the missing repeated stuff if necessary (I'm talking basic compression algorithm here, such as: "11111111" -> "#81" and viceversa), store in memory and ready to feed the actual inputs, then do it at each frame. I think adding too much to do at every frame could introduce some significant delay thus making it impossible to keep up with the proper timing.

I'm using 2019.2 and I'm not even sure I could amend this myself. Is there any other solution?

 

Thank you Samuel for your reply. It saved a lot of time because I just tested and it does the job.

Thanks to all of you for that thread as well.