Question Input when the game does not have focus (for integration tests)

Hi!

I'm running some automated tests, and I'm using the new Input system's test setup (InputTestFixture) to test things.

It's pretty great - being able to say "this is what needs to happen when the jump button is pressed" rather than "this is what needs to happen when the jump method is called" allows the game code to have the proper access modifiers, while still working with tests.

But I have a rather large problem. It seems like that if the game view (or Unity in general) doesn't have focus, the inputs doesn't get piped through to the game. This makes sense in general - if I tab away from the game and write stuff, I don't want the player to run around.

But for tests - especially when there's a lot of them - I want to be able to start them and then tab away. But now all my tests fail, because the input's not going through!


Is there a way to work around this? I was trying to look for settings that had to do with focus, but I couldn't find them. Ideally, this is something we could turn on in tests only.

 

One instance where this is really annoying is when I'm attaching my debugger and trying to step through the code during tests. Since the inputs are not active, I can't really do that, which makes figuring out stuff a lot harder.

 

When I'm testing, it seems like it's been attempted to be fixed. If I start the tests and then click into a different window before the first test runs, all the tests work. It's only if I alt-tab or click after the tests has started that things stop working properly.


The code looks like this. I've trimmed it down to only be a single unit-test, and removed a ton of extraneous details:

Code (csharp):

1.  
2. namespace Tests {
3. public class TestLedgeGrab {
4.     private bool sceneReady;
5.     private bool referencesSetup;
6.  
7.     private Gamepad gamePad;
8.     private InputTestFixture inputTester = new InputTestFixture();
9.  
10.     [OneTimeSetUp]
11.     public void OneTimeSetup() {
12.         EditorSceneManager.LoadSceneInPlayMode("Assets/Scenes/Unit Test Scenes/TestLedgeGrab.unity", new LoadSceneParameters(LoadSceneMode.Single));
13.         SceneManager.sceneLoaded += OnSceneLoaded;
14.  
15.         gamePad = InputSystem.AddDevice<Gamepad>();
16.  
17.         Time.captureFramerate = 60;
18.     }
19.  
20.     [OneTimeTearDown]
21.     public void OneTimeTearDown() {
22.         InputSystem.RemoveDevice(gamePad);
23.     }
24.  
25.     [TearDown]
26.     public void TearDown() {
27.         inputTester.Set(gamePad.leftStick, Vector2.zero);
28.         inputTester.Release(gamePad.buttonSouth);
29.     }
30.  
31.     void OnSceneLoaded(Scene scene, LoadSceneMode mode) {
32.         sceneReady = true;
33.     }
34.  
35.     [UnityTest]
36.     public IEnumerator TestLedgeHandling() {
37.         yield return new WaitWhile(() => sceneReady == false);
38.  
39.         var playerGO = Instantiate(...); // details skipped for brevity
40.  
41.         inputTester.PressAndRelease(gamePad.buttonSouth); //makes the player jump
42.  
43.         yield return new WaitForSeconds(.5f);
44.  
45.         Assert.IsTrue(PlayerHasGrabbedLedge()); // details skipped for brevity
46.     }
47.  

When testing, it's clear that PressAndRelease gets called, but the code that listens doesn't get any calls. Here's the listener code, from a MonoBehaviour attached to the instantiated player object:

Code (csharp):

1.     //The type Teslagrad2Inputs is generated from an input actions asset with "Generate C# class" enabled.
2.     private Teslagrad2Inputs teslagrad2Inputs;
3.  
4.     private void OnEnable() {
5.         if (teslagrad2Inputs == null) {
6.             teslagrad2Inputs = new Teslagrad2Inputs();
7.             teslagrad2Inputs.Player.SetCallbacks(this);
8.         }
9.  
10.         teslagrad2Inputs.Player.Enable();
11.     }
12.  
13. ...
14.  
15.     public void OnJump(InputAction.CallbackContext context) {
16.         // handle jump! This does not get called if the game loses focus during tests
17.     }
18.  

 

Oh, and:
Unity 2020.1.2f1
Input System 1.0.0
Test Framework 1.1.18

 

Bug's still there when I activate 1.1.0-preview.1 in the package manager.

I can look into creating a repro project if you think this'll be considered a bug.

 

Thanks, I'll give it a try tomorrow!

Am I supposed to call those methods in [SetUp]/[TearDown] or [OneTimeSetup]/[OneTimeTeardown]?

 

Welp, that broke everything!

I added this to my test setup:

Code (csharp):

1.  
2.     [SetUp]
3.     public void SetUp() {
4.         inputTester.Setup();
5.     }
6.  
7.     [TearDown]
8.     public void TearDown() {
9.         inputTester.TearDown();
10.     }
11.  

When that's there, interacting with the inputTester during tests causes nullRefs deep in the input system. This line of code:

Code (csharp):

1. inputTester.PressAndRelease(gamePad.buttonSouth);

Gives this error:

Code (csharp):

1.  
2. TestLedgeHandling_Blink1 (0,316s)
3. ---
4. System.NullReferenceException : Object reference not set to an instance of an object
5. ---
6. at UnityEngine.InputSystem.LowLevel.InputStateBuffers+DoubleBuffers.GetFrontBuffer (System.Int32 deviceIndex) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\State\InputStateBuffers.cs:73
7.   at UnityEngine.InputSystem.LowLevel.InputStateBuffers.GetFrontBufferForDevice (System.Int32 deviceIndex) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\State\InputStateBuffers.cs:126
8.   at UnityEngine.InputSystem.InputControl.get_currentStatePtr () [0x00000] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\Controls\InputControl.cs:788
9.   at UnityEngine.InputSystem.LowLevel.DeltaStateEvent.From (UnityEngine.InputSystem.InputControl control, UnityEngine.InputSystem.LowLevel.InputEventPtr& eventPtr, Unity.Collections.Allocator allocator) [0x00074] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\Events\DeltaStateEvent.cs:88
10.   at UnityEngine.InputSystem.InputTestFixture.Set[TValue] (UnityEngine.InputSystem.InputControl`1[TValue] control, TValue state, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x000b4] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:457
11.   at UnityEngine.InputSystem.InputTestFixture.Press (UnityEngine.InputSystem.Controls.ButtonControl button, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:347
12.   at UnityEngine.InputSystem.InputTestFixture.PressAndRelease (UnityEngine.InputSystem.Controls.ButtonControl button, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:359
13.   at Tests.TestLedgeGrab+<TestLedgeHandlingForSpawnPoint>d__24.MoveNext () [0x00186] in E:\Teslagrad2\Assets\Scripts\Editor\Unit Tests\Runtime Tests\TestLedgeGrab.cs:89
14.   at UnityEngine.TestTools.TestEnumerator+<Execute>d__6.MoveNext () [0x0004c] in E:\Teslagrad2\Library\PackageCache\com.unity.test-framework@1.1.18\UnityEngine.TestRunner\NUnitExtensions\Attributes\TestEnumerator.cs:36
15.  

I also managed to break Unity completely when doing things. I did it by messing up stuff when experimenting, but it's a very easy repro; run this code:

Code (csharp):

1.  
2.     [MenuItem("Test/Break Unity Kinda")]
3.     public static void EnoughToBreakUnity() {
4.         new InputTestFixture().TearDown();
5.     }
6.  

And then recompile anything. Unity will start spamming two errors outside of play mode until the editor is restarted completely:

Error 1:

Code (csharp):

1.  
2. TypeInitializationException during event processing of Editor update; resetting event buffer
3. UnityEngine.InputSystem.LowLevel.<>c__DisplayClass7_0:<set_onUpdate>b__0(NativeInputUpdateType, NativeInputEventBuffer*)
4. UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate(NativeInputUpdateType, IntPtr)
5.  

Error 2:

Code (csharp):

1.  
2. NullReferenceException: Object reference not set to an instance of an object
3. UnityEngine.InputSystem.InputSystem.InitializeInEditor (UnityEngine.InputSystem.LowLevel.IInputRuntime runtime) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputSystem.cs:2999)
4. UnityEngine.InputSystem.InputSystem..cctor () (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputSystem.cs:2916)
5. Rethrow as TypeInitializationException: The type initializer for 'UnityEngine.InputSystem.InputSystem' threw an exception.
6. UnityEngine.InputSystem.InputAnalytics.OnStartup (UnityEngine.InputSystem.InputManager manager) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputAnalytics.cs:25)
7. UnityEngine.InputSystem.InputManager.OnUpdate (UnityEngine.InputSystem.LowLevel.InputUpdateType updateType, UnityEngine.InputSystem.LowLevel.InputEventBuffer& eventBuffer) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputManager.cs:2537)
8. UnityEngine.InputSystem.LowLevel.NativeInputRuntime+<>c__DisplayClass7_0.<set_onUpdate>b__0 (UnityEngineInternal.Input.NativeInputUpdateType updateType, UnityEngineInternal.Input.NativeInputEventBuffer* eventBufferPtr) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/NativeInputRuntime.cs:60)
9. UnityEngine.InputSystem.LowLevel.<>c__DisplayClass7_0:<set_onUpdate>b__0(NativeInputUpdateType, NativeInputEventBuffer*)
10. UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate(NativeInputUpdateType, IntPtr)
11.  

So right now it looks to me like any call to InputTestFixture.SetUp or TearDown will break the tests, and also maybe Unity.

 

Thanks for all the help so far! I've changed the setup code:

Code (csharp):

1.  
2.     private bool sceneReady;
3.  
4.     private Gamepad gamePad;
5.     private InputTestFixture inputTester;
6.  
7.     [OneTimeSetUp]
8.     public void OneTimeSetup() {
9.         EditorSceneManager.LoadSceneInPlayMode("Assets/Scenes/Unit Test Scenes/TestLedgeGrab.unity", new LoadSceneParameters(LoadSceneMode.Single));
10.         SceneManager.sceneLoaded += OnSceneLoaded;
11.     }
12.  
13.     [SetUp]
14.     public void Setup() {
15.         inputTester = new InputTestFixture();
16.         inputTester.Setup();
17.         gamePad = InputSystem.AddDevice<Gamepad>();
18.     }
19.  
20.     [TearDown]
21.     public void TearDown() {
22.         inputTester.TearDown();
23.     }
24.  
25.     void OnSceneLoaded(Scene scene, LoadSceneMode mode) {
26.         sceneReady = true;
27.     }
28.  

This made me run into a new problem:


If I run a bunch of tests, the first one succeeds, and the other ones fail. They fail in a pretty strange way, though: the scene doesn't have any objects anymore. For all tests after the first one, SceneManager.GetActiveScene().GetRootGameObjects() is an empty list. My tests rely on finding spawn points through GameObject.Find and instantiating the player prefab there, so all the tests after the first one is failing on that.

Commenting out inputTester.TearDown(); causes the issue to disappear (and the tests to work), so it seems like inputTester.TearDown somehow breaks the loaded scene or something?

 

That's good to hear! I was aware that the code was nonsensical, I wrote it originally by commenting out one line of code and forgetting to comment out another one. The error was expected, but having to restart the editor by calling methods in an unexpected order should probably not happen

 

Hah, yes that would break our tests Thanks for the help again!

I think I'll make a subclass of InputTestFixture that overrides TearDown and just... doesn't do that. That shouldn't lead to any problems, right?

 

Hacky hacky: I noticed that the TestFixture only deletes objects with HideFlags set to 0, so I simply set all the hide flags to something else, run base.TearDown(), and restore the hide flags.

... With a big, fat, comment about how this will all be fixed in the future!

Code (csharp):

1.  
2.     public override void TearDown() {
3.         var rootGameObjects = SceneManager.GetActiveScene().GetRootGameObjects();
4.  
5.         var hideFlags = rootGameObjects.Select(go => go.hideFlags).ToArray();
6.         rootGameObjects.Each(go => go.hideFlags = HideFlags.DontSave);
7.  
8.         base.TearDown();
9.  
10.         rootGameObjects.EachIndex(i => rootGameObjects[i].hideFlags = hideFlags[i]);
11.     }
12.  

 

Wanted to drop by and say thank you for this little clue! I've been trying to figure out why some of my tests were failing when running from CLI but could never get them to fail when running from the editor. I feel silly that I didn't notice

InputTestFixture

has it's own

[SetUp]

that needs to be called. Appreciate your feedback on this thread!