Search Unity

Question Input when the game does not have focus (for integration tests)

Discussion in 'Input System' started by Baste, Oct 20, 2020.

  1. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    Hi!

    I'm running some automated tests, and I'm using the new Input system's test setup (InputTestFixture) to test things.

    It's pretty great - being able to say "this is what needs to happen when the jump button is pressed" rather than "this is what needs to happen when the jump method is called" allows the game code to have the proper access modifiers, while still working with tests.

    But I have a rather large problem. It seems like that if the game view (or Unity in general) doesn't have focus, the inputs doesn't get piped through to the game. This makes sense in general - if I tab away from the game and write stuff, I don't want the player to run around.

    But for tests - especially when there's a lot of them - I want to be able to start them and then tab away. But now all my tests fail, because the input's not going through!


    Is there a way to work around this? I was trying to look for settings that had to do with focus, but I couldn't find them. Ideally, this is something we could turn on in tests only.
     
  2. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    One instance where this is really annoying is when I'm attaching my debugger and trying to step through the code during tests. Since the inputs are not active, I can't really do that, which makes figuring out stuff a lot harder.
     
  3. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    Could you post the code for a test that is affected by this? Would like to have a closer look. I assume it's a [UnityTest]?

    Overall,
    InputTestFixture
    is set to ignore focus. In the editor, it will always toggle "Lock Input to Game View" on and
    InputTestRuntime
    has its own focus status (which is true by default and only changes when set explicitly in code) that is not tied to
    Application.isFocused
    (though looking at it, we don't currently expose control over that through InputTestFixture; something to add).

    But... for [UnityTest] tests, InputTextFixture hooks itself into the player loops... which is indeed affected by GameView focus. So my guess is this is where the problem is originating from. If that is indeed the case, it's probably something that should be solved in the Unity test runner. IMO runs in the Test Runner window within the editor shouldn't be affected by GameView focus. If this is indeed the source of trouble, I'll go have a chat about it with the test framework people.
     
  4. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    When I'm testing, it seems like it's been attempted to be fixed. If I start the tests and then click into a different window before the first test runs, all the tests work. It's only if I alt-tab or click after the tests has started that things stop working properly.


    The code looks like this. I've trimmed it down to only be a single unit-test, and removed a ton of extraneous details:

    Code (csharp):
    1.  
    2. namespace Tests {
    3. public class TestLedgeGrab {
    4.     private bool sceneReady;
    5.     private bool referencesSetup;
    6.  
    7.     private Gamepad gamePad;
    8.     private InputTestFixture inputTester = new InputTestFixture();
    9.  
    10.     [OneTimeSetUp]
    11.     public void OneTimeSetup() {
    12.         EditorSceneManager.LoadSceneInPlayMode("Assets/Scenes/Unit Test Scenes/TestLedgeGrab.unity", new LoadSceneParameters(LoadSceneMode.Single));
    13.         SceneManager.sceneLoaded += OnSceneLoaded;
    14.  
    15.         gamePad = InputSystem.AddDevice<Gamepad>();
    16.  
    17.         Time.captureFramerate = 60;
    18.     }
    19.  
    20.     [OneTimeTearDown]
    21.     public void OneTimeTearDown() {
    22.         InputSystem.RemoveDevice(gamePad);
    23.     }
    24.  
    25.     [TearDown]
    26.     public void TearDown() {
    27.         inputTester.Set(gamePad.leftStick, Vector2.zero);
    28.         inputTester.Release(gamePad.buttonSouth);
    29.     }
    30.  
    31.     void OnSceneLoaded(Scene scene, LoadSceneMode mode) {
    32.         sceneReady = true;
    33.     }
    34.  
    35.     [UnityTest]
    36.     public IEnumerator TestLedgeHandling() {
    37.         yield return new WaitWhile(() => sceneReady == false);
    38.  
    39.         var playerGO = Instantiate(...); // details skipped for brevity
    40.  
    41.         inputTester.PressAndRelease(gamePad.buttonSouth); //makes the player jump
    42.  
    43.         yield return new WaitForSeconds(.5f);
    44.  
    45.         Assert.IsTrue(PlayerHasGrabbedLedge()); // details skipped for brevity
    46.     }
    47.  
    When testing, it's clear that PressAndRelease gets called, but the code that listens doesn't get any calls. Here's the listener code, from a MonoBehaviour attached to the instantiated player object:

    Code (csharp):
    1.     //The type Teslagrad2Inputs is generated from an input actions asset with "Generate C# class" enabled.
    2.     private Teslagrad2Inputs teslagrad2Inputs;
    3.  
    4.     private void OnEnable() {
    5.         if (teslagrad2Inputs == null) {
    6.             teslagrad2Inputs = new Teslagrad2Inputs();
    7.             teslagrad2Inputs.Player.SetCallbacks(this);
    8.         }
    9.  
    10.         teslagrad2Inputs.Player.Enable();
    11.     }
    12.  
    13. ...
    14.  
    15.     public void OnJump(InputAction.CallbackContext context) {
    16.         // handle jump! This does not get called if the game loses focus during tests
    17.     }
    18.  
     
  5. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    Oh, and:
    Unity 2020.1.2f1
    Input System 1.0.0
    Test Framework 1.1.18
     
  6. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    Thanks for the details. Much appreciated.

    Just to make sure it's not that, could you give 1.1-preview.1 a try? There was a fix for `[UnityTest]` tests running over more than a single frame in there and I'd like to make sure it's not that the same issue.
     
  7. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    Bug's still there when I activate 1.1.0-preview.1 in the package manager.

    I can look into creating a repro project if you think this'll be considered a bug.
     
  8. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    Thanks for giving it a go.

    This just caught my eye

    Looking only at the test setup here, this will not isolate the input system. Instantiating
    InputTestFixture
    is not enough. It needs to have its
    Setup()
    and
    TearDown()
    methods called. My guess is you're actually running with just input system as is and thus get the full impact of focus switching and such.

    Note that calling
    InputTestFixture.TearDown()
    also makes it unnecessary to do any input-related cleanup. All devices and custom registrations will be thrown away automatically.

    With those changes, my guess would be the problem you're seeing will go away. (note you'll still need 1.1-preview.1 for the run-UnityTests-over-multiple-frames fix)

    For 1.1, we've clarified this better (hopefully) in the docs.
     
    willwolfram18 likes this.
  9. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    Thanks, I'll give it a try tomorrow!

    Am I supposed to call those methods in [SetUp]/[TearDown] or [OneTimeSetup]/[OneTimeTeardown]?
     
  10. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    Depends a bit on what you're going for. Personally, would recommend setting up a test environment for each test and tearing it down after each one so that each test gets clean, known state and there's no bleeding from one test into another. In that case it'd be [SetUp] and [TearDown]. But the choice is up to you. The fixture should do its job just fine sticking around for an entire test run.
     
  11. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    Welp, that broke everything!

    I added this to my test setup:

    Code (csharp):
    1.  
    2.     [SetUp]
    3.     public void SetUp() {
    4.         inputTester.Setup();
    5.     }
    6.  
    7.     [TearDown]
    8.     public void TearDown() {
    9.         inputTester.TearDown();
    10.     }
    11.  
    When that's there, interacting with the inputTester during tests causes nullRefs deep in the input system. This line of code:

    Code (csharp):
    1. inputTester.PressAndRelease(gamePad.buttonSouth);
    Gives this error:

    Code (csharp):
    1.  
    2. TestLedgeHandling_Blink1 (0,316s)
    3. ---
    4. System.NullReferenceException : Object reference not set to an instance of an object
    5. ---
    6. at UnityEngine.InputSystem.LowLevel.InputStateBuffers+DoubleBuffers.GetFrontBuffer (System.Int32 deviceIndex) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\State\InputStateBuffers.cs:73
    7.   at UnityEngine.InputSystem.LowLevel.InputStateBuffers.GetFrontBufferForDevice (System.Int32 deviceIndex) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\State\InputStateBuffers.cs:126
    8.   at UnityEngine.InputSystem.InputControl.get_currentStatePtr () [0x00000] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\Controls\InputControl.cs:788
    9.   at UnityEngine.InputSystem.LowLevel.DeltaStateEvent.From (UnityEngine.InputSystem.InputControl control, UnityEngine.InputSystem.LowLevel.InputEventPtr& eventPtr, Unity.Collections.Allocator allocator) [0x00074] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\InputSystem\Events\DeltaStateEvent.cs:88
    10.   at UnityEngine.InputSystem.InputTestFixture.Set[TValue] (UnityEngine.InputSystem.InputControl`1[TValue] control, TValue state, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x000b4] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:457
    11.   at UnityEngine.InputSystem.InputTestFixture.Press (UnityEngine.InputSystem.Controls.ButtonControl button, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:347
    12.   at UnityEngine.InputSystem.InputTestFixture.PressAndRelease (UnityEngine.InputSystem.Controls.ButtonControl button, System.Double time, System.Double timeOffset, System.Boolean queueEventOnly) [0x00001] in E:\Teslagrad2\Library\PackageCache\com.unity.inputsystem@1.1.0-preview.1\Tests\TestFixture\InputTestFixture.cs:359
    13.   at Tests.TestLedgeGrab+<TestLedgeHandlingForSpawnPoint>d__24.MoveNext () [0x00186] in E:\Teslagrad2\Assets\Scripts\Editor\Unit Tests\Runtime Tests\TestLedgeGrab.cs:89
    14.   at UnityEngine.TestTools.TestEnumerator+<Execute>d__6.MoveNext () [0x0004c] in E:\Teslagrad2\Library\PackageCache\com.unity.test-framework@1.1.18\UnityEngine.TestRunner\NUnitExtensions\Attributes\TestEnumerator.cs:36
    15.  

    I also managed to break Unity completely when doing things. I did it by messing up stuff when experimenting, but it's a very easy repro; run this code:

    Code (csharp):
    1.  
    2.     [MenuItem("Test/Break Unity Kinda")]
    3.     public static void EnoughToBreakUnity() {
    4.         new InputTestFixture().TearDown();
    5.     }
    6.  
    And then recompile anything. Unity will start spamming two errors outside of play mode until the editor is restarted completely:

    Error 1:
    Code (csharp):
    1.  
    2. TypeInitializationException during event processing of Editor update; resetting event buffer
    3. UnityEngine.InputSystem.LowLevel.<>c__DisplayClass7_0:<set_onUpdate>b__0(NativeInputUpdateType, NativeInputEventBuffer*)
    4. UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate(NativeInputUpdateType, IntPtr)
    5.  
    Error 2:
    Code (csharp):
    1.  
    2. NullReferenceException: Object reference not set to an instance of an object
    3. UnityEngine.InputSystem.InputSystem.InitializeInEditor (UnityEngine.InputSystem.LowLevel.IInputRuntime runtime) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputSystem.cs:2999)
    4. UnityEngine.InputSystem.InputSystem..cctor () (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputSystem.cs:2916)
    5. Rethrow as TypeInitializationException: The type initializer for 'UnityEngine.InputSystem.InputSystem' threw an exception.
    6. UnityEngine.InputSystem.InputAnalytics.OnStartup (UnityEngine.InputSystem.InputManager manager) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputAnalytics.cs:25)
    7. UnityEngine.InputSystem.InputManager.OnUpdate (UnityEngine.InputSystem.LowLevel.InputUpdateType updateType, UnityEngine.InputSystem.LowLevel.InputEventBuffer& eventBuffer) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/InputManager.cs:2537)
    8. UnityEngine.InputSystem.LowLevel.NativeInputRuntime+<>c__DisplayClass7_0.<set_onUpdate>b__0 (UnityEngineInternal.Input.NativeInputUpdateType updateType, UnityEngineInternal.Input.NativeInputEventBuffer* eventBufferPtr) (at Library/PackageCache/com.unity.inputsystem@1.1.0-preview.1/InputSystem/NativeInputRuntime.cs:60)
    9. UnityEngine.InputSystem.LowLevel.<>c__DisplayClass7_0:<set_onUpdate>b__0(NativeInputUpdateType, NativeInputEventBuffer*)
    10. UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate(NativeInputUpdateType, IntPtr)
    11.  
    So right now it looks to me like any call to InputTestFixture.SetUp or TearDown will break the tests, and also maybe Unity.
     
  12. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    The one time setup is interferring with the per-test setup. Would recommend going one way or the other. What the mix does is run the one-time setup which adds the Gamepad to the *non-isolated* system and then the per-test setup saves and wipes the system state to put input in isolation mode. Thus the exception when you access the Gamepad that belongs to the non-isolated system.

    Sorry, but that code doesn't make sense :) InputTestFixture is touching global system state in its setup and teardown work, so yes, invoking them arbitrarily will wreak havoc on the global system state. That's expected. (////EDIT: I do believe that in 1.1-preview.2 there's now some check that will, as a side-effect, make TearDown not do any work if Setup wasn't called)
     
  13. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    Thanks for all the help so far! I've changed the setup code:

    Code (csharp):
    1.  
    2.     private bool sceneReady;
    3.  
    4.     private Gamepad gamePad;
    5.     private InputTestFixture inputTester;
    6.  
    7.     [OneTimeSetUp]
    8.     public void OneTimeSetup() {
    9.         EditorSceneManager.LoadSceneInPlayMode("Assets/Scenes/Unit Test Scenes/TestLedgeGrab.unity", new LoadSceneParameters(LoadSceneMode.Single));
    10.         SceneManager.sceneLoaded += OnSceneLoaded;
    11.     }
    12.  
    13.     [SetUp]
    14.     public void Setup() {
    15.         inputTester = new InputTestFixture();
    16.         inputTester.Setup();
    17.         gamePad = InputSystem.AddDevice<Gamepad>();
    18.     }
    19.  
    20.     [TearDown]
    21.     public void TearDown() {
    22.         inputTester.TearDown();
    23.     }
    24.  
    25.     void OnSceneLoaded(Scene scene, LoadSceneMode mode) {
    26.         sceneReady = true;
    27.     }
    28.  
    This made me run into a new problem:
    upload_2020-10-21_16-33-46.png

    If I run a bunch of tests, the first one succeeds, and the other ones fail. They fail in a pretty strange way, though: the scene doesn't have any objects anymore. For all tests after the first one, SceneManager.GetActiveScene().GetRootGameObjects() is an empty list. My tests rely on finding spawn points through GameObject.Find and instantiating the player prefab there, so all the tests after the first one is failing on that.

    Commenting out inputTester.TearDown(); causes the issue to disappear (and the tests to work), so it seems like inputTester.TearDown somehow breaks the loaded scene or something?
     

    Attached Files:

  14. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    That's good to hear! I was aware that the code was nonsensical, I wrote it originally by commenting out one line of code and forgetting to comment out another one. The error was expected, but having to restart the editor by calling methods in an unexpected order should probably not happen :p
     
  15. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    Ah, I kinda feared that would become an issue :) And yup, it's InputTestFixture messing around. In particular, this code. Pretty sure the fixture shouldn't do that. Makes sense for our own tests but not necessarily for other test suites. Think it's the kind of thing that just crept in there and managed to stick around.

    I'll go see if I can manage to sneak a PR into 1.1-preview.2 last second to get this addressed in some form (even if for now it's just a way to toggle the behavior off).

    As a workaround, temporarily adding an empty scene while calling TearDown() should probably prevent the code from doing damage for now.
     
  16. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    Hah, yes that would break our tests :) Thanks for the help again!

    I think I'll make a subclass of InputTestFixture that overrides TearDown and just... doesn't do that. That shouldn't lead to any problems, right?
     
  17. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    You'd still probably want the rest of what the method does. If TearDown() isn't called but Setup() is, the input system is left hanging in an isolated state and the "real" system state won't get swapped back in.
     
  18. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    Hacky hacky: I noticed that the TestFixture only deletes objects with HideFlags set to 0, so I simply set all the hide flags to something else, run base.TearDown(), and restore the hide flags.

    ... With a big, fat, comment about how this will all be fixed in the future!

    Code (csharp):
    1.  
    2.     public override void TearDown() {
    3.         var rootGameObjects = SceneManager.GetActiveScene().GetRootGameObjects();
    4.  
    5.         var hideFlags = rootGameObjects.Select(go => go.hideFlags).ToArray();
    6.         rootGameObjects.Each(go => go.hideFlags = HideFlags.DontSave);
    7.  
    8.         base.TearDown();
    9.  
    10.         rootGameObjects.EachIndex(i => rootGameObjects[i].hideFlags = hideFlags[i]);
    11.     }
    12.  
     
  19. willwolfram18

    willwolfram18

    Joined:
    Feb 20, 2021
    Posts:
    1
    Wanted to drop by and say thank you for this little clue! I've been trying to figure out why some of my tests were failing when running from CLI but could never get them to fail when running from the editor. I feel silly that I didn't notice
    InputTestFixture
    has it's own
    [SetUp]
    that needs to be called. Appreciate your feedback on this thread!