Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only. On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live. Read our full announcement for more information and let us know if you have any questions.

Run tests before play mode and prevent enter play mode if tests are failing?

Discussion in 'Testing & Automation' started by Xarbrough, Oct 7, 2020.

  1. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    Is it possible with the testing API to run a set of tests before entering play mode? I'd like to do so with only a few tests and if they fail, prevent entering play mode to avoid users having to wait through unity freezing and several loading screens just to realize that something basic broke.

    I've actually implemented this as a custom tool without using the test API. When exiting edit mode, it searches the scene for missing mandatory references, and if any issues are found, the enter play mode is cancelled and an error is logged. This has helped my team in a lot of cases and we'd like to continue expanding on this tool.

    My idea now was to incorporate this is a test that could be run manually via the test runner, but also runs automatically before entering the scene (best as a user preference).

    If this is not possible at the moment, that's my feature suggestion. ;)
     
    JoNax97 likes this.
  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,670
    Well, you can run tests via the TestRunnerApi, and you can make them happen synchronously by setting the execution settings appropriately - I demoed this technique in the talk at Unite Copenhagen.

    What I'm less sure about is how to integrate it with the process of entering play mode. You could try subscribing to EditorApplication.playModeStateChanged, but I'm not sure if this callback would fire before entering play mode or after it.
     
  3. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    Thanks for the pointers! I think that’s enough for me to get this working. I’ll try it out tonight and post my results here. :)
     
  4. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    It's working out pretty well:

    Code (CSharp):
    1. using NUnit.Framework;
    2. using UnityEditor;
    3. using UnityEditor.TestTools.TestRunner.Api;
    4. using UnityEngine;
    5.  
    6. namespace Tests
    7. {
    8.     public class SceneValidationTest
    9.     {
    10.         [InitializeOnLoadMethod]
    11.         private static void Initialize()
    12.         {
    13.             EditorApplication.playModeStateChanged += OnPlayModeStateChanged;
    14.         }
    15.  
    16.         private static void OnPlayModeStateChanged(PlayModeStateChange change)
    17.         {
    18.             if (change == PlayModeStateChange.ExitingEditMode)
    19.             {
    20.                 var testRunner = ScriptableObject.CreateInstance<TestRunnerApi>();
    21.                 var filter = new Filter()
    22.                 {
    23.                     testMode = TestMode.EditMode,
    24.                     categoryNames = new[] { "SceneValidation" }
    25.                 };
    26.                 var resultHandler = new ResultHandler();
    27.                 testRunner.RegisterCallbacks(resultHandler);
    28.                 testRunner.Execute(new ExecutionSettings(filter)
    29.                 {
    30.                     runSynchronously = true
    31.                 });
    32.                 if (resultHandler.result.TestStatus == TestStatus.Failed)
    33.                 {
    34.                     Debug.LogError("A validation test has failed. Aborting enter play mode.");
    35.                     EditorApplication.isPlaying = false;
    36.                 }
    37.                 else
    38.                 {
    39.                     Debug.Log("All tests were passed.");
    40.                 }
    41.  
    42.                 Object.DestroyImmediate(testRunner);
    43.             }
    44.         }
    45.  
    46.         private class ResultHandler : ICallbacks
    47.         {
    48.             public ITestResultAdaptor result { get; private set; }
    49.  
    50.             public void RunFinished(ITestResultAdaptor result)
    51.             {
    52.                 this.result = result;
    53.             }
    54.  
    55.             // No interface segregation or separate events?
    56.             public void RunStarted(ITestAdaptor testsToRun) { }
    57.             public void TestFinished(ITestResultAdaptor result) { }
    58.             public void TestStarted(ITestAdaptor test) { }
    59.         }
    60.  
    61.         [Test]
    62.         [Category("SceneValidation")]
    63.         public void SceneValidation_AlwaysFails()
    64.         {
    65.             Assert.Fail("This test is only a demo and always fails.");
    66.             //Assert.Pass();
    67.         }
    68.     }
    69. }
    70.  
    This does exactly what I want in only a few clean lines of code, so I'm really happy with this solution.

    I'm still wondering why the callback listener is an interface and I have to create this container object with the stub implementations for all callbacks, instead of subscribing a specific listener to an event. Not complaining but I'm curious what the design reason is.
     
    Last edited: Oct 9, 2020
  5. FICHEKK

    FICHEKK

    Joined:
    Feb 6, 2016
    Posts:
    6
    Hey, I know this is an old thread, but I tried your solution and unfortunately I am having some problems. I am on Unity 2021.2.11 and when I make code changes and recompile, clicking on "play" button the first time, I don't get notified about exiting edit-mode. Do you have this issue and how did you fix it?
     
  6. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    I don't see any issue, I've tested in Untiy 2021.2.5. Sorry, I don't have 2021.2.11 installed.

    Can you elaborate what you mean "notified about exiting edit-mode"?

    The code I posted defines an edit-mode test which always fails. The code in OnPlayModeStateChanged then checks if you are exiting play mode and runs all tests. The example test will then fail. If the test results indicate a failure, I set EditorApplication.isPlaying = false to immediately abort enter playmode. It then logs an error to the console and you should still be in edit mode.
     
    FICHEKK likes this.
  7. FICHEKK

    FICHEKK

    Joined:
    Feb 6, 2016
    Posts:
    6
    Well, my problem is pretty weird. This is what happens:
    1. I make some code changes and go to Unity.
    2. Unity automatically recompiles my new code.
    3. When I click on "play" button, it does "Reload Script Assemblies", which in turn erases the state and with that my subscription made with EditorApplication.playModeStateChanged += OnPlayModeStateChanged.
    4. Game runs and I don't get notified that I am exiting edit-mode, so therefore no tests are run.
    5. However, when I run it the second time (without making any code changes), it works properly. In order words, this happens: PLAY (doesn't run my test), STOP, PLAY (now it works).

    And it is always like that. It never works from the first try when I make some code changes, I always have to run it once first, stop the game and run again. It's really annoying and I don't want to use it like that. So my question is, does it work the first time for you? Also, do you have automatic recompilation when code changes are made? And do you have domain-reload disabled maybe?
     
  8. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    Yea, it works as expected for me. I tried with automatic refresh and manual (CTRL + R keypress), both work fine. I have domain-reload enabled, but it should also work without reloading the domain.

    Have you tried my example script exactly as it is without modifications?

    You write that
    EditorApplication.playModeStateChanged += OnPlayModeStateChanged
    is erased, but in my script I have that inside of
    [InitializeOnLoadMethod]
    so it will always be called when the domain is reloaded. If you have domain reloading disabled, it should still work, because then you don't lose any static references.

    Despite potential copy/paste errors, there could be a new bug in your Unity version which prevents InitializeOnLoad from being called at the right time. It could be that the new domain reload features introduced issues with the order of things, etc. But you can simply test by going back to Unity 2021.2.5f1, I confirmed it working there at least.
     
    FICHEKK likes this.
  9. FICHEKK

    FICHEKK

    Joined:
    Feb 6, 2016
    Posts:
    6
    Hey, I have found the issue and it was my mistake. The issue was that I was calling a method on a logger (I am using custom logger) that has not been initialized at that point in time and was getting a null-reference exception. I really have no idea how I missed that exception - probably because I was logging a lot of other stuff with it and it slipped.

    The thing that probably made me miss the bug is that it suddenly worked every second time, which made me think it must be an issue with Unity execution order or something. Interestingly enough, running the game for the second time preserves the initialized logger state (it is a bunch of static variables, one Action<string> for each logging type: info, warning and error) which makes the tests run on the second "play" click.

    Anyway, it works perfectly now and I want to thank you so much for your help.
     
  10. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    Great to hear!

    Btw, I still use the approach of running tests before play mode at my company, but I've scaled it down to configurable options:
    • Run all tests (takes some time and can be annoying)
    • Only run scene validation (missing references in open scenes)
    • Run validation at runtime after scene load (to detect missing references)
    • Disable everything to avoid annoying testers
     
    FICHEKK likes this.
  11. FICHEKK

    FICHEKK

    Joined:
    Feb 6, 2016
    Posts:
    6
    I will definitely add some configurable options so user can enable/disable certain tests (especially if test is expensive and takes a long time to execute). Thank you for the suggestion!