Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Running tests consecutively

Discussion in 'Testing & Automation' started by jessfdm_sky, Nov 22, 2019.

  1. jessfdm_sky

    jessfdm_sky

    Joined:
    Nov 19, 2018
    Posts:
    12
    Hi,

    I've got a load of tests that I need to run consecutively and not concurrently. When I select them individually and run them they run fine, and if I select a few of them (say a fixture or a namespare with a couple of fixtures below it) then they run fine as well. But if I select more they seem to run consecutively but some of my plugins don't work (DOTween if it helps to know), and if I ask all of them to run then they all start to run at the same time and clash with each other causing the results to flicker back and forth across random tests.

    Is it possible to run all the tests but force it to run all the tests concurrently and such that each test is completely isolated (as if I entered or exited playmode)? And is there a way to do this via the command line?

    Cheers!
     
    Whatever560 likes this.
  2. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    What do you mean by consecutively? all tests are being executed one after the other, never concurrently.
     
  3. jessfdm_sky

    jessfdm_sky

    Joined:
    Nov 19, 2018
    Posts:
    12
    Mine don't seem to, they seem to run all at the same time and their results change flicker back and forth between each pass/fail.

    (These are playmode tests btw).

    Running them individually means that they pass fine though.

    I'm loading and unloading a scene for some of the tests and for those I'm using a superclass which has a [UnitySetUp] and [UnityTearDown] to load in my scene and then load back the test scene respectively, which all the other fixtures derive from. That's the only thing I think that could be causing the issue.

    Are we also allowed to "yield break" at the end of tests if the tests should be completable within a single frame?
     
  4. StewedHarry

    StewedHarry

    Joined:
    Jan 20, 2020
    Posts:
    45
    I am having the same issue when running tests consecutively. Did you find the problem?
     
  5. MajeureX

    MajeureX

    Joined:
    May 4, 2020
    Posts:
    13
    I'm guessing you're using the [UnityTest] attribute for your test methods? If so, your tests are run as coroutines, which might explain the behaviour. You can still use the [Test] attribute even in PlayMode tests, so if your test doesn't need to use yield then use the [Test] attribute instead.
     
    Last edited: Sep 12, 2020
  6. StewedHarry

    StewedHarry

    Joined:
    Jan 20, 2020
    Posts:
    45
    My tests do require yields unfortunately
     
  7. andrew-fray

    andrew-fray

    Joined:
    Jul 19, 2012
    Posts:
    158
    Gonna bump this thread. My problem is that I have a legacy codebase, which likes to leave around a ton of DontDestroyOnLoad gameobjects, and probably some static state too. Running my playmode tests in isolation work fine, but once I try to chain two, the lingering objects and state starts causing issues. If I could make my tests each end with a proper play mode exit, things would be sufficiently cleaned up. Is there a straightforward way to do this? I realise I'm sacrificing the ability to run these in a build.

    The alternative I could try is restructuring them as edit mode tests, and have each architect a call to play mode enter/exit. Does that sound feasible?
     
    Whatever560 likes this.
  8. andrew-fray

    andrew-fray

    Joined:
    Jul 19, 2012
    Posts:
    158
    (in the end I bit the bullet and made my levels play nicely when flowing from one to the other. That's obviously the Proper Fix and worth the investment)
     
  9. StewedHarry

    StewedHarry

    Joined:
    Jan 20, 2020
    Posts:
    45
    You might need to properly setup and teardown each test:

    Code (CSharp):
    1.  
    2.         [SetUp]
    3.         public void Init()
    4.         {
    5.             // loads a new scene for each test
    6.             SceneManager.LoadScene("TestScene", LoadSceneMode.Single);
    7.        
    8.         }
    9.  
    10.         [TearDown]
    11.         public void TearDown()
    12.         {
    13.             // disposes training instance from the Academy singleton
    14.             Academy.Instance.Dispose();
    15.         }
    I also ran into trouble when setting up tests due to the immediate calling of the delegate passed to the event:
    Academy.Instance.OnEnvironmentReset
     
  10. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    BTW this fails and it should not if you run both tests at the same time. If you run them one by one they pass.

    This is a real issue, none of NUnit and MSTests (nor any test framework ever) are working this way...

    This should be documented and have an easy work around ...

    Code (CSharp):
    1.     [TestFixture]
    2.     public class PlayModeCleanContextTest
    3.     {
    4.         [UnityTest]
    5.         public IEnumerator TestA()
    6.         {
    7.             yield return new EnterPlayMode();
    8.             var a = new GameObject("A");
    9.             Object.DontDestroyOnLoad(a);
    10.             yield return new ExitPlayMode();
    11.         }
    12.         [UnityTest]
    13.         public IEnumerator TestB()
    14.         {
    15.             yield return new EnterPlayMode();
    16.             var a = Object.FindObjectOfType<GameObject>();
    17.             if(a) Assert.AreNotEqual("A", a.name);
    18.         }
    19.     }
     
  11. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    This is not really feasible with heavy integration tests though. Also InputSystem Setup/TearDown is flawed and errors stack up after each couple of tests.
     
  12. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    You can do
    yield return new Enter/ExitPlayMode(), but it's not cleaning anything as you can see in the previous example
     
  13. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    I tried adding this in the SetUp of tests. It just breaks the InputSystem even more with an error flood

    Code (CSharp):
    1.             for (var i = 0; i < SceneManager.sceneCount; ++i)
    2.             {
    3.                 foreach (var rootGameObject in SceneManager.GetSceneAt(i).GetRootGameObjects())
    4.                 {
    5.                     UnityEngine.Object.Destroy(rootGameObject);
    6.                 }
    7.             }
    8.  
     
  14. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    53
    There's really no easy way to clear static state (you should just avoid it in general).

    However, for destroying all GameObjects, you might want to take a look at how Zenject test utilities do it.
     
    Whatever560 likes this.
  15. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    Yeah, I already cleaned up any static code. My issues actually come from unity own modules (InputSystem) that get all wacky once an error occured in a test. It comes from the fact that they use static based code with a push/pop mecanism.

    I'll try Zenject code though. Thanks
     
  16. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    I pulled my hair out on this one, do not forgot to NOT destroy the "ResourceManagerCallbacks" gameObject. If using the addressable package or async operation will not finish and hang the test silently. I'll add a PR to Zenject
     
    sbergen likes this.
  17. MilanAssuied

    MilanAssuied

    Unity Technologies

    Joined:
    Sep 14, 2021
    Posts:
    4
    Hi there,

    I am bumping this thread because I am encountering the exact same issue, @Whatever560 did you finally manage to solve it ?
     
    Meatloaf4 likes this.
  18. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    I'm using the workaround I wrote about based on Zenject unit test cleaner. Basically I also filtered out the "ResourceManagerCallbacks" object that is in the DontDestroyOnLoad scene. So between tests both the Test Runner and the Addressable manager are kept.

    Is it clear enough ?

    I'm pretty sure I've filled the bug report on the subject.

    The PR on the Zenject project, not using Zenject myself, just the cleaner for now https://github.com/modesttree/Zenject/pull/249
     
    Andy_Ves likes this.
  19. MilanAssuied

    MilanAssuied

    Unity Technologies

    Joined:
    Sep 14, 2021
    Posts:
    4
    Hi @Whatever560

    I went a different path by changing my tests to Playmode tests, then went back to Editor tests, and could not reproduce the issue.

    I marked the commit which had the issue, and made no changes so far, I will try to investigate the differences between the two commit to understand better what is causing it
     
  20. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    Andy_Ves likes this.
  21. MilanAssuied

    MilanAssuied

    Unity Technologies

    Joined:
    Sep 14, 2021
    Posts:
    4
    I ran back in the problem again, and with the new test framework I get more details on the error:

    ArgumentException: An item with the same key has already been added. Key: <test name using TestCaseSource here>



    I can still run the tests class by class.
    I don't understand why in some version it works fine, and in some other it does not, the code is identical, it's the location of the tests that differs (Editor vs Runtime)

    I am going to try with the Zenject code now
     
  22. MilanAssuied

    MilanAssuied

    Unity Technologies

    Joined:
    Sep 14, 2021
    Posts:
    4
    So it seems to work with a subset of the code snipped fetched from Zenject.

    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEngine.SceneManagement;
    3.  
    4. namespace Tests.Tools.Runtime
    5. {
    6.     public static class TestRunnerCleaner
    7.     {
    8.         private const string UnitTestRunnerGameObjectName = "Code-based tests runner";
    9.  
    10.         public static void DestroyEverythingExceptTestRunner()
    11.         {
    12.             var testRunner = GameObject.Find(UnitTestRunnerGameObjectName);
    13.  
    14.             if (null == testRunner)
    15.             {
    16.                 return;
    17.             }
    18.        
    19.             Object.DontDestroyOnLoad(testRunner);
    20.  
    21.             // We want to clear all objects across all scenes to ensure the next test is not affected
    22.             // at all by previous tests
    23.             for (var i = 0; i < SceneManager.sceneCount; i++)
    24.             {
    25.                 foreach (var obj in SceneManager.GetSceneAt(i).GetRootGameObjects())
    26.                 {
    27.                     Object.DestroyImmediate(obj);
    28.                 }
    29.             }
    30.  
    31.  
    32.         }
    33.     }
    34. }
    Now, what is REALLY surprising, is that ...
    testRunner 
    is always null and the function actually do nothing.
    I tried to debug it, and I never reach the
    DonDestroyOnLoad
    call.

    But ... now the tests passes.
    So what did I do ? I removed the call to this function. And the test still passes. No other code change.

    If I change the function to make sure the
    SceneManager
    part run even if the
    testRunner
    is null, it fails again.

    So my guess is that there's something in Unity that don't clean properly even after the test runner is completed but that this behaviour is not deterministic, which explains the erratic previously noticed behaviour.

    Also, test runner 2.0.1-pre.18 is required, it will not work with version 1.1
     
    Last edited: Mar 9, 2022
  23. Andy_Ves

    Andy_Ves

    Joined:
    Nov 3, 2022
    Posts:
    1
    Hi @Whatever560 , thanks alot for postings and the link to your PR, much appreciated. My question is:
    Do you still (like me) run into the problem that the InputSystem breaks after a failed test? Or is some additional workaround necessary? (besides your modified code from Zenject?)
     
  24. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    I added a lot of contextual conditions, (setting a static var at test runs to detect we're in test and deactivate some onscreen or other behaviour related to inputs) but It is eventually breaking on most cases. It's been a while now, I've be driven away from using unity playmode tests because of a crunchy "just do the feature" period along with several regular issues :

    - Unable to launch a test bed without failing at some point. Really can't launch tens of play mode tests individually. Even thought of creating my own test runner that is really launching test atomically with a clear cleanup, but guess it'll wait for the industrialization phase.

    - Unity updates brought up more error messages (probably for a reason) but then didn't had time to work on the regressions.

    The ability to launch tests in their own sandbox is the n°1 feature a test framework should have. Unless this happens it will always feel very hacky.

    I have 1000+ tests running regularly on my c# backend with MSTest and it wouldn't be manageable if tests were not able to run in an atomic fashion (and in parallel, but I'm not asking as much as multithreading in unity tests)
     
    Last edited: Dec 14, 2022
    Andy_Ves likes this.
  25. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    Test runner name was changed, did you get this part regarding the last test pkg? That would explain your null ref.
     
    Last edited: Dec 14, 2022
  26. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    The last version I'm using for the Zenject Fork in test fwk 2.X.X
    Code (CSharp):
    1. public static class ZenjectTestUtil
    2.     {
    3.         public const string UnitTestRunnerGameObjectName = "tests runner";
    4.         /// <summary>
    5.         /// Addressable resource manager
    6.         /// </summary>
    7.         public const string ResourceManagerCallbacks     = "ResourceManagerCallbacks";
    8.  
    9.         // public static GameObject TestRunner;
    10.         /// <summary>
    11.         /// Will destroy everything in scene
    12.         /// </summary>
    13.         /// <param name="immediate"></param>
    14.         public static void DestroyEverythingExceptTestRunner(bool immediate)
    15.         {
    16.             // if (Application.isPlaying)
    17.             // {
    18.             //     TestRunner = GameObject.Find(UnitTestRunnerGameObjectName);
    19.             //     Assert.IsNotNull(TestRunner);
    20.             //     Object.DontDestroyOnLoad(TestRunner);
    21.             // }
    22.             // else
    23.             // {
    24.             //     TestRunner = null;
    25.             // }
    26.  
    27.             if(Application.isPlaying.Not() || AppInjectorBh.HasAppGlobalInjector().Not())
    28.             {
    29.             // We want to clear all objects across all scenes to ensure the next test is not affected
    30.             //at all by previous tests
    31.             for (int i = 0; i < SceneManager.sceneCount; i++)
    32.                 {
    33.                     foreach (var obj in SceneManager.GetSceneAt(i).GetRootGameObjects())
    34.                     {
    35.                         if (obj.name.Equals(UnitTestRunnerGameObjectName) ||
    36.                             obj.name.Equals(ResourceManagerCallbacks)
    37.                             ) continue;
    38.  
    39.                         if (immediate)
    40.                         {
    41.                             Object.DestroyImmediate(obj);
    42.                         }
    43.                         else
    44.                         {
    45.                             Object.Destroy(obj);
    46.                         }
    47.                     }
    48.                 }
    49.             }
    50.  
    51.  
    52.  
    53.             if (Application.isPlaying && AppInjectorBh.HasAppGlobalInjector())
    54.             {
    55.                 var dontDestroyOnLoadRoots = AppInjectorBh.GetCurrent()
    56.                                                           .Select(g => g.gameObject.scene
    57.                                                                         .GetRootGameObjects())
    58.                                                           .OrElseGet(() => new GameObject[]{});
    59.                                                        
    60.          
    61.                 foreach (var rootObj in dontDestroyOnLoadRoots)
    62.                 {
    63.                     if (rootObj.name != UnitTestRunnerGameObjectName &&
    64.                         rootObj.name != ResourceManagerCallbacks)
    65.                     {
    66.                         if (immediate)
    67.                         {
    68.                             Object.DestroyImmediate(rootObj);
    69.                         }
    70.                         else
    71.                         {
    72.                             Object.Destroy(rootObj);
    73.                         }
    74.                     }
    75.                 }
    76.                 //AppInjectorBh.ClearInjectorDebug();
    77.             }
    78.             IndusLogger.Debug(nameof(DestroyEverythingExceptTestRunner),$"Cleaned up for test : {TestContext.CurrentContext.Test.FullName}");
    79.         }
    80.     }
     
    Last edited: Dec 14, 2022
  27. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    Some example of semi-functional "workaround" to disable some input that might create consecutive errors after a failed tests :

    Code (CSharp):
    1.  
    2. public IEnumerator SetUp()
    3. {
    4.     MyStaticVars.IsAutomatedTests = true;
    5.     DOTween.useSafeMode             = false;
    6.     InputTest.Setup();
    7.     _gamepadDevice = InputSystem.AddDevice<Gamepad>();
    8.     ZenjectTestUtil.DestroyEverythingExceptTestRunner(Application.isPlaying.Not()); //from the post above
    9.  
    10.     if (EditorApplication.isPlaying)
    11.     {
    12.         yield return null; //to be sure destroys or whatever needs a frame was called
    13.     }
    14.     ...
    15. }
    16.  
    17. public IEnumerator TearDown(bool ignoreTearDownErrors = true)
    18. {
    19.     if (ignoreTearDownErrors) LogAssert.ignoreFailingMessages = true;
    20.     //1. Load empty scene
    21.     yield return Addressables.LoadSceneAsync(EScenes.IconsDummyScene + ".unity");
    22.  
    23.     Application.logMessageReceivedThreaded -= IgnoreUnloadErrors;
    24.     //2. Destroy all
    25.     UnityTestUtils.DestroyEverythingExceptTestRunner();
    26.  
    27.     //3. Let a frame for destroys to be performed
    28.     yield return null;
    29.  
    30.     //4. Disable inputs // REMARK doc says no need to clean up ... https://docs.unity3d.com/Packages/com.unity.inputsystem@1.3/manual/Testing.html
    31.     // IconsInputMap.Disable();
    32.     // IconsInputMap.Dispose();
    33.     InputTest.TearDown();
    34.  
    35.     LogAssert.ignoreFailingMessages = false;
    36.     IndusLogger.Debug(nameof(TearDown), TestContext.CurrentContext.Test.FullName);
    37.     AdvancedMonoBehaviour.IsAutomatedTests = false;
    38. }
    39.  
    40. private void IgnoreUnloadErrors(string condition, string stacktrace, LogType type)
    41. {
    42.     if (type == LogType.Error) LogAssert.Expect(type, stacktrace);
    43. }
    44.  
    A on screen button that was failing at cleanup

    Code (CSharp):
    1.  /// <summary>
    2.     /// This is the on screen control added by <see cref="InputActionInteraction"/> and that handles click interactions
    3.     /// </summary>
    4.     public class InputActionButtonControl : OnScreenControl, IHasOnCursorClick
    5.     {
    6.         [SerializeField]
    7.         private InputActionReference _inputActionRef;
    8.  
    9.     ...
    10.  
    11.         protected override void OnEnable()
    12.         {
    13.             //Bug in automated tests
    14.             if (AdvancedMonoBehaviour.IsAutomatedTests) return;
    15.  
    16.             try
    17.             {
    18.                 base.OnEnable();
    19.             }
    20.             catch (Exception e)
    21.             {
    22.                 IndusLogger.Warning(nameof(InputActionButtonControl), $"[IMPORTANT] While enabling on screen button control <{this.GetPathInScene()}>, probably due to faulty disable : \r\n{e}");
    23.             }
    24.         }
    25.  
    26.         /// <summary>
    27.         /// Let this as we may need it for debugging issue with the input system.
    28.         /// </summary>
    29.         protected override void OnDisable()
    30.         {
    31.        
    32.             //Bug in automated tests
    33.             if (AdvancedMonoBehaviour.IsAutomatedTests) return;
    34.  
    35.             try
    36.             {
    37.                 //        
    38.                 //Remark : This is causing errors in tests tear downs. Is it necessary ?
    39.                 base.OnDisable();
    40.             }
    41.             catch (Exception e)
    42.             {
    43.                 IndusLogger.Warning(nameof(InputActionButtonControl), $"[IMPORTANT] While disabling on screen button control <{this.GetPathInScene()}> : \r\n{e}");
    44.             }
    45.         }
    46.  
    47.         public void OnCursorClick()
    48.         {
    49.             //Button controls are float InputControl
    50.             SendValueToControl<float>(0);
    51.             SendValueToControl<float>(1);
    52.         }
    53.     }
    Another class that was enabling some inputs
    Code (CSharp):
    1.         public override void PostAwake()
    2.         {
    3. #if UNITY_INCLUDE_TESTS || UNITY_EDITOR //BUG : Crash in playmode tests
    4.             if (IsAutomatedTests) return;
    5. #endif
    6.  
    7.             //Seems necessary in builds as the Inputs are not enabled by default. Not needed in editor ...
    8.  
    9.             _inputActionReferences.Select(r => r.action.actionMap).Distinct().ForEach(m => m.Enable());
    10.  
    11.             _inputActionReferences.ForEach(m => m.action.Enable());
    12.         }
    13.  
    Hope this helps
     
    Last edited: Jan 12, 2023
    RGV and Andy_Ves like this.