Running tests consecutively

Hi,

I've got a load of tests that I need to run consecutively and not concurrently. When I select them individually and run them they run fine, and if I select a few of them (say a fixture or a namespare with a couple of fixtures below it) then they run fine as well. But if I select more they seem to run consecutively but some of my plugins don't work (DOTween if it helps to know), and if I ask all of them to run then they all start to run at the same time and clash with each other causing the results to flicker back and forth across random tests.

Is it possible to run all the tests but force it to run all the tests concurrently and such that each test is completely isolated (as if I entered or exited playmode)? And is there a way to do this via the command line?

Cheers!

 

What do you mean by consecutively? all tests are being executed one after the other, never concurrently.

 

Mine don't seem to, they seem to run all at the same time and their results change flicker back and forth between each pass/fail.

(These are playmode tests btw).

Running them individually means that they pass fine though.

I'm loading and unloading a scene for some of the tests and for those I'm using a superclass which has a [UnitySetUp] and [UnityTearDown] to load in my scene and then load back the test scene respectively, which all the other fixtures derive from. That's the only thing I think that could be causing the issue.

Are we also allowed to "yield break" at the end of tests if the tests should be completable within a single frame?

 

I am having the same issue when running tests consecutively. Did you find the problem?

 

I'm guessing you're using the [UnityTest] attribute for your test methods? If so, your tests are run as coroutines, which might explain the behaviour. You can still use the [Test] attribute even in PlayMode tests, so if your test doesn't need to use yield then use the [Test] attribute instead.

 

My tests do require yields unfortunately

 

Gonna bump this thread. My problem is that I have a legacy codebase, which likes to leave around a ton of DontDestroyOnLoad gameobjects, and probably some static state too. Running my playmode tests in isolation work fine, but once I try to chain two, the lingering objects and state starts causing issues. If I could make my tests each end with a proper play mode exit, things would be sufficiently cleaned up. Is there a straightforward way to do this? I realise I'm sacrificing the ability to run these in a build.

The alternative I could try is restructuring them as edit mode tests, and have each architect a call to play mode enter/exit. Does that sound feasible?

 

(in the end I bit the bullet and made my levels play nicely when flowing from one to the other. That's obviously the Proper Fix and worth the investment)

 

You might need to properly setup and teardown each test:

Code (CSharp):

1.  
2.         [SetUp]
3.         public void Init()
4.         {
5.             // loads a new scene for each test
6.             SceneManager.LoadScene("TestScene", LoadSceneMode.Single);
7.        
8.         }
9.  
10.         [TearDown]
11.         public void TearDown()
12.         {
13.             // disposes training instance from the Academy singleton
14.             Academy.Instance.Dispose();
15.         }

I also ran into trouble when setting up tests due to the immediate calling of the delegate passed to the event:

Academy.Instance.OnEnvironmentReset

 

BTW this fails and it should not if you run both tests at the same time. If you run them one by one they pass.

This is a real issue, none of NUnit and MSTests (nor any test framework ever) are working this way...

This should be documented and have an easy work around ...

Code (CSharp):

1.     [TestFixture]
2.     public class PlayModeCleanContextTest
3.     {
4.         [UnityTest]
5.         public IEnumerator TestA()
6.         {
7.             yield return new EnterPlayMode();
8.             var a = new GameObject("A");
9.             Object.DontDestroyOnLoad(a);
10.             yield return new ExitPlayMode();
11.         }
12.         [UnityTest]
13.         public IEnumerator TestB()
14.         {
15.             yield return new EnterPlayMode();
16.             var a = Object.FindObjectOfType<GameObject>();
17.             if(a) Assert.AreNotEqual("A", a.name);
18.         }
19.     }

 

This is not really feasible with heavy integration tests though. Also InputSystem Setup/TearDown is flawed and errors stack up after each couple of tests.

 

You can do
yield return new Enter/ExitPlayMode(), but it's not cleaning anything as you can see in the previous example

 

I tried adding this in the SetUp of tests. It just breaks the InputSystem even more with an error flood

Code (CSharp):

1.             for (var i = 0; i < SceneManager.sceneCount; ++i)
2.             {
3.                 foreach (var rootGameObject in SceneManager.GetSceneAt(i).GetRootGameObjects())
4.                 {
5.                     UnityEngine.Object.Destroy(rootGameObject);
6.                 }
7.             }
8.  

 

There's really no easy way to clear static state (you should just avoid it in general).

However, for destroying all GameObjects, you might want to take a look at how Zenject test utilities do it.

 

Yeah, I already cleaned up any static code. My issues actually come from unity own modules (InputSystem) that get all wacky once an error occured in a test. It comes from the fact that they use static based code with a push/pop mecanism.

I'll try Zenject code though. Thanks

 

I pulled my hair out on this one, do not forgot to NOT destroy the "ResourceManagerCallbacks" gameObject. If using the addressable package or async operation will not finish and hang the test silently. I'll add a PR to Zenject

 

I'm using the workaround I wrote about based on Zenject unit test cleaner. Basically I also filtered out the "ResourceManagerCallbacks" object that is in the DontDestroyOnLoad scene. So between tests both the Test Runner and the Addressable manager are kept.

Is it clear enough ?

I'm pretty sure I've filled the bug report on the subject.

The PR on the Zenject project, not using Zenject myself, just the cleaner for now https://github.com/modesttree/Zenject/pull/249

 

I'm trying the new version of the test FWK (2.0.1, at https://docs.unity3d.com/Packages/com.unity.test-framework@2.0/manual/index.html), and it seems that I just need to clean up my own "Don't destroy on load" objects now. No need to tag the "tests runner" as dont destroy on load anymore (beware of its new name, was previously "code-base tests runner")

 

Hi @Whatever560 , thanks alot for postings and the link to your PR, much appreciated. My question is:
Do you still (like me) run into the problem that the InputSystem breaks after a failed test? Or is some additional workaround necessary? (besides your modified code from Zenject?)

 

I added a lot of contextual conditions, (setting a static var at test runs to detect we're in test and deactivate some onscreen or other behaviour related to inputs) but It is eventually breaking on most cases. It's been a while now, I've be driven away from using unity playmode tests because of a crunchy "just do the feature" period along with several regular issues :

- Unable to launch a test bed without failing at some point. Really can't launch tens of play mode tests individually. Even thought of creating my own test runner that is really launching test atomically with a clear cleanup, but guess it'll wait for the industrialization phase.

- Unity updates brought up more error messages (probably for a reason) but then didn't had time to work on the regressions.

The ability to launch tests in their own sandbox is the n°1 feature a test framework should have. Unless this happens it will always feel very hacky.

I have 1000+ tests running regularly on my c# backend with MSTest and it wouldn't be manageable if tests were not able to run in an atomic fashion (and in parallel, but I'm not asking as much as multithreading in unity tests)

 

Test runner name was changed, did you get this part regarding the last test pkg? That would explain your null ref.

 

The last version I'm using for the Zenject Fork in test fwk 2.X.X

Code (CSharp):

1. public static class ZenjectTestUtil
2.     {
3.         public const string UnitTestRunnerGameObjectName = "tests runner";
4.         /// <summary>
5.         /// Addressable resource manager
6.         /// </summary>
7.         public const string ResourceManagerCallbacks     = "ResourceManagerCallbacks";
8.  
9.         // public static GameObject TestRunner;
10.         /// <summary>
11.         /// Will destroy everything in scene
12.         /// </summary>
13.         /// <param name="immediate"></param>
14.         public static void DestroyEverythingExceptTestRunner(bool immediate)
15.         {
16.             // if (Application.isPlaying)
17.             // {
18.             //     TestRunner = GameObject.Find(UnitTestRunnerGameObjectName);
19.             //     Assert.IsNotNull(TestRunner);
20.             //     Object.DontDestroyOnLoad(TestRunner);
21.             // }
22.             // else
23.             // {
24.             //     TestRunner = null;
25.             // }
26.  
27.             if(Application.isPlaying.Not() || AppInjectorBh.HasAppGlobalInjector().Not())
28.             {
29.             // We want to clear all objects across all scenes to ensure the next test is not affected
30.             //at all by previous tests
31.             for (int i = 0; i < SceneManager.sceneCount; i++)
32.                 {
33.                     foreach (var obj in SceneManager.GetSceneAt(i).GetRootGameObjects())
34.                     {
35.                         if (obj.name.Equals(UnitTestRunnerGameObjectName) ||
36.                             obj.name.Equals(ResourceManagerCallbacks)
37.                             ) continue;
38.  
39.                         if (immediate)
40.                         {
41.                             Object.DestroyImmediate(obj);
42.                         }
43.                         else
44.                         {
45.                             Object.Destroy(obj);
46.                         }
47.                     }
48.                 }
49.             }
50.  
51.  
52.  
53.             if (Application.isPlaying && AppInjectorBh.HasAppGlobalInjector())
54.             {
55.                 var dontDestroyOnLoadRoots = AppInjectorBh.GetCurrent()
56.                                                           .Select(g => g.gameObject.scene
57.                                                                         .GetRootGameObjects())
58.                                                           .OrElseGet(() => new GameObject[]{});
59.                                                        
60.          
61.                 foreach (var rootObj in dontDestroyOnLoadRoots)
62.                 {
63.                     if (rootObj.name != UnitTestRunnerGameObjectName &&
64.                         rootObj.name != ResourceManagerCallbacks)
65.                     {
66.                         if (immediate)
67.                         {
68.                             Object.DestroyImmediate(rootObj);
69.                         }
70.                         else
71.                         {
72.                             Object.Destroy(rootObj);
73.                         }
74.                     }
75.                 }
76.                 //AppInjectorBh.ClearInjectorDebug();
77.             }
78.             IndusLogger.Debug(nameof(DestroyEverythingExceptTestRunner),$"Cleaned up for test : {TestContext.CurrentContext.Test.FullName}");
79.         }
80.     }

 

Some example of semi-functional "workaround" to disable some input that might create consecutive errors after a failed tests :

Code (CSharp):

1.  
2. public IEnumerator SetUp()
3. {
4.     MyStaticVars.IsAutomatedTests = true;
5.     DOTween.useSafeMode             = false;
6.     InputTest.Setup();
7.     _gamepadDevice = InputSystem.AddDevice<Gamepad>();
8.     ZenjectTestUtil.DestroyEverythingExceptTestRunner(Application.isPlaying.Not()); //from the post above
9.  
10.     if (EditorApplication.isPlaying)
11.     {
12.         yield return null; //to be sure destroys or whatever needs a frame was called
13.     }
14.     ...
15. }
16.  
17. public IEnumerator TearDown(bool ignoreTearDownErrors = true)
18. {
19.     if (ignoreTearDownErrors) LogAssert.ignoreFailingMessages = true;
20.     //1. Load empty scene
21.     yield return Addressables.LoadSceneAsync(EScenes.IconsDummyScene + ".unity");
22.  
23.     Application.logMessageReceivedThreaded -= IgnoreUnloadErrors;
24.     //2. Destroy all
25.     UnityTestUtils.DestroyEverythingExceptTestRunner();
26.  
27.     //3. Let a frame for destroys to be performed
28.     yield return null;
29.  
30.     //4. Disable inputs // REMARK doc says no need to clean up ... https://docs.unity3d.com/Packages/com.unity.inputsystem@1.3/manual/Testing.html
31.     // IconsInputMap.Disable();
32.     // IconsInputMap.Dispose();
33.     InputTest.TearDown();
34.  
35.     LogAssert.ignoreFailingMessages = false;
36.     IndusLogger.Debug(nameof(TearDown), TestContext.CurrentContext.Test.FullName);
37.     AdvancedMonoBehaviour.IsAutomatedTests = false;
38. }
39.  
40. private void IgnoreUnloadErrors(string condition, string stacktrace, LogType type)
41. {
42.     if (type == LogType.Error) LogAssert.Expect(type, stacktrace);
43. }
44.  

A on screen button that was failing at cleanup

Code (CSharp):

1.  /// <summary>
2.     /// This is the on screen control added by <see cref="InputActionInteraction"/> and that handles click interactions
3.     /// </summary>
4.     public class InputActionButtonControl : OnScreenControl, IHasOnCursorClick
5.     {
6.         [SerializeField]
7.         private InputActionReference _inputActionRef;
8.  
9.     ...
10.  
11.         protected override void OnEnable()
12.         {
13.             //Bug in automated tests
14.             if (AdvancedMonoBehaviour.IsAutomatedTests) return;
15.  
16.             try
17.             {
18.                 base.OnEnable();
19.             }
20.             catch (Exception e)
21.             {
22.                 IndusLogger.Warning(nameof(InputActionButtonControl), $"[IMPORTANT] While enabling on screen button control <{this.GetPathInScene()}>, probably due to faulty disable : \r\n{e}");
23.             }
24.         }
25.  
26.         /// <summary>
27.         /// Let this as we may need it for debugging issue with the input system.
28.         /// </summary>
29.         protected override void OnDisable()
30.         {
31.        
32.             //Bug in automated tests
33.             if (AdvancedMonoBehaviour.IsAutomatedTests) return;
34.  
35.             try
36.             {
37.                 //        
38.                 //Remark : This is causing errors in tests tear downs. Is it necessary ?
39.                 base.OnDisable();
40.             }
41.             catch (Exception e)
42.             {
43.                 IndusLogger.Warning(nameof(InputActionButtonControl), $"[IMPORTANT] While disabling on screen button control <{this.GetPathInScene()}> : \r\n{e}");
44.             }
45.         }
46.  
47.         public void OnCursorClick()
48.         {
49.             //Button controls are float InputControl
50.             SendValueToControl<float>(0);
51.             SendValueToControl<float>(1);
52.         }
53.     }

Another class that was enabling some inputs

Code (CSharp):

1.         public override void PostAwake()
2.         {
3. #if UNITY_INCLUDE_TESTS || UNITY_EDITOR //BUG : Crash in playmode tests
4.             if (IsAutomatedTests) return;
5. #endif
6.  
7.             //Seems necessary in builds as the Inputs are not enabled by default. Not needed in editor ...
8.  
9.             _inputActionReferences.Select(r => r.action.actionMap).Distinct().ForEach(m => m.Enable());
10.  
11.             _inputActionReferences.ForEach(m => m.action.Enable());
12.         }
13.  

Hope this helps