Search Unity

  1. Are you interested in providing feedback directly to Unity teams? Sign up to become a member of Unity Pulse, our new product feedback and research community.
    Dismiss Notice

Running tests consecutively

Discussion in 'Testing & Automation' started by jessfdm_sky, Nov 22, 2019.

  1. jessfdm_sky

    jessfdm_sky

    Joined:
    Nov 19, 2018
    Posts:
    12
    Hi,

    I've got a load of tests that I need to run consecutively and not concurrently. When I select them individually and run them they run fine, and if I select a few of them (say a fixture or a namespare with a couple of fixtures below it) then they run fine as well. But if I select more they seem to run consecutively but some of my plugins don't work (DOTween if it helps to know), and if I ask all of them to run then they all start to run at the same time and clash with each other causing the results to flicker back and forth across random tests.

    Is it possible to run all the tests but force it to run all the tests concurrently and such that each test is completely isolated (as if I entered or exited playmode)? And is there a way to do this via the command line?

    Cheers!
     
    Whatever560 likes this.
  2. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,504
    What do you mean by consecutively? all tests are being executed one after the other, never concurrently.
     
  3. jessfdm_sky

    jessfdm_sky

    Joined:
    Nov 19, 2018
    Posts:
    12
    Mine don't seem to, they seem to run all at the same time and their results change flicker back and forth between each pass/fail.

    (These are playmode tests btw).

    Running them individually means that they pass fine though.

    I'm loading and unloading a scene for some of the tests and for those I'm using a superclass which has a [UnitySetUp] and [UnityTearDown] to load in my scene and then load back the test scene respectively, which all the other fixtures derive from. That's the only thing I think that could be causing the issue.

    Are we also allowed to "yield break" at the end of tests if the tests should be completable within a single frame?
     
  4. StewedHarry

    StewedHarry

    Joined:
    Jan 20, 2020
    Posts:
    45
    I am having the same issue when running tests consecutively. Did you find the problem?
     
  5. MajeureX

    MajeureX

    Joined:
    May 4, 2020
    Posts:
    13
    I'm guessing you're using the [UnityTest] attribute for your test methods? If so, your tests are run as coroutines, which might explain the behaviour. You can still use the [Test] attribute even in PlayMode tests, so if your test doesn't need to use yield then use the [Test] attribute instead.
     
    Last edited: Sep 12, 2020
  6. StewedHarry

    StewedHarry

    Joined:
    Jan 20, 2020
    Posts:
    45
    My tests do require yields unfortunately
     
  7. andrew-fray

    andrew-fray

    Joined:
    Jul 19, 2012
    Posts:
    147
    Gonna bump this thread. My problem is that I have a legacy codebase, which likes to leave around a ton of DontDestroyOnLoad gameobjects, and probably some static state too. Running my playmode tests in isolation work fine, but once I try to chain two, the lingering objects and state starts causing issues. If I could make my tests each end with a proper play mode exit, things would be sufficiently cleaned up. Is there a straightforward way to do this? I realise I'm sacrificing the ability to run these in a build.

    The alternative I could try is restructuring them as edit mode tests, and have each architect a call to play mode enter/exit. Does that sound feasible?
     
    Whatever560 likes this.
  8. andrew-fray

    andrew-fray

    Joined:
    Jul 19, 2012
    Posts:
    147
    (in the end I bit the bullet and made my levels play nicely when flowing from one to the other. That's obviously the Proper Fix and worth the investment)
     
  9. StewedHarry

    StewedHarry

    Joined:
    Jan 20, 2020
    Posts:
    45
    You might need to properly setup and teardown each test:

    Code (CSharp):
    1.  
    2.         [SetUp]
    3.         public void Init()
    4.         {
    5.             // loads a new scene for each test
    6.             SceneManager.LoadScene("TestScene", LoadSceneMode.Single);
    7.        
    8.         }
    9.  
    10.         [TearDown]
    11.         public void TearDown()
    12.         {
    13.             // disposes training instance from the Academy singleton
    14.             Academy.Instance.Dispose();
    15.         }
    I also ran into trouble when setting up tests due to the immediate calling of the delegate passed to the event:
    Academy.Instance.OnEnvironmentReset
     
  10. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    221
    BTW this fails and it should not if you run both tests at the same time. If you run them one by one they pass.

    This is a real issue, none of NUnit and MSTests (nor any test framework ever) are working this way...

    This should be documented and have an easy work around ...

    Code (CSharp):
    1.     [TestFixture]
    2.     public class PlayModeCleanContextTest
    3.     {
    4.         [UnityTest]
    5.         public IEnumerator TestA()
    6.         {
    7.             yield return new EnterPlayMode();
    8.             var a = new GameObject("A");
    9.             Object.DontDestroyOnLoad(a);
    10.             yield return new ExitPlayMode();
    11.         }
    12.         [UnityTest]
    13.         public IEnumerator TestB()
    14.         {
    15.             yield return new EnterPlayMode();
    16.             var a = Object.FindObjectOfType<GameObject>();
    17.             if(a) Assert.AreNotEqual("A", a.name);
    18.         }
    19.     }
     
  11. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    221
    This is not really feasible with heavy integration tests though. Also InputSystem Setup/TearDown is flawed and errors stack up after each couple of tests.
     
  12. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    221
    You can do
    yield return new Enter/ExitPlayMode(), but it's not cleaning anything as you can see in the previous example
     
  13. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    221
    I tried adding this in the SetUp of tests. It just breaks the InputSystem even more with an error flood

    Code (CSharp):
    1.             for (var i = 0; i < SceneManager.sceneCount; ++i)
    2.             {
    3.                 foreach (var rootGameObject in SceneManager.GetSceneAt(i).GetRootGameObjects())
    4.                 {
    5.                     UnityEngine.Object.Destroy(rootGameObject);
    6.                 }
    7.             }
    8.  
     
  14. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    23
    There's really no easy way to clear static state (you should just avoid it in general).

    However, for destroying all GameObjects, you might want to take a look at how Zenject test utilities do it.
     
    Whatever560 likes this.
  15. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    221
    Yeah, I already cleaned up any static code. My issues actually come from unity own modules (InputSystem) that get all wacky once an error occured in a test. It comes from the fact that they use static based code with a push/pop mecanism.

    I'll try Zenject code though. Thanks
     
  16. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    221
    I pulled my hair out on this one, do not forgot to NOT destroy the "ResourceManagerCallbacks" gameObject. If using the addressable package or async operation will not finish and hang the test silently. I'll add a PR to Zenject
     
    sbergen likes this.
unityunity