Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Restart player between playmode tests?

Discussion in 'Testing & Automation' started by eric_rdg, Jun 17, 2020.

  1. eric_rdg

    eric_rdg

    Joined:
    Nov 12, 2019
    Posts:
    6
    We use Unity test framework for gameplay automation testing, where (at least some of) our tests load the game as a player would, manipulate some data, and then carry out scripted interactions to exercise various pieces of functionality and assert/etc based on the outcomes. It's been super useful even just to make sure that no exceptions/errors trigger during user flows, much higher "bugs avoided per line of test code" value than any of our more tightly-focused unit tests, for sure.

    The problem is, while these tests are super easy to develop in isolation (when running them individually, manually), running them one after another is super painful and in some cases almost impossible, because of various pieces of state that linger between runs. There's the easy stuff to manage, like our own singletons and any static state (which of course is not great to have around anyways) and DoNotDestroyOnLoad GOs, but much more difficult to deal with are things outside of our control, like third party libraries and scene preloading (if you happen to end a test when a scene has already started preloading, the next test that needs to load a scene blows up, and there doesn't seem to be any way to cancel a preload). For another example, our audio library does not tolerate being torn down and recreated, so we had to add checks into our game so that we don't restart it when we enter our "first load" scene on a second or third test, and shipping logic that only exists to make tests work is a bad smell.

    The other problem with state that persists between tests is that accidental dependencies on that state can slip into tests, and that's never good, either.

    Is there any way to tag a test as requiring a completely fresh execution context and player, or a way to hack something similar together?
     
    Julian_FW likes this.
  2. Julian_FW

    Julian_FW

    Joined:
    Oct 21, 2021
    Posts:
    3
    Running into the same problem with persistent game objects. Using
    EditorSceneManager.UnloadSceneAsync("Assets/Tests/PlayMode/MainScene.unity");
    after each test doesn't work because of Game Objects that DontDestroyOnLoad between scenes...

    Have you found any workarounds?
     
  3. eric_rdg

    eric_rdg

    Joined:
    Nov 12, 2019
    Posts:
    6
    Unfortunately we've just resorted to a bunch of nasty code that explicitly manages teardown and resetting. It is not perfect and we still end up with order-dependence of tests, but we're just working around it for now. We've gone in two directions: lean more on proper unit tests (which don't require a scene) to test pure code logic as much as possible, and coalesce most of our playmode stuff into a single uber-test which does a single pass covering most of what we need for a smoke test (including using cheats after testing certain things to reset user state and move onto the next thing to check, etc.).

    We're not doing very extreme TDD or anything, though, we really only lean on tests to make sure that a few happy paths succeed without exceptions. It still catches a *ton* of stuff, so despite some quirks it remains very useful.
     
    Julian_FW likes this.