Search Unity

Play mode tests - SceneHandling

Discussion in 'Testing & Automation' started by Kruemelchen, Sep 26, 2019.

  1. Kruemelchen

    Kruemelchen

    Joined:
    Aug 26, 2019
    Posts:
    27
    Hi,

    First of all, I'm a big fan of unity's test framework. Now as I work with it a lot, I came to the point were I'm repeating myself over and over again by building up my test scenes, loading prefabs, making some tests and cleaning up afterwards. In my case this gets a little bit annoying. For example take a look at this example:
    Image you want to test a weapon shooting projectiles. The test looks something like this:
    * Spawn weapon prefab
    * Trigger Weapon.Shoot()
    * Count projectile-GameObjects in scene
    So far so good. But now you have to manually delete all projectiles. In this case that's mostly simple as you know all projectile GameObjects. But imagine that the weapon does not only spawn projectiles but also particle systems, decals, etc. You have always to bear in mind that these artifacts CAN influence your other tests.
    E.g. you want to test another weapon. Depending on which test was executed first, the test environment changes and you might find GameObjects in the scene that were produced by other tests.
    In my opinion, each test should run in its own scene. Of course, counting GameObjects might be not the smartest way to test your behaviors. There are better approaches like faking/mocking/subs etc. But there will be probably cases were scene environment matters.

    Therefore I wrote a little helper class that sets up a new scene every time, a tests get executed to ensure a fresh, non-manipulated environment for each individual test of each individual test script.

    Code (CSharp):
    1.  
    2. public static class TestFrameworkHelper
    3. {
    4.     private static GameObject _testRunner;
    5.        
    6.     public static IEnumerator InitTestScene()
    7.     {
    8.          var activeScene = SceneManager.GetActiveScene();
    9.          var rootGameObjects = activeScene.GetRootGameObjects();
    10.          if (_testRunner == null)
    11.              _testRunner = rootGameObjects.FirstOrDefault(go =>
    12.                     go.GetComponents(typeof(MonoBehaviour)).Any(c => c.GetType().Name == "PlaymodeTestsController"));
    13.          var newScene = SceneManager.CreateScene("TestScene" + DateTime.Now.Millisecond);
    14.          SceneManager.MoveGameObjectToScene(_testRunner, newScene);
    15.          Object.DontDestroyOnLoad(_testRunner);
    16.          yield return SceneManager.UnloadSceneAsync(activeScene);
    17.      }
    18. }
    19.  
    A little bit hacky but that's due to the internal accessibility of PlaymodeTestsController.

    What do you think? Just call TestFrameworkHelper.InitTestScene() at the beginning of each play mode test and don't bother with cleaning up!
     
  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    What I tend to do is to create a test scene ahead of time. This could be empty, or it could contain pre-setup objects - e.g. for your weapon example, I might have the weapon prefab already in the scene. Then you'd call
    SceneManager.LoadScene("TestScene", SceneMode.Single)
    in a [Setup] or [OneTimeSetUp] in your fixture. The new
    ITestPlayerBuildModifier
    interface can be used to make sure the test scene is pulled into any test players you build.
     
    liortal likes this.
  3. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    Is it safe to use SceneMode.Single? When you run a Play Mode test, a bootstrapping scene is loaded with a single GameObject that contains a PlaymodeTestsController component. What is the purpose of this? Loading a different test scene with SceneMode.Single destroys the bootstrapping scene and the PlaymodeTestsController component. Is that ok?
     
  4. floky

    floky

    Joined:
    Oct 6, 2010
    Posts:
    260
    Yes this confused me as well.

    Examples of using the Unity test framework are all over the place in various different ways.
    For example, I'm personally still not clear as to what's the correct way to load a predefined custom scene in a play mode test (that's supposed to work in a player also).
    Some examples I've seen modify the "EditorBuildSettings.scene" in a prebuild step and do the manual cleanup in post build other examples use the "ITestPlayerBuildModifier" interface.
     
    neilsarkar and DrummerB like this.
  5. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    Some more detailed examples would certainly be nice. It's clear how to set up some basic unit tests, but I'm not sure about what's a good setup for more complex integration tests, i.e. preparing different tests scenes, loading a specific scene for a test etc.

    I think you're supposed to use ITestPlayerBuildModifier for adding test scenes to the build and then simply use SceneManager.LoadScene.
     
    waldgeist likes this.
  6. floky

    floky

    Joined:
    Oct 6, 2010
    Posts:
    260
    Yeah I saw the "ITestPlayerBuildModifier" interface. The entire flow seems so cumbersome just to load a test scene or more and run a few test methods for them. You have to implement a dozen interfaces and write a lot of boilerplate code to just load some scenes.
    And not to mention you always have to hardcode the full scene path. That hinders a lot of future flexibility or re-usability in other similar projects for a certain test base you work at.
    Oh well, it's still a WIP project I guess.

    For example I didn't find any example of how to load a ScriptableObject asset in a run-time test without having to create custom scene to bind its reference to it. And not just a "ScriptableObject", but any other type of asset. Maybe you just want to load a prefab without creating an extra scene for it. You already have the generated scene for the run-time test.

    Sry for pouring these issues here, just hoping someone would maybe read and help shed some light on them...one day.
     
  7. floky

    floky

    Joined:
    Oct 6, 2010
    Posts:
    260
    Would be great to be able to easily just create and setup a ScriptableObject config with input params for a test or a batch of tests where you can drag an drop the references to the desired custom test scenes and other separate objects you may want to access at run-time in your tests. The test-framework would automatically take that scriptableobject config and add the custom scenes to the build and the objects you set up for it.
     
  8. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    I think you can abstract that away. I wrote a custom attribute that loads a specific scene before the test and unloads it afterward. Then you can just have something like this:

    Code (CSharp):
    1. [Test, Scene("Intro")]
    2. public void SceneHasPlayer()
    3. {
    4.     Assert.That(Object.FindObjectOfType<Player>(), Is.Not.Null);
    5.     Assert.That(Player.Instance, Is.Not.Null);
    6. }
    It's not yet completely finished, but I could upload it somewhere when it's done. I still have to figure out how to include the scene in a standalone test build.

    I don't understand the issue about ScritableObjects and Prefabs. Can you not just find them in the assets and create an instance or instantiate the prefab in the scene?

    That could work. You can just search for all scriptable objects in the project and generate test cases for each of them. This would work in plain NUnit, but I'm not sure if the Unity variant supports dynamic test case generation.
     
    SolidAlloy and floky like this.
  9. Desoxi

    Desoxi

    Joined:
    Apr 12, 2015
    Posts:
    191
    How do you load the scene? I'm trying to load the scene in [OneTimeSetUp] but when I the try to find objects or scripts in the loaded scene its always null. Am I missing something obvious? Im trying to find the objects in the same setup method and in [Test] methods as well. Both do return null :/
     
    CashMonkey, soccerbob97 and melkior like this.
  10. melkior

    melkior

    Joined:
    Jul 20, 2013
    Posts:
    199
    Just trying to use unit testing in Unity for the first time myself and having this same problem. I have a "ClientManager" game object with a few scripts that exists in the scene before the game starts and the Unit tests can not find it (always returns null).

    Google brought me here .. still trying to figure out how to solve this.
     
  11. Desoxi

    Desoxi

    Joined:
    Apr 12, 2015
    Posts:
    191
    Hey @melkior, a while ago I managed to get it working and now I'm writing tests for all my scenes.
    There are a few to keep in mind for it to work, so I will try to list them here:

    • You can load any scene via
      Code (CSharp):
      1. SceneManager.LoadScene("Scene Name", LoadSceneMode.Single);
    • However, because it takes some time to load the scene, you can not access the components and objects inside this scene immediately. So you have to wait for it to load completely. Here is my code to make it work after the scene loads and load it only once and set the references to any scripts or objects once (and not on every single test):

      Code (CSharp):
      1.         bool sceneLoaded;
      2.         bool referencesSetup;
      3.         SomeComponent someComponentReference;
      4.  
      5.        
      6.         [OneTimeSetUp]
      7.         public void OneTimeSetup()
      8.         {
      9.             SceneManager.sceneLoaded += OnSceneLoaded;
      10.             SceneManager.LoadScene("SceneName", LoadSceneMode.Single);
      11.         }
      12.  
      13.         void OnSceneLoaded(Scene scene, LoadSceneMode mode)
      14.         {
      15.             sceneLoaded = true;
      16.         }
      17.  
      18.         void SetupReferences()
      19.         {
      20.             if (referencesSetup)
      21.             {
      22.                 return;
      23.             }
      24.  
      25.             Transform[] objects = Resources.FindObjectsOfTypeAll<Transform>();
      26.             foreach (Transform t in objects)
      27.             {
      28.                 if (t.name == "Name of Gameobject")
      29.                 {
      30.                     someComponentReference = t.GetComponent<SomeComponent>();
      31.                 }
      32.             }
      33.            
      34.             referencesSetup = true;
      35.         }
      36.        
      37.         [UnityTest]
      38.         public IEnumerator TestReferencesNotNullAfterLoad()
      39.         {
      40.             yield return new WaitWhile(() => sceneLoaded == false);
      41.             SetupReferences();
      42.             Assert.IsNotNull(someComponentReference);
      43.             //Add all other references as well for quick nullref testing
      44.             yield return null;
      45.         }

    Each of my UnityTests are looking more or less the same. I always call these two lines
    Code (CSharp):
    1. yield return new WaitWhile(() => sceneLoaded == false);
    2. SetupReferences();
    before I start testing and traversing through my scene flow.

    The reason why I had to do it like this is because the OneTimeSetup attribute does not allow the used method to be a IEnumerator (Coroutine), and so I've written this workaround.

    The finding process of your gameobjects and components consists of 2 parts.

    • Finding gameobjects which are active in hierarchy: here you can use the simple GameObject.Find("Name of object") method and get its components via GetComponent<Type>()
    • Finding inactive gameobjects: Here you cant use GameObject.Find because it only finds active objects. You could use tags maybe (didn't test FindByTag), but I didn't want to create Tags for each and every gameobject in all my scenes. For inactive objects you can use this snippet:
      Code (CSharp):
      1. Transform[] objects = Resources.FindObjectsOfTypeAll<Transform>();
      2.             foreach (Transform t in objects)
      3.             {
      4.                 if (t.name == "Name of Gameobject")
      5.                 {
      6.                     someComponentReference = t.GetComponent<SomeComponent>();
      7.                 }
      8.             }
      This way you have an array with ALL objects in your scene which have a transform component. Then you just need to check for its name (beware of whitespace at the end of your names! The name may look the same but "Name " (whitespace at the end) is not the same like "Name".
    I hope I could help you and happy testing!
     
  12. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,579
    Any good way to work around the scene having to be in the build settings?

    I really don't want to have all our tests scenes in there, but otherwise I'm not allowed to load them. That's fine for all usual cases, but not here.

    I tried the obvious hack;

    Code (csharp):
    1.  
    2.     private EditorBuildSettingsScene[] buildSettingsScenesBackup;
    3.  
    4.     [OneTimeSetUp]
    5.     public void OneTimeSetup() {
    6.         buildSettingsScenesBackup = EditorBuildSettings.scenes;
    7.         var ledgeGrab = new EditorBuildSettingsScene("Assets/Scenes/Unit Test Scenes/Scene.unity", true);
    8.         var newScenes = EditorBuildSettings.scenes.Append(ledgeGrab).ToArray();
    9.         EditorBuildSettings.scenes = newScenes;
    10.  
    11.         SceneManager.LoadScene("Scene", LoadSceneMode.Single);
    12.         SceneManager.sceneLoaded += OnSceneLoaded;
    13.     }
    14.  
    15.     [OneTimeTearDown]
    16.     public void OneTimeTearDown() {
    17.         EditorBuildSettings.scenes = buildSettingsScenesBackup;
    18.     }
    19.  
    But that doesn't work. I'm guessing that it has to be in the list before you enter play mode?
     
  13. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
  14. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,579
    That sounds like it'd work for running tests in builds - or is it possible to use for running in the editor as well?
     
  15. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    I don't think that works for in the Editor, but you do have the option of using EditorSceneManager there instead (behind a #if directive).
     
  16. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,579
    Last time I tried, EditorSceneManager errors out if you try to load scenes in play mode, so that's probably a no-go. Unless there's a way to make it not do that?
     
  17. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    @superpig @Baste I'm relatively sure that it works in the Editor. I'm running tests both in Editor and Standalone. I call the following functions from IPrebuildSetup.Setup and IPostBuildCleanup.Cleanup

    According to the documentation these should be called before any test run.

    Code (CSharp):
    1. /// Add all scenes to the build settings that are in the test scene folder.
    2. public static void AddTestScenesToBuildSettings()
    3. {
    4.     #if UNITY_EDITOR
    5.     var scenes = new List<EditorBuildSettingsScene>();
    6.     var guids = AssetDatabase.FindAssets("t:Scene", new[] {TestSceneFolder});
    7.     if (guids != null)
    8.     {
    9.         foreach (string guid in guids)
    10.         {
    11.             var path = AssetDatabase.GUIDToAssetPath(guid);
    12.             if (!string.IsNullOrEmpty(path) && File.Exists(path))
    13.             {
    14.                 var scene = new EditorBuildSettingsScene(path, true);
    15.                 scenes.Add(scene);
    16.             }
    17.         }
    18.     }
    19.  
    20.     Debug.Log("Adding test scenes to build settings:\n" + string.Join("\n", scenes.Select(scene => scene.path)));
    21.     EditorBuildSettings.scenes = EditorBuildSettings.scenes.Union(scenes).ToArray();
    22.     #endif
    23. }
    24.  
    25. /// Remove all scenes from the build settings that are in the test scene folder.
    26. public static void RemoveTestScenesFromBuildSettings()
    27. {
    28.     #if UNITY_EDITOR
    29.     EditorBuildSettings.scenes = EditorBuildSettings.scenes
    30.         .Where(scene => !scene.path.StartsWith(TestSceneFolder)).ToArray();
    31.     #endif
    32. }
     
  18. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    Baste likes this.
  19. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,579
    EditorSceneManager.LoadSceneInPlayMode indeed works! Thank you.

    Though, is the behaviour when the path is wrong correct? Right now passing it a path that doesn't point to a scene just causes it to create and load a temp scene with that name instead. I would've preferred it if I got an error if the path doesn't exist, so I can get a good error if some folders gets moved instead.
     
  20. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    Looking at the code, I think that's probably a bug - the scene load is done by creating a new scene and then loading the scene file 'into' it, so my guess is that if you request a scene path that does not exist then the second part fails but the first part still ran. Should be fairly easy to validate the path before starting the whole process. Could you file a bug report?
     
    Last edited: Sep 15, 2020
  21. outbreaker99

    outbreaker99

    Joined:
    Aug 19, 2018
    Posts:
    1
    this is how i handled the situation:
    this is the testFixture class:


    [OneTimeSetUp]
    public void Setup()
    {
    SceneManager.LoadSceneAsync("Level1").completed += SetupTests_completed;
    }

    private void SetupTests_completed(AsyncOperation obj)
    {
    TestEvents.FireSceneLoaded(this, true);
    }


    and in a specific test class:


    [OneTimeSetUp]
    public void Setup()
    {
    TestEvents.SceneLoadedListeners += SetupTestsAfterLoadScene;
    }
     
  22. fafase

    fafase

    Joined:
    Jul 3, 2012
    Posts:
    157
    Just wanted to add to the topic, I figured out the best way for me so far is to use:

    Code (CSharp):
    1. EditorSceneManager.LoadSceneAsyncInPlayMode(path, new LoadSceneParameters(LoadSceneMode.Single));
    where path has to be the full path as "Asset/<scene_path>/sceneName.unity" and not just the scene name as it would in SceneManager.LoadScene.

    I was shortly using the IPrebuildSetup/IPostBuildCleanup but as it is meant to, it would actually load and unload the scene in Jenkins jobs and I really want to avoid that even if it does not go in the build. For instance, my scene was referring to an asset bundle and the build would break for the bundle was missing in build.

    Now the next thing I am after is the cleaning of the test scenes that are created. Each launch makes a new TestScenelonghashvalue in the asset folder. I currently manually remove them or could set a quick script to run and delete them but I would think there may be a Unity way to do that.
     
    bobbaluba likes this.
  23. bobbaluba

    bobbaluba

    Joined:
    Feb 27, 2013
    Posts:
    79
    For the show don't tell coders out there:

    If you just want to load a single scene once before running all the tests

    Code (CSharp):
    1. public class Tests
    2. {
    3.     [OneTimeSetUp] public void OneTimeSetup() => EditorSceneManager.LoadSceneInPlayMode("Assets/Tests/TestScene.unity", new LoadSceneParameters(LoadSceneMode.Single));
    4.  
    5.     [UnityTest]
    6.     public IEnumerator Test1()
    7.     {
    8.         var a = GameObject.Find("a");
    9.         Debug.Assert(a != null);
    10.         yield return null;
    11.     }
    12. }
    13.  
    And if you want/need a clean state, i.e. reload the scene before each test case:

    Code (CSharp):
    1. public class Tests
    2. {
    3.     [UnitySetUp]
    4.     public IEnumerator Setup()
    5.     {
    6.         yield return EditorSceneManager.LoadSceneAsyncInPlayMode("Assets/Tests/TestScene.unity", new LoadSceneParameters(LoadSceneMode.Single));
    7.     }
    8.  
    9.     [UnityTest]
    10.     public IEnumerator Test1()
    11.     {
    12.         var a = GameObject.Find("a");
    13.         Debug.Assert(a != null);
    14.         Object.Destroy(a);
    15.         yield return null;
    16.     }
    17.  
    18.     [UnityTest]
    19.     public IEnumerator Test2()
    20.     {
    21.         var a = GameObject.Find("a");
    22.         Debug.Assert(a != null);
    23.         Object.Destroy(a);
    24.         yield return null;
    25.     }
    26. }
    27.  
    Finally, if you want a different scene per test:

    Code (CSharp):
    1. public class Tests
    2. {
    3.     [UnityTest]
    4.     [LoadScene("Assets/Tests/TestA.unity")]
    5.     public IEnumerator Test1()
    6.     {
    7.         var a = GameObject.Find("a");
    8.         Debug.Assert(a != null);
    9.         Object.Destroy(a);
    10.         yield return null;
    11.     }
    12.  
    13.     [UnityTest]
    14.     [LoadScene("Assets/Tests/TestB.unity")]
    15.     public IEnumerator Test2()
    16.     {
    17.         var b = GameObject.Find("b");
    18.         Debug.Assert(b != null);
    19.         Object.Destroy(b);
    20.         yield return null;
    21.     }
    22. }
    23.  
    24. public class LoadSceneAttribute : NUnitAttribute, IOuterUnityTestAction
    25. {
    26.     private string scene;
    27.  
    28.     public LoadSceneAttribute(string scene) => this.scene = scene;
    29.  
    30.     IEnumerator IOuterUnityTestAction.BeforeTest(ITest test)
    31.     {
    32.         Debug.Assert(scene.EndsWith(".unity"));
    33.         yield return EditorSceneManager.LoadSceneAsyncInPlayMode(scene, new LoadSceneParameters(LoadSceneMode.Single));
    34.     }
    35.  
    36.     IEnumerator IOuterUnityTestAction.AfterTest(ITest test)
    37.     {
    38.         yield return null;
    39.     }
    40. }
     
  24. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    1,303
    Is there a way to do this without introducing a dependency (that's irrelevant to the test) on external files? (which is a very bad practice in testing).

    (out of interest - how long are people seeing in execution times? The whole 'load a scene and wait' approach seems pretty bizarre to me - the one thing tests shouldn't be is slow to run, and this seems guaranteed to make my colleagues and I stop running tests. Unity is already waaaay to slow for running play mode tests although I know there are moves to fix that)
     
  25. floky

    floky

    Joined:
    Oct 6, 2010
    Posts:
    260
    Awesome summary @bobbaluba! Really appreciate it! The docs weren't so clear as your neatly crash-course post.

    Do you happen to know of a way to detect if Unity is preparing to start running tests before the "EditorApplication.playModeStateChanged" event gets fired?
    I'm using an editor tool that in the "playModeStateChanged" event is setting the "EditorSceneManager.playModeStartScene" to start the play mode with a specific startup scene. This messes up Unity's automated tests flow which apparently makes use of the same property when running the tests.

    Thanks again for taking the time!
     
    Last edited: May 25, 2021
  26. steinbitglis

    steinbitglis

    Joined:
    Sep 22, 2011
    Posts:
    210
    Seems to have been fixed by now. The newer Unity versions will print an error message.
     
    superpig likes this.
  27. Vishal0703

    Vishal0703

    Joined:
    Jul 5, 2020
    Posts:
    1
    I know this thread has become about scene handling, but referring to the start of thread where the problem was gameobjects from one test interfering with other, this simple function does the cleanup

    [TearDown]
    public void TearDown()
    {
    Object.FindObjectsOfType<GameObject>().ToList()
    .ForEach(go => GameObject.DestroyImmediate(go));
    }

    I was afraid it was gonna remove the CodeRunner gameobject which has the PlaymodeTestsController scripts etc attached, but it doesn't... It only removes gameobject that you instantiated... so pretty neat....

    Reference :
     
    steinbitglis likes this.
unityunity