Search Unity

Play mode tests - SceneHandling

Discussion in 'Testing & Automation' started by Kruemelchen, Sep 26, 2019.

  1. Kruemelchen

    Kruemelchen

    Joined:
    Aug 26, 2019
    Posts:
    27
    Hi,

    First of all, I'm a big fan of unity's test framework. Now as I work with it a lot, I came to the point were I'm repeating myself over and over again by building up my test scenes, loading prefabs, making some tests and cleaning up afterwards. In my case this gets a little bit annoying. For example take a look at this example:
    Image you want to test a weapon shooting projectiles. The test looks something like this:
    * Spawn weapon prefab
    * Trigger Weapon.Shoot()
    * Count projectile-GameObjects in scene
    So far so good. But now you have to manually delete all projectiles. In this case that's mostly simple as you know all projectile GameObjects. But imagine that the weapon does not only spawn projectiles but also particle systems, decals, etc. You have always to bear in mind that these artifacts CAN influence your other tests.
    E.g. you want to test another weapon. Depending on which test was executed first, the test environment changes and you might find GameObjects in the scene that were produced by other tests.
    In my opinion, each test should run in its own scene. Of course, counting GameObjects might be not the smartest way to test your behaviors. There are better approaches like faking/mocking/subs etc. But there will be probably cases were scene environment matters.

    Therefore I wrote a little helper class that sets up a new scene every time, a tests get executed to ensure a fresh, non-manipulated environment for each individual test of each individual test script.

    Code (CSharp):
    1.  
    2. public static class TestFrameworkHelper
    3. {
    4.     private static GameObject _testRunner;
    5.        
    6.     public static IEnumerator InitTestScene()
    7.     {
    8.          var activeScene = SceneManager.GetActiveScene();
    9.          var rootGameObjects = activeScene.GetRootGameObjects();
    10.          if (_testRunner == null)
    11.              _testRunner = rootGameObjects.FirstOrDefault(go =>
    12.                     go.GetComponents(typeof(MonoBehaviour)).Any(c => c.GetType().Name == "PlaymodeTestsController"));
    13.          var newScene = SceneManager.CreateScene("TestScene" + DateTime.Now.Millisecond);
    14.          SceneManager.MoveGameObjectToScene(_testRunner, newScene);
    15.          Object.DontDestroyOnLoad(_testRunner);
    16.          yield return SceneManager.UnloadSceneAsync(activeScene);
    17.      }
    18. }
    19.  
    A little bit hacky but that's due to the internal accessibility of PlaymodeTestsController.

    What do you think? Just call TestFrameworkHelper.InitTestScene() at the beginning of each play mode test and don't bother with cleaning up!
     
  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,657
    What I tend to do is to create a test scene ahead of time. This could be empty, or it could contain pre-setup objects - e.g. for your weapon example, I might have the weapon prefab already in the scene. Then you'd call
    SceneManager.LoadScene("TestScene", SceneMode.Single)
    in a [Setup] or [OneTimeSetUp] in your fixture. The new
    ITestPlayerBuildModifier
    interface can be used to make sure the test scene is pulled into any test players you build.
     
    liortal likes this.
  3. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    135
    Is it safe to use SceneMode.Single? When you run a Play Mode test, a bootstrapping scene is loaded with a single GameObject that contains a PlaymodeTestsController component. What is the purpose of this? Loading a different test scene with SceneMode.Single destroys the bootstrapping scene and the PlaymodeTestsController component. Is that ok?
     
    dan_ginovker likes this.
  4. zalogic

    zalogic

    Joined:
    Oct 6, 2010
    Posts:
    273
    Yes this confused me as well.

    Examples of using the Unity test framework are all over the place in various different ways.
    For example, I'm personally still not clear as to what's the correct way to load a predefined custom scene in a play mode test (that's supposed to work in a player also).
    Some examples I've seen modify the "EditorBuildSettings.scene" in a prebuild step and do the manual cleanup in post build other examples use the "ITestPlayerBuildModifier" interface.
     
    dan_ginovker, neilsarkar and DrummerB like this.
  5. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    135
    Some more detailed examples would certainly be nice. It's clear how to set up some basic unit tests, but I'm not sure about what's a good setup for more complex integration tests, i.e. preparing different tests scenes, loading a specific scene for a test etc.

    I think you're supposed to use ITestPlayerBuildModifier for adding test scenes to the build and then simply use SceneManager.LoadScene.
     
    dan_ginovker and waldgeist like this.
  6. zalogic

    zalogic

    Joined:
    Oct 6, 2010
    Posts:
    273
    Yeah I saw the "ITestPlayerBuildModifier" interface. The entire flow seems so cumbersome just to load a test scene or more and run a few test methods for them. You have to implement a dozen interfaces and write a lot of boilerplate code to just load some scenes.
    And not to mention you always have to hardcode the full scene path. That hinders a lot of future flexibility or re-usability in other similar projects for a certain test base you work at.
    Oh well, it's still a WIP project I guess.

    For example I didn't find any example of how to load a ScriptableObject asset in a run-time test without having to create custom scene to bind its reference to it. And not just a "ScriptableObject", but any other type of asset. Maybe you just want to load a prefab without creating an extra scene for it. You already have the generated scene for the run-time test.

    Sry for pouring these issues here, just hoping someone would maybe read and help shed some light on them...one day.
     
  7. zalogic

    zalogic

    Joined:
    Oct 6, 2010
    Posts:
    273
    Would be great to be able to easily just create and setup a ScriptableObject config with input params for a test or a batch of tests where you can drag an drop the references to the desired custom test scenes and other separate objects you may want to access at run-time in your tests. The test-framework would automatically take that scriptableobject config and add the custom scenes to the build and the objects you set up for it.
     
  8. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    135
    I think you can abstract that away. I wrote a custom attribute that loads a specific scene before the test and unloads it afterward. Then you can just have something like this:

    Code (CSharp):
    1. [Test, Scene("Intro")]
    2. public void SceneHasPlayer()
    3. {
    4.     Assert.That(Object.FindObjectOfType<Player>(), Is.Not.Null);
    5.     Assert.That(Player.Instance, Is.Not.Null);
    6. }
    It's not yet completely finished, but I could upload it somewhere when it's done. I still have to figure out how to include the scene in a standalone test build.

    I don't understand the issue about ScritableObjects and Prefabs. Can you not just find them in the assets and create an instance or instantiate the prefab in the scene?

    That could work. You can just search for all scriptable objects in the project and generate test cases for each of them. This would work in plain NUnit, but I'm not sure if the Unity variant supports dynamic test case generation.
     
    SolidAlloy and zalogic like this.
  9. Desoxi

    Desoxi

    Joined:
    Apr 12, 2015
    Posts:
    195
    How do you load the scene? I'm trying to load the scene in [OneTimeSetUp] but when I the try to find objects or scripts in the loaded scene its always null. Am I missing something obvious? Im trying to find the objects in the same setup method and in [Test] methods as well. Both do return null :/
     
  10. melkior

    melkior

    Joined:
    Jul 20, 2013
    Posts:
    199
    Just trying to use unit testing in Unity for the first time myself and having this same problem. I have a "ClientManager" game object with a few scripts that exists in the scene before the game starts and the Unit tests can not find it (always returns null).

    Google brought me here .. still trying to figure out how to solve this.
     
  11. Desoxi

    Desoxi

    Joined:
    Apr 12, 2015
    Posts:
    195
    Hey @melkior, a while ago I managed to get it working and now I'm writing tests for all my scenes.
    There are a few to keep in mind for it to work, so I will try to list them here:

    • You can load any scene via
      Code (CSharp):
      1. SceneManager.LoadScene("Scene Name", LoadSceneMode.Single);
    • However, because it takes some time to load the scene, you can not access the components and objects inside this scene immediately. So you have to wait for it to load completely. Here is my code to make it work after the scene loads and load it only once and set the references to any scripts or objects once (and not on every single test):

      Code (CSharp):
      1.         bool sceneLoaded;
      2.         bool referencesSetup;
      3.         SomeComponent someComponentReference;
      4.  
      5.        
      6.         [OneTimeSetUp]
      7.         public void OneTimeSetup()
      8.         {
      9.             SceneManager.sceneLoaded += OnSceneLoaded;
      10.             SceneManager.LoadScene("SceneName", LoadSceneMode.Single);
      11.         }
      12.  
      13.         void OnSceneLoaded(Scene scene, LoadSceneMode mode)
      14.         {
      15.             sceneLoaded = true;
      16.         }
      17.  
      18.         void SetupReferences()
      19.         {
      20.             if (referencesSetup)
      21.             {
      22.                 return;
      23.             }
      24.  
      25.             Transform[] objects = Resources.FindObjectsOfTypeAll<Transform>();
      26.             foreach (Transform t in objects)
      27.             {
      28.                 if (t.name == "Name of Gameobject")
      29.                 {
      30.                     someComponentReference = t.GetComponent<SomeComponent>();
      31.                 }
      32.             }
      33.            
      34.             referencesSetup = true;
      35.         }
      36.        
      37.         [UnityTest]
      38.         public IEnumerator TestReferencesNotNullAfterLoad()
      39.         {
      40.             yield return new WaitWhile(() => sceneLoaded == false);
      41.             SetupReferences();
      42.             Assert.IsNotNull(someComponentReference);
      43.             //Add all other references as well for quick nullref testing
      44.             yield return null;
      45.         }

    Each of my UnityTests are looking more or less the same. I always call these two lines
    Code (CSharp):
    1. yield return new WaitWhile(() => sceneLoaded == false);
    2. SetupReferences();
    before I start testing and traversing through my scene flow.

    The reason why I had to do it like this is because the OneTimeSetup attribute does not allow the used method to be a IEnumerator (Coroutine), and so I've written this workaround.

    The finding process of your gameobjects and components consists of 2 parts.

    • Finding gameobjects which are active in hierarchy: here you can use the simple GameObject.Find("Name of object") method and get its components via GetComponent<Type>()
    • Finding inactive gameobjects: Here you cant use GameObject.Find because it only finds active objects. You could use tags maybe (didn't test FindByTag), but I didn't want to create Tags for each and every gameobject in all my scenes. For inactive objects you can use this snippet:
      Code (CSharp):
      1. Transform[] objects = Resources.FindObjectsOfTypeAll<Transform>();
      2.             foreach (Transform t in objects)
      3.             {
      4.                 if (t.name == "Name of Gameobject")
      5.                 {
      6.                     someComponentReference = t.GetComponent<SomeComponent>();
      7.                 }
      8.             }
      This way you have an array with ALL objects in your scene which have a transform component. Then you just need to check for its name (beware of whitespace at the end of your names! The name may look the same but "Name " (whitespace at the end) is not the same like "Name".
    I hope I could help you and happy testing!
     
    levexis, glenneroo, Kokowolo and 6 others like this.
  12. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,336
    Any good way to work around the scene having to be in the build settings?

    I really don't want to have all our tests scenes in there, but otherwise I'm not allowed to load them. That's fine for all usual cases, but not here.

    I tried the obvious hack;

    Code (csharp):
    1.  
    2.     private EditorBuildSettingsScene[] buildSettingsScenesBackup;
    3.  
    4.     [OneTimeSetUp]
    5.     public void OneTimeSetup() {
    6.         buildSettingsScenesBackup = EditorBuildSettings.scenes;
    7.         var ledgeGrab = new EditorBuildSettingsScene("Assets/Scenes/Unit Test Scenes/Scene.unity", true);
    8.         var newScenes = EditorBuildSettings.scenes.Append(ledgeGrab).ToArray();
    9.         EditorBuildSettings.scenes = newScenes;
    10.  
    11.         SceneManager.LoadScene("Scene", LoadSceneMode.Single);
    12.         SceneManager.sceneLoaded += OnSceneLoaded;
    13.     }
    14.  
    15.     [OneTimeTearDown]
    16.     public void OneTimeTearDown() {
    17.         EditorBuildSettings.scenes = buildSettingsScenesBackup;
    18.     }
    19.  
    But that doesn't work. I'm guessing that it has to be in the list before you enter play mode?
     
    dan_ginovker likes this.
  13. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    135
  14. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,336
    That sounds like it'd work for running tests in builds - or is it possible to use for running in the editor as well?
     
  15. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,657
    I don't think that works for in the Editor, but you do have the option of using EditorSceneManager there instead (behind a #if directive).
     
  16. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,336
    Last time I tried, EditorSceneManager errors out if you try to load scenes in play mode, so that's probably a no-go. Unless there's a way to make it not do that?
     
  17. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    135
    @superpig @Baste I'm relatively sure that it works in the Editor. I'm running tests both in Editor and Standalone. I call the following functions from IPrebuildSetup.Setup and IPostBuildCleanup.Cleanup

    According to the documentation these should be called before any test run.

    Code (CSharp):
    1. /// Add all scenes to the build settings that are in the test scene folder.
    2. public static void AddTestScenesToBuildSettings()
    3. {
    4.     #if UNITY_EDITOR
    5.     var scenes = new List<EditorBuildSettingsScene>();
    6.     var guids = AssetDatabase.FindAssets("t:Scene", new[] {TestSceneFolder});
    7.     if (guids != null)
    8.     {
    9.         foreach (string guid in guids)
    10.         {
    11.             var path = AssetDatabase.GUIDToAssetPath(guid);
    12.             if (!string.IsNullOrEmpty(path) && File.Exists(path))
    13.             {
    14.                 var scene = new EditorBuildSettingsScene(path, true);
    15.                 scenes.Add(scene);
    16.             }
    17.         }
    18.     }
    19.  
    20.     Debug.Log("Adding test scenes to build settings:\n" + string.Join("\n", scenes.Select(scene => scene.path)));
    21.     EditorBuildSettings.scenes = EditorBuildSettings.scenes.Union(scenes).ToArray();
    22.     #endif
    23. }
    24.  
    25. /// Remove all scenes from the build settings that are in the test scene folder.
    26. public static void RemoveTestScenesFromBuildSettings()
    27. {
    28.     #if UNITY_EDITOR
    29.     EditorBuildSettings.scenes = EditorBuildSettings.scenes
    30.         .Where(scene => !scene.path.StartsWith(TestSceneFolder)).ToArray();
    31.     #endif
    32. }
     
  18. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,657
    Baste likes this.
  19. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,336
    EditorSceneManager.LoadSceneInPlayMode indeed works! Thank you.

    Though, is the behaviour when the path is wrong correct? Right now passing it a path that doesn't point to a scene just causes it to create and load a temp scene with that name instead. I would've preferred it if I got an error if the path doesn't exist, so I can get a good error if some folders gets moved instead.
     
  20. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,657
    Looking at the code, I think that's probably a bug - the scene load is done by creating a new scene and then loading the scene file 'into' it, so my guess is that if you request a scene path that does not exist then the second part fails but the first part still ran. Should be fairly easy to validate the path before starting the whole process. Could you file a bug report?
     
    Last edited: Sep 15, 2020
    dan_ginovker likes this.
  21. outbreaker99

    outbreaker99

    Joined:
    Aug 19, 2018
    Posts:
    2
    this is how i handled the situation:
    this is the testFixture class:


    [OneTimeSetUp]
    public void Setup()
    {
    SceneManager.LoadSceneAsync("Level1").completed += SetupTests_completed;
    }

    private void SetupTests_completed(AsyncOperation obj)
    {
    TestEvents.FireSceneLoaded(this, true);
    }


    and in a specific test class:


    [OneTimeSetUp]
    public void Setup()
    {
    TestEvents.SceneLoadedListeners += SetupTestsAfterLoadScene;
    }
     
  22. fafase

    fafase

    Joined:
    Jul 3, 2012
    Posts:
    163
    Just wanted to add to the topic, I figured out the best way for me so far is to use:

    Code (CSharp):
    1. EditorSceneManager.LoadSceneAsyncInPlayMode(path, new LoadSceneParameters(LoadSceneMode.Single));
    where path has to be the full path as "Asset/<scene_path>/sceneName.unity" and not just the scene name as it would in SceneManager.LoadScene.

    I was shortly using the IPrebuildSetup/IPostBuildCleanup but as it is meant to, it would actually load and unload the scene in Jenkins jobs and I really want to avoid that even if it does not go in the build. For instance, my scene was referring to an asset bundle and the build would break for the bundle was missing in build.

    Now the next thing I am after is the cleaning of the test scenes that are created. Each launch makes a new TestScenelonghashvalue in the asset folder. I currently manually remove them or could set a quick script to run and delete them but I would think there may be a Unity way to do that.
     
    bobbaluba likes this.
  23. bobbaluba

    bobbaluba

    Joined:
    Feb 27, 2013
    Posts:
    81
    For the show don't tell coders out there:

    If you just want to load a single scene once before running all the tests

    Code (CSharp):
    1. public class Tests
    2. {
    3.     [OneTimeSetUp] public void OneTimeSetup() => EditorSceneManager.LoadSceneInPlayMode("Assets/Tests/TestScene.unity", new LoadSceneParameters(LoadSceneMode.Single));
    4.  
    5.     [UnityTest]
    6.     public IEnumerator Test1()
    7.     {
    8.         var a = GameObject.Find("a");
    9.         Debug.Assert(a != null);
    10.         yield return null;
    11.     }
    12. }
    13.  
    And if you want/need a clean state, i.e. reload the scene before each test case:

    Code (CSharp):
    1. public class Tests
    2. {
    3.     [UnitySetUp]
    4.     public IEnumerator Setup()
    5.     {
    6.         yield return EditorSceneManager.LoadSceneAsyncInPlayMode("Assets/Tests/TestScene.unity", new LoadSceneParameters(LoadSceneMode.Single));
    7.     }
    8.  
    9.     [UnityTest]
    10.     public IEnumerator Test1()
    11.     {
    12.         var a = GameObject.Find("a");
    13.         Debug.Assert(a != null);
    14.         Object.Destroy(a);
    15.         yield return null;
    16.     }
    17.  
    18.     [UnityTest]
    19.     public IEnumerator Test2()
    20.     {
    21.         var a = GameObject.Find("a");
    22.         Debug.Assert(a != null);
    23.         Object.Destroy(a);
    24.         yield return null;
    25.     }
    26. }
    27.  
    Finally, if you want a different scene per test:

    Code (CSharp):
    1. public class Tests
    2. {
    3.     [UnityTest]
    4.     [LoadScene("Assets/Tests/TestA.unity")]
    5.     public IEnumerator Test1()
    6.     {
    7.         var a = GameObject.Find("a");
    8.         Debug.Assert(a != null);
    9.         Object.Destroy(a);
    10.         yield return null;
    11.     }
    12.  
    13.     [UnityTest]
    14.     [LoadScene("Assets/Tests/TestB.unity")]
    15.     public IEnumerator Test2()
    16.     {
    17.         var b = GameObject.Find("b");
    18.         Debug.Assert(b != null);
    19.         Object.Destroy(b);
    20.         yield return null;
    21.     }
    22. }
    23.  
    24. public class LoadSceneAttribute : NUnitAttribute, IOuterUnityTestAction
    25. {
    26.     private string scene;
    27.  
    28.     public LoadSceneAttribute(string scene) => this.scene = scene;
    29.  
    30.     IEnumerator IOuterUnityTestAction.BeforeTest(ITest test)
    31.     {
    32.         Debug.Assert(scene.EndsWith(".unity"));
    33.         yield return EditorSceneManager.LoadSceneAsyncInPlayMode(scene, new LoadSceneParameters(LoadSceneMode.Single));
    34.     }
    35.  
    36.     IEnumerator IOuterUnityTestAction.AfterTest(ITest test)
    37.     {
    38.         yield return null;
    39.     }
    40. }
     
  24. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    1,933
    Is there a way to do this without introducing a dependency (that's irrelevant to the test) on external files? (which is a very bad practice in testing).

    (out of interest - how long are people seeing in execution times? The whole 'load a scene and wait' approach seems pretty bizarre to me - the one thing tests shouldn't be is slow to run, and this seems guaranteed to make my colleagues and I stop running tests. Unity is already waaaay to slow for running play mode tests although I know there are moves to fix that)
     
  25. zalogic

    zalogic

    Joined:
    Oct 6, 2010
    Posts:
    273
    Awesome summary @bobbaluba! Really appreciate it! The docs weren't so clear as your neatly crash-course post.

    Do you happen to know of a way to detect if Unity is preparing to start running tests before the "EditorApplication.playModeStateChanged" event gets fired?
    I'm using an editor tool that in the "playModeStateChanged" event is setting the "EditorSceneManager.playModeStartScene" to start the play mode with a specific startup scene. This messes up Unity's automated tests flow which apparently makes use of the same property when running the tests.

    Thanks again for taking the time!
     
    Last edited: May 25, 2021
    tokar_dev and booferei like this.
  26. steinbitglis

    steinbitglis

    Joined:
    Sep 22, 2011
    Posts:
    254
    Seems to have been fixed by now. The newer Unity versions will print an error message.
     
    superpig likes this.
  27. Vishal0703

    Vishal0703

    Joined:
    Jul 5, 2020
    Posts:
    2
    I know this thread has become about scene handling, but referring to the start of thread where the problem was gameobjects from one test interfering with other, this simple function does the cleanup

    [TearDown]
    public void TearDown()
    {
    Object.FindObjectsOfType<GameObject>().ToList()
    .ForEach(go => GameObject.DestroyImmediate(go));
    }

    I was afraid it was gonna remove the CodeRunner gameobject which has the PlaymodeTestsController scripts etc attached, but it doesn't... It only removes gameobject that you instantiated... so pretty neat....

    Reference :
     
  28. Fangh

    Fangh

    Joined:
    Apr 19, 2013
    Posts:
    274
    Since you are using "EditorSceneManager", will it work for PlayMode Tests ?
     
  29. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    1,933
    I like the idea.

    But ... This is incorrect. It will work for most situations, then it will fail in others (i.e. break your tests in bad, hard to detect, ways). When testing: the word "most" should have you filled with terror and thinking: "Nooooo!". If you're going to use it, please read the API docs and at least set the final arg to "true".

    ... more generally: FindObjectsOfType / FindObjectsOfTypeAll are dangerous API calls and should not be used for anything that will run in Editor, unless you literally have no choice - they don't do what they say they do, they have undocumented failure cases, and are highly unpredictable - they will work today and fail tomorrow for no obvious reason. In particular: FindObjectsOfTypeAll (recommended from the docs page for FindObjectsOftype) has never worked (it only works in Runtime, not in Editor), and Unity's current policy is that they won't fix it now or in future. Avoid!
     
  30. N8W1nD

    N8W1nD

    Joined:
    Sep 3, 2020
    Posts:
    9

    Thanks alot!! You made my tests a lot cleaner and faster!!

    This is a really smart solution, I was thinking about using WaitForSeconds, but this is so much better o_O

    <3
     
  31. darbotron

    darbotron

    Joined:
    Aug 9, 2010
    Posts:
    352
    @bobbaluba OMG this is pure gold 10/10.

    The LoadScene attribute is inspired - I've had to solve similar but not identical problems with tests over the years and I had no idea that writing custom attributes would offer such an elegant solution... thanks for sharing!!
     
    gigelgigel likes this.
  32. gigelgigel

    gigelgigel

    Joined:
    Jul 20, 2018
    Posts:
    1
    Woah, this is truly brilliant. Not sure why that's not a builtin annotation in Unity!
     
  33. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    613
    I am writing tests for Unity packages, and there is no support for adding scenes to the build settings which is really limiting for testing. I guess you can set up test scenes in samples? But that's really ugly.

    What's the best way to support this workflow when working with packages? OP's solution in this case seems best.
     
  34. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,336
    Does anyone know of a way to do this without requiring the test to be a UnityTest? I want to have a tests that runs code on a bunch of objects in a scene that can run instantly, for speed reasons. Ie. what I want to do is:

    Code (csharp):
    1.  
    2.     [OneTimeSetUp] public void OneTimeSetup() => EditorSceneManager.LoadSceneInPlayMode("Assets/Tests/TestScene.unity", new LoadSceneParameters(LoadSceneMode.Single));
    3.  
    4.     [Test]
    5.     public void Test1()
    6.     {
    7.         var a = GameObject.Find("a");
    8.         Debug.Assert(a != null);
    9.     }
    10. }
    11.  
    But if I do that, the object isn't found. There's something going on internally such that after OneTimeSetup runs, the normal [Test] tests are run before the scene is completely loaded yet.

    The fact that this even works:
    Code (csharp):
    1.  
    2. [UnityTest]
    3. public IEnumerator MyTest() {
    4.     // actual, instant test
    5.     yield return null;
    6. }
    7.  
    Means that this is due to how the UnityTests are run, not that there has to be a yield there.


    So I have found one way to work around this, which is probably the stupidest hack I have written for Unity as of yet.

    Code (csharp):
    1.  
    2. [OneTimeSetUp]
    3. public void LoadScene() {
    4.     EditorSceneManager.LoadSceneInPlayMode("Assets/Tests/TestScene.unity", new(LoadSceneMode.Single));
    5. }
    6.  
    7. [UnityTest] public IEnumerator AAAA() { yield return null; }
    8.  
    9. [Test] public void Test1() { ... }
    10. [Test] public void Test2() { ... }
    11. [Test] public void Test3() { ... }
    12. [Test] public void Test4() { ... }
    13.  
    The existence of a single UnityTest that runs before the normal tests means that we get a yield before the tests run and then we're off to the races. The problem here, though, is that I have to run all of the tests in a class once every time instead of having the ability to run only one, but this still runs faster than running two [UnityTest] tests because they wait at minimum one frame.

    But this is incredibly dumb! All of it is! It can't be that hard to write a functioning test runner for Unity where you don't have to run through insane hoops to test things in a scene without it taking ages.
     
    AldeRoberge likes this.
  35. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    613
    I believe it has to do with how scenes load, there actually needs to be one frame where the scene in a way “initializes” (goes through awake all the way up to destroy) before it allows methods like GameObject.Find.

    In the tutorials for the test framework you can see that when they load a scene you have to wait for one frame.

    You can try doing an
    async await Yield();
    instruction but I am unsure if async OneTimeSetup is supported. Unfortunately, somewhere that yield needs to exist, but the bottomline is that it is not the test frameworks fault, this is just something that happens with scene loading.
     
  36. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,336
    [async await Yield() sadly doesn't work - you can have an
    async Task OneTimeSetup
    , but that just makes the "running tests" wait bar stay up forever, and cancelling it doesn't work, so the editor has to be force-quit.

    I tried downgrading - we've been on test framework 2, so I wondered if the 1.3 version might have a way to handle this, but downgrading to that makes our project not compile at all - we get a bunch of esoteric compiler errors that look like this:

    upload_2023-9-19_10-39-24.png

    And the only way I found to recover the project was to delete the library, enter safe mode, and then trigger a compile twice.

    I think I'm done trying to make the test framework work. It seems like it's going to be significantly less work and headache to just write our own thing. Switching to a scene, entering play mode, running a bunch of functions in order, and recording their result should not be something you need years of engineering effort to achieve.
     
    RDeluxe likes this.
  37. MikePhill

    MikePhill

    Joined:
    Jun 8, 2014
    Posts:
    1
    Hi, did you managed to solve the issue? I faced the same problem having default Scene load from playModeStartScene (which i certainly need, so i can't get rid of it), so sharing kinda solution i did. It's not fully automated, but it makes my life much easier :)

    It solves following problems:
    1. Sets playModeStartScene to null with a Ctrl+Shift+T hotkey, you can run any tests after. You need to press the hotkey before each run
    2. Returns playModeStartScene to default value (EditorBuildSettings.scenes[0] in my case) during InitializeOnLoad procedure, which means any Run, code update or Editor restart. This helps me to not be bothered to return the default scene value back
    3. Deletes garbage Scenes during InitializeOnLoad. The garbage Scenes creation happens when you forget to press the Ctrl+Shift+T hotkey before the test run, so the run jumps out of the TestScene to the Scene defined in playModeStartScene, which causes TestScene's save automatically

    Anyway, here's the code i've got:
    Code (CSharp):
    1. using System.IO;
    2. using UnityEditor;
    3. using UnityEditor.SceneManagement;
    4. using UnityEngine;
    5.  
    6. [InitializeOnLoad]
    7. public class EditorInit
    8. {
    9.     public static string openedScene;
    10.     static EditorInit()
    11.     {
    12.         openedScene = EditorSceneManager.GetActiveScene().path;
    13.         var pathOfFirstScene = EditorBuildSettings.scenes[0].path;
    14.         var sceneAsset = AssetDatabase.LoadAssetAtPath<SceneAsset>(pathOfFirstScene);
    15.         EditorSceneManager.playModeStartScene = sceneAsset;
    16.         Debug.Log(pathOfFirstScene + " was set as default play mode scene. To disable it for test execution press Ctrl+Shift+T");
    17.         DeleteGarbageFiles("InitTestScene");
    18.     }
    19.  
    20.  
    21.     [MenuItem("Tools/Editor/Play From Scene/Set Start Screen Scene as Main", false, 0)]
    22.     public static void SetDefaultScene()
    23.     {
    24.         var pathOfFirstScene = EditorBuildSettings.scenes[0].path;
    25.         var sceneAsset = AssetDatabase.LoadAssetAtPath<SceneAsset>(pathOfFirstScene);
    26.         if (sceneAsset != null)
    27.         {
    28.             EditorSceneManager.playModeStartScene = sceneAsset;
    29.             Debug.Log(pathOfFirstScene + " was set as default play mode scene. To disable it for test execution press Ctrl+Shift+T");
    30.         }
    31.         else
    32.             Debug.Log("Scene not saved!");
    33.     }
    34.  
    35.     [MenuItem("Tools/Editor/Play From Scene/Unset Main Scene (for tests execution) ^#t", false, 1)]
    36.     public static void UnsetDefaultScene()
    37.     {
    38.         EditorSceneManager.playModeStartScene = null;
    39.         Debug.Log("null was set as default play mode scene. Default scene will be applied after any run");
    40.     }
    41.  
    42.     private static void DeleteGarbageFiles(string stringPart)
    43.     {
    44.         string[] garbageScenes = AssetDatabase.FindAssets(stringPart);
    45.         foreach (string guid in garbageScenes)
    46.         {
    47.             var path = AssetDatabase.GUIDToAssetPath(guid);
    48.             File.Delete(path);
    49.         }
    50.     }
    51. }