Search Unity

Resolved How to run Editor testing in multiple scene

Discussion in 'Testing & Automation' started by Max_power1965, Dec 6, 2017.

  1. Max_power1965

    Max_power1965

    Joined:
    Oct 31, 2013
    Posts:
    77
    Hello guys,
    I have a quick question.
    I wrote a couple of tests using the unity test runner tool. All of them are Editor testing, so no need to run in play mode.

    Now I was wondering if is possible to run those testing in multiple scenes. Because now, when I press play, only the loaded scene is been tested, but I want to run those test on multiple scenes instead of manually switch scene every time.

    Maybe by using some terminal tool is possible to tell Unity to do us... any ideas?

    Thanks
     
  2. tonemcbride

    tonemcbride

    Joined:
    Sep 7, 2010
    Posts:
    1,015
  3. Max_power1965

    Max_power1965

    Joined:
    Oct 31, 2013
    Posts:
    77
    Thanks, but how do I use it?
    Assume I have 10 scene to test, I load one scene and the test starts automatically. But when the test is finished, how do i move to next scene to test?

    I need something that can iterate through a list of scenes and run the test in each scene.

    Thanks.
     
  4. tonemcbride

    tonemcbride

    Joined:
    Sep 7, 2010
    Posts:
    1,015
    I'm not sure how you're running your tests. Do you use an editor script to do that or some kind of external tool?
     
  5. Fido789

    Fido789

    Joined:
    Feb 26, 2013
    Posts:
    343
    You can use ValueSource attribute like this:

    Code (CSharp):
    1.     public class LevelValidationTests
    2.     {
    3.         public static IEnumerable<string> LevelTestCases
    4.         {
    5.             get { return new List<string> {"Level-1", "Level-2"}; }
    6.         }
    7.  
    8.         [UnityTest]
    9.         public IEnumerator LevelIsValid([ValueSource("LevelTestCases")] string levelName)
    10.         {
    11.             yield return LoadLevel(levelName);
    12.  
    13.             Assert.IsTrue(true);
    14.         }
    15.    
    16.         [TearDown]
    17.         public void UnloadLevel()
    18.         {
    19.             SceneManager.UnloadSceneAsync(sceneToUnload);
    20.         }
    21.  
    22.         private string sceneToUnload;
    23.  
    24.         private IEnumerator LoadLevel(string levelName)
    25.         {
    26.             sceneToUnload = levelName;
    27.  
    28.             var loadSceneOperation = SceneManager.LoadSceneAsync(levelName);
    29.             loadSceneOperation.allowSceneActivation = true;
    30.  
    31.             while (!loadSceneOperation.isDone)
    32.                 yield return null;
    33.         }
    34.     }
    The code should run fine in both edit and play modes.

    Also scene names doesn't have to be hardcoded, LevelTestCases can contain any code you wish. I am for example getting my scene names from scriptable object.
     
    Last edited: Dec 7, 2017
    Whatever560 likes this.
  6. Max_power1965

    Max_power1965

    Joined:
    Oct 31, 2013
    Posts:
    77
    Thanks again for the answer
    I run my test through an editor script, here a simple example class:

    Code (CSharp):
    1. using UnityEngine;
    2. using NUnit.Framework;
    3.  
    4. public class PlayerTester
    5. {
    6.  
    7.     [Test]
    8.     public void OnlyOnePlayerInTheScene()
    9.     {
    10.         Assert.IsTrue(GameObject.FindGameObjectsWithTag(ProjectIds.TagIDs.Player).Length == 1, "There are " + GameObject.FindGameObjectsWithTag(ProjectIds.TagIDs.Player).Length + " player in the scene");
    11.     }
    12.  
    13.     [Test]
    14.     public void OnlyOnePlayerMoving()
    15.     {
    16.         Assert.IsTrue(GameObject.FindObjectsOfType<PlayerMoving>().Length == 1, "Error: There are " + GameObject.FindObjectsOfType<PlayerMoving>().Length + " player in the scene");
    17.     }
    18. }
    19.  
    Then I use test runner in edit mode to run the test

    Any idea how to run this code in every scene and in case something fails, how to know which one is the scene who failed?
    Thanks again
     
  7. Max_power1965

    Max_power1965

    Joined:
    Oct 31, 2013
    Posts:
    77
    Thanks for the answer, I see your point on load one or more scene, but the question is how can I run my tests on the scenes that I've loaded?
    I have 5 classes for example, and each one performs some kind of test. I want to load a scene and run those test, then if everything is all right, move to the next scene an do the same.
     
  8. Fido789

    Fido789

    Joined:
    Feb 26, 2013
    Posts:
    343
    Can't you change your flow and run one test on every scene and then move to the next test?
     
  9. Max_power1965

    Max_power1965

    Joined:
    Oct 31, 2013
    Posts:
    77
    Assume I have 50 scenes, is an incredible waste of time to run all the tests manually XD.
    There must be a way to automate the workflow...
     
  10. Fido789

    Fido789

    Joined:
    Feb 26, 2013
    Posts:
    343
    You CAN run all the tests on all the scenes automatically using the solution I sent you. You just have to at the start of each test to load a scene and after the test is finished to unload it. And that is all right, because each test should be atomic.

    test runner.png
     
  11. Max_power1965

    Max_power1965

    Joined:
    Oct 31, 2013
    Posts:
    77
    Hello,
    si I finally manage to try your code, but I have an error:

    EditMode test can only yield null
    UnityEditor.EditorApplication:Internal_CallUpdateFunctions()


    Maybe is not possible in edit mode?
    I just found this article: https://blogs.unity3d.com/2017/08/18/verifying-the-scripting-docs-fun-with-editortests/

    Maybe we have to use the [TestCaseSource] directive
     
    Last edited: Dec 11, 2017
  12. Fido789

    Fido789

    Joined:
    Feb 26, 2013
    Posts:
    343
    It should be possible. There is nothing in my code that yields anything else than null. Could you post your test code?
     
  13. Max_power1965

    Max_power1965

    Joined:
    Oct 31, 2013
    Posts:
    77
    Found the solution, here an example that works in editor mode:

    Code (CSharp):
    1. public static readonly string[] ScenesToTest = new string[]
    2.    {
    3.         "Assets/SquareMaze/Scene/TestScene/TestGameScene.unity", "Assets/SquareMaze/Scene/TestScene/TestGameScene2TMP.unity"
    4.    };
    5.  
    6. [TearDown]
    7.     public void UnloadLevel()
    8.     {
    9.         EditorSceneManager.UnloadSceneAsync(sceneToUnload);
    10.     }
    11.  
    12.     string sceneToUnload;
    13.  
    14.     void LoadLevel(string levelName)
    15.     {
    16.         sceneToUnload = levelName;
    17.         EditorSceneManager.OpenScene(sceneToUnload);
    18.     }
    19.  
    20. public void ObjectAreCenteredInCell([ValueSource("ScenesToTest")] string levelName)
    21. {
    22.         LoadLevel(levelName);
    23.        ///DO YOUR TESR HERE
    24. }
     
  14. Fido789

    Fido789

    Joined:
    Feb 26, 2013
    Posts:
    343
    Ok, glad you got it working.
     
  15. benbenmushi

    benbenmushi

    Joined:
    Jan 29, 2013
    Posts:
    33
    Note that this code:
    Code (CSharp):
    1.         public class LevelValidationTests
    2.         {
    3.             public static IEnumerable<string> LevelTestCases
    4.             {
    5.                 get { return new List<string> {"Level-1", "Level-2"}; }
    6.             }
    7.    
    8.             [UnityTest]
    9.             public IEnumerator LevelIsValid([ValueSource("LevelTestCases")] string levelName)
    10.             {
    11.                 yield return LoadLevel(levelName);
    12.    
    13.                 Assert.IsTrue(true);
    14.             }
    15.      
    16.             [TearDown]
    17.             public void UnloadLevel()
    18.             {
    19.                 SceneManager.UnloadSceneAsync(sceneToUnload);
    20.             }
    21.    
    22.             private string sceneToUnload;
    23.    
    24.             private IEnumerator LoadLevel(string levelName)
    25.             {
    26.                 sceneToUnload = levelName;
    27.    
    28.                 var loadSceneOperation = SceneManager.LoadSceneAsync(levelName);
    29.                 loadSceneOperation.allowSceneActivation = true;
    30.    
    31.                 while (!loadSceneOperation.isDone)
    32.                     yield return null;
    33.             }
    34.         }
    35.  
    doesnt work with Unity 5.6.4f1
    but works fine with Unity2017.3
     
unityunity