Search Unity

How to pass PlayMode test early instead of waiting for timeout?

Discussion in 'Testing & Automation' started by Xarbrough, Jun 3, 2019.

  1. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    375
    In NUnit we can simply call Assert.Pass, but Unity does not seem to handle the SuccessException. Is there anything I need to configure to make this work or is it simply not supported?

    Are there other ways of ending a coroutine test during PlayMode early and count the test as passed?
     
  2. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    17
    Hey. How is your test structured? Normally we do handle Assert.Pass etc, even in playmode. But if you start up your code in a separate coroutine, then the test framework does not get the exception.
     
    Xarbrough likes this.
  3. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    375
    I can post the complete test later, but for now, maybe I already found the issue: I have been calling Assert.Pass within a lambda callback hooked up to the SceneManager.sceneLoaded event. I want to test if a special scene is actually loaded within a timeout of e.g. 5 Seconds. Instead of having to wait the entire time, I wanted to pass the test early when I receive the loaded event with the correct scene name, but maybe this is the reason why it did not catch the exception.
     
  4. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    375
    Here is simplified version of the issue:

    Code (CSharp):
    1. public class SceneLoaderTests
    2. {
    3.     [UnityTest]
    4.     public IEnumerator DummyTest_LoadSceneAsync_MainSceneIsLoaded()
    5.     {
    6.         // When a scene was loaded, check if we can quit the test early.
    7.         SceneManager.sceneLoaded += (scene, mode) =>
    8.         {
    9.             // If the correct scene was loaded, simply stop the test successfully.
    10.             // However, this does not work and only logs "SuccessException",
    11.             // which fails the test.
    12.             if (scene.name == "Main" && mode == LoadSceneMode.Single)
    13.                 Assert.Pass();
    14.         };
    15.  
    16.         SceneManager.LoadSceneAsync("Main");
    17.  
    18.         // We can wait the full timeout duration and assert, which works
    19.         // but wastes a lot of time waiting for nothing.
    20.         yield return new WaitForSeconds(5f);
    21.         Assert.IsTrue(SceneManager.GetSceneByName("Main").isLoaded);
    22.         yield return null;
    23.     }
    24.  
    25.     [UnityTest]
    26.     public IEnumerator DummyTest_SimplePass()
    27.     {
    28.         // This test passes as expected.
    29.         Assert.Pass();
    30.         yield return null;
    31.     }
    32. }
    The DummyTest_SimplePass demonstrates, that the Assert.Pass call indeed works in this simple case, however in the test above, when called within the SceneManager.sceneLoaded callback, it does not pass the test. Instead, it logs the SuccessException and fails the test.

    SuccessException
    NUnit.Framework.Assert.Pass (System.String message, System.Object[] args) (at <59819be142c34115ade688f6962021f1>:0)
    NUnit.Framework.Assert.Pass () (at <59819be142c34115ade688f6962021f1>:0)
    Tests.SceneLoaderTests+<>c.<DummyTest_LoadSceneAsync_MainSceneIsLoaded>b__0_0 (UnityEngine.SceneManagement.Scene scene, UnityEngine.SceneManagement.LoadSceneMode mode) (at Assets/PlayModeTests/SceneLoaderTests.cs:17)
    UnityEngine.SceneManagement.SceneManager.Internal_SceneLoaded (UnityEngine.SceneManagement.Scene scene, UnityEngine.SceneManagement.LoadSceneMode mode) (at C:/buildslave/unity/build/Runtime/Export/SceneManager/SceneManager.cs:244)
     
  5. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    268
    You should use this instead of that "WaitForSeconds(5f)".

    Note that with the below message, the call to replace that would be:
    Code (CSharp):
    1.  
    2. yield return StartCoroutine(WaitFor(() => SceneManager.GetSceneByName("Main").isLoaded), 5f);
    3.  
    This will wait for up to 5 seconds for the main scene to be loaded. Or increase the timeout value, but it offers a dynamic wait. Please note that this is part my Trilleon Automation/Unit Test Framework so there will need to be some edits if you use the following code directly in your current framework.

    If you switched to Trilleon for your unit testing and automation, none of this will be a problem at all for you. Give it a try. Compare it to what you are using now. I think it will be much easier.

    Code (CSharp):
    1.         /// <summary>
    2.         /// Waits for a single value to not be null or default. Waits for a list of values to return more than 0. Waits for object(s) to be active, visible, and interactable.
    3.         /// </summary>
    4.         /// <param name="propertyExpression">Lamba expression representing the check you wish to perform with each iteration of the loop. This is syntactically as simple as "() => SomeCondition && SomeOtherCondition", for example.</param>
    5.         public IEnumerator WaitFor(Func<bool> condition, string optionalOnFailMessage = "", float timeout = TIMEOUT_DEFAULT, params int[] testCaseIds) {
    6.  
    7.             PreCommandCheck();
    8.  
    9.             isDeepDive = true;
    10.             float timer = 0;
    11.             while(!condition.Invoke() && timer <= timeout) {
    12.  
    13.                 yield return StartCoroutine(WaitRealTime(1f));
    14.                 timer++;
    15.  
    16.             }
    17.             if(timer > timeout) {
    18.  
    19.                 if(!isTry) {
    20.  
    21.                     yield return StartCoroutine(Q.assert.Fail(string.IsNullOrEmpty(optionalOnFailMessage) ? "Conditional wait (WaitFor) timed out." : optionalOnFailMessage));
    22.  
    23.                 }
    24.  
    25.             }
    26.             isDeepDive = false;
    27.             yield return StartCoroutine(Q.game.WaitForNoLoadingIndicators());
    28.  
    29.             if(!string.IsNullOrEmpty(optionalOnFailMessage)) {
    30.  
    31.                 yield return StartCoroutine(Q.assert.Pass(optionalOnFailMessage));
    32.  
    33.             }
    34.  
    35.             if(testCaseIds.Length > 0) {
    36.  
    37.                 Q.assert.MarkTestRailsTestCase(timer > timeout, testCaseIds);
    38.  
    39.             }
    40.  
    41.             PostCommandCheck();
    42.             yield return null;
    43.  
    44.         }
     
  6. Stormy102

    Stormy102

    Joined:
    Jan 17, 2014
    Posts:
    480
    I know this won't fix this issue in general but you can use a yield statement
    Code (CSharp):
    1. yield return SceneManager.LoadAsync()
    to wait until the level isn't loaded. An error will be thrown anyway if the level isn't loaded anyway e.g. build settings.
     
    Xarbrough likes this.