Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug: Unity Test Runner swallows exceptions from child coroutines

Discussion in 'Testing & Automation' started by Kleptine, Jan 12, 2021.

  1. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    274
    I've seen this bug now twice, so I thought that I would mention it. Currently if you have a coroutine like:
    Code (CSharp):
    1. public IEnumerator Parent(){
    2.  
    3.     yield return LoadScene("Foo.unity");
    4.     SceneManager.GetSceneByName("Foo").GetRoots(); // Test runner reports invalid scene error.
    5. }
    6.  
    7. public IEnumerator LoadScene(string name) {
    8.     yield return null;
    9.     Assert.AreNotEqual(name, "Foo.unity", "Please don't load foo"); // Test runner swallows this assert.
    10.     yield return SceneManager.LoadSceneAsync(name);
    11. }
    The exception thrown by the child LoadScene will be swallowed. Instead, you only get an error in Parent(), line 2, that the loaded scene is invalid.

    In the TestRunner package, this is because the function ExecuteEnumerableAndRecordExceptions only records one exception at a time. New exceptions overwrite the old exceptions recorded, rather than maintaining a list. When LoadScene() throws an exception, it doesn't stop the parent coroutine from running (which it probably should).

    This bug makes the test errors panel un-useful and means test errors (especially in a CI machine) are very hard to parse / track down.
     
    neonblitzer and booferei like this.
  2. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    53
    Kleptine likes this.
  3. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    274
    Argh, has that been open since 2018? Is anyone actively working on the testing package?
     
    dan_ginovker likes this.
  4. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    505
    Man, nested coroutine is a mess to debug ...
     
  5. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    274
    Yes, I ended up writing my own Coroutine runner and stack trace debugging system. >.>
     
  6. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    505
    [Edit: Fixed see next post]

    Would you mind sharing some thoughts here ?

    Here is some of my code :

    Code (CSharp):
    1.         /// <summary>
    2.         /// Test each game scene to check if fallback context works correctly
    3.         /// </summary>
    4.         /// <returns></returns>
    5.         [UnityTest, TimeoutDebug]
    6.         public IEnumerator DirectSceneTest()
    7.         {
    8.             yield return BaseAppScenesTest.DirectSceneTest();
    9.         }
    Code (CSharp):
    1.  public IEnumerator DirectSceneTest()
    2.         {
    3.             var dummyScene = "DummyScene.unity";
    4.             string[] scenes =
    5.             {
    6.                 "SceneA",
    7.                 "SceneB"
    8.             };
    9.      
    10.             yield return new WaitForEndOfFrame();
    11.  
    12.             foreach (var scene in scenes)
    13.             {
    14.                 yield return Addressables.LoadSceneAsync(scene);
    15.  
    16.                 Logger.Debug(nameof(DirectSceneTest), "Scene loaded Async");
    17.  
    18.                 yield return null;
    19.  
    20.                 //Wait for any on going transition to finish. This is internal magic
    21.                 yield return AppInjectorBh.CurrentExporter
    22.                              .GetExportedValue<IStateMachine>()
    23.                              .GetCurrentStateTransitionAsync().AsUniTask().ToCoroutine();
    24.          
    25.                 //Clean up by loading an empty scene and destroy things Zenject style
    26.                 yield return Addressables.LoadSceneAsync(dummyScene);
    27.                 UnityTestUtils.DestroyEverythingExceptTestRunner();
    28.  
    29.                 yield return null;
    30.             }
    31.         }
    I don't get why but when I debug, I go through the loop once without issue. But then on Scene B load, It stops on L.14 and never continues on the second yield.

    The scenes are correctly loaded but the test stops silently, the log message never gets printed, I enabled all exceptions in the debugger but nothing is thrown.
     
    Last edited: Sep 14, 2021
  7. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    505
  8. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    274
    It's probably because L14 throws an exception, which is then caught and swallowed by the coroutine runner you're using (this is running via the Test Runner?). I would:
    - Step into L14 to figure out which of the code within it throws the error.
    - Check the test output to see if you see the exception.
     
  9. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    274
    Ahh, haha. Awesome! Oof, silent hangs are rough. Should have though of that given that Addressables are very asynchronous.
     
    odaimoko and Whatever560 like this.
  10. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    505
    Yeah unfortunatelly none of it was working. It just occured to me that I was seeing some GO poping in the scene list, and after investigating on its role found out it was a part of Addressable workflow. I just wild guessed it was the issue. And it was.