Search Unity

Question Is it possible to pause a test and explore the scene in the editor?

Discussion in 'Testing & Automation' started by AcademyOfFetishes, Mar 3, 2019.

  1. AcademyOfFetishes

    AcademyOfFetishes

    Joined:
    Nov 16, 2018
    Posts:
    219
    I can't figure out a way to pause a test while it's running and explore the scene in the editor. If I could do this, I think I could have saved 16 hours of time on this project so far. Is there any way to do this?

    Whenever a [UnityTest] gives me an unexpected error, I could save a bunch of time if I could just look at the scene and figure out what's going on. Someone taught me one trick:

    Code (CSharp):
    1. while (true) yield return null;
    This loop will allow me to see what's happening in the game view. I can put that right above a failing assertion and see what's rendered on the screen. That helps in certain situations, but in other situations, nothing beats being able to explore the hierarchy and look at the inspector. Is there any way to do that?
     
    a436t4ataf likes this.
  2. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,334
    I guess you could run the unit test method from an editor script instead of through the test runner?
     
    AcademyOfFetishes likes this.
  3. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,657
    So, you can kinda do this in Playmode tests today, by calling Debug.Break() in your test. It's fiddly though - the break will only take effect at the next 'yield', and the test needs to not conclude before that happens (either from an assertion failure, or from reaching the end of your method). This test does sort of what you're looking for:

    Code (CSharp):
    1.         [UnityTest]
    2.         public IEnumerator DirtyingUnityTest()
    3.         {
    4.             new GameObject("Some scene object");
    5.             yield return null;
    6.  
    7.             try
    8.             {
    9.                 Assert.Fail();
    10.             }
    11.             catch (AssertionException ex)
    12.             {
    13.                 LogAssert.ignoreFailingMessages = true;
    14.                 Debug.LogException(ex);
    15.                 Debug.Break();
    16.             }
    17.  
    18.             yield return null;
    19.         }
    When I run this as a Playmode test inside the Editor, it pauses and I'm able to inspect "Some scene object" in the Inspector. This is clearly pretty messy, though!

    For Editmode tests, sadly the same trick doesn't work. Perhaps you could temporarily mark your tests as playmode tests (by just enabling them to run on at least one more platform) and use this trick.

    I think it's a good idea, though! We should build 'pause on error' functionality into the framework directly...
     
    Xarbrough, akuno, neilsarkar and 2 others like this.
  4. AcademyOfFetishes

    AcademyOfFetishes

    Joined:
    Nov 16, 2018
    Posts:
    219
    That's an interesting idea. But if I do that, wouldn't the scene be all messed up from the changes in my test?
     
  5. AcademyOfFetishes

    AcademyOfFetishes

    Joined:
    Nov 16, 2018
    Posts:
    219
    I've never successfully got a playmode test to work. You're saying all I have to do is check one more of these?


    I think I'd have to uncheck the Editor box, too, but I'm not sure.

    I would love that, though all I really would need is
    Debug.Break();
    to work in edit mode tests. Or, if there were some way to say, "don't reset the scene after the test" that would work, too. I could just put a return statement right before the failing assert. Of course, I'd need a way to clean up the mess I've made when I'm done inspecting the situation. I'm happy to create a feature request around this if that helps.
     
    a436t4ataf likes this.
  6. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    1,933
    8 months later ... did this make it into a future build (it's not in 2019.2 - or am I looking in the wrong place?)

    Also, incidentally ... Unity deleted the TestRunner docs in 2019.2 onwards ( I'm using the 2019.1 archived version here - https://docs.unity3d.com/2019.1/Documentation/Manual/testing-editortestsrunner.html ) although the 2019.1 docs seem to be valid for my experience in 2019.2
     
  7. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    I'd also be interested in a feature that allowed inspecting the scene state for failed tests.

    I used Superpig's suggestion to debug my test, but it's really fiddly setting this up for edit mode tests. I feel like most of the time it would be enough to inspect the scene after a test has finished if that makes it easier to implement.
     
  8. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,657
    Nope, we've not looked at this internally yet.

    They moved to the package docs here: https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/index.html
     
    Xarbrough likes this.
  9. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    1,933
    Well, it's now 18 months since I asked the question :), and I stopped using EditorTests entirely - I don't think anyone else I work with uses them either. Without basic functionality like this, there's not much point to them (yes, they're much faster to startup, but ... as soon as one fails, you then spend more time than you saved in total, by trying to debug the un-debuggable).

    I hate having to do this multiple times an hour, but its still quicker to use Superpig's method and change:

    Code (CSharp):
    1. [Test]
    2. public void TestAThing()
    3. {
    4.    // do stuff
    5.    // do more stuff
    6. }
    to:

    Code (CSharp):
    1. [[B]Unity[/B]Test]
    2. public [B]IEnumerator[/B] TestAThing()
    3. {
    4.    // do stuff
    5.    [B]Debug.Break();
    6.    yield return null;[/B]
    7.    // do more stuff
    8. }
    (and change it back again after you fix the test ... otherwise your testrunner will pause every. Time. You. Run. Any. Test ...than to waste time using your mental powers to try and guess what has changed/broken in a failing test.

    From what I recall ... the current plan is to make Play tests run super fast, implying EditorTests are going to be deleted / obsoleted anyway? I don't mind, so long as we get something that works, and I can stop having to delete/undelete those 3 changes over and over again.

    (it's possible to convert everything to UnityTests, and have a "yield break;" at the end of every test - and if I had a large team of first-time junior engineers who'd never learned to unit testing before, I'd probably force them to go that way - but then you have to teach everyone the subtelties that 'everything is now a coroutine' and teach them to stop using the standards for test control ([Setup] etc), otherwise things break in non-obvious ways. ... and you still need to keep manually adding two lines everytime a test fails, and then deleting them as soon as it passes.)
     
  10. RamzaB

    RamzaB

    Joined:
    Nov 4, 2012
    Posts:
    42
    My workaround was to save the scene created during the test into a file, and open it after.
    Sounds silly, but it worked for me.

    Code (CSharp):
    1. bool saveOK = EditorSceneManager.SaveScene(EditorSceneManager.GetActiveScene(), "Assets/test.unity");
     
    UnLapinDeBois, gwelkind and Xarbrough like this.