Search Unity

Feedback Why the regressions to Test Runner in Unity 5.6?

Discussion in 'Testing & Automation' started by CanisLupus, May 18, 2017.

  1. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    We have recently moved our project from Unity 5.5.0 to Unity 5.6.1 and noticed the Test Runner was changed and now supports play mode tests and updates the results as it executes. That's great! However, we also noticed that:

    1. Tests are running much slower now. For few lengthy tests, about 3 times slower (of course it depends); for many quick tests it's actually much worse by orders of magnitude (see post below).
    2. There are no icons to select only tests that failed or that were not run yet. This was very useful!
    3. There is no number that counts the number of tests (just a detail).

    Why these regressions? Will those be fixed in the future?

    Thanks and cheers,
    Daniel
     
    Last edited: Jun 14, 2017
  2. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    It seems that due to the results updating while executing, there is a large interval between running each test. If you have 1000 very quick tests they will now take a very long time, whereas before they finished almost instantly.

    In fact, I tried this with 1000 tests that look like this:
    Code (CSharp):
    1. [Test] public static void Test1() { Assert.True(true); }
    2. [Test] public static void Test2() { Assert.True(true); }
    3. [Test] public static void Test3() { Assert.True(true); }
    4. [Test] public static void Test4() { Assert.True(true); }
    They took about 21s on my machine (i7-3632QM), having only trivial code as seen above.

    Most of our actual test scripts went from taking a few dozen ms each in Unity 5.5 to taking hundreds of ms in 5.6. And as I've just now optimized them, they should run in a few ms but take about 6 seconds total.

    Can we return to the old test scheme somehow? As it stands this is having a hit on our development time.

    - Daniel
     
    Last edited: May 20, 2017
  3. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    Just reported this as a bug. At the moment I have no suggestions for other people running 5.6... :( I haven't tried to get old versions of the unit testing tools into our project. Maybe that could work it in the meantime...?

    EDIT: just tried the tests in Unity 2017 (2017.1.0b6) and it's even worse. Hangs for a while before even starting to run the tests, depending on how many you have. Also reported that. D:
    UPDATE: for the hang problem in Unity 2017, it had already been reported by someone else: this is the page. The general slowness in 5.6 is still a problem, however.
     
    Last edited: Jun 14, 2017
  4. catplaystudios

    catplaystudios

    Joined:
    Oct 7, 2015
    Posts:
    1
    I would like to note that I have the exact same problem. We have around 1400 unit tests in our project and recently upgraded from 5.3.41f to 5.6.1f. The unit tests that formerly took a few seconds now take over a minute (!!) to run. :(
     
  5. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    Ouch! That's even worse than our case. :( Let's hope this gets fixed. If I end up trying an alternative I'll let everyone here know.
     
  6. sybo_jon

    sybo_jon

    Joined:
    May 29, 2017
    Posts:
    1
    I'm seeing the same problem. As soon as you start using just a few [Values(...)] attributes or similar, as the amount of permutations grows it becomes unbearably slow.

    It's easy to see this with the following test:
    Code (CSharp):
    1. using NUnit.Framework;
    2.  
    3. public class PerformanceTest
    4. {
    5.     [Test]
    6.     public void ManyTests(
    7.         [Range(0,1000)] int testInt)
    8.     {
    9.         Assert.IsTrue(true);
    10.     }
    11. }
    This is just 1000 executions that should be sub-millisecond each, but in Unity 5.6 this takes ~18 seconds for me.

    Also, the lack of filters is super annoying -- we used the category filter all the time.
     
  7. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    281
    Also, is there any way of starting the test runner from script? I just recently implemented ITestRunnerCallback / Batch.RunTests() into our build script, it worked nicely and allowed us to check the unit tests on each build. All those classes appear to have been removed in 5.6, is there any replacement for this use case (running tests from scripts)?
     
  8. steinjak

    steinjak

    Joined:
    Jan 31, 2017
    Posts:
    3
    We're facing the slowness issue as well, but it seems to vary between machines, even after clean installs of Unity (including deletion of all settings).

    In addition, we've noticed the following regression: The unity test runner no longer treats NUnit's Explicit attribute correctly. These tests are supposed to be run if selected by a category, but now they only run if executed individually - both in batch mode (build server) and in the editor. This is the same issue as noted by @sybo_jon above, about the category dropdown disappearing from the test runner GUI. For us, this is a big deal, as we have a significant number of tests that are unity-independent and run in the IDE, so we need a way to run different sets of tests in different environments.
     
  9. LavaPatrik

    LavaPatrik

    Joined:
    Feb 21, 2017
    Posts:
    4
    I just want to say that we're experiencing the exact same problems. Its also very annoying that you can't alt-tab away from Unity while the test runner is running (it will stop and wait until you tab back).

    We've gone from being able to run all our tests in a few seconds, to it taking 5-8 minutes.

    I've noticed that this has increased the amount of failing tests, since developers don't want to wait that long for them to run.

    I think its time for Unity to tell us what the plan is, and when we can expect improvements.
     
  10. steinjak

    steinjak

    Joined:
    Jan 31, 2017
    Posts:
    3
    Another one: The XML format generated by the test runner is no longer valid NUnit XML, because the required root element is missing. Build servers tend to love that...
     
    einWikinger likes this.
  11. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    281
    Anybody at Unity care to comment? I think @Tomek-Paszek was doing related work..?
     
  12. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    The comments everyone is writing just make me realize this is even worse than I thought! Would also really appreciate if someone from Unity could talk about this a bit.
     
  13. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    673
    I reported that months ago in a 5.6 beta and it's still active... I had to workaround it in Jenkins rewriting output XML...

    It's annoying that something so easy to solve takes months to fix.
     
  14. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    673
    I also think that Unity should create a subforum for test runner because this threads get lost...
     
  15. deemen

    deemen

    Joined:
    Dec 29, 2013
    Posts:
    17
    The new test runner also doesn't support tests in managed plugins (DLL assemblies). We use DLLs exclusively for our project and this is making our tests totally unusable.

    It's clear to me the new Test Runner is not quite ready for prime-time. Can someone from Unity comment about this and potentially revert back to the old Test Runner somehow?
     
  16. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    673
    Also documentation regarding batch mode is outdated and I had to submit bug report to learn the undocumented new parameters for play mode tests.
     
  17. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Ad 1. We understand the tests became slower. The reason behind it is because we changes the way the runner is executing tests. Before it used to run all tests in one frame. We improved the runner to make it possible to yield/skip frames from tests and therefor we had run each test in a different frame (which runs in the same framerate as EditorApplication.update). We will try to look at improving the execution time at some point while keeping all the new features.
    Ad 2 and 3. Lack of UI filter is a regression. We simply didn't have enough resources to focus on reimplementing that feature. It's again something we will hopefully fix.

    There is no public API for it but you can reflect the Batch class and invoke it like that (keep in mind the API may change)

    We are also aware of this issue. For now I suggest postprocessing the result file before parsing it.

    That, again, is on our TODO list.

    Apologies for regressing in funcionality in some of the areas but we needed to rewrite some parts of the runner in order to make it even better and we simply lack resources to fix it on time. Personally I don't work on the test runner anymore but I have notified people that will look into the problems.
     
  18. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    First of all, thanks for your reply. :)

    I thought that was the case, but it really is a huge difference for non-frame dependent tests (or non-Unity-dependent tests, for that matter). Though I realize the following may be a little maintenance nightmare, I suggest adding a toggle so that we can run tests sequentially and without breaks, using the old system (while the new one isn't up to par).

    Secondly, do you know of a way that we can, right now, run the tests in the old Editor Test Runner - or similar tool - while keeping Unity 5.6? I believe everyone in this thread would like to know this. :) Anything Unity Technologies can provide in separate, like the old tests module itself?

    Thanks!
     
    Last edited: May 22, 2019
  19. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    281
    Thanks for replying. I could use the Batch-class with reflection but what about the callback interface, is that still around somewhere? In 5.5 I had a nice progress dialog that listed the number of tests that succeeded, failed etc. so I need the callback to be able to do that.
     
  20. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    673
    @Tomek-Paszek reading you I'm thinking that there is no dedicated team working on test runner because you broke so many things months ago and there are still broken...
     
  21. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Unfortunetlly it's not possbile. Updating the test runner came with quite a few internal changes and nunit version update.

    It hasn't been exposed either which means we don't support it but... see my previous answer :)

    How many people you can count as a team :)?
     
  22. peterhorsley

    peterhorsley

    Joined:
    Jun 26, 2017
    Posts:
    2
    the performance regression in the test runner is huge, we do need a workaround or for this to be fixed in an upcoming beta, as it affects our ability to deliver high-quality products with Unity. Can this please be prioritized?

    EDIT: Just to be clear, the tests themselves actually run quickly, but the problem we are seeing is that they take a very long time to actually start running (from the time 'Run All' button is clicked) - Unity.exe takes 100% CPU of one core for a long time, before showing the modal 'Running tests' dialog. For example, we have 477 unit tests, which take 10 seconds to run, but Unity.exe hangs at 100% cpu for 4 mins and 30 seconds prior to executing them. This is on Windows 7 and Mac using Unity 2017.1.0b1 (64 bit). It happens when we select to run all tests or even just 1.
     
    Last edited: Jun 26, 2017
    Whatever560 likes this.
  23. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Hi @peterhorsley
    Can you please file a bug report for that issue? Do you have any special test setup in the code? I didn't manage to reproduce the problem. Also, which Unity version are you using?
     
  24. peterhorsley

    peterhorsley

    Joined:
    Jun 26, 2017
    Posts:
    2
    Thanks @Tomek-Paszek, we are using Unity 2017.1.0f1 (released a few days ago) but in fact it turned out it was our bug. We are using nUnit's TestCaseSource attribute to generate a lot of tests which yield return each test case, and i discovered that some code that deserializes a large json file was being run during creating of every test case. So instead I just made a static field with the deserialized result so that it was only done once, and the 4.5 mins reduced to about 15 seconds. Sorry for the false report.
     
  25. Heathro

    Heathro

    Joined:
    Mar 3, 2016
    Posts:
    3
    This new test runner was clearly rushed out. It should not have been released with so many regressions. Very disappointing and causing headaches for many teams. Not having enough resources is no excuse for releasing something so broken that is vital to many teams. :mad:
     
    mh114 likes this.
  26. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,265
    Could you guide me how to access the Batch class via reflection? I did something like this and could not find either Batch class or UnityEditor.EditorTests

    Code (CSharp):
    1.         Assembly unityEditorAssembly = Assembly.Load("UnityEditor");
    2.  
    3.         StringBuilder sb = new StringBuilder();
    4.         foreach (Type type in unityEditorAssembly.GetTypes())
    5.         {
    6.             sb.AppendLine(type.FullName);
    7.         }
    8.         Debug.Log(sb.Length);
    9.  
    10.         EditorGUIUtility.systemCopyBuffer = sb.ToString();
     
  27. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    281
    I haven't actually tried yet, with more important tasks at hand. But I was going to use ILSpy to find out where the classes were hiding and work from there. Trying it now, I can't seem to find them either. :( Maybe @Tomek-Paszek can point us in the right direction?
     
    5argon likes this.
  28. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    Wait, the massive delay before running any number of unit tests hasn't been fixed for the final release of Unity 2017.1?

    In our project I click a test and it takes almost one minute of high CPU thrashing before it runs any test (I have no idea what Unity is doing with it). Then it happily says that tests took a few hundred ms themselves. I reported it during the 2017.1 beta, and it was apparently already reported by someone else during beta version b1, here:

    https://issuetracker.unity3d.com/issues/editortest-running-a-single-edit-mode-test-is-really-slow

    I'm baffled, really. The Test Runner is completely unusable now. This is why we shouldn't update to new versions, really... Why even have betas if critical problems are left unfixed for months?

    Sorry, I'm really mad at this.
     
    Last edited: May 22, 2019
  29. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @CanisLupus
    Hi Canis,
    I understand your frustrations. Sorry, for making you wait for a fix so long. The issue has been fixed but not backported to 2017.1 yet. I will see what I can do.
    Tomek
     
  30. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    Hi Tomek,

    Thank you for replying to my angry venting moment. :| Personally, we don't need play mode tests, so seeing everything getting worse performance (in this case a bug) each version is really maddening for me. When you have more news about this please do tell us.

    Cheers,
    Daniel
     
  31. einWikinger

    einWikinger

    Joined:
    Jul 30, 2013
    Posts:
    31
    Just to chime in, the issues still haven't been fixed. I'm like filing bug reports since May or so, but nothing has improved so far. We had to go for weeks without getting test reports back from our CI until I took my time to postprocess the results XML file in a sensible way. We have about 6000 (edit-mode) test cases and running those in the editor just takes ages... (2017.1.0p1).

    Also: https://forum.unity3d.com/threads/unity-test-runner-batch-mode-broken-workaround.482145/
     
    CanisLupus likes this.
  32. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,265
    Yeah, it is a shame that I am just getting into testing practice too. Currently I have to stick to only PlayMode test because I can afford to wait in that mode. I wonder when will the EditMode test become sensible again. (Did you tried 2017.2 beta?)
     
  33. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @einWikinger @5argon A performance issue was addressed and backported to 2017.1 p1. Can you verify please with an updated version? If it's still an issue, can you please send me the bug id for it?
     
    5argon likes this.
  34. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    I suspect the issue is still the original problem that made me create this thread, which is that the new Tests Runner is very inferior to the old one, performance- and feature-wise. Where tests ran almost instantly before, now there's a few frames between each one. In a project with 6000 tests this is now probably an insane wait (likely takes minutes).

    I haven't tested p1, but it says this: "Editor: Fixed a performance regression that cause very long startup time when invoking a test run." This seems related to the bug that appeared in 2017.1; not the runner regression.
     
  35. spfund

    spfund

    Joined:
    Sep 4, 2017
    Posts:
    1
    We also had the problem that the test runner doesn't support tests in DLL assemblies and made a testproject for this case. It's reported under Case 948723.
     
  36. MHatfull

    MHatfull

    Joined:
    Aug 23, 2017
    Posts:
    7
    @Tomek-Paszek we have recently moved to TDD however we require to upgrade our unity project to use 2017 (we were previously using 5.4.5). I've just spent nearly a week upgrading all our various plugins to play nice together and with our project. I've now been fixing our tests (having finally made the thing compile). I double click a test, wait 3 mins and then it runs. The test took a few ms. We are using 2017.1.0f3. Should I be using a different version? This performance hit will seriously hit our team where it hurts.
     
  37. dtaTrifork

    dtaTrifork

    Joined:
    Aug 21, 2017
    Posts:
    3
    We used to utilize 'Application.temporaryCachePath' for some of our unit tests, this also seem to have regressed. Is unit tests not performed on the main thread any more?
     
  38. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    Any news on this, @Tomek-Paszek? I was needing to do more tests and it all takes soooo much time to run (using 2017.1 and 2017.2 alike).

    This thread was created 6 months ago, and apart from Unity fixing the performance bug introduced in 2017.1.0, nothing seems to have changed in this regard (to my knowledge, since I haven't tried the 2017.3 beta). How is this issue going?

    Cheers!
     
  39. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    As far as I know no work was done to optimise this. If I remember correctly it happens because we started to run reach test in a separate frame. Maybe @HaraldNielsen can chip in with more information?
     
  40. mh114

    mh114

    Joined:
    Nov 17, 2013
    Posts:
    281
    @Tomek-Paszek What about the Batch-class and ITestRunnerCallback, I couldn't find them to use with reflection (as you suggested)? Where are the classes these days? They shouldn't have been removed if there is no replacement..
     
  41. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    98
    Simply to reply to this as we are using unity latest LTS 2017.4.1f1 and it seems have been not fixed.

    Running tests is pretty quick in itself though there are still 2 big heavy issue :
    - Startup always takes around 15s each time. (before the progress bar appearing)
    - After test are run (we can see results in the window) unity hangs for 60s !

    This is a pretty unbearable situation, this is reproductible whatever number of test we are running, even 1.

    60 runs of test in a day equals 1h15 minutes of work lost hanging. Right now its one of our most major caveat.

    As you are working on the LTS it should really get fixed for it to be production ready
     
    Last edited: Apr 12, 2018
  42. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    98
    As additional info. When hitting the cancel button (for instance when making an infinite yield return null test) the 60s of hanging are still there.
     
  43. CanisLupus

    CanisLupus

    Joined:
    Jul 29, 2013
    Posts:
    257
    I haven't tested 2017.4 yet, but the general slowness and waiting frames between tests is still there.

    Now there's a progress bar so you can't even do anything else in the editor, but that's okay. However, if that is the case why not run every Edit mode test at once in the same Editor frame and skip all the waiting? Ever since Unity 5.5 the performance of Edit mode tests never recovered. It ran almost instantly and now takes a very long time.

    Maybe give us the option of running a configurable number of tests before updating the progress bar, such as 10, 100 or infinity. So if you want to see the progress bar you can choose, but we value our time much higher. :)

    Daniel
     
  44. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,265
    There is a regression again in transition from 2017.3.1 to 2017.4/2018.1 beta. EditMode test has a very long start up and tear down time. Running multiple tests is slow only before the first test and after the final test. By slow it is like 5-10 seconds, depending on how large your project is. (I think it depends on how many scripts you have, even if they are not related to the tests) This make rapid iteration impossible and I would rather test manually.

    PlayMode test is not affected by this. I have 2 related bug report submitted with no reply yet
    https://fogbugz.unity3d.com/default.asp?1010794_q8u3dlr4095r6dqm
    https://fogbugz.unity3d.com/default.asp?1021076_uhcsjltr5gpv1g97
     
    CanisLupus and Whatever560 like this.
  45. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    98
    Our issue is definitely this one. As our project has a lot of scripts it may be what takes 60s for tear Down. Please we need a fix
     
  46. unity_sxF4kWUhnH3bzw

    unity_sxF4kWUhnH3bzw

    Joined:
    Feb 16, 2018
    Posts:
    6
    Another "still an issue" issue with the testRunner is the lack of automation. The "run on recompile" option was removed and has not been restored. The only reference I see to this issue is this thread, now nearly a year old, stating that the option would be restored soon. There's a suggestion in the thread to reimplement it in an editor script... except there's absolutely no reference as to what API calls can be used to run the tests in a script! I've been digging around in the available source code and the assembly browser but I can't find anything useful for this purpose.