Feedback Test runner API questions / feedback

I am going over the documentation for Unity Test Runner and some things aren't totally clear to me. here's my feedback:

1. Play mode vs. edit mode tests
What is the technical distinction between the two types? what is allowed to be done in edit mode which is not allowed in play mode tests ? (and vice versa). i also do not understand the motivation for selecting either type (when should i use each of these?)

In Unity test tools (which is now deprecated), unit tests could not perform certain things (e.g: create new game objects). They were also executed as editor code (e.g: not in "play mode") so they could not start coroutines and such things.

With these 2 new test types, it's not clear anymore (to me).

2. Editor restart after enabling play mode tests
After enabling play mode tests, the editor must be restarted. kind of a nitpick, but that's pretty annoying, and i don't recall any other feature that behaves the same. wasn't there any other solution for this? (why is it necessary anyway ?). Also, it is mentioned that this adds some overhead to the game at runtime - can someone clarify what does this add and whether i should actually worry about enabling this? (if it adds 10kb to the build i am totally fine with it).

3. PrebuildSetupAttribute
(what an awful name BTW) - why should i prefer this over NUnit's SetUp attribute? what is the difference between the two and which one should i prefer to use ?

4. MonoBehaviourTest
while i believe the intention was good (probably due to lots of feedback about how it's hard to test MonoBehaviour code), i don't really see how this thing could be useful for real world scenarios.Also, it "asks" you to mix your runtime (MonoBehaviour) code with your tests (IMonoBehaviourTest interface).

5. Command line execution
does this work in conjunction with -batchmode or -nographics? we would like to run editor / play mode tests from our build server. so far i haven't been able to execute it via command line.Also, it can be nice to have more control over which tests will get executed (by name, category, etc).

6. Improve the documentation
documentation on the docs page (linked above) is not clear at times. also links coming from this page to various attributes or other code related stuff is very under-documented and missing examples or includes confusing descriptions.

7. Please Open source the test runner
I think we (the dev community) can benefit from it as well as improve on it and send it back as PRs.

 

It's pretty funny that nobody replied to this important thread. I'm currently looking into TDD on Unity but there's so little to read.

 

That's highly subjective.
TDD isn't first class citizen in game development for better or for worse, that's matter of taste. Personally I don't like TDD.

 

Do you dislike TDD at all or only in Gamedev? Do you want to argue about it? I'm quite interested to know other people opinions and experience.

disclosure: I use TDD or a edulcorated version of it in all my development projects.

 

I dislike TDD everywhere.

I'm an enterprise software engineer, so in my day-time job, I write tests, after I wrote the actual code to ensure I don't mistakenly change something. But I don't write BS tests in advance.

I value the functional tests much more, since I usually work on applications, where the only thing matters is what the users experience. If the application is working properly, we're good at the moment.
This applies to the game development as well. I value QA manual tests more than artificial unit tests. (Although obviously for an API you don't really have a choice).

 

Thanks @DavidGeoffroy, some time has passed since i originally wrote this post.
I do realize now what's the different between edit and playmode tests. it seems that playmode tests are more suitable to what i'm after (as i am testing our game's components rather than some editor code that we created).

Having worked with the test framework a bit more, there are more things that i find missing, off the top of my head:

1. No editor API for running tests
The only ways i know are either from the editor UI, or by launching the editor using the relevant command line args.
We are using an automated build system so we do not manually interact with the UI. The only option we have is running Unity with the -runtests command line arg.

Our build consists of running tests first and then building the player. This means we have to launch the editor twice (once for tests, once for building the player). This slows down our build process. i wish there was an editor API (e.g: TestRunner.ExecuteTests) that will allow me to run all tests.

2. UI changes to test runner window
There are a lot of visual improvements that can be done to this window, for example:
- Show the number of tests (executes, passed, failed, etc)
- Allow grouping by categories (built-in support in NUnit using the [Category] attribute
- Better marking for ignored tests (also, it seems that ignored tests take so much time to skip over, even if they are not run)

3. Playmode tests don't work with -nographics
Not sure this is 100% correct, but our CI system runs these tests and it cannot execute playmode tests on Windows based machines, since when running them it requires access to the "desktop" since Unity cannot be run with -nographics (our build agents run as a service that doesn't have that access). Why can't i run Unity without any graphics, but only logically simulate running those tests?

4. Allow building a Playmode test build
Playmode tests can be executed on a target device, but this only works if you build and run it through the test runner UI. it could be more flexible to be able to run a complete build that will be generated to run playmode tests, and then install that build wherever i want (any device, device farm, etc).

5. All tests are run in the same "isolated" scene
When starting test execution, a new scene is created. this is often times not enough, as we'd like a "clean slate" for every test fixture or even every test that is run. It could be great to have that baked right into the framework (currently i have to implement this myself).

I have a lot of other feedback points, i really wish there were more updates about the test runner UI and framework (open sourcing it, updates, etc).

 

Maybe @ElvisAlistar can comment on any upcoming updates to the test runner?

 

Great news. Thanks for replying.
I will definitely be waiting for all of that.

 

Is there any progress? We need to have C# API to run tests from the editor. At least could you provide an example of how to use reflection to call Run All tests? This is very useful to do before any commit to the repository. Thanks.

 

@Warnecke there's a lot of obvious bugs in the test-framework docs (from typos, to broken/incorrect/impossible code examples, to missing/incomprehensible explanations/sentences, to incorrectly-tagged HTML headings/links).

Is there a way to report / correct these? Since it's outside the main docs, the normal links for "there's incorrect information here" aren't present. And there's no contact information included.

 

I just post my feedback / comments here on the forums.

 

OK. @Warnecke : Current "causes problems multiple times for at least one team member on a day to day basis" issues in the framework / docs:

1. The testRunner window keeps resizing everytime you reload assemblies. Specifically: the detail area (bottom of the window) keeps shrinking to a size that is useless: it stops at precisely the height BEFORE you can see the line number of any assertion-fail. If it has to be hardcoded to something (it shouldn't!) then at least make it a useful position .

2. Assert.AreEqual : has no override for Quaternion, Vector2/3/4 ... these are very common things to need to compare equality on, I'm not sure why you'd leave them out? We wrote our own, and it seems everyone else has to also.

3. Assert.AreNotEqual : almost useless. Since Unity Engine's architecture uses float everywhere for almost everything, this method needs to have a version that matches Assert.AreEqual's variant with the extra "delta" argument. Again: wrote our own to fix the gap.

4. TestRunner has no "right click a test name to select the class in project view" (which exists for all materials, shaders, scripts, etc in Unity, in multiple places throughout the editor). When you have a few toy tests (5, 10) then it's fine, no-one cares; but if you're actually using the Test Framework properly, on a real project, with thousands of tests, then this is a HUGE missing feature.

5. Missing docs: MonoBehaviourTest<T> ... ???? ... This thing has 3 sentences of doc that I can't understand, and no-one else seems to either, and a single code example that equally makes no sense - what is this for? Why would you do that? On forums, most people seem to make the same assumption: it's "something" related to mocking MB's, but ... how? (and is that even true? I don't think so - I think the name is misleading?)

6. The "Overview" is missing critical background information, for instance: what happens with Scenes? (Trial and error: TestRunner auto-creates a scene, which is shared for all tests within a run (or is it all tests within a namespace? Who knows!). But it also allows you to override that and create your own scene (if needed). But the scene it creates isn't a proper Unity Scene (it has no name! Which breaks some Unity APIs that need a name or a path) and confuses the Unity SceneManager/EditorSceneManager classes).

(there are some detailed forums posts here from other people who've trial-and-error tried to understand the missing docs around scene-creation with UTF. Personally, once I realised how confusing it is, I just rewrote our game code to fit UTF's useage of scenes - it's a bit ugly, but depending on undocumented behaviour in Unity UTF was a bigger risk)

7. Frontpage of project declares that Parameterized Tests aren't supported except for ValueSource. This is false (and it would be a disaster if it were true! Trying to use ValueSource without Sequential is a nightmare. But Sequential actually works fine, which is a relief.)

8. The official manual page for tests and testing has been deleted, and there's no link / reference to the new UTF. Team members upgrading from Unity 2019.1 or 2018.4 were lost. Google indexes the 2019.1 and 2018.3 pages as the top hits - and because Unity didn't link the UTF docs into the main manual, that's all Google values, unless you know the magic words to search for. As they say: "A URL is for life, not just for Christmas" (i.e.: the testing page shouldn't be deleted from the manual, it should be updated to point to the UTF sure, but not just wiped out!)

... there's also typos, grammatical errors, and missing docs for key classes / attributes, but they are minor compared to the above.

 

But - to be clear - overall it's very useable and we're getting genuine benefit out of it. Things that we've been unable to test with Unity in the past (because of missing features in unity's unit-testing) and had been manually testing (ugh!) and/or using custom editor scripts for ... I've mostly been able to port across and they run fine in Unity TestRunner.

Also: I've been pleasantly suprised by how fast the testrunner runs. There's an annoying delay on startup for every test-run (3-5 seconds), but where I was afraid it would increase with the number of tests and test classes it seems fairly constant so far, and I've been throwing larger and larger numbers of tests at it.

I've only had one crash - I used 6 ValueSource variables, each with 6 values, and forgot to mark Sequential. The UnityEditor hung. It was attempting to render 46,000 tests in the TestRunner, I guess? I consider that a bug - it meant I had to kill the process and lose unsaved scene edits (fortunately very little in this situation - but in future that could bite someone) - so it would be better if the pre-process step that generates the set of tests especially for combinatorial ValueSource setups had some kind of early-out backstop that detected an insane number of tests and output a warning / required a manual override in the testrunner (an "Are you SURE you want to run 10,000 combinatorial tests?" button or something like that).

(also ... 46,000 tests isn't actually that much. I am disappointed that the Editor can't cope. But I know that the UnityEditor's GUI has some very bad bugs when it comes to rendering large numbers of items - it's taken many years for the Unity Profiler to stop hanging on simple profiles, and it's still slow for medium sized game projects, so I appreciate this is probably an issue with the Editor rather than with TestRunner)

 

First, a correction: the namespace collision between Unity's proprietary class "Assert" and the same-named methods with NUnit's class "Assert" + methods caused me to be wrong in one of the complaints above.

(NB: that is IMHO another annoying bug. Why has Unity decided to create a name collision, while both libraries are packaged together in Unity by default, and both are live at once? Since Unity's version is NOT identical (but has some idenitcally named methods!) the whole thing becomes a huge mess)

In reality, Unity does have an AssertNotEqual method that works on float/double, but they decided to break the naming convention of NUnit (again ... why?!! This just makes life harder for all of us ) and so I didn't know it existed.

Now that I've pushed more tests over to using Unity's Assert class, I'm finding it's significantly worse than NUnit's . For instance ... The implementation of Assert.AreEqual ... is not good. The message it outputs is:

Code (CSharp):

1. UnityEngine.Assertions.AssertionException : Assertion failure. Values are not equal.
2. Expected: 0 == 10

What does that mean?

(hint: it doesn't mean what it says . It definitely DID NOT expect 0)

Digging in the decompiled code, the problem seems to be "UnityEngine.Assertions.AssertionMessageUtil", which has this method:

Code (CSharp):

1.  
2. public static string GetEqualityMessage(object actual, object expected, bool expectEqual)
3. {
4.   return AssertionMessageUtil.GetMessage(UnityString.Format("Values are {0}equal.", !expectEqual ? (object) "" : (object) "not "), UnityString.Format("{0} {2} {1}", actual, expected, !expectEqual ? (object) "!=" : (object) "=="));
5. }
6.  

...the problem is that that method has no text saying which value is which, and it's called from a context that prefixes it with the text (from same class):

Code (CSharp):

1. private const string k_Expected = "Expected:";

So the net effect is that Unity outputs:

Code (CSharp):

1. Expected: [I](not the expected value)[/I] == [I](the expected value)[/I]

In contrast, the NUnit implementations of Assert.AreEqual output simple, clear, messages that are wholly correct and don't confuse the reader. IT would be great if Unity dropped their messages and used the NUnit ones.

 

Another issue (bug?):

1. Create ValueSource array that's pre-filled with ScriptableObject instances
2. The first test you run will work
3. All other tests now hang the TestRunner, you cannot use TestRunner again with these tests until you reload assemblies

(the test never ends - neither succeeds nor fails - and Unity returns to the original scene as though the test had run, but it hasn't been executed (verified by outputting Debug.Log during the test))

 

Also ... for reference for anyone combining unit-testing with performance-tests, it's worth being aware of this undocumented feature of the Profiler: https://forum.unity.com/threads/profiling-recorder.432880/#post-5081726 (which I think Alexey is getting added to the profiler docs in a future release, but is currently not recorded anywhere).

TL;DR: if you want to include performance as a failure-condition for a UTF test, you need to use the Unity Recorder class slightly differently than if you're using interactive profiling with the Profiler window - but it works great once you do.

(e.g. I'm using this to have some integration-tests that check if specific high-level algorithms ever suffer a big spike in execution time. It's great to use UTF for this - if you suspect the game is getting slower, you re-run the performance-tests, and double-check. And/or you can load them up on a different computer and check performance in-editor while debugging/improving your source)

 

Coming back to this a year later, and... great that the window bug got fixed . Thoughts in 2021:

1. EditorTests can't be paused or breakpointed? Can do it with playmode tests (if you make them [UnityTest] and do a yield-return-null).

EDIT: 1.5. EditorTests are wonderfully fast though!

2. No quick/easy way to run an EditorTest as PlaymodeTest or vice versa? (even though they are often identical source code. Nothing is different, just the magic asmdefs. Which cannot be changed quickly.)

This matters a lot given: Playmode tests are EXTREMELY slow to run (for obvious reason: they have to wait for the Editor to (de/re)serialize the entire scene/new test scene) so you want to use them "never" if possible and only add them when you have to (use Editor tests for everything, including game-logic, because otherwise your development is very slow and your colleagues will hate you - or simply refuse to run any tests ... then create Playmode tests for the poor unfortunate souls who are going to be forced to do the QA runs), but Editor tests don't allow debugging of the test (see above).

3. Still disappointing that there's no support for e.g. Vectors, Quaternions having an "approximately equal" (or even a basic "equal"!)

 

(to put this into perspective: I'm currently having to copy/paste tests back and forth between a pair of class files every time there is a bug, just so that Unity will 'allow' me to debug the test. As far as I can tell, Unity's set this up so that tests simply cannot be debugged by default - which is ironic: mandating that programmers must write perfect code first time every time, for the piece of code that only exists because programmers do NOT write perfect code )

It takes so long, and it's so frustrating, that I can see why so many people still give up on doing unit-testing/system-testing/automated-testing with Unity: you have to fight the Unity editor/tools all the way .

I hope there's some (undocumented) workaround that makes this work like in all other toolchains, where automated testing has been lightning fast and built-in for many many years now.

 

...and I don't understand why Fail is still marked private - we need to use this frequently to add the missing core methods (e.g. Assert.ApproximatelyEqual( Vector3, Vector3 )).

private static void Fail(string message, string userMessage)

 

Argh! OK, thanks - I'll dig into this some more, see if it's just this project.

NB: how I've previously debugged such tests is "Debug.Break(); yield return null;" which has worked fine with [UnityTest] tests running as coroutines - sounds correct, right?

Sounds awesome! I look forwards to it

NB: I would recommend someone linking that frmo the "manual" / front page, because it's completely non-obvious. It's a core concept that should be on the front page. I would never know to even look for them unless I already knew they existed (which I do now, thanks!). I noticed that we could provide custom comparers - which made sense - which I thought was great for compatibility with C# standard practices, but in most Unity-related cases it's quicker/easier to provide our own methods that do everything in one-liner (just like NAssert! Just like the rest of Unity's Asserts!).

The comparers solve the problem. But they makes the Assert.xxx( ...... ) line of code much longer (no examples on the docs page, but I think it's: Assert.Equals( myQuat, other.thing.reference.Quat, "These should be the same", QuaternionEqualityComparer.Instance)". Sounds like a small complaint, but ... I've found with tests we have to make the assert methods as user-friendly as possible or else people fall back on (mis)using the Assert.Equals().

I'll defintely start using the comparers (but probably just dump them inside custom AssertQuaternionEquals(..) method etc).

 

I just have to say YES PLEASE for this! Currently most of our editmode tests are such just for the faster execution, and having to have pairs of test assemblies just for this reason is really clunky.

 

I'm a fool. I was trying to use Debug.Break(); and/or yield return null to pause an editor test. Of course that's not going to work - you can't pause the editor! Breakpointing is working after I restarted and re-attached debugger.

So, to rephrase this, as something that is a recurring challenge when trying to use editortests (and a primary reason why I'd prefer to run playmode tests even though they're so slow):

As far as I know, there's no easy way to debug the data in an editor test: you can use your debugger, but that stops you from having access to the whole of the UnityEditor, Inspector, Scene view etc that allow you to rapidly inspect/debug your test data.

By contrast: playmode tests you can pause the editor, inspect the scene + objects + data directly, then unpause / step through and watch the test progress.

 

After trying these for a few days ... some are OK, but others are a waste of time.

Sure, it's using a C# standard Comparer - but that's not the problem here: the real problem is "comparing and formatting".

e.g. try comparing two Quaternions.

You get ... an error message that is of no use to most people:

" .... (meaningless numbers) != (more meaningless numbers). ... "

... . Unity itself doesn't present Quaternions as quads, and the assertion library shouldn't either.

It should be saying "Quaternion differs by X degrees rotated around Vector Y" ... or ... "Quaterion ( euler-angles ) isn't Quaternion (euler-angles)" ... or similar.

Using an EqualityComparer here makes that - as far as I'm aware - not possible to solve. So we're back to square one: replace the Unity Assert class with a more sensible and appropriate one that includes support for he Unity core datatypes.

 

Thanks - good point about the version problem too! Must make it a lot harder