Search Unity

Feedback Test runner API questions / feedback

Discussion in 'Testing & Automation' started by liortal, Mar 31, 2018.

  1. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,231
    I am going over the documentation for Unity Test Runner and some things aren't totally clear to me. here's my feedback:

    1. Play mode vs. edit mode tests
    What is the technical distinction between the two types? what is allowed to be done in edit mode which is not allowed in play mode tests ? (and vice versa). i also do not understand the motivation for selecting either type (when should i use each of these?)

    In Unity test tools (which is now deprecated), unit tests could not perform certain things (e.g: create new game objects). They were also executed as editor code (e.g: not in "play mode") so they could not start coroutines and such things.

    With these 2 new test types, it's not clear anymore (to me).

    2. Editor restart after enabling play mode tests
    After enabling play mode tests, the editor must be restarted. kind of a nitpick, but that's pretty annoying, and i don't recall any other feature that behaves the same. wasn't there any other solution for this? (why is it necessary anyway ?). Also, it is mentioned that this adds some overhead to the game at runtime - can someone clarify what does this add and whether i should actually worry about enabling this? (if it adds 10kb to the build i am totally fine with it).

    3. PrebuildSetupAttribute
    (what an awful name BTW) - why should i prefer this over NUnit's SetUp attribute? what is the difference between the two and which one should i prefer to use ?

    4. MonoBehaviourTest
    while i believe the intention was good (probably due to lots of feedback about how it's hard to test MonoBehaviour code), i don't really see how this thing could be useful for real world scenarios.Also, it "asks" you to mix your runtime (MonoBehaviour) code with your tests (IMonoBehaviourTest interface).

    5. Command line execution
    does this work in conjunction with -batchmode or -nographics? we would like to run editor / play mode tests from our build server. so far i haven't been able to execute it via command line.Also, it can be nice to have more control over which tests will get executed (by name, category, etc).

    6. Improve the documentation
    documentation on the docs page (linked above) is not clear at times. also links coming from this page to various attributes or other code related stuff is very under-documented and missing examples or includes confusing descriptions.

    7. Please Open source the test runner
    I think we (the dev community) can benefit from it as well as improve on it and send it back as PRs.
     
  2. Psicomante

    Psicomante

    Joined:
    Mar 18, 2013
    Posts:
    2
    It's pretty funny that nobody replied to this important thread. I'm currently looking into TDD on Unity but there's so little to read.
     
  3. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    4,266
    That's highly subjective.
    TDD isn't first class citizen in game development for better or for worse, that's matter of taste. Personally I don't like TDD.
     
  4. Psicomante

    Psicomante

    Joined:
    Mar 18, 2013
    Posts:
    2
    Do you dislike TDD at all or only in Gamedev? Do you want to argue about it? I'm quite interested to know other people opinions and experience.

    disclosure: I use TDD or a edulcorated version of it in all my development projects.
     
  5. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    4,266
    I dislike TDD everywhere.

    I'm an enterprise software engineer, so in my day-time job, I write tests, after I wrote the actual code to ensure I don't mistakenly change something. But I don't write BS tests in advance.

    I value the functional tests much more, since I usually work on applications, where the only thing matters is what the users experience. If the application is working properly, we're good at the moment.
    This applies to the game development as well. I value QA manual tests more than artificial unit tests. (Although obviously for an API you don't really have a choice).
     
  6. DavidGeoffroy

    DavidGeoffroy

    Unity Technologies

    Joined:
    Sep 9, 2014
    Posts:
    283
    Hi @liortal,
    I'm nowhere near a test expert, but I can give you some info based on my personal experience writing tests for Unity code.

    I use edit mode tests to test Unity APIs that are part of UnityEditor, or code that sits in Editor folders. Since these tests use Editor code, they will not compile as playmode tests if you try to run them anywhere but in the Editor.

    Playmode tests run in playmode / standalone, so if you use them in combination with the [UnityTest] attribute, you can test the interaction between your code and the rest of the Player loop in one coroutine, which might be possible with Editor tests, but will be more complicated. So this makes them a bit more suited to Integration tests.

    There are a lot of simpler tests that you can write as either type, but you need to keep in mind that Editor tests will not run outside of the Editor, so you will miss platform coverage by writing your tests as Editor tests. That might not be an issue if you write strict unit tests, but as soon as you start interacting with Unity APIs, having multi-platform coverage is always good.

    Finally, Edit mode tests don't have the game loop running, so there are less variables involved. If what you are testing doesn't depend on the game loop running, or MonoBehaviours executing, then you can definitely implement them as Editor Tests.
     
    Last edited: Feb 19, 2019
  7. DavidGeoffroy

    DavidGeoffroy

    Unity Technologies

    Joined:
    Sep 9, 2014
    Posts:
    283
    Just dug up some more info:

    This is a setup that's invoked in the Editor, before the playmode test builds a standalone player to execute your tests on-platform. This is useful if you need to test things that require generating assets/scenes in the editor.
     
  8. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,231
    Thanks @DavidGeoffroy, some time has passed since i originally wrote this post.
    I do realize now what's the different between edit and playmode tests. it seems that playmode tests are more suitable to what i'm after (as i am testing our game's components rather than some editor code that we created).

    Having worked with the test framework a bit more, there are more things that i find missing, off the top of my head:

    1. No editor API for running tests
    The only ways i know are either from the editor UI, or by launching the editor using the relevant command line args.
    We are using an automated build system so we do not manually interact with the UI. The only option we have is running Unity with the -runtests command line arg.

    Our build consists of running tests first and then building the player. This means we have to launch the editor twice (once for tests, once for building the player). This slows down our build process. i wish there was an editor API (e.g: TestRunner.ExecuteTests) that will allow me to run all tests.

    2. UI changes to test runner window
    There are a lot of visual improvements that can be done to this window, for example:
    - Show the number of tests (executes, passed, failed, etc)
    - Allow grouping by categories (built-in support in NUnit using the [Category] attribute
    - Better marking for ignored tests (also, it seems that ignored tests take so much time to skip over, even if they are not run)

    3. Playmode tests don't work with -nographics
    Not sure this is 100% correct, but our CI system runs these tests and it cannot execute playmode tests on Windows based machines, since when running them it requires access to the "desktop" since Unity cannot be run with -nographics (our build agents run as a service that doesn't have that access). Why can't i run Unity without any graphics, but only logically simulate running those tests?

    4. Allow building a Playmode test build
    Playmode tests can be executed on a target device, but this only works if you build and run it through the test runner UI. it could be more flexible to be able to run a complete build that will be generated to run playmode tests, and then install that build wherever i want (any device, device farm, etc).

    5. All tests are run in the same "isolated" scene
    When starting test execution, a new scene is created. this is often times not enough, as we'd like a "clean slate" for every test fixture or even every test that is run. It could be great to have that baked right into the framework (currently i have to implement this myself).

    I have a lot of other feedback points, i really wish there were more updates about the test runner UI and framework (open sourcing it, updates, etc).
     
  9. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,231
    Maybe @ElvisAlistar can comment on any upcoming updates to the test runner?
     
  10. ElvisAlistar

    ElvisAlistar

    Unity Technologies

    Joined:
    Oct 2, 2013
    Posts:
    221
    I would refer you to Christian's reply here: https://forum.unity.com/threads/feedback-for-logassert-class.530539/#post-4248829

    Looking to hire one more person internally to look after this feature. Also, working hard to getting it released as a package and very close to that.

    At a minimum, you will have access to the source code and do anything you like with it. We plan to address most feedback and concerns in the future, but it's going much slower than we would like to.

    We're also going to set up a separate forum section dedicated to testing and we're going to be more closely monitoring that.
     
    liortal likes this.
  11. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,231
    Great news. Thanks for replying.
    I will definitely be waiting for all of that.
     
  12. starabanaf

    starabanaf

    Joined:
    Jun 26, 2019
    Posts:
    1
    Is there any progress? We need to have C# API to run tests from the editor. At least could you provide an example of how to use reflection to call Run All tests? This is very useful to do before any commit to the repository. Thanks.
     
  13. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    31
    Yes. The test runner api has been released a few weeks ago. More details here: https://forum.unity.com/threads/extending-unity-test-framework.723773/