Search Unity

  1. Unity 2019.1 is now released.
    Dismiss Notice

The mother of all unit/automated testing threads

Discussion in 'Scripting' started by liortal, Oct 17, 2012.

  1. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    Hey all,

    In the enterprise development world, there seems to be a few handful leading frameworks for doing unit tests (referred to as the xUnit family of frameworks such as NUnit, xUnit, etc).

    When approaching this subject with Unity3d, it seems there are multiple different options.

    A short Google search reveals many different frameworks for unit testing with Unity3d, however it is unclear whether any of these are still actively being developed/used by developers.

    I would like to enlist (with your help) the current available unit testing frameworks for Unity3d (or automated testing frameworks, as some other forms of testing may be required that are "larger" in scope than unit testing).

    Also, i am interested in knowing what sort of automated testing are you doing for your games?

    Things like proper "unit testing" where taking a single method, invoking and asserting against some conditions in an isolated way, testing of interaction between different objects (without having to start a "scene" and simluate using the engine), throwing "test" objects into real game scenes to test certain things, creating specific test scenes to check code, etc.

    Any help will be of value as i'm trying to create a single and coherent source of information on this subject for myself and for other developers usage.

    Frameworks/tools i've stumbled upon were:
    *Comments/opinion/feedback on these will also be great.

    • NUnitLite
    • UUnit
    • SharpUnit
    • Test Star
     
    Last edited: Jan 11, 2017
  2. Banderous

    Banderous

    Joined:
    Dec 25, 2011
    Posts:
    669
    I am actively developing Uniject, which is a testability framework and not merely a test runner.

    Uniject lets you write genuinely testable code because it actually mocks UnityEngine. It lets you execute and verify almost all your code without even firing up the Unity editor.

    Every other testing framework I've seen merely shoehorns an XUnit runner into the editor, without addressing the more fundamental problem of the impossibility of unit testing code that has dependencies on concrete types.

    Take a look at my most recent post that shows some of the goodness on offer.

    On the more general question of what unit testing practices are prevalent in the community, I have tried to answer the very same question.

    My conclusion to the lack of response was that almost nobody is unit testing.

    I believe this is because of my earlier point - out of the box Unity scripts are fundamentally untestable without a framework like Uniject.
     
  3. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    Thanks for replying :)

    I will definitely check out your project, and am also willing to participate during my free time to donate to the project in case i find it fits my needs :)

    In reply to your assumption that people don't unit test, and that unity scripts are non unit testable -- My question was more broad in the sense that i am not strictly aiming at unit tests, but in fact i am interested in all types of automated testing frameworks that are available to Unity (this includes integration testing for example, which is not aiming to isolate a particular method or a piece of logic).

    Lior
     
  4. Banderous

    Banderous

    Joined:
    Dec 25, 2011
    Posts:
    669
    Integration testing may be more prevalent. Even with Uniject I find myself writing integration tests more often than unit tests, since I'm normally interested in the interactions between two or more of my own entities.

    Uniject makes integration testing very straightforward because it can instantiate any portion of your codebase from a single component to an entire scene - as ever, without any dependence on Unity.
     
    Last edited: Oct 22, 2012
  5. _Shockwave

    _Shockwave

    Joined:
    Sep 2, 2012
    Posts:
    21
    I'm very interested in Uniject. I'm come from TDD mindset and have always followed standards and best practices to write software.
    I was shocked knowing that the game development practices do not highlight testing. I actively asked famous indie game devs (Jonathan Blow, Tommy Refenes ...etc.) about testing their games and they basically said that they only do build testing and that's about it!

    I'm more than willing to even donate money to help you out with this project!

    I have few things that I would like to ask:

    1- Can you run the tests from the command line? Effectively this would be useful to hook up with a CI server such as Jenkins or Bamboo.
    2- Do you have examples of complex interactions that are tested with your test framework? You only have a simple example on your website.
    3- Can you test network behaviour as well? e.g. Loading an asset from a server then using the asset in a scene.
    4- Does your test framework generate reports after the test has finished? what kind of formats for reports does it output? (i.e HTML, JSON, junit xml ...etc.)
    5- Do you have documentation for using your test framework?
    6- Does it support a specific Unity3d version? or works on all ?


    Thanks! Looking forward to your response.
    Cheers.
     
  6. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    Out of curiosity - do traditional mocking/isolation tools work with Unity? (for example, Moq, Rhino mocks, Typemock, NSubstitute, etc).

    Lior
     
  7. ericknajjar

    ericknajjar

    Joined:
    Sep 12, 2013
    Posts:
    8
    I've used unit level testing on a couple projects with varying degrees of success, most of them where turn base/strategic games, so most of the logic was not real time, what is a pain to test. I've developed my own tool, and you can find it here: http://forum.unity3d.com/threads/200210-UTest-Simplified-and-easy-unit-test-framework

    I find the approach of mocking away all the engine very unhealthy. TDD aim not only in having a tested code base in the end, but also a code base that is better coded because you had to decouple and think about how to get your code testable in the first place. The theory is that code that is easy to test is a good code altogether. By mocking all the engine, you are not enforcing any decoupling between the engine and your code, not that you're build a multi-engine game or something, but decoupling your code brings a lot of other good stuff to your code base(among them, testability). In the end, you may have a bunch of tests testing your code against the mocked up unity3D, but a lot of behaviors may be different between the mock and the real deal, and since you're not decoupled, you rely on those behaviors... It's better than no tests.

    The approach that I used, and that my tool supports is decoupling your game code from the engine. No business logic class should inherit from MonoBehaviour, unity is just the presentation layer of your application, decouple it the best you can. If you got to mock up 300 classes just to test a component, you're not doing unit testing at all.
     
  8. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    Having totally decoupled code (not even relying on MonoBehaviour) makes your code testable, however it gives your more concerns to deal with in order to execute your tests...

    For example - Unity automatically calls Awake, Start, etc for you. In order to "simulate" this behavior you'd have to perform these yourself for your test.

    That's just one example, there are probably many others.

    Also, while unit tests are important, the bigger picture (integration between different parts to form the game) is also important, what techniques did you use for these sorts of tests?
     
  9. ericknajjar

    ericknajjar

    Joined:
    Sep 12, 2013
    Posts:
    8
    It's important to keep in mind why are you writing automated tests in the first place. For me, it is to be able to change things easily and rest assured I didn't brake anything(refactoring). 90% of the bugs should be catch-able on the unit level of testing and using TDD, if not, you probably are not decoupled enough. For that 10% that go thru, generally they are 'cause I did something wrong with my TDD, forget to write a test or something, so I write a unit test for that case and go on with my life. But there are some bugs, few, disastrous and hard to catch bugs that just don't go down with simple unit tests, on that rare occasions, I do one of 2 things:

    1 - Hardcode a scenario; be it a unity scene, a giant "unit" test, an integration test or end to end test
    2 - Trust my QA team to catch it =)

    The problem is that people forget that most of your tests should be on the unit level and try to come up clever ways to do integration tests and end to end tests instead of making their code unit testable. It would all be fine if were not for the fact that end to end and integration tests make your code more brittle, every change you do, you have to basically rewrite most of you integration and end to end tests. That fact hurts the main purpose for what I do testing, LESS of this tests is better. So I only do tests that are not unit tests when I admit that I'm not good enough to crack that nut, I've failed, so I write an integration test. I do so little "other" tests that I don't need a special tool to do it.
     
  10. Julianobsg

    Julianobsg

    Joined:
    Jul 14, 2012
    Posts:
    4
    I used uunit for some time to notice the problems in unit testing monobehaviour, then to make it work I´ve decoupled test game logic from monobehaviours. That works for most of situations, with some exceptions, like updates and movements, coroutines with wait for seconds, and the biggest problem the WWW call.

    Never used uniject but I guess it will be hard to convince my team to use a framework that decouples from unity to unit test a game, for me unit testing is just like Erick said, this week I´ve decided to create my own unit test framework, and just today had the time to start the job.

    Never used another framework, and go interested in the utest(it will be hard to convince a company to buy, even by the price of just 10 dollars, is hard to convince a company that unit test will work great in games, mainly because anybody saw any efficiency in it).

    Yet there are some features needed that is missing, and monobehaviour is my greatest problem.

    I'll be glad if anyone can suggest any new stuff.
     
  11. gfoot

    gfoot

    Joined:
    Jan 5, 2011
    Posts:
    548
    Things like WaitForSeconds and WWW should always be mocked for client testing, you need to check that your code is calling them correctly and responds correctly to various outputs (e.g. server errors) but you don't need to check that your actual server behaves correctly, at least not through unit testing from the Unity client - you should have a separate test suite for the server.
     
  12. trumpets

    trumpets

    Joined:
    Mar 1, 2013
    Posts:
    8
    How did you guys achieved mocking in Unity? Most of the mocking frameworks I use don't work with .NET 2.0 :/
     
  13. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
  14. Glader

    Glader

    Joined:
    Aug 19, 2013
    Posts:
    357
    I wish I knew Uniject existed, may have helped during development of a tool I worked on. I spent a couple months working on something called Testity: https://github.com/HelloKitty/Testity which I've decided to temporarily cancel since it was so large. The idea was to create interfaces for all components and services in the engine, some are implemented, and write adapters for the Unity3D types.

    Then on dll import it used Rosyln code generation to build out MonoBehaviour equivalents and boxed/adapted the Unity3D types into the adapters and injected them into the classes you'd write with Testity that were sort of like MonoBehaviours but without the dependency of the UnityEngine. This let you write everything against interfaces and unit test with mocks.

    I stopped because it was just too large in scope and I ran into some cases where I wasn't sure how I should handle the code generation for things such as: collections and how to handle classes marked [Serializable] as single fields or collections.

    Most things were handled, even UnityEvent, but the above was left unfinished and the Unity3D components and services weren't fully fleshed out.

    Edit: Why can't Unity Technologies add interfaces to their components so we can mock them? Would solve this problem much easier.
     
    Last edited: Feb 1, 2016
  15. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    Thought it would be a good place to also add a link to my last post on "what's new with automated testing with Unity" here: www.tallior.com/whats-new-automated-testing-unity/

    As part of that post i'm also holding a survey to see other devs' testing habits, feel free to join in: https://www.surveymonkey.com/r/Y9FYVHZ

    Lastly, if anyone has any interesting topics related to unit (or any other automated) testing with Unity, feel free to suggest them here!
     
  16. Joseph-Ferano

    Joseph-Ferano

    Joined:
    Jul 18, 2008
    Posts:
    165
    So I've recently been getting into automated testing with Unity. I was a bit disappointed with the built in unit testing because it can't be run in playmode. Maybe it's because I don't come from a previous domain where unit testing had its philosophies, but I don't see the point to mocking things out to that extent. Why not run tests you can yield in playmode?

    I bought both uUnit and FUnit and will be trying both out. I have a new state machine class I build and I might try testing with both classes. I'm keeping NSubstitute around but moving it out of the Editor directory. I'm also going to keep the integration tests tools and encourage our non-programmer devs to learn those tools. However, for unit testing, I want to run in playmode. I don't have to jump through hoops to have the game logic completely decouple, at least not when it doesn't make sense to. Sure, I'll keep an eye out and try to make as many of my classes POCOs, but I've seen some examples of people going to extremes of separating MonoBehaviours from the logic when clearly their logic is mostly time based.

    I'd love to hear people's opinions on either FUnit or uUnit.
     
  17. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    What are uUnit and FUnit? can you add links to these? are they still being actively maintained?

    Regarding unit tests / integration tests -- i know that there's going to be a new kind of built-in test called a "play mode" test. It had surfaced somewhere around the 5.5 betas, but i guess it's still not ready for use. Basically i think it's going to be support for running NUnit still tests, only during play mode.

    You can achieve something similar to this already today, by using the Unity Test tools package (it's called "integration tests"). These are tests that are made of a scene, game objects and code that asserts for specific conditions. You can check out my posts on the matter here:

    http://www.tallior.com/introduction-unity-test-tools/
    http://www.tallior.com/whats-new-automated-testing-unity/
     
  18. Joseph-Ferano

    Joseph-Ferano

    Joined:
    Jul 18, 2008
    Posts:
    165
    @liortal

    Thanks so much for the reply. This will be a large post, I'll add some headers...

    FUnit and uUnit

    So I don't think either are maintained anymore, but they work. However, I tested both last night. Here are the packages;

    https://www.assetstore.unity3d.com/en/#!/content/3727
    https://www.assetstore.unity3d.com/en/#!/content/27388

    The promise of both is the ability to do "yielding" tests. They run in playmode and you can use yield statements to control the timing of the tests. This is great, cause now you can more easily test MonoBehaviours.

    So both have their issues; uUnit is the more fully featured of the two, having a better assertion library and more attributes. However, it has a major flaw; you can't select which tests you want to run, it seems to just run all tests found in an assembly. I don't know if I can just return a namespace, I could investigate that further. If I can get around that issue, I would probably stick to uUnit, because obviously, when we get into tests ranging in the hundreds and some are time based yielding tests, that's going to be a big issue. There's a manager that uses "System.Reflection.Assembly[]" in order to find all unit tests.

    FUnit is a little bit more bare, although it's more fully featured as a unit testing and assertion library compared to Integraion Testing. You can select which tests to run, so that at least gets around uUnits problem. Another thing I didn't like as much is that you have to separate out unit tests and yielding tests into different classes. You can get around that by simply yield breaking at the end of a yielding tests, so it's treated like a unit test, but it's still lacking in features compared to uUnit and even more so NUnit.

    Verdict

    I think I'm going to use all test tools from here on out; for non MonoBehaviours, I'll stick to NUnit and the editor test runner. For MonoBehaviour unit tests, I'll use FUnit, since I can pick which tests I want to run. For true integration level tests (multiple components interacting with each other), I'll use Unity's integration tests.

    My take on MonoBehaviours and Unit Testing

    I am of the opinion that while you can have your game logic not implement MonoBehaviours, in many instances, it just doesn't make sense. Like how do you test an FSM that uses coroutines in Enter/Exit states? Or a Motor with a MoveTo method? Sure, I can jump through hoops to separate out the Unity part, but why? The idea behind unit testing is to save time and become more productive in the long run, plus ensure a higher quality product. Why not test a real world MonoBehaviour in it's natural environment and not heavily modify either an existing code base or change the way you program to include more classes and more overhead to implement code? The former is the big one; how do you deal with a legacy codebases (defined as non-unit tested code)? It's one thing to start a brand new project in TDD fashion and do all the Unity separation from the start, another is to refactor legacy code.

    Potential Playmode tests

    So the Playmode tests sound promising but I couldn't find any information on it. Just found this thread;
    https://forum.unity3d.com/threads/t...de-tests-and-code-coverage-with-nunit.447941/

    I'd appreciate any other info you could provide. Running unit tests in playmode sounds great, but I really hope that it's easy to do the equivalent "yielding" tests in the tools mentioned. The issue I see with Integration Test Tools in Unity is that the code based API just isn't friendly or as full featured as NUnit, making it a poor choice for code based unit testing, even if it has the playmode option. So hopefully playmode tests will be that sweet in-between pure unit tests and multi-component integration tests. Maybe as Unity developers we simply have to redefine automated testing by adding a new category of testing to the testing pyramid.
     
    Last edited: Jan 11, 2017
  19. Joseph-Ferano

    Joseph-Ferano

    Joined:
    Jul 18, 2008
    Posts:
    165
    I just noticed that in the Editor Test Runner, in 5.5.0 f3, there's PlayMode and EditMode tests. Any idea what this is about? Is there any documentation on the PlayMode tests? I'd love to get started working on those, maybe I don't need FUnit anymore.
     
  20. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    This is not functional yet. it's been since early 5.5 betas i believe. I think it's some sort of mixture between editor ("unit") tests and integration tests (that run in a scene, using game objects, etc).

    Not quite ready yet, but definitely sounds interesting.
     
  21. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    BTW: I think you can start trying it, by going to the "about" window and then typing the letters "internal"
    This activates the "internal" mode which probably adds a few hidden stuff you shouldn't really use :)
     
    Circool likes this.
  22. Joseph-Ferano

    Joseph-Ferano

    Joined:
    Jul 18, 2008
    Posts:
    165
    Thanks for the info. Where is this "about" window? I can't find it. I'm on Mac
     
  23. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
  24. Joseph-Ferano

    Joseph-Ferano

    Joined:
    Jul 18, 2008
    Posts:
    165
    Oh, I see, you actually just type it out. I didn't see anything new. I'm pretty sure there must be some kind of attribute that you add to tests or something in order to distinguish them, I can't wait till this feature goes live. FUnit is nice but it's no longer actively developed on. If playmode tests can use NUnit and NSubstitute, that's going to be amazing.
     
  25. Voxel-Busters

    Voxel-Busters

    Joined:
    Feb 25, 2015
    Posts:
    659
    Is there any possibility now for testing coroutines?
     
  26. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,166
    Last edited: May 6, 2018
  27. unity_fEKUP1-WOAJKUA

    unity_fEKUP1-WOAJKUA

    Joined:
    Jul 23, 2018
    Posts:
    3
    Now that it's possible to run the tests using the command line, does anybody know if it's possible to print the output of the runner (passing tests, failing tests, stack trace, logs in the console) in the terminal while the tests are running?
     
  28. remi-sormain-viond

    remi-sormain-viond

    Joined:
    Aug 29, 2018
    Posts:
    1
    Hello there!
    Does anyone know of any plans to upgrade the NUnit framework packed with Unity to the latest version ? Or if it is possible to upgrade it manually ?
    I'm currently embracing the async/await support with the new .NET version but the Unity NUnit version lacks the latest async support and has known bugs :(.