Search Unity

  1. Click here to see what's on sale for the "Best of Super Sale" on the Asset Store
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

What options do I have for Automation and Unit Testing In Unity?

Discussion in 'Testing & Automation' started by tsibiski, May 22, 2019.

  1. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    I wanted to create this topic as it does not seem that anyone has done so yet. There seems to be very little discussion of ANY automation tools in Unity outside of the Unity Test Tools. People may not realize that there are now SEVERAL choices to automate UI testing in Unity, along with additional options for unit testing!

    **Full disclosure! I created and maintain the Trilleon framework mentioned below**

    Here are some categories I will mark with a or an X if the frameworks provide them:
    Automation Testing
    Unit Testing
    Full Framework
    (Meaning that it fully replicates something like Selenium, instead of extending an existing framework, or representing a shell of one to build off of.)
    CI Integration
    External Driver
    Report Generation
    Parallel Execution
    Custom Editor Window(s)


    -- In alphabetical order --

    AltUnity Tester

    It's an end-to-end UI test automation tool allowing you to run tests on real devices (mobile, PCs, etc.) or inside the Unity Editor. It supports tests written in C#, Python and Java.

    It'll help you to find elements and get all their properties and also to simulate any kind of device input.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration:
    External Driver: (it can use Appium to interact with the game)
    Report Generation: (test results and reports inside unity editor for C#)
    Parallel Execution: X ( s possible, but no special implementation in the tool for it)
    Custom Editor Window(s):
    Multiplayer Testing:

    asset store link: https://assetstore.unity.com/packages/tools/utilities/altunity-tester-ui-test-automation-112101
    gitlab: https://gitlab.com/altom/altunity/altunitytester
    wiki: https://altom.gitlab.io/altunity/altunitytester/
    discord server for questions and support: https://discord.gg/uZrah5U


    Minefield


    Not even a wrapper to Unity's own Test Runner, it is just an extension library and guidelines to help you write pure Unity Test Runner test cases. Its focus is that you should be able to code up a scene navigation test fast enough that you could do it while creating the scene without disrupting your creation flow. It is possible with help from "test beacon" components added to the scene as a metadata and a various yieldable methods to help click on those beacons in the test. Easy installation via Unity Package Manager. Because you are still writing Unity tests, of course you could use it together with something like Performance Testing package to create a performance test of scenes while navigating around.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration: X
    External Driver: X
    Report Generation: X (possible with performance testing package)
    Parallel Execution: X
    Custom Editor Window(s): X

    github: https://github.com/5argon/Minefield
    wiki: https://github.com/5argon/Minefield (README.md)​


    Responsible


    Responsible is a reactive asynchronous testing utility primarily designed for, but not limited to be used in high level system tests. It's inspired by Rx, and uses UniRx under the hood, but can be used without any Rx knowledge. It was designed to run as part of Unity Test Runner execution, but should also work with any other test runner that supports coroutines or observables.

    The primary benefits of using Responsible are:
    • Detailed output on test failures and timeouts (operation statuses and async "stack traces")
    • Declarative, composable, and reusable test code
    • Using Responsible will circumvent a Unity Test Runner bug, where test execution will continue after errors within nested coroutines, sometimes even hiding the first exception.
    Automation Testing:
    Unit Testing: (Not the primary use case)
    Full Framework: X
    CI Integration: X
    External Driver: X
    Report Generation: X
    Parallel Execution: X
    Custom Editor Window(s): X
    github: https://github.com/sbergen/Responsible
    wiki:
    n/a​


    The Puppetry

    The Puppetry gives ability to test a game in Unity Editor and when built and deployed to a device. This is done via additional integration with a device's driver (like Appium for mobile, SDK for consoles or SDK for VR glasses). It is cross platform framework primarily written in C#.

    The Puppetry doesn't have dependencies from any specific technology so it can be combined with any TestRunner (Nunit, XUnit, MSTest), any CI system, any reporting tool set, and other required libraries or frameworks.

    The main purpose of driver approach is to separate the tests from the application code base, so that you don't worry about how changing tests will affect the application.

    And finally, The Puppetry supports parallel execution of tests right out of the box. So you can run your tests simultaneously on different Unity Editors or builds.

    Automation Testing:
    Unit Testing: X (Perhaps possible, but not explicitly designed for it)
    Full Framework:
    CI Integration: X (Ostensibly capable, but not included)
    External Driver: X (Custom driver w/ Appium for devices)
    Report Generation: X (Could not find sign of reporting. Likely has basic result output.)
    Parallel Execution:
    Custom Editor Window(s):

    github: https://github.com/TestUnitLab/Puppetry
    wiki: ttps://github.com/GameUnitLab/Puppetry/wiki


    Trilleon

    A complete, internal automation framework that is entirely self-contained in your game binary. Communicates with CI process (Jenkins, Bamboo, Pipelines) over simple pubsub messaging framework and sockets. Accepts commands to run tests, and then takes care of the rest. Entirely white box testing solution. Adds massive number of additional options for controlling the order that tests run, along with dependencies between tests and test classes. Extensive editor window, "The Nexus", where you can organize, launch tests, review test results, and customize the framework.

    Just as easy to use as a Unit test framework alternative too. Trilleon can handle any Unit test logic that requires the use of Coroutines, or any context specific game code that cannot be tested on compile time!​

    Automation Testing:
    Unit Testing: (Atypical; not run at compile time.)
    Full Framework:
    CI Integration: (Scripts supplied fully for Mac, partially for Windows)
    External Driver: X (Uses Appium to load apps onto devices; no further driver usage)
    Report Generation: (Friendly Html/Js/Css report, Standard XML report, In-editor results window)
    Parallel Execution: (Supports through Selenium Grid)
    Custom Editor Window(s):



    Unity Test Runner

    The built-in way of running unit tests, and scenario-based automation tests. The latter requires test scenes, and does not test the real scene, on devices, as a user would.

    Automation Testing: X (Yes & no; it is not true automation tests, but more like a Unit-Integration/Automation hybrid)
    Unit Testing:
    Full Framework:
    CI Integration: (CLI provided here)
    External Driver: X
    Report Generation: (Standard xml report, In-editor results window.)
    Parallel Execution: X(Not for automation tests)
    Custom Editor Window(s):


    wiki: https://docs.unity3d.com/Manual/testing-editortestsrunner.html


    Unium

    Unium is an experimental library for the purposes of facilitating automated testing and tool development for your Unity games.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration: X (Ostensibly capable, but not explicitly)
    External Driver: X (Facilities use of external drivers. No dependencies on them.)
    Report Generation: X (Facilitates integration of existing testing frameworks; ex: mocha, jest, pytest, specflow, etc...)
    Parallel Execution: ? (Unknown)
    Custom Editor Window(s):

    github: https://github.com/gwaredd/unium
    wiki: n/a

    Unity UI Test Automation

    This is a basic wrapper for running ui automation tests that works with the Unity Test Runner. The information below describes what this tool offers only, and not what it offers in addition to the Unity Test Runner.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration: X
    External Driver: X
    Report Generation: X
    Parallel Execution: X
    Custom Editor Window(s):
    github: https://github.com/taphos/unity-uitest
    wiki: n/a


    *If you are an owner of one of these repositories, feel free to provide me an updated elevator pitch for your toolset. Correct me on any mistakes I make. Additionally, if you have an alternative tool to add to this list, let me know as well.*
     
    Last edited: Nov 20, 2020
    TziAllstar, mulova, sonnyb and 5 others like this.
  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,372
    Unity Test Runner is usable in CI - as documented you can launch it test runs from the command line.
     
    seblaf1 and liortal like this.
  3. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    Thanks, @superpig

    I've updated the post.
     
  4. jGate99

    jGate99

    Joined:
    Oct 22, 2013
    Posts:
    1,203
    Much needed topic, It should be Pinned
     
  5. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,511
    Great work!

    I just want to say that I am also starting a new play mode testing library called Minefield. This completely depends on Unity Test Runner, it is just an extension that adds some useful methods. Though it is still in development currently it could navigate the scene the way I want to. Maybe it is still not that fully fledged like other on your list yet ..but it is getting there.
     
    tsibiski likes this.
  6. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    Write me a primer like those above, and I will add you to the list.
     
  7. gwaredd

    gwaredd

    Joined:
    Oct 22, 2012
    Posts:
    8
    Hey, great list :).

    A small correction, whilst you certainly can use unium and appium togther, there's no dependency there. Appium is not required neither is unium a wrapper for it in any way - it shares the some of the same philosophy in it's approach but none of the implementation details.

    Also, for clarity there is no report generation - none, zero, zip, nada ;)

    Rather - it facilitates existing testing frameworks (mocha, jest, pytest, specflow, etc...) and all the goodness that comes with them.
     
    Last edited: Jun 15, 2019
    tsibiski likes this.
  8. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    Thanks for the corrections and updates. Just added them.
     
  9. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,511
    Alright, I just hit a milestone on my game with Minefield and quite confident of the state of the API now.

    MINEFIELD

    Not even a wrapper to Unity's own Test Runner, it is just an extension library and guidelines to help you write pure Unity Test Runner test cases. Its focus is that you should be able to code up a scene navigation test fast enough that you could do it while creating the scene without disrupting your creation flow. It is possible with help from "test beacon" components added to the scene as a metadata and a various yieldable methods to help click on those beacons in the test. Easy installation via Unity Package Manager. Because you are still writing Unity tests, of course you could use it together with something like Performance Testing package to create a performance test of scenes while navigating around.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration: X
    External Driver: X
    Report Generation: X (possible with performance testing package)
    Parallel Execution: X
    Custom Editor Window(s): X
    github: https://github.com/5argon/Minefield
    wiki: https://github.com/5argon/Minefield (README.md)
     
    Last edited: Jun 27, 2019
  10. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    Added. Thanks.
     
    5argon likes this.
  11. shiroto

    shiroto

    Joined:
    Nov 14, 2012
    Posts:
    38
    Hey, thanks for this list! Its just what we were looking for. While looking for testing tools, we also found this one: https://www.gamedriver.io/
    Have you tried it by chance? Maybe you can add it to the list :)
     
  12. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    So without being able to freely view the code for gamedriver.io on something like Github, I cannot verify things in the same way as the examples in my first post.

    Also, at the moment, I am not going through the the process to get a free 30 day trial and access to the software.

    I am more than happy to add a profile for them if a representative of gamedriver.io can write up a pitch for me, and give me at least some limited visibility into their product to reasonably confirm the claims.
     
  13. rhino_potato

    rhino_potato

    Joined:
    Nov 10, 2016
    Posts:
    1
    @tsibiski Trilleon sounds like something I would want to try. How do we use Trilleon on a closed source Unity project and comply with section 4.d of the LGPL? Specifically, how do we provide a way for users to re-link different versions of the Trilleon library? I don't think there is a way to provide that when building the Unity game to Android or iOS. What do you think about MIT, BSD, Apache licenses to help increase adoption rates? Projects using those licenses still get plenty of pull requests to help improve the software.
     
  14. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    Hi @chrislouierhino
    Unfortunately, I did not choose the license and don't have control over what license is used. I think that aspect of its use was probably a mistake to omit when choosing a license, rather than indicative of any intent to restrict its use. The intent was that no one took the project and claimed it as their own creation, or tried to make money off of it. There was no other intent behind the license used.

    I will try to contact DisruptorBeam to ask for a change to the license, but I have not worked there in 4 years now. So I can't make any guarantees. Given that I know the intent, I personally would not hesitate to use it in the scenario you supplied above. For what it's worth.
     
  15. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    I contacted someone from the C level, and they also said that it is not intended to be that restrictive. But they will confirm everything and, if agreed, update it. Likely to be MIT if so.
     
    rhino_potato likes this.
  16. timeapusok

    timeapusok

    Joined:
    Apr 15, 2020
    Posts:
    22
    Hi,
    Great topic, this list is very helpful.

    I'd also like to suggest a tool.
    AltUnity Tester, is a UI test automation tool allowing you to run tests on real devices (mobile, PCs, etc.) or inside the Unity Editor.
    We started developing it a few years ago, after struggling to interact with objects from Unity Games.

    Hope it will help some of you. It's open-source, free and it can be downloaded from the Asset Store: https://assetstore.unity.com/packages/tools/utilities/altunity-tester-ui-test-automation-112101
     
  17. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301

    Could you do me a favor and write up a description of it in the same format as the existing tools above? (Hopefully marking the relevant features like the other tools do). Thanks!
     
    timeapusok likes this.
  18. timeapusok

    timeapusok

    Joined:
    Apr 15, 2020
    Posts:
    22
    @tsibiski here's the description in the format you asked for. Let me know if you need more info.

    AltUnity Tester

    It's an end-to-end UI test automation tool allowing you to run tests on real devices (mobile, PCs, etc.) or inside the Unity Editor. It supports tests written in C#, Python and Java.

    It'll help you to find elements and get all their properties and also to simulate any kind of device input.


    Automation Testing:

    Unit Testing: X

    Full Framework: X

    CI Integration:

    External Driver: ( it can use Appium to interact with the game)

    Report Generation: ( test results and reports inside unity editor for C#)

    Parallel Execution: X ( is possible, but no special implementation in the tool for it)

    Custom Editor Window(s):

    Multiplayer Testing:



    asset store link: https://assetstore.unity.com/packages/tools/utilities/altunity-tester-ui-test-automation-112101

    gitlab: https://gitlab.com/altom/altunity/altunitytester

    wiki: https://altom.gitlab.io/altunity/altunitytester/

    gitter room for questions and support: https://gitter.im/AltUnityTester/Lobby
     
  19. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    Thanks, I added your tool.
     
    timeapusok likes this.
  20. mfvnlrxu

    mfvnlrxu

    Joined:
    May 20, 2020
    Posts:
    1
    Hey, thanks for making this list. I'm overloaded with choice!

    We are new to automated testing and not sure which is the best one to use. We are working on a mobile puzzle game and we want to have the game run through the levels overnight and check everything is working.

    Which ones have people had success with and would recommend?
     
  21. timeapusok

    timeapusok

    Joined:
    Apr 15, 2020
    Posts:
    22
    @mfvnlrxu If you're just getting started, here you can find few example projects.
    There are tests written in Java, C# and Python.
    Hope it helps!
     
    Last edited: Jun 3, 2020
  22. sbordenTurner

    sbordenTurner

    Joined:
    Nov 5, 2019
    Posts:
    11
    I've been research automated testing and just came across AirTest. I haven't seen many people talking about it so I thought I should share. The really interesting thing about it is that it uses image recognition to find elements to interact with, without requiring changes to your game. If you are willing to add a plugin to your game they have a library called Poco that adds more functionality. It was developed by NetEase (a Chinese mobile game company) and announced in partnership with Google at GDC 2018. I found it because I was looking at Google's Firebase Test Lab cloud mobile test tools, which this works with.
     
  23. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    This is called image recognition testing, and is highly unreliable, inflexible, and extremely difficult and time consuming to maintain. It is the original way to test games. Baseline images are required in enormous numbers, and if your game changes at all after they are taken, it requires a lot of updates the images. Additionally, you have to account for very different scenarios based on gameplay resolution and the size and scaling of objects. Not only that, but it is almost useless outside of extremely static games with primarily UI components.

    That is the last type of automation I would recommend for video games. It will ultimately be a miserable nightmare. I am an automation engineer with 10 years experience, and 5 of those in the video game industry if that matters for my opinion.
     
    timeapusok likes this.
  24. sbordenTurner

    sbordenTurner

    Joined:
    Nov 5, 2019
    Posts:
    11
    Thanks for you input @tsibiski, I don't have any experience with image recognition testing so it's helpful to hear that it is unreliable. I'd still be curious what you think of their Poco library that uses scene hierarchy.
     
    timeapusok likes this.
  25. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    From a cursory look, I'd say Poco is not really all the different than many of the solutions above, so it sounds like a viable option.
     
  26. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    4,035
    ok so let's say I'm making a game on console and need to auto play it for hours or days to check for edge cases, leaks etc...
    Which package allows me to do that without me writing a line of code?
    Ideally monkeying me playing the game for a while then repeating the input over and over and if the game crashes, log how long that took, memory at the time etc...
     
  27. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,457
    See my reply on the other thread :) i don't think such a tool exists. But depending on your game you may be able to come up with something like that yourself.
     
    laurentlavigne likes this.
  28. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    As mentioned above, there is no such thing. Games aren't like web browsers where you can set up a crawler and have it successfully navigate around various pages. You can try something like that, but the chances of it working well are slim unless the game is incredibly simple, and acts a lot like a web app where you can jump from one area to a completely different part of the app without causing continuity errors.

    If your game is super simple, then you can use most of these options, and set up the code for something that eternally runs a single test, looking for buttons to click, fields to type stuff in, etc. But it isn't a simple thing, and you will need to write code to tailor it to your game to make it work.
     
  29. jgrahamPokemon

    jgrahamPokemon

    Joined:
    Mar 20, 2020
    Posts:
    2
    This is a slightly different topic, but is there any automation that works with tools made in Unity that just use the UI (a.k.a you don't hit run to use them. They are UI extensions of the Unity framework).

    I guess I'm looking for something along the lines of the EditorApplication class, but to be able to manipulate our custom UI of the tool we made.

    https://docs.unity3d.com/ScriptReference/EditorApplication.html
     
    Last edited: Jun 23, 2020
  30. gwaredd

    gwaredd

    Joined:
    Oct 22, 2012
    Posts:
    8
    Hey, the general approach to deliver what you're asking for is something called Imitation Learning which is a form of reinforcement machine learning. However this is cutting edge stuff and at the moment out of reach in a production environment to all but the most sophisticated AAA studios.

    Some people write systems to record and playback controller input but given that games are generally not deterministic, the game state gets out of sync with the controller input pretty quickly. Although if you are just after a soak test, sometimes randomly sending input commands could be enough (depending on the game).

    Best approach is to write a player AI and get the game to play itself.

    Is there an out-of-the-box solution which involves no code right now? Not to my knowledge. Although I suspect it wouldn't be too hard to write something that can record controller input to navigate unity menus automatically which can get you halfway there.
     
    Last edited: Jun 27, 2020
    laurentlavigne likes this.
  31. gwaredd

    gwaredd

    Joined:
    Oct 22, 2012
    Posts:
    8
    Hey, unium can help you here. Out-of-the-box it is setup to work for games at runtime but it is relatively easy to modify it to work in the editor as well. If you create an issue on the github page I can show you how.
     
  32. timeapusok

    timeapusok

    Joined:
    Apr 15, 2020
    Posts:
    22
  33. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    Done
     
    timeapusok likes this.
  34. TziAllstar

    TziAllstar

    Joined:
    Nov 18, 2020
    Posts:
    1
    Great post!
    though I noticed rather old.
    What are the current methods for automation testing in Unity/C# these days?
    Would like to know more
     
  35. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    These are still your options. And if someone else created a framework, they unfortunately never googled "Automation Unity" or searched in the forum about it, or else they definitely would have seen this post and then hopefully would have notified me to update it (as some people did since this was created). So I would say that there is a high probability that your options are only what you see here.

    Some of these projects are actively updated or maintained. Mine, Trilleon, is only updated when someone contacts me with an issue. I recently updated it to that it worked with the most recent version of Unity. Check them out if you are looking for the most recent and maintained option(s).
     
    Last edited: Nov 20, 2020
  36. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    9
    I guess I'll need to chime in then :)

    Responsible

    Responsible is a reactive asynchronous testing utility primarily designed for, but not limited to be used in high level system tests. It's inspired by Rx, and uses UniRx under the hood, but can be used without any Rx knowledge. It was designed to run as part of Unity Test Runner execution, but should also work with any other test runner that supports coroutines or observables.

    The primary benefits of using Responsible are:
    • Detailed output on test failures and timeouts (operation statuses and async "stack traces")
    • Declarative, composable, and reusable test code
    • Using Responsible will circumvent a Unity Test Runner bug, where test execution will continue after errors within nested coroutines, sometimes even hiding the first exception.
    Automation Testing:
    Unit Testing: (Not the primary use case)
    Full Framework: X
    CI Integration: X
    External Driver: X
    Report Generation: X
    Parallel Execution: X
    Custom Editor Window(s): X
    github: https://github.com/sbergen/Responsible
    wiki:
    n/a
     
  37. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    301
    Added.
     
    sbergen likes this.
unityunity