What options do I have for Automation and Unit Testing In Unity?

I wanted to create this topic as a registry, listing all of the prominent options for automating UI testing in Unity. I want to give people a high level comparison of what each package offers. People may not realize that there are now many choices to automate UI testing in Unity, along with additional alternative options for unit testing!

**Full disclosure! I created and maintain the Trilleon framework mentioned below. I am also a contributor to the Automated QA package.**

Here are some categories I will mark with a ✓ or an X if the frameworks provide them:
Automation Testing
Unit Testing
Full Framework (Meaning that it fully replicates something like Selenium, instead of extending an existing framework, or representing a shell of one to build off of.)
CI Integration
External Driver
Report Generation
Parallel Execution
Custom Editor Window(s)

-- In alphabetical order --
​

AltTester

It's an end-to-end UI test automation tool allowing you to run tests on real devices (mobile, PCs, etc.) or inside the Unity Editor. It supports tests written in C#, Python and Java.It'll help you to find elements and get all their properties and also to simulate any kind of device input. The desktop app has recording feature, allowing you to generate tests automatically based on recorded actions.​

Automation Testing: ✓
Unit Testing: X
Full Framework: X
CI Integration: ✓
External Driver: ✓ (it can use Appium to interact with the game)
Report Generation: ✓ (test results and reports inside unity editor for C#)
Parallel Execution: ✓
Custom Editor Window(s): ✓
Multiplayer Testing: ✓​

website & download link: https://alttester.com/tools/
github: https://github.com/alttester
wiki: https://alttester.com/docs/sdk/latest/#
discord server for questions and support: https://discord.gg/uZrah5U​

Automated QA (Unity)
[Shelved / Development Paused]

The Automated QA package enables users to record and playback touch or drag interactions with the UI of a Unity Project and optionally use recordings to drive Unity Tests - in the editor, on a local iOS or Android device, or on an Android device managed by Unity in the cloud.

See the full alpha announcement post here.

Automation Testing: ✓ (Record & Playback w/ Customization of generated C# tests)
Unit Testing: ✓X (Full Unit Testing in the associated Unity Test Framework; see below)
Full Framework: ✓X (Links to Unity Test Tools & ML Agents to make larger test ecosystem) (Development in progress)
CI Integration: ✓ (Development in progress)
Cloud & Device Farm: ✓ (Development in progress)
External Driver: ✓X (Uses Appium in Cloud Device Farm test runs to interact with devices)
Report Generation: ✓
Custom Editor Window(s): ✓
Multiplayer Testing: X (On the horizon)

asset store link: Disabled...
github: Disabled...
wiki: Disabled...
product page: Disabled...
​

Minefield

Not even a wrapper to Unity's own Test Framework, it is just an extension library and guidelines to help you write pure Unity Test Framework test cases. Its focus is that you should be able to code up a scene navigation test fast enough that you could do it while creating the scene without disrupting your creation flow. It is possible with help from "test beacon" components added to the scene as a metadata and a various yieldable methods to help click on those beacons in the test. Easy installation via Unity Package Manager. Because you are still writing Unity tests, of course you could use it together with something like Performance Testing package to create a performance test of scenes while navigating around.

Automation Testing: ✓
Unit Testing: X
Full Framework: X
CI Integration: X
External Driver: X
Report Generation: X (possible with performance testing package)
Parallel Execution: X
Custom Editor Window(s): X

github: https://github.com/5argon/Minefield
wiki: https://github.com/5argon/Minefield (README.md)​

Responsible

Responsible is a reactive asynchronous testing utility primarily designed for, but not limited to be used in high level system tests. It's inspired by Rx, and uses UniRx under the hood, but can be used without any Rx knowledge. It was designed to run as part of Unity Test Framework execution, but should also work with any other test framework that supports coroutines or observables.

The primary benefits of using Responsible are:

• Detailed output on test failures and timeouts (operation statuses and async "stack traces")
• Declarative, composable, and reusable test code
• Using Responsible will circumvent a Unity Test Framework bug, where test execution will continue after errors within nested coroutines, sometimes even hiding the first exception.

Automation Testing: ✓
Unit Testing: ✓ (Not the primary use case)
Full Framework: X
CI Integration: X
External Driver: X
Report Generation: X
Parallel Execution: X
Custom Editor Window(s): X
github: https://github.com/sbergen/Responsible
wiki: n/a​

The Puppetry

The Puppetry gives ability to test a game in Unity Editor and when built and deployed to a device. This is done via additional integration with a device's driver (like Appium for mobile, SDK for consoles or SDK for VR glasses). It is cross platform framework primarily written in C#.

The Puppetry doesn't have dependencies from any specific technology so it can be combined with any test runner(Nunit, XUnit, MSTest), any CI system, any reporting tool set, and other required libraries or frameworks.

The main purpose of driver approach is to separate the tests from the application code base, so that you don't worry about how changing tests will affect the application.

And finally, The Puppetry supports parallel execution of tests right out of the box. So you can run your tests simultaneously on different Unity Editors or builds.

Automation Testing: ✓
Unit Testing: X (Perhaps possible, but not explicitly designed for it)
Full Framework: ✓
CI Integration: ✓X (Ostensibly capable, but not included)
External Driver: ✓X (Custom driver w/ Appium for devices)
Report Generation: X (Could not find sign of reporting. Likely has basic result output.)
Parallel Execution: ✓
Custom Editor Window(s): ✓

github: https://github.com/TestUnitLab/Puppetry
wiki: ttps://github.com/GameUnitLab/Puppetry/wiki​

Trilleon

A complete, internal automation framework that is entirely self-contained in your game binary. Communicates with CI process (Jenkins, Bamboo, Pipelines) over simple pubsub messaging framework and sockets. Accepts commands to run tests, and then takes care of the rest. Entirely white box testing solution. Adds massive number of additional options for controlling the order that tests run, along with dependencies between tests and test classes. Extensive editor window, "The Nexus", where you can organize, launch tests, review test results, and customize the framework.

Just as easy to use as a Unit test framework alternative too. Trilleon can handle any Unit test logic that requires the use of Coroutines, or any context specific game code that cannot be tested on compile time!​

Automation Testing: ✓
Unit Testing: ✓ (Atypical; not run at compile time.)
Full Framework: ✓
CI Integration: ✓ (Scripts supplied fully for Mac, partially for Windows)
External Driver: ✓X (Uses Appium to load apps onto devices; no further driver usage)
Report Generation: ✓ (Friendly Html/Js/Css report, Standard XML report, In-editor results window)
Parallel Execution: ✓ (Supports through Selenium Grid)
Custom Editor Window(s): ✓​

github: https://github.com/disruptorbeam/trilleon
wiki: https://github.com/disruptorbeam/trilleon/wiki/Welcome-to-Trilleon​

Unity Test Framework
(formerly Unity Test Runner)

The built-in way of running unit tests, and scenario-based automation tests. The latter requires test scenes, and does not test the real scene, on devices, as a user would.

Automation Testing: ✓X (Yes & no; it is not true automation tests, but more like a Unit-Integration/Automation hybrid)
Unit Testing: ✓
Full Framework: ✓
CI Integration: ✓ (CLI provided here)
External Driver: X
Report Generation: ✓ (Standard xml report, In-editor results window.)
Parallel Execution: X✓ (Not for automation tests)
Custom Editor Window(s): ✓

wiki: https://docs.unity3d.com/Packages/com.unity.test-framework@1.0/manual/index.html
​

Unium

Unium is an experimental library for the purposes of facilitating automated testing and tool development for your Unity games.

Automation Testing: ✓
Unit Testing: X
Full Framework: X
CI Integration: ✓X (Ostensibly capable, but not explicitly)
External Driver: X (Facilities use of external drivers. No dependencies on them.)
Report Generation: X (Facilitates integration of existing testing frameworks; ex: mocha, jest, pytest, specflow, etc...)
Parallel Execution: ? (Unknown)
Custom Editor Window(s): ✓​

github: https://github.com/gwaredd/unium
wiki: n/a

​

Unity UI Test Automation

This is a basic wrapper for running ui automation tests that works with the Unity Test Framework. The information below describes what this tool offers only, and not what it offers in addition to the Unity Test Framework.

Automation Testing: ✓
Unit Testing: X
Full Framework: X
CI Integration: X
External Driver: X
Report Generation: X
Parallel Execution: X
Custom Editor Window(s): ✓
github: https://github.com/taphos/unity-uitest
wiki: n/a

​

*If you are an owner of one of these repositories, feel free to provide me an updated elevator pitch for your toolset. Correct me on any mistakes I make. Additionally, if you have an alternative tool to add to this list, let me know as well.*

 

Thanks, @superpig

I've updated the post.

 

Much needed topic, It should be Pinned

 

Great work!

I just want to say that I am also starting a new play mode testing library called Minefield. This completely depends on Unity Test Runner, it is just an extension that adds some useful methods. Though it is still in development currently it could navigate the scene the way I want to. Maybe it is still not that fully fledged like other on your list yet ..but it is getting there.

 

Write me a primer like those above, and I will add you to the list.

 

Hey, great list .

A small correction, whilst you certainly can use unium and appium togther, there's no dependency there. Appium is not required neither is unium a wrapper for it in any way - it shares the some of the same philosophy in it's approach but none of the implementation details.

Also, for clarity there is no report generation - none, zero, zip, nada

Rather - it facilitates existing testing frameworks (mocha, jest, pytest, specflow, etc...) and all the goodness that comes with them.

 

Thanks for the corrections and updates. Just added them.

 

Alright, I just hit a milestone on my game with Minefield and quite confident of the state of the API now.

MINEFIELD

Not even a wrapper to Unity's own Test Runner, it is just an extension library and guidelines to help you write pure Unity Test Runner test cases. Its focus is that you should be able to code up a scene navigation test fast enough that you could do it while creating the scene without disrupting your creation flow. It is possible with help from "test beacon" components added to the scene as a metadata and a various yieldable methods to help click on those beacons in the test. Easy installation via Unity Package Manager. Because you are still writing Unity tests, of course you could use it together with something like Performance Testing package to create a performance test of scenes while navigating around.

Automation Testing: ✓
Unit Testing: X
Full Framework: X
CI Integration: X
External Driver: X
Report Generation: X (possible with performance testing package)
Parallel Execution: X
Custom Editor Window(s): X
github: https://github.com/5argon/Minefield
wiki: https://github.com/5argon/Minefield (README.md)

 

Added. Thanks.

 

Hey, thanks for this list! Its just what we were looking for. While looking for testing tools, we also found this one: https://www.gamedriver.io/
Have you tried it by chance? Maybe you can add it to the list

 

So without being able to freely view the code for gamedriver.io on something like Github, I cannot verify things in the same way as the examples in my first post.

Also, at the moment, I am not going through the the process to get a free 30 day trial and access to the software.

I am more than happy to add a profile for them if a representative of gamedriver.io can write up a pitch for me, and give me at least some limited visibility into their product to reasonably confirm the claims.

 

@tsibiski Trilleon sounds like something I would want to try. How do we use Trilleon on a closed source Unity project and comply with section 4.d of the LGPL? Specifically, how do we provide a way for users to re-link different versions of the Trilleon library? I don't think there is a way to provide that when building the Unity game to Android or iOS. What do you think about MIT, BSD, Apache licenses to help increase adoption rates? Projects using those licenses still get plenty of pull requests to help improve the software.

 

Hi @chrislouierhino
Unfortunately, I did not choose the license and don't have control over what license is used. I think that aspect of its use was probably a mistake to omit when choosing a license, rather than indicative of any intent to restrict its use. The intent was that no one took the project and claimed it as their own creation, or tried to make money off of it. There was no other intent behind the license used.

I will try to contact DisruptorBeam to ask for a change to the license, but I have not worked there in 4 years now. So I can't make any guarantees. Given that I know the intent, I personally would not hesitate to use it in the scenario you supplied above. For what it's worth.

 

I contacted someone from the C level, and they also said that it is not intended to be that restrictive. But they will confirm everything and, if agreed, update it. Likely to be MIT if so.

 

Hi,
Great topic, this list is very helpful.

I'd also like to suggest a tool.
AltUnity Tester, is a UI test automation tool allowing you to run tests on real devices (mobile, PCs, etc.) or inside the Unity Editor.
We started developing it a few years ago, after struggling to interact with objects from Unity Games.

Hope it will help some of you. It's open-source, free and it can be downloaded from the Asset Store: https://assetstore.unity.com/packages/tools/utilities/altunity-tester-ui-test-automation-112101

 

Could you do me a favor and write up a description of it in the same format as the existing tools above? (Hopefully marking the relevant features like the other tools do). Thanks!

 

@tsibiski here's the description in the format you asked for. Let me know if you need more info.

AltUnity Tester

It's an end-to-end UI test automation tool allowing you to run tests on real devices (mobile, PCs, etc.) or inside the Unity Editor. It supports tests written in C#, Python and Java.

It'll help you to find elements and get all their properties and also to simulate any kind of device input.


Automation Testing: ✓

Unit Testing: X

Full Framework: X

CI Integration: ✓

External Driver: ✓ ( it can use Appium to interact with the game)

Report Generation: ✓ ( test results and reports inside unity editor for C#)

Parallel Execution: X ( is possible, but no special implementation in the tool for it)

Custom Editor Window(s): ✓

Multiplayer Testing: ✓



asset store link: https://assetstore.unity.com/packages/tools/utilities/altunity-tester-ui-test-automation-112101

gitlab: https://gitlab.com/altom/altunity/altunitytester

wiki: https://altom.gitlab.io/altunity/altunitytester/

gitter room for questions and support: https://gitter.im/AltUnityTester/Lobby

 

Thanks, I added your tool.

 

Hey, thanks for making this list. I'm overloaded with choice!

We are new to automated testing and not sure which is the best one to use. We are working on a mobile puzzle game and we want to have the game run through the levels overnight and check everything is working.

Which ones have people had success with and would recommend?

 

@mfvnlrxu If you're just getting started, here you can find few example projects.
There are tests written in Java, C# and Python.
Hope it helps!

 

I've been research automated testing and just came across AirTest. I haven't seen many people talking about it so I thought I should share. The really interesting thing about it is that it uses image recognition to find elements to interact with, without requiring changes to your game. If you are willing to add a plugin to your game they have a library called Poco that adds more functionality. It was developed by NetEase (a Chinese mobile game company) and announced in partnership with Google at GDC 2018. I found it because I was looking at Google's Firebase Test Lab cloud mobile test tools, which this works with.

 

This is called image recognition testing, and is highly unreliable, inflexible, and extremely difficult and time consuming to maintain. It is the original way to test games. Baseline images are required in enormous numbers, and if your game changes at all after they are taken, it requires a lot of updates to the baseline images. Additionally, you have to account for very different scenarios based on gameplay resolution and the size and scaling of objects. Not only that, but it is almost useless outside of extremely static games with primarily UI components.

That is the last type of automation I would recommend for video games. It will ultimately be a miserable nightmare. I am an automation engineer with 10 years experience, and 5 of those in the video game industry if that matters for my opinion.

 

Thanks for you input @tsibiski, I don't have any experience with image recognition testing so it's helpful to hear that it is unreliable. I'd still be curious what you think of their Poco library that uses scene hierarchy.

 

From a cursory look, I'd say Poco is not really all that different than many of the solutions above, so it sounds like a viable option.

 

ok so let's say I'm making a game on console and need to auto play it for hours or days to check for edge cases, leaks etc...
Which package allows me to do that without me writing a line of code?
Ideally monkeying me playing the game for a while then repeating the input over and over and if the game crashes, log how long that took, memory at the time etc...

 

See my reply on the other thread i don't think such a tool exists. But depending on your game you may be able to come up with something like that yourself.

 

EDIT: The Unity Automated QA package has a GameCrawler that I made for it. It works reasonably well. However, that package is currently shelved and does not have support from Unity at this time.

As mentioned above, there is no such thing. Games aren't like web browsers where you can set up a crawler and have it successfully navigate around various pages. You can try something like that, but the chances of it working well are slim unless the game is incredibly simple, and acts a lot like a web app where you can jump from one area to a completely different part of the app without causing continuity errors.

If your game is super simple, then you can use most of these options, and set up the code for something that eternally runs a single test, looking for buttons to click, fields to type stuff in, etc. But it isn't a simple thing, and you will need to write code to tailor it to your game to make it work.

 

This is a slightly different topic, but is there any automation that works with tools made in Unity that just use the UI (a.k.a you don't hit run to use them. They are UI extensions of the Unity framework).

I guess I'm looking for something along the lines of the EditorApplication class, but to be able to manipulate our custom UI of the tool we made.

https://docs.unity3d.com/ScriptReference/EditorApplication.html

 

Hey, the general approach to deliver what you're asking for is something called Imitation Learning which is a form of reinforcement machine learning. However this is cutting edge stuff and at the moment out of reach in a production environment to all but the most sophisticated AAA studios.

Some people write systems to record and playback controller input but given that games are generally not deterministic, the game state gets out of sync with the controller input pretty quickly. Although if you are just after a soak test, sometimes randomly sending input commands could be enough (depending on the game).

Best approach is to write a player AI and get the game to play itself.

Is there an out-of-the-box solution which involves no code right now? Not to my knowledge. Although I suspect it wouldn't be too hard to write something that can record controller input to navigate unity menus automatically which can get you halfway there.

 

Hey, unium can help you here. Out-of-the-box it is setup to work for games at runtime but it is relatively easy to modify it to work in the editor as well. If you create an issue on the github page I can show you how.

 

Hey @tsibiski,
Could you please make a quick update to AltUnity tester?
Please replace: "gitter room for questions and support: https://gitter.im/AltUnityTester/Lobby" with "discord server for questions and support: https://discord.gg/uZrah5U"

We're no longer active on gitter, so it would be useful if you could update this. Thanks a lot!

 

Done

 

Great post!
though I noticed rather old.
What are the current methods for automation testing in Unity/C# these days?
Would like to know more

 

These are still your options. And if someone else created a framework, they unfortunately never googled "Automation Unity" or searched in the forum about it, or else they hopefully would have seen this post and then hopefully would have notified me to update it (as some people did since this was created). So I would say that there is a high probability that your options are only what you see here.

Some of these projects are actively updated or maintained. Mine, Trilleon, is normally only updated when someone contacts me with an issue. I recently updated it to that it worked with the most recent version of Unity. Although I sometimes make additional updates based on my continued usage. Check them out if you are looking for the most recent and maintained option(s).

 

I guess I'll need to chime in then

Responsible

Responsible is a reactive asynchronous testing utility primarily designed for, but not limited to be used in high level system tests. It's inspired by Rx, and uses UniRx under the hood, but can be used without any Rx knowledge. It was designed to run as part of Unity Test Runner execution, but should also work with any other test runner that supports coroutines or observables.

The primary benefits of using Responsible are:

• Detailed output on test failures and timeouts (operation statuses and async "stack traces")
• Declarative, composable, and reusable test code
• Using Responsible will circumvent a Unity Test Runner bug, where test execution will continue after errors within nested coroutines, sometimes even hiding the first exception.

Automation Testing: ✓
Unit Testing: ✓ (Not the primary use case)
Full Framework: X
CI Integration: X
External Driver: X
Report Generation: X
Parallel Execution: X
Custom Editor Window(s): X
github: https://github.com/sbergen/Responsible
wiki: n/a

 

Added.

 

Holy moly what a great post! Thank you for your work and keeping things up to date! Now i've got some exploring to do

 

Hi tsibiski,
I'd be happy to get you what you need to verify GameDriver. I'll send you a DM too.

 

Added a new option to the list of automation tools. This one is straight from Unity. A new tool to help automate the QA process.

-------------------------------

Automated QA (Unity)
[Alpha Release]

The Automated QA package enables users to record and playback touch or drag interactions with the UI of a Unity Project and optionally use recordings to drive Unity Tests - in the editor, on a local iOS or Android device, or on an Android device managed by Unity in the cloud.

See the full alpha announcement post here.

Automation Testing: ✓ (Record & Playback)
Unit Testing: ✓X (Full Unit Testing in the associated Unity Test Runner; see Unity Test Runner in list)
Full Framework: ✓X (Links to Unity Test Tools and ML Agents to make larger test ecosystem) (Development in progress)
CI Integration: ✓ (Development in progress)
Cloud & Device Farm: ✓ (Development in progress)
External Driver: N/A (Not determined yet. Development in progress)
Report Generation: X
Custom Editor Window(s): ✓
Multiplayer Testing: X

asset store link: Coming Soon...
github: Coming Soon...
wiki: https://docs.unity3d.com/Packages/com.unity.automated-testing@latest​

 

Hello @tsibiski, and thank you for the effort of maintaining this very helpful thread.
I am researching the automation topic for the first time, and it is my understanding that
Unity Test Runner is now known as Unity Test Framework (https://docs.unity3d.com/Packages/com.unity.test-framework@1.0/manual/index.html)
I believe it would be helpful, to reflect that in the List - to avoid any confusion, you could keep the old name like the official Unity documentation does:

Unity Test Framework package (formerly the “Unity Test Runner”) ​

 

Thanks. I've updated cases where it is referred to the Unity Test Runner to Unity Test Framework. I left a note about it being formerly called that. And I left one mention of a "Test Runner" that is a general use of the term and not referring directly to Unity's framework.

 

Regarding edge case testing, image recognition and ML, check this out https://openai.com/blog/universe/

 

Hello,
The project that I am working on is huge and now team decided that we need something for automatic testing, Could someone tell me if we would use lets say AltUnity Tester does every object in our game needs a unique name/ID for AltUnity Tester to work because renaming all objects in our project would take a lot of time and we would like to avoid that. From what I read here image recognition is unreliable so are there any other options if im not mistaken about renaming ?
Thanks

 

Thanks for bringing this to my attention @timsibiski!

Hey @Vaelse! Besides ID's AltUnity Tester allows you to find objects by name, tag, component, path, etc. Here you can find all the selectors it supports: https://altom.gitlab.io/altunity/altunitytester/pages/commands.html#by-selector
Also, it has an easy way to add IDs to the objects when you create the test build. More info about this here: https://altom.gitlab.io/altunity/altunitytester/pages/commands.html#altid

 

Hey all, figured this would be the place to ask, but does anybody have any good resources on how to go about writing automated tests for custom editor windows? I have a custom editor that is built atop the GraphView API and I am trying to figure out if there is a way to simulate user actions within the window and check results similar to how one would go about doing so with something like Selenium webdriver.

Thanks in advance!

 

As long as it inherets from UIElements.VisualElement, then you can automate it. But it will require customization and creating your own driver class or own driver methods to interact with those elements - unless one of the above options already has wrappers for that. My guess is that the only one that might is AltUnityTester. But even then I am not so sure. You can certainly make your own driver methods to accomplish this quite easily in Trilleon, for example, though.

 

Following @Io-94 request here.

I'm developping custom editor windows but it's quite difficult to automate their tests with Unity Test Runner : at best I can open the window but it's not repainted neither updated (window is completely white) until the test is finished. I have to test everything in Editor mode (not Play mode), of course. Didn't found how to press a button yet either, nothing seems to work lol (even with integrated windows, like "Preferences").

AltUnityTester (free version) is not on the Asset store anymore and the installation required dependencies that are not linked so it's also a pain in the ass to install manually.

I didn't find a way to run test in editor mode with Trilleon either as

Code (CSharp):

1. TrilleonAutomation.AutomationMaster.Initialize();

requires to be run in Play mode too, as far as I understand it. The tool looks complete tho, but I'm not sure if it can help my case.

@tsibiski, do you have experience in the matter ? Maybe you can provide me a sample test or redirect me to any resource ? Not much stuff online for this particular subject I'm afraid. Thank you mate

 

Unfortunately, all of these solutions, to my knowledge, require a game to be started/in play mode. And none of them allow for testing of an editor window, rather than actual game content. In fact, you will need something that has a scope outside of the game, like a traditional automation framework that could interact with the editor itself. You may want to try Appium, and see if it can start the editor, and there is a tool in Windows that lets you inspect any application to get every element's details. I forget what it's called, but search for tools that let you inspect compiled applications to find elements your automation can interact with.