Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

What options do I have for Automation and Unit Testing In Unity?

Discussion in 'Testing & Automation' started by tsibiski, May 22, 2019.

  1. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    274
    I wanted to create this topic as it does not seem that anyone has done so yet. There seems to be very little discussion of ANY automation tools in Unity outside of the Unity Test Tools. People may not realize that there are now SEVERAL choices to automate UI testing in Unity, along with additional options for unit testing!

    **Full disclosure! I created and maintain the Trilleon framework mentioned below**

    Here are some categories I will mark with a or an X if the frameworks provide them:
    Automation Testing
    Unit Testing
    Full Framework
    (Meaning that it fully replicates something like Selenium, instead of extending an existing framework, or representing a shell of one to build off of.)
    CI Integration
    External Driver
    Report Generation
    Parallel Execution
    Custom Editor Window(s)


    -- In alphabetical order --
    MINEFIELD

    Not even a wrapper to Unity's own Test Runner, it is just an extension library and guidelines to help you write pure Unity Test Runner test cases. Its focus is that you should be able to code up a scene navigation test fast enough that you could do it while creating the scene without disrupting your creation flow. It is possible with help from "test beacon" components added to the scene as a metadata and a various yieldable methods to help click on those beacons in the test. Easy installation via Unity Package Manager. Because you are still writing Unity tests, of course you could use it together with something like Performance Testing package to create a performance test of scenes while navigating around.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration: X
    External Driver: X
    Report Generation: X (possible with performance testing package)
    Parallel Execution: X
    Custom Editor Window(s): X
    github: https://github.com/5argon/Minefield
    wiki: https://github.com/5argon/Minefield (README.md)​


    THE PUPPETRY

    The Puppetry gives ability to test a game in Unity Editor and when built and deployed to a device. This is done via additional integration with a device's driver (like Appium for mobile, SDK for consoles or SDK for VR glasses). It is cross platform framework primarily written in C#.

    The Puppetry doesn't have dependencies from any specific technology so it can be combined with any TestRunner (Nunit, XUnit, MSTest), any CI system, any reporting tool set, and other required libraries or frameworks.

    The main purpose of driver approach is to separate the tests from the application code base, so that you don't worry about how changing tests will affect the application.

    And finally, The Puppetry supports parallel execution of tests right out of the box. So you can run your tests simultaneously on different Unity Editors or builds.

    Automation Testing:
    Unit Testing: X (Perhaps possible, but not explicitly designed for it)
    Full Framework:
    CI Integration: X (Ostensibly capable, but not included)
    External Driver: X (Custom driver w/ Appium for devices)
    Report Generation: X (Could not find sign of reporting. Likely has basic result output.)
    Parallel Execution:
    Custom Editor Window(s):
    github: https://github.com/TestUnitLab/Puppetry
    wiki: ttps://github.com/GameUnitLab/Puppetry/wiki


    TRILLEON

    A complete, internal automation framework that is entirely self-contained in your game binary. Communicates with CI process (Jenkins, Bamboo, Pipelines) over simple pubsub messaging framework and sockets. Accepts commands to run tests, and then takes care of the rest. Entirely white box testing solution. Adds massive number of additional options for controlling the order that tests run, along with dependencies between tests and test classes. Extensive editor window, "The Nexus", where you can organize, launch tests, review test results, and customize the framework.

    Just as easy to use as a Unit test framework alternative too. Trilleon can handle any Unit test logic that requires the use of Coroutines, or any context specific game code that cannot be tested on compile time!​

    Automation Testing:
    Unit Testing: (Atypical; not run at compile time.)
    Full Framework:
    CI Integration: (Scripts supplied fully for Mac, partially for Windows)
    External Driver: X (Uses Appium to load apps onto devices; no further driver usage)
    Report Generation: (Friendly Html/Js/Css report, Standard XML report, In-editor results window)
    Parallel Execution: (Supports through Selenium Grid)
    Custom Editor Window(s):



    UNITY TEST RUNNER

    The built-in way of running unit tests, and scenario-based automation tests. The latter requires test scenes, and does not test the real scene, on devices, as a user would.

    Automation Testing: X (Yes & no; it is not true automation tests, but more like a Unit-Integration/Automation hybrid)
    Unit Testing:
    Full Framework:
    CI Integration: (CLI provided here)
    External Driver: X
    Report Generation: (Standard xml report, In-editor results window.)
    Parallel Execution: X(Not for automation tests)
    Custom Editor Window(s):


    wiki: https://docs.unity3d.com/Manual/testing-editortestsrunner.html


    UNIUM

    Unium is an experimental library for the purposes of facilitating automated testing and tool development for your Unity games.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration: X (Ostensibly capable, but not explicitly)
    External Driver: X (Facilities use of external drivers. No dependencies on them.)
    Report Generation: X (Facilitates integration of existing testing frameworks; ex: mocha, jest, pytest, specflow, etc...)
    Parallel Execution: ? (Unknown)
    Custom Editor Window(s):

    github: https://github.com/gwaredd/unium
    wiki: n/a

    UNITY UI TEST AUTOMATION

    This is a basic wrapper for running ui automation tests that works with the Unity Test Runner. The information below describes what this tool offers only, and not what it offers in addition to the Unity Test Runner.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration: X
    External Driver: X
    Report Generation: X
    Parallel Execution: X
    Custom Editor Window(s):
    github: https://github.com/taphos/unity-uitest
    wiki: n/a


    *If you are an owner of one of these repositories, feel free to provide me an updated elevator pitch for your toolset. Correct me on any mistakes I make. Additionally, if you have an alternative tool to add to this list, let me know as well.*
     
    Last edited: Jul 15, 2019
    5argon and jGate99 like this.
  2. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,200
    Unity Test Runner is usable in CI - as documented you can launch it test runs from the command line.
     
    liortal likes this.
  3. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    274
    Thanks, @superpig

    I've updated the post.
     
  4. jGate99

    jGate99

    Joined:
    Oct 22, 2013
    Posts:
    898
    Much needed topic, It should be Pinned
     
    tsibiski likes this.
  5. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,294
    Great work!

    I just want to say that I am also starting a new play mode testing library called Minefield. This completely depends on Unity Test Runner, it is just an extension that adds some useful methods. Though it is still in development currently it could navigate the scene the way I want to. Maybe it is still not that fully fledged like other on your list yet ..but it is getting there.
     
    tsibiski likes this.
  6. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    274
    Write me a primer like those above, and I will add you to the list.
     
  7. gwaredd

    gwaredd

    Joined:
    Oct 22, 2012
    Posts:
    6
    Hey, great list :).

    A small correction, whilst you certainly can use unium and appium togther, there's no dependency there. Appium is not required neither is unium a wrapper for it in any way - it shares the some of the same philosophy in it's approach but none of the implementation details.

    Also, for clarity there is no report generation - none, zero, zip, nada ;)

    Rather - it facilitates existing testing frameworks (mocha, jest, pytest, specflow, etc...) and all the goodness that comes with them.
     
    Last edited: Jun 15, 2019
    tsibiski likes this.
  8. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    274
    Thanks for the corrections and updates. Just added them.
     
  9. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,294
    Alright, I just hit a milestone on my game with Minefield and quite confident of the state of the API now.

    MINEFIELD

    Not even a wrapper to Unity's own Test Runner, it is just an extension library and guidelines to help you write pure Unity Test Runner test cases. Its focus is that you should be able to code up a scene navigation test fast enough that you could do it while creating the scene without disrupting your creation flow. It is possible with help from "test beacon" components added to the scene as a metadata and a various yieldable methods to help click on those beacons in the test. Easy installation via Unity Package Manager. Because you are still writing Unity tests, of course you could use it together with something like Performance Testing package to create a performance test of scenes while navigating around.

    Automation Testing:
    Unit Testing: X
    Full Framework: X
    CI Integration: X
    External Driver: X
    Report Generation: X (possible with performance testing package)
    Parallel Execution: X
    Custom Editor Window(s): X
    github: https://github.com/5argon/Minefield
    wiki: https://github.com/5argon/Minefield (README.md)
     
    Last edited: Jun 27, 2019
  10. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    274
    Added. Thanks.
     
    5argon likes this.