Trillion - QA Automation Testing Framework - Suggestions/Ideas?

Hello,

I have developed a QA automation framework that is showing a lot of promise (tentatively named Trillion). I intend to build important plugins to extend the framework that I am developing and make the entire automation test-writing process reliable, simple, and I'd even say - fun.

I would put this in the Work in Progress forum, but I don't have anything I wish to demo at this time. I am less trying to show it off as I am trying to get input at this juncture.

Below, I have put the overview of the project, and would appreciate your thoughts. I would like to gauge interest for this as I would need to do some extra work to make this all available beyond my team's projects on the Asset Store. Also, I appreciate all advice, and am happy to answer questions if anything is not elaborated on enough, missing, etc.

Trillion
A Complete QA Test Automation Package​

-Components-​

Framework

The framework is the core library that bridges the gap between test automation code and Unity game code. It provides the necessary resources to make a game test itself. This test code mimics actual player input using Unity's own tools. It wraps and extends these resources in a manner that makes writing a test easy, quick, and intuitive. The more a consumer extends the framework in ways custom to their own products, the faster their test development process is. It is inherently scalable and meets the demand of flexible and consistent code that produces reliable and valuable results.


Test Guide

The test guide allows inspection of any game object in the hierarchy. All relevant details about the object are displayed to allow for an informed decision on the next step in a test's development. A user can choose to interact with an element and the game in a number of ways that allow for extremely accurate approaches to developing the perfect automation test. After choosing how you intend to interact with the object in your test, the test guide will generate a line of code that meets your specifications and requirements. In this manner, a user may write entire tests with minimal manual code writing and a greatly reduced need to debug.


Recorder

The recorder is a tool that, when activated, records every action you take while playing the game in editor mode. These actions are converted into code that accomplishes exactly what you just executed. In this manner, an entire automation test will be completed and ready for service the moment you stop recording.

-Drill Down-​

Test Guide​

--Main Tab--

[Fields]

Step Type – This is a drop down of code step types. For example, Act On Object, Wait Real Time, Make Assertion etc. Based on the selection, other fields will appear or disappear. Assuming one selects Act On Object, the following fields will appear:

Object – Drag object from hierarchy or Assets into this field. Automatically populates all relevant fields with data usable to find this object during an automation test
Object Fields[] – Shows all data pertaining to the object that is relevant for locating an object with your test script

Action– What action do you want to do to this object? (Click, Set Variable, Send Text To, Drag etc.)

By – Dropdown of all options from Object Fields that have valid identifiers/values. This will be how your test step will find this object during script execution

WaitFor – Has default value. If a value greater than zero is entered, the code will wait for the requested object to exist, or for the requested condition to be true, before performing the requested action. This means that if you have a 1 second animation between two steps, the code for the second step will not fail because it activated before the animation completed.

Assert – Optional assertion to do on this object or a property of this object.

Generate – Copies the calculated line(s) of code for this entire step to your clipboard. This code is based on the automation framework provided

**Note, there will be further explanation of fields for other Step Type choices, but they are more straightforward in the options provided.**

Recorder​

--Main Tab--

[Buttons]

Start – Begin recording
Pause – Pause recording (essentially a stop that can be restarted)

Delete Last – Removes the most recent recorded step (mistaken action?)

Stop – Stop recording

*Generate – Saves recording to a temp text file and copies to clipboard (appears after stop clicked if recording was active)

*New Test – Creates a new test class in your set directory that adds this as a new test

**Class Name – Name of new test class (Appears after clicking New Test)
**Test Name – Name of new test (Appears after clicking New Test)

**Create – Creates new test with class name and test name (Appears after clicking New Test)


[Fields]
Last Step – Describes last recorded step in plain language (Option to make action a try action which means, "don't fail the test if we can't complete this action".)

Status – Displays the status of the recorder (Recording, Paused, Stopped)


--Settings Tab--

New Tests Path – Sets the directory path under Assets folder where new tests are saved

Allow Direct Code Edit – Check box that sets whether the Code Tab can be edited directly while recording


--Code Tab--

Code – Shows every line of code and allows certain edits for each line


--Components--

Listener– Listens for actions while active, and records every action of substance that the player takes

Translator – Determines what the last action was and writes code to replicate the action

Settings Manager –Takes user supplied and default settings to drive other components and build a final test script upon recording stop

Framework​

I cannot drill down into the framework at this time, although it is essentially complete. It provides the ability to listen for external commands on a socket (Pubnub channel? / Launch via Jenkins through cloud device farm!) along with cheat console integration for launching automation tests. It wraps Unity code in a manner very similar, and indeed inspired by, Selenium. It makes writing your automation scripts for Unity as simple as writing web application tests in Selenium.

 

Sounds cool. But is it quicker and cheaper then hiring a spotty teenager and keeping him supplied with pizza?

Seems like on any complex game you'd spend more time developing the tests then you would doing traditional QA.

 

That's a good question that almost every developer who considers automation tests grapples with before they finally decide to move beyond or go with an automation framework.

Automation is critical in games that use continuous delivery models (not necessarily games with expansions etc.). A game that sells itself by, or relies on, being continuously updated with new events, content updates, etc. over the span of many months or several years needs to strongly consider an automation framework.

There is a risk to continuous development, and constantly refactoring or extending our existing code. We never know for sure, until a tester regression tests every last bit of functionality, that we haven't broken the live game with our newest round of code (and even then...).

Now, many developers hire an army of manual testers under contract, with many being released when the game is released. However, this model doesn't exactly work for continuous development models. A game that is forever evolving needs a full time group of manual testers to regression test everything after ever build. As the game grows more complex, this becomes a battle of diminishing returns with any existing team. Covering the most critical and fragile areas of the game with automation tests that can be launched by a Jenkins build increases in value with every sprint/cycle/iteration.

Starting early with automation tests will pay dividends later. Now, for a traditional game's development, a full automation suite seems like a pretty tricky idea, since the suite will be complete when the game is complete. However, even traditional development schemes can benefit greatly from a very limited group of smoketest automation tests.

If you have more or less completed a tutorial sequence that uses inherited classes from the main game, then any change to the rest of the game could potentially affect the tutorial. If you write a small set of automation tests to cover it, you will not spend much time making them, and you will know the moment you have made any part of your tutorial behave differently.

So automation does not need to share the spirit of Unit tests; with end-to-end coverage. It can do so if needed, but it can be a single test. And with the recorder tool, someone with limited time or will to devote to testing coverage can quickly get a small set of tests spun up to cover the most critical part of the game. Manual testers will appreciate knowing that part of the game has an extra set of security covering their eyes, and they will be ostensibly able to focus on other, more complex and less repetitive tasks.

 

tsibiski,

Your project sounds extremely useful - especially as you seen to have modeled it after the very popular (and super useful in past projects) Selenium framework. Looking forward to anything you release in the future along this path. Any expected release date (either in beta or final form)? If you are looking for additional 'testers', please reach out via PM! Thanks for your efforts!

 

Hi Jeff,

I can't say with certainty when I will be able to show this off. My company owns this property that I am making, and I know they are on board with sharing the product, but I need to work through that situation first.

Then, I need to decide how I will implement it. I may begin by offering the test library for people to start building tests manually. I might then update it as soon as the Test Guide and Recorder are correct and bug free.

Or, it may be that I must wait until all three pieces of the product are complete before I can make it available.

It ultimately depends on how much time outside of work I can devote to this - essentially overtime given the situation. In any event, I'd expect to see a very solid and presentable version ready by the end of this year.

I will definitely keep people apprised of my progress, though. I will certainly be open to people taking the product for a test drive before its ready, and if I am allowed to, I will contact those interested, including yourself.

Thanks!

 

First thought: Trillian is already a really popular chat client, and this is pronounced the same. I'd avoid the name.

 

Very interested in this. Any recent news?

 

Hi bitbutter,

The framework is "tentatively complete" in its first form. It is essentially in its alpha/beta form, and I hope to get permission to announce it in its entirety ASAP.