Assets [RELEASED] Trilleon - Full Integration Automation and Unit Test Framework

Trilleon
Built and owned by DisruptorBeam! (www.DisruptorBeam.com)
Further supported by GSN (Game Show Network) Games! (www.worldwinner.com & www.gsn.com)

Download Now: v1.1.4

- Please DO NOT send any reports of issues to DisruptorBeam or GSN. Send them to me, or post them on the Trilleon Github: https://github.com/disruptorbeam/trilleon/issues.
- The minimum requirement to begin working with Trilleon is to insert the Automation folder into your Assets folder, and add "TrilleonAutomation.AutomationMaster.Initialize()" to an appropriate bootstrap method that begins loading your game.
- I will be making some tutorial videos and updating the Trilleon wiki over the near future.

New! (Building) Trilleon Wiki (I need to update the wiki with the link to our github, it currently says "coming soon")

Traditional QA Test Automation Framework AND Unit Test Framework

The Trilleon test automation and unit test framework is a comprehensive library of code built for C# in the Unity engine. Modeled (*1) "spiritually" after Selenium, it is the first comprehensive Unity toolset that allows you to build a complete automation (integration) test framework inside of Unity. Additionally, it allows you to write non-standard unit tests that can test code that cannot be tested at compile time (such as IEnumerators or context-specific logic).


The Short and Sweet List – What do I get with Trilleon?

- On Devices, Test Execution Both Locally and In the Cloud (Mac, Android, IOS, WebGL - Execution via a server using Windows will currently require you to modify the script, developed on MacOS, to work in the Windows shell)
- Plays Your Game As Close As Currently Possible To How Any Real User Would
- Near Plug-and-Play Setup Experience
- Selenium-inspired, although notable differences exist
- Actively Sends Test Results and Performance Metrics So You Can Tune Your Game
- Unit Test/Unit-like Support. (Does Not Run Tests At Compile Time. It Is Effectively a Hybrid of Unit Tests and Integration Automation Tests)
- Record and Playback (Beta state)
- Out-Of-The-Box, Jenkins-Ready Server Framework (Appium/Python/Shell Scripts Supplied)
- Framework Impact On Memory Usage Negligible (Great Even For Mobile Games!)
- Several Editor Windows For Easy Development and Launching Of Tests
- Reactive, Visually-Appealing, Informative Test Reports (Html Reports and Simple JUnit Xml). I have formerly done full stack web engineer work, and have used this experience to make all html and css-based logic as appealing as possible.
- Game-Agnostic Client Code Library (C#)
- Wide Array Of Options That Allow Complete Control Over Test Interactions And Dependencies
- Easy To Strip In Production Build Process, But Lightweight If You Choose Not To
- Approximately ~110 files in framework, 28,000 lines of code, and with a footprint of 8.5mb (not including third party code)
- Abstraction For Easy Integration Of Developer-Specific Code (Ex: TextMeshProUGUI?)
- Auto-Generates Test Function Stubs For Interacting With Selected Hierarchy Objects (Soft-disabled in 1.0.0 to receive extensive refactor and upgrades)
- IL2CPP and Mono2x Ready
- Does not work with Web Player platform (No longer supported by Unity)
- Unity 5.x+ Tested; Likely Works With No Or Minimal Modifications 4.x+
- Works In Both .Net 2.0 and .Net 2.0 (Subset)
- Considerate Of Earlier Unity Version Foibles (ex. No ForEach Usage - fixed in Unity 5.5p3+)
- FREE and OPEN SOURCE! The entire core and extension libraries are fully available for your edification. Make any changes you like, anywhere. Submit changes you think others will benefit from in the GitHub repository.
- Trilleon is designed as a basis for automation in all types of games. But you can design and add your own Starter Kits (must be free and open source), such as modified driver classes that are highly specialized for specific genres of games, such as FPS Shooters, Turn-based Strategy RPG's, or Mobile Scrollers.
- Currently Supports Single Player Or Turn-Based Multiplayer Testing (Multiplayer not recently used on my end due to lacking a multiplayer app, and thus this may need several small updates to work properly)
- You can finally Unit Test your Coroutines, and anything else that cannot be validated at compile time.
- Offers two default options for server <--> client communications:
----- (default) Can use a simple python <--> c# socket and threading strategy as a simple pubsub service.
----- Can use a free implementation of ©Pubnub as a pubsub service for communicating between a server and client. ©Pubnub is arguably the best pubsub service available for Unity, and has supported the process Trilleon uses to communicate between Jenkins and Unity client applications. Simple, initial implementation is free, and can be scaled to a paid model as needed.
----- However, you can implement any service you already use, easily, using the provided wrapper classes.


What Are My Options?

Unity Test Tools is the only other full QA test integration framework available to Unity developers. However, it does not provide a substantial footing for development of traditional test automation suites. Unity Test Tools primarily allows you to build scenarios, and to test those scenarios around situations that approximate the game as a user would experience it. The ultimate result becomes an amalgamation of unit testing and automation testing. Additionally, the ability to run such automation outside of the Unity Editor (on devices, and playable builds), as part of an automatic build process, is limited or non-existent.

For years, people have tried developing traditional automation frameworks for their video games that can interact with any game using the same engine. Some have resorted to broadcasting xy coordinates to an external application through a debug log. The external test driver would then parse, interpret, and click the specified coordinates for an intended target. In this way, your automation only has insight into information you explicitly tell the game code to broadcast; setting such a framework up demands that the game be built around this requirement every step of the way. This isn’t a timely, developer-friendly, or scalable approach. Not to mention that heavy use of logging can have a satistically-significant effect on performance.

Other people have built frameworks around image recognition; programmatically comparing baseline images to what the game is rendering to find coordinates of buttons, or confirm text on screen. This process is incredibly unreliable. Your baseline images must be replaced whenever the interface elements change. These baselines must account for aspect ratio, device resolution, and asset quality variants. Anything dynamic or unexpected will throw off the process. Tests developed in this way cannot be relied on as part of a build process.

Enter Trilleon: With the new Disruptor Beam® framework, you will find the tried and true automation approaches of browser applications and standard native applications have been ported into the world of video game development.


Why Should I Use Trilleon?

Trilleon is modeled syntactically after Selenium, offering a multitude of tools built to interact with every aspect of your game, while remaining independent of the actual game code (with the exception of Page/Test Objects that wrap game code interactions in a single, organized place).

Selenium relies heavily on open source support to develop and maintain driver executables. This means that an additional application needs to be run to broker interaction between a test suite and most browsers or applications.

In web and native application automation, this external driver has access to a DOM consisting of XML formatted in child-parent hierarchical relationships. This simple, but highly-ordered structure is easy for an external process to read through and find the object that it needs to interact with. Both the application and driver develop a mutual understanding of what object is being referenced, and the driver in turn is delivered the information needed to simulate an interaction. Video games are a completely different animal. The volatile and mutative nature of a video game is not conducive to a DOM. Therefore is no easy way to retrieve information on game objects from outside the game. Indeed, if there were, it could be a security hazard.

Because of this “middle man” driver in web and native app automation, it becomes difficult to completely rely on the results of your automation, such that you can integrate your automation suite into a Jenkins build step that marks the build as unstable if any test failures occur. What this ultimately means is that, in very large automation suites, it is almost guaranteed that one or more tests will fail simply because the driver encountered a transient error that is out of your control. Because of this, gated builds relying on automation results are never as reliable as unit tests are.

This is a liability that Trilleon eliminates; there is no true driver for interaction between tests and the game (Although appium is used within Jenkins CI to load and launch the game, and dismiss device level alerts. But it does not execute any test code). With Trilleon, your game tests itself. The only thing that prevents you from investing full confidence in the results of your tests is your confidence in the quality of the test code (and how it interacts with your game).

Trilleon provides a diverse selection of test-manipulation tools; some of which even Selenium does not offer. An example of this supra-Selenium functionality is the Dependency Architecture system. With this option, you can map relationships between tests throughout your framework, or exert complete control over the order tests run within the context of the entire suite, linked test classes, or individual test classes. The framework has several layers of validation that prevents and eliminates circular dependencies; supporting you as you scale your framework from a few dozen to a few hundred tests. You can even view an interactive, graphical representation of these dependencies in the form of a custom editor window. Another functionality example is the BuddySystem Architecture which allows you to test game code that relies on more than one client (multiplayer [Turn-based currently supported, and real time support is a WIP]). More information on these topics can be found below.

Trilleon offers easy setup with cloud device farm integration for your automation suite. The code you will need is all there; simply plug it in and customize as needed. In no time, you will be ready to run your tests in TestDroid, AWS Device Farm, Xamarin, or any other provider that offers appium-python integration options. Just as important, full instructions/code are provided to set up launching of test runs on local devices through a Jenkins build step.

There are several Unity Editor Window tools included in Trilleon. For example, there’s the “Assistant” tool which allows inspection of elements in the hierarchy window of the editor. This displays all automation-relevant details, including links to the objects and scripts that reference it, and allows you to choose an action or assertion before generating code stubs that can be pasted into your tests. This code accounts for references to the selected game object, and provides a direct link to that reference. In cases where no references are found, a generic search script will use the finder functionality to locate it based on the requested attribute.

Additionally, a Record and Playback tool has just been developed that catches automation-relevant actions taken in your editor, generating a full test script that you can paste into a C# file and immediately playback. In this manner, no manual test-writing will be required to make automation tests. Therefore people of any technical skill level can generate complete, flexible, and reliable automation test scripts.

Try it out, and decide for yourself if you are ready to take advantage of the contribution to quality that a scalable automation test suite provides.

Here is the original trailer demo (more videos to come shortly):

Important Info For Initial Release!
Note: Trilleon is in Beta. It is not available yet from the Asset Store. It is initially only available on Github @ https://github.com/disruptorbeam/trilleon/releases. Nothing can be guaranteed flawless under all scenarios, so please help me make it better by reporting issues to me here or on our Github page. I will react to all reported problems as quickly as possible. Documentation is far from complete, so feel free to ask me how to do things not defined in https://github.com/disruptorbeam/trilleon/wiki/Welcome-to-Trilleon. Please note that I have added "CUSTOMIZE:" comments throughout the code that is prime for customization and extension. Much of what is not yet spelled out on the wiki will have some helpful information in these comments to allow you to explore all of the features in Trilleon.

Things that are advertised as part of Trilleon that need special consideration on their current status:

1) Trilleon was primarly developed on MacOS. Because of this, the server scripts were developed for MacOS, and will not work out of the box for a Windows machine! I will get access to the resources necessary to set this up on a Windows server machine, but you will need to make modifications in the mean time. The main modifications will need to center around shell script commands that differ in name and usage between the two operating systems. Also, some of the visual aspects of the Nexus editor window need to be updated to look more appealing on Windows. For example, font sizes need to be different for icons between the two operating systems. I will work on updating that for the next release.

2) Record And Playback: (It Works!) I spent a lot of time trying to make this tool as easy to use, helpful, and functional under as wide an array of game situations as possible. It has worked well for me in my testing, but I do not have a massive library of games to test the tool out on. Given the nature of this tool, I expect it to have some hiccups in situations that I could not test it under, or in contexts that I did not consider. So if you encounter any problems, please let me know and I will be happy to improve the tool! It is also very important to note that if you want this feature to work at all, you will likely need to customize the fields in GameMaster.cs between the region defined as "#region Customize With Company And Project-specific Interactable Components". This is because Trilleon, by default, only accounts for objects that Unity provides in UnityEngine, such as Button, InputField, Toggle, etc. Any custom scripts that you use to create interactable objects in your game must be explicitly declared in these fields. Once that is done, AutomationListener components will be applied to these objects, allowing for Record and Playback to detect them.

3) Code Stub Generation: (Disabled!) This feature was one of the first support tools that I created for Trilleon. The code is some of the oldest within Trilleon, and due to a major set of updates around Record and Playback, the shared logic involved needs to be refactored and fleshed out further to be fully functional. Beacuse of this, I disabled the code. The code is still there, and can be manually activated, but the resulting dynamic code will not be compile-ready. I have delayed revisiting this because I did not consider it as high of a priority as polishing the Record And Playback tool, along with other needed features. It is on my short list to introduce in the next minor version of the framework.

4) Multiplayer Testing in Trilleon; aka Buddy System: (Exists/Has Worked/Status Currently Unknown) This framework has been utilized and worked well in the past. However, due to my personal situation, I lost access to source code for games that require or utilize this type of gameplay. Thus, I have not tested the feature in many months. With massive updates to the framework, I expect the Buddy System to encounter a few hurdles for those first implementing it into their suite of Trilleon automation tests. Please let me know whatever issues you encounter, workarounds you implement, etc. I will fix whatever I can immediately, and I will endevour to get ahold of applications that I can implement the Buddy System on, so that I can handle and resolve issues myself.

*1 "Spiritually" Framework was inspired by Selenium, but certainly is not related to it, and differs from it in many ways. Certain approaches to wrapping and normalizing common interactions with apps were mimicked as closely as possible to generate an experience that was not so different from Selenium that it is a whole new experience understanding the syntax and approach to automation.

 

but how can we get it?

 

It is not currently available. I will update the post to mention that in more places/more clearly, and the ambiguity in any timeline.

The product is very solid - even now, but to offer the asset on the asset store, we have to be confident about how we can provide support for it, and what we want consumers to expect. We need to determine this. Meanwhile, the product will continue to expand and become better and more powerful that shown above.

Another important factor is that our company 's priority is the high quality launch of our next game, so all resources are devoted to that at this time. After that is over, we can potentially make better estimations for Trilleon.

I also just noticed that I didn't put [WIP] in the title, which I will rectify asap/

 

just announce it when it's ready...

 

As soon as it is ready, we will make a post in the Asset Store forum. For now, we wanted to show a preview the functionality through the Work-In-Progress forum. Thanks for your interest!

 

I appreciate input of any kind. One thing in particular that I am looking for is suggestions for features not mentioned, along with opinions on which of the features are most interesting, especially for incomplete features.

For example, here are several incomplete features:
1) Record and Playback test writing (In Progress)
2) Real-time Multiplayer Testing (Not started)
3) Unit Testing and Performance Testing framework integration (Not Started)

Also, knowing if there is interest in this framework will potentially help speed up the period before it is made available.

 

UPDATE:

Record and Playback is functional, and working well! Automation relevant actions are recorded using a system that tags relevant objects with an automation listener. Actions are recorded, and scripts are generated!

Single Window Accessibility Template (SWAT) model has been integrated into Trilleon! This functions much like a Single Page Application (SPA) in web development. Most automation editor windows are joined into a single, appealing, and reactive window.

Here is a preview:

 

Record and Playback GIF Example In Action:

Develop full automation scripts in seconds simply by showing the automation what you want it to do.

- Add explicit waits directly where they make sense
- Toggle actions between asserted action and "Try" (non-assertive) actions
- Edit recorded actions on the fly
- Reorder or delete actions easily
- Stopping recording generates full code snippet to duplicate all actions taken

FULL SIZE GIF LINK:
https://gifyu.com/images/RecordAndPlayback.gif

PREVIEW:

 

Definitely going to pick this up.

 

I had the misfortune of stumbling upon this thread the same morning that I saw on Facebook that Disruptor Beam had layoffs a few hours ago (always lame, sorry they happened). With that in mind, what's the status of this project now? Browsing through it, it seems like a pretty incredible automation framework for Unity projects.

 

Hi Addison,

I was included in the layoffs. However, I, David Cham, and our CEO Jon Radoff, are very interested in continuing this in a way that it is made available and supported by me.

I am excited, because the framework is ready to rock and roll. It will take a little while for me to solidify my situation, and for DB to give me the green light. There is licensing consideration to make, as it is their product.

But to assuage any concern. This is not dead. It is very much alive. It is very much ready, at least on a level that provides you a firm and reliable foundation.

DB and I just need to hash a few things out since I no longer work for them.

 

Just an update. We are very close to making this available. The work is:

A) Github setup
B) Asset Store submittal

Looking forward to working with everyone to get this set up for your projects!

 

Just an update and an apology.

I'm sorry; I am still waiting on the green light from DB to go ahead with this. They want to post the code base from their own account, and then I will be given privileges to update freely. This means that I am not able to simply post the code up on the web myself. The guy who will be pulling the code and putting out into the public sphere is currently doing the job of many people all by himself, so he is taxed thin, and this is not a company priority.

They want this out there, but it is secondary to their success, especially during this unfortunate period after the layoffs. So again I apologize for the delay. But the benefit of the delay is that I have been working on it all the time since, and have made significant upgrades to logic and the functionality available.

I have significantly extended the validation of the framework - what amounts to something like Unit tests for the logic and test runner. I have refactored a lot of logic, cleaned up a lot of the code, reorganized things, and added a bunch of comments to the code to make it easier to work with.

A random example of a new feature I added - The "WaitForBlahBlah" logic just got a nifty upgrade. You can now pass in a lambda expression for super simple implicit waiting. It is particularly useful if you want to use a property as a condition in wait logic. For example, "SomeClassTestObject.someGameObjectProperty". So now, you can wait by saying

Code (CSharp):

1. yield return StartCoroutine(Q.driver.WaitFor(() => SomeClassTestObject.someGameObjectProperty.isActive));

It takes one line of code, and technically can make the explicit wait "WaitForSeconds(1f)" obsolete.

I also simplified the usage of "Try" commands and "Soft" asserting. Using Try will not cause a failure or interruption to test execution if a command cannot be completed as expected. Using Soft will mark a test as failing on an assertion, but the test will continue to execute instead of instantly being stopped.

Code (CSharp):

1.  
2. Q.driver.Click(SomeObject); // Can't click? Test failure is asserted.
3. Q.driver.Try.Click(SomeObject); // Can't click? Moving on to the next step of the test.
4. Q.assert.IsTrue(SomeCondition); // Fails? Test execution is immediately stopped.
5. Q.assert.Soft.IsTrue(SomeCondition); // Fails? Test execution continues, but test is marked as failed on this assertion for reporting purposes.
6.  

I am also working on a tool that simplifies manipulating the hierarchy and inspector windows and searching for game objects with a myriad of conditions so much easier. I am calling it the Gadget. Get it... Inspector ... Gadget? Anyways, that will be a tool that can be opened both independently of the Trilleon test run window, but also is included in it if you have that open (So that you can just use that without the overhead of the Trilleon stuff that you might not need at that moment in time).

 

Thanks for the update tsibiski! Looking for a testing tool in Unity and this sounds promising!

 

I've created a wiki for Trilleon. It isn't using a wiki farm because the developer limitations on the ones I tried were driving me bonkers. Since I do web development anyway, I just wanted to create it from scratch. Particularly because it isn't initially going to be open for public edits. Although that may change, and I will cross that bridge when I come to it.

 

Hello,
interesting project. Can you provide any estimation for release ? We are planning to start a project in March ( now we are in requirment specifcation and exploring new assets ) and you tool would be nice.

 

Hi jlocke. I apologize; I cannot give a firm release. It is out of my hands as far as an exact, or even reasonably close date. A few posts above, I mention how I was laid off from my company, and they own the rights to this - and they have making it open source on their list of things to do. And I have heard that it is now second on their list, so it **should** be very soon.

That being said, "soon" is open to interpretation, and I don't think it would be fair of me to assert how that fits in a specific time period. Or, in other words, I am not comfortable giving much of a date if completing something is completely out of my control. But knowing my previous company, and the people involved in making this public, I can assure you that they are working their butts off, and will give me the permission to forge ahead as soon as is reasonable and possible!

That being said, I would be floored if it was not ready FAR before March.

 

Hi @tsibiski

I was sorry to hear about your layoffs, and wish you the best luck.

I spent some time this week looking for UI automation framework for Unity.
Yesterday I came across this project and it sounds very promising.

Something I wanted to ask:
Is this framework will support Unity app for android, running on VR 3D glasses? (Samsung Gear VR)

the app actually running on the phone screen like standard Unity android app, and the glasses showing the content in 3D.
UI operations are controlled from the remote. (press/move)

Thanks,
Avi.

 

This fully supports Android and iOS runs. There are even scripts to quickly and easily set your machine up to handle these platforms. It's one script that knows how and what to do based on the few parameters you send it from your builds.

Now, as far as support for VR, I have done nothing in VR myself. The framework will not have any VR-specific driver commands, but it is designed to be very easy to implement your own.

Here is an example scenario. Perhaps you have a shooter game, and you want to add a driver command "AimAndShoot();", which accepts the target game object, and directs the player character to aim and shoot at the provided target. That functionality does not exist in Trilleon by default. It is a framework for you to build on, with a bunch of various click, sendkey, and wait functionality. But classes, such as the driver, are extendable. You can add whatever you need as needed.

The automation cannot command your remote to send a command to the headset. Or, at least, no such functionality exists. If you can find out how, implementing that might be a great idea. Instead, Trilleon triggers user-like actions on the apps in test by using whatever mock option Unity inherently offers - most of which is at the top level of the program, at the point where the app receives an external "key down" or "button pressed" command.

 

Great news! Trilleon should be available sometime this week. I also just completed the initial native Cheat/Command Console feature that allows you to write out cheat commands and execute them from the code, or via the GUI console that you open by pressing (defaults) Tilde( ` ) or three-finger swipe up on devices. I am also working to flesh out TrilleonAutomation.wiki so that it has the bare minimum necessary to help with your initial setup and exploration of features offered.

Few Notes:
1) It will only initially be available from GitHub
2) It will be made available in the Unity Asset Store in the near future
3) GitHub will be updated with every minor/major version, while the Asset Store will only receive major version updates

 

@tsibiski Hi, do you have any news related to the availability of the Trilleon?

 

Hi Sorry for the extra delay. Right when I was sure it was going to be uploaded, a few extra steps were added to the process. A testing team is now reviewing the framework. It shouldn't be long, but my expectations are clearly not in lockstep with DB's. I apologize for not being accurate in my previous statement. That being said, extra QA is great.

If it is any consolation, I have redownloaded the Space Arcade Shooter tutorial game from Unity, and I am making a small suite of tests for it. I intend to include it as an example of implementation for various features of the framework. That is mostly complete.

 

Hi,

I want to again apologize for delays. I know there are significant number of people who have publicly or privately expressed interest in this asset. I really, really, really appreciate that you are interested. You have no idea!

So... we near the release date (sorry, I will refrain from making suggested likely timeframes for release until I know for sure, as my excitement and anticipation has been unfair to you. I know some are setting some dates for when they can begin implementing things, and that is serious business.):

I want to express a few things for what everyone should expect from Trilleon at launch.

1) Trilleon, when viewed as an entire package, is in beta (until it is released as a major/stable version in the asset store, as opposed to the initial GitHub release). That being said, I expect test setup and test execution to run flawlessly. If you have difficulties in setup or execution of tests, I want you to let me know, and they will be my highest priority. Weekday or weekend.

2) The same goes for any compilation errors. I expect Trilleon to be drag and drop for 5.x +, full stop. I also suspect that 4.x + should have minimal or no compilation errors. If 4.x - 5.0 compilation errors occur, I am very happy to do my best to fix them. Please let me know, and give me all the details you can. If 5.x + compilation errors occur, let me know and I will also make them my highest priority.

3) Everything else in the framework should be considered Beta mode. While other features I have expressed work perfectly for me, under the conditions I have put them, that does not mean that you have a scenario I have not thought of to test. There are literally hundreds of files involved in this framework. It is a complete automation framework (the real deal). If any errors occur, please let me know primarily in the soon-to-be-created Asset Forum Thread, or alternatively at trilleonautomation@gmail.com (will check as often as possible). For example, I expect the Record and Playback feature to fail for someone, somewhere, somewhen. Such a feature is not conducive to perfect execution under all scenarios, to be honest.

My goal is to provide approximately 75% - 90% of the work needed to develop an automation framework for a Unity game. Your effort should only have to be in completing game/company-specific wrappers and the actual tests themselves. I appreciate everyone's interest in this, and I want to impress you, and make your jobs easier. The intent is to make it possible to easily submit your own updates, improvements, and add-ons as well. I can tell you that any and all support you provide me to fix, update, and support the Trilleon framework will be greatly appreciated. I really mean that. Thanks!

 

@tsibiski This looks like an awesome project. Is there a specific website or something, where you guys will be announcing the launch of the product?

 

Hi @ssreflekt. I am glad you are interested in Trilleon!

I will be announcing the release of the framework on www.trilleonautomation.wiki.

I will also forum-pm all of the people that showed interest in the product to notify them as soon as it is released.

I will post here as well, so watching this thread will help. And finally I will be posting in the Asset forum, proper, once it is available for download @ GitHub.

 

Awesome to see another tool for Unity automation =)

I did it in another way. For my tool it looks like Appium but for Unreal/Unity, Full selenium support, full support of ios/android device. Beta release scheduled for end of May.
I will be happy to follow your progress also

So, my first post here =)

 

Hey, Inoishadow,

That is awesome. It sounds like our solutions won't be competing directly, because your's offers an interface with existing frameworks. Mine tries to be a completely new framework that exists inside of Unity.

I hope our two frameworks can complement each other. If yours is open source, there may be an opportunity to integrate functionality into a single framework that provides both full Selenium integration, or in-game automation in the same framework.

Trilleon will be fully open source, so if you see an opportunity to combine forces, let's keep in touch. Even if our frameworks are not mutually inclusive, I suspect there is always a way for us to share info in a way that makes both frameworks a whole lot more useful to everyone!

 

I've just completed support for Unit Tests!

You can finally unit test IEnumerators and context-specific game logic!

Now, you can run Unit Tests/Unit-like tests. These tests do not run at compile or launch time like traditional Unit tests. Instead, they run in the same way as integration automation tests run in Trilleon. With this support, you can test your API's and game logic directly to catch bugs early.

Unit tests and Integration tests logic, organization, and support work together seamlessly in Trilleon. QA and Devs will be using the same interface and logical design to write out their tests. You can share your knowledge between one another and it will all cross-apply.

 

@tsibiski Waiting to try out with our unity application. When can we expect the availability of Trillion in GitHub?

 

Hi @vijayravindran

It is not yet available. But, this will be available on Github at first, in a Beta form. As soon as I am given the permission I need, I will post here, and you can access the link to our GitHub download via www.TrilleonAutomation.wiki (in the top left corner of that website).

I cannot guarantee an exact date, or a general date, unfortunately. The green light for distribution is out of my hands entirely. But, the framework is 100% ready for you and everyone. However, I do not own the framework. I was paid by DisruptorBeam to make the framework, so they own it 100%.

However, DisruptorBeam has been amazing to me, and they continue to collaborate with me to make this framework completely public and open source. We are at a point where the final hurdle is brand image. It is critical to make sure that Trilleon and any problems that ever occur are not thrust onto DisruptorBeam. Anything negative should be applied to me, and I will correct them.

It has recently passed the first major hurdle, and is now privately on github. You will still not see it anywhere at this time, until it can be switched to public.

--General Comments--
I am working with DB to boost their confidence, and I want to stress that DisruptorBeam should get 100% of the glory, but 0% of the bug reports and complaints. I alone developed this framework for them. It exists solely because they funded it, and gave me the opportunity to work for them. If any bugs occur, suggestions arise, etc, they should be directed towards me!

Also of note, I recently created an entire (but basic) game involving mathematical equations flying up the screen, and I developed a full suite of tests for this game to demonstrate how to set up the framework in your own games. This demo game/implementation will be included in the supplied, free Trilleon framework under "Examples", which can be extracted into a separate client folder, and deleted as soon as you want.

 

@tsibiski Wow. Thanks for the detailed update.

 

What is the status of this? I can tell you I would be all over using this if I could

 

We are on what I currently understand is the final step before it is opened to the public on Github. This step is a final review by the programming manager at DB (who has been my point of contact this entire time). He is on vacation this week, but I have high hopes that he can do this review next week. All of the necessary licensing information is in place, and the framework seems to be quite stable. It is on Github and all that is needed after the review is done is a flip of the switch.

 

@tsibiski Can the tests be run from the command line under linux?

 

I've never tried. But I understand that you can call a method from command line, and pass in parameters? (Although that is only while the project is in editor? If that is the case, then it shouldn't be hard to do. But that would only be for non-device/non web-gl launches, I suspect.

If you can direct me to how that all is done, I will see what I need to do to accomodate it, and add the functionality.

 

https://effectiveunity.com/articles/making-most-of-unitys-command-line/

Looks like this might be the ticket. @zdolezal, can you confirm this is what you mean? Something where you might say this (pseudo bash):

Unity -executeMethod TrilleonAutomation.LaunchTestsFromCommandLine -testsToRun "all"

 

Well, it's been a long time coming. It's been a hoot. It's been the bees knees so far.

But the wait is coming to an end. We intend to release this today!

 

You can get trilleon here, now! https://github.com/disruptorbeam/trilleon/releases/tag/v1.0.2

Release thread: https://forum.unity.com/threads/trilleon-full-integration-automation-and-unit-test-framework.528699/

 

Great. A bit surprised by the license though. Not fully modeled after selenium after all

Update: I misread GPL instead of LGPL. Doh. Forget what I said

 

I am unable to get to the effectiveunity site - is it still up? When I go there, it asks for a username/password, even on the root site... there doesn't seem to be any way to create an account?

 

Yeah that website seems to now be down. But this is the approximate code to do this:

Code (CSharp):

1. -quit -batchmode -projectPath $WORKSPACE/app -executeMethod MyStaticClass.MyStaticTestLaunchMethod

And that method would launch a suite of tests directly from the editor, for example. I just need to add in-framework support for it, but it is so simple that it can be manly added as needed.

 

@tsibiski Sendkeys is not working in Unity 2017.3.0f3. I tried with script and record script. In both the case i'm unable to send the text value to input field?

 

@vijayravindran

Are you getting any errors? Or it simply does nothing? What class is your object using? Is it using UnityEngine.UI.InputField? Is it using TMPro.TMP_InputField? Or is it using a custom class?

In Driver.cs, line 132, can you put a breakpoint on this:

Code (CSharp):

1.             if(field != null && keysToSend != null && field.isActiveAndEnabled) {
2.  

Then, can you step in and see if the field object is null, or if it thinks the field is not active and enabled? That would prevent it from trying to send the keys.

 

@tsibiski Thanks for the update. I have missed out the coroutine method for sending keys to input field. Now it is working fine.

 

@tsibiski: I have a single class file with multiple [Automation("")] attribute Ienumerator method. How to execute them in a priority order ?. Currently it is executed in alphabetical order as shown in Nexus->Launch category.

 

@tsibiski I'm unable to click a gameobject present in Scroll view. When I click I get following error in the editor

It shows an error in AutomationListener.cs (173 line)

 

@vijayravindran

To execute in an order you determine, use the attributes "DependencyClass/DependencyTest" or "DependencyWeb". You can read more about that here: http://www.trilleonautomation.wiki/attribute-usage

Also, examples exist under Automation/Engine/Tests and in the example project included with the github codebase in the top level "examples" folder.

For the error, can you provide me the full callstack? That line is a call to another method, so it doesn't help me localize the issue.

 

@tsibiski Click event is not working properly when the child has multiple events. For eg, I Have a parent and multiple child attached to parent object. If one of the child object (click event is not attached) we use pointerdown and another child has Toggle select. When I try to click a child the trillion automatically search for the click event under the parent and select the toggle instead of another child object.

 

@vijayravindran So are you saying that you have a custom script with a pointerdown event handler on one object? And then another child object in the same level has a toggle?

The way it currently works is that, when you pass a game object into the click handler, if it does not have a click script, automation bubbles up the hierarchy trying to find the first parent in the tree that does (if any do).

If two child objects on the same level have their own trigger scripts, you would need to initiate a click on both objects. You could also take the current driver functionality in Driver.cs and extend it in GameDriver.cs to allow you to pass in an array/list of gameobjects that should all be triggered at approximately the same moment.

Am I understanding this use case correctly?

 

New release! Socket communication implemented for talking between server and client. It is an alternative to Pubnub, and also the default setup now.
Download Now: v1.1.1