Assets [RELEASED] Trilleon - Full Integration Automation and Unit Test Framework

A new Trilleon update has been released that fixes several potential bugs: V1.1.2

Additionally, I have created the first demo video for Trilleon that explains all of the Nexus editor window functionality!

 

Hi Ya'll,

I just wanted to gauge how many people are using the Trilleon framework, to determine how much of my free time I should be devoting to Trilleon. I was recently laid off again, and I do not expect to be working within Unity in my next position.

The response to Trilleon has been incredibly luke-warm. Before release, it seemed that there was a strong interest in the framework, but since then, I have seen almost zero interest in the Trilleon automation framework. Both in the forum threads and via Github.

I would love to know who is using it, and a quick poke on how it has helped you. If there seems to be a reasonable number of people using it, I will continue to develop for it and improve it.

Thanks to any responses.

Regards,
Tim

 

I'd like to spend some time in the near future to upgrade Trilleon. Among those that are invested in Trilleon, can anyone give me a heads up on what they would like to see most. It can be anything from better documentation to a specific feature!

 

Can't access any page on your wiki:

 

The wiki is back up!

 

Hi There.

I am rewriting the CI scripts to work in Linix in addition to Mac OS. While I am doing this, is anyone else interested in ANTHING Trilleon? I have been only slowly attending to it due to a, seemingly, extremely small interest in the framework. If you are actively using it and/or wish to use it, please let me know what features you want most (or just let me know that you are using it). If I get no replies to this, I am only going to assume that the interest is light, and won't get to it in a timely fashion.

Thanks!

 

Hi @tsibiski ,
First, let me thank you for a great work so far. I would like to run a stress test with lots and lots of instances (low hundreds). I would love to have windows support as well, but i think linux support will do as well

 

I am actually working on a Linux version of the server scripts this weekend but can do Windows after that.

 

Hi, any news on Linux/Windows version?

 

Hello @tsibiski I only just found this project, but I see you're reaching out looking for interest. I can tell you I'm about to integrate it into a project here at work. I'll let you know how it goes, but I'm very excited for it. Thank you in advance for all the work you've put in

 

Hi all. I have had some important life events that have made me ghost. I am going to try my darnedest to work on these PowerShell scripts this weekend.

 

Great news!

 

I've got a first-pass/rough pass of the powershell scripts. I am going to test them step by step tomorrow to whenever I am happy with them. But it is important to reiterate that I do not have the ability to test them end to end. So there may be some commands that hiccup. We can work them out together once I post the first draft of PowerShell scripts.

 

The first, and largest of the scripts is fully converted from a bash script to powershell
https://github.com/tsibiski/trilleon/blob/master/server/RunAppiumLocal.ps1

Though there may be some issues - once I get the other shell scripts into ps1, and you hook it all together.

There also may be some permissioning issues around PowerShell. Note in this file that I do two things that require admin privileges to do.

First, I kill process that are on explicit port numbers using Stop-Process. Additionally, Start-Process requires similar privileges. The latter can be fixed by manually opening PowerShell in Admin mode and doing "Set-ExecutionPolicy Unrestricted", although this may not apply to PowerShell scripts launched by a CI process such as Jenkins.

Also, Stop-Process requires the shell to be running with full admin rights, so it may not work in this context. This may require something fancy that I just haven't figured out yet.

 

Thanks to @AdamSingle for notifying me of this bug: A newer version of Unity changed things such that the "Recorder" logic could not find objects marked as "DontDestroyOnLoad". This is fixed, and I've deployed a new version on Github that has this fix.

 

@tsibiski sorry for the radio silence. Another short term project jumped up at work and budget has been pulled away from Trilleon integration for now. You know how it is. How did you go with the windows version of the server?

 

The Powershell scripts are tentatively complete, and available here: https://github.com/disruptorbeam/trilleon/tree/master/server/sh

Tentatively, in this case, meaning that I cannot test them end to end currently. But each command was tested individually. Please report any issues you encounter Thanks!

 

After review, It looks as if every command in the current shell scripts for Mac should work on Linux. Please give it a try, and tell me if any commands do not work.

 

Thank you @tsibiski
Is the process still to follow the set up instructions for the mac on PC? as in using npm to install appium etc

 

Good question. That is definitely a place where things could differ. Linux uses apt-get commands for downloading certain things. Anywhere that I use Homebrew /"brew" commands would be replaced by apt-get.

For example, if you come to "brew install python", try "apt-get install python3.6". A quick google search should bridge any gaps in that process. I will add modifications to give alternatives for Linux asap.

But any command that uses npm will require Nodejs as normal. Once you download that for Linux, npm commands will be the same.

If you come to a command that does not work with minor changes like those mentioned above, please let me know, and I will update the documentation.

 

Hello @tsibiski. I am very excited to learn about Trilleon. At this moment I'm following this example (

) but I can't find the WaitForCondition () function.
Thanks for your help!

 

Hi @dsgrassg . The WaitForCondition() method was deprecated before the first public release of the framework, so I removed it entirely from the framework. Instead, use "WaitUntil()", and pass any custom condition in as an argument.

So as an example:

Code (CSharp):

1. yield return WaitUntil(() => MyGameObject.IsVisibleAndInteractable());

Before, using WaitForCondition, I had a convoluted wait of setting a condition, and it was more like Seleniums conditional waits. It would look more like this:

Code (CSharp):

1. //Deprecated and deleted.
2. yield return WaitForCondition(Condition.Visible, MyGameObject);

This was too limiting as I needed to define every possible condition and define that conditional check in WaitForCondition. It was too limiting and inflexible. So I just decided to make WaitUntil and let you pass the condition that you want to check for as a predicate/anonymous function.

 

I just found this and I am impressed! Looks awesome! I have to try it on something Are you still working on it?

 

Hi Mikael-H,

I have not been actively working on it. Before I released it, a lot of people were providing feedback and comments that showed a good amount of interest. So I made sure it was open source after I had it working for my employer. There was a surprising lack of interest after I released it, though.

And since I had around 2 people that I knew were using it in any capacity for a year after its release, I couldn't convince myself to devote too much of my time to it after I left the game industry (working for non gaming software companies now).

In specific scenarios, I am very happy to add things and help out. But without much public interest, there's no way I can convince myself to expand and actively update it. And granted, there may be quite a bit of interest that I don't know about, but if there is, they never reached out to me to express that.

Sorry, that's a lot more than your bargained for with your question, I'm, sure. Just thought this was a good time/place to put all that out there!

 

No that was exactly the type of information I was looking for, thanks for taking the time to answer!

I suspect an asset like this has a very high value for a few people as opposed to others that has a little value to lots of people. The amount of devs who take their work seriously enough to do rigorous atutomated testing is probably small, but the ones who do probably really like this

Thanks for open sourcing it!

 

@tsibiski is the wiki down? Could you add the documentation to the repo maybe? then you don't need to maintain the site. I'd be fine with that

 

http://trilleonautomation.wiki/

The site is up. Give it a try. Tell me if you can't access it still.

 

I can access it with that link. But some days with some links it is very hit and miss. The link in the original post in this thread doesn't work for me. And the link on the git repo readme doesn't work. Maybe there is a subtle spelling issue in them. Do they work for you?
Let me paste the the url from my browser for the one that worked above:
http://trilleonautomation.wiki/
And the one it goes to for the links that don't work for me:
http://www.trilleonautomation.wiki/

hmm, that's interesting. In all the cases where it doesn't work for me, there is the www in there, although I can't see that in the browser, in both cases it just shows trilleonautomation.wiki but copying and pasting from the browser to somewhere else shows the above.

 

Oh, I know the problem. Thank you for bringing this up. Will try to fix it asap.

 

@tsibiski Will Trilleon work for unity UI automation testing in webgl build?

 

It has code that I've used in the past to handle WebGL. So it definitely should. I am answering your issue in the Github repo now.

 

Is there any way to run tests in Real device/ or emulating the device and run tests from Unity?

 

Hi Jay,

Yes, it is specifically designed to do that. Take a look at https://github.com/disruptorbeam/trilleon/wiki/Welcome-to-Trilleon. It explains the process and gives you scripts needed (Scripts are also in the server folder).

 

Hi Just touching base on this project. I'm looking to do some Unit testing for some new changes coming to our game and found Trilleon. Just wondering if it's still supported and being worked on (I.e. if we find bugs).

 

It technically is, but on an as-needed basis (bugs/issues, incomplete features that people express a need for). Not enough people expressed interest in it to warrant much more work, considering I had 2 consecutive video game company layoffs that saw me return to the non-video game software industry (which kept me from upgrading this as a paid job as well).

But I have updated it at times, such as when the need arose for a PowerShell version of the CI scripts.

 

Thanks. Do you have any other possible suggestions for Unity Unit Testing (That's all we really need) that isn't Unity's built in testing framework?

 

Unfortunately, I don't know of any other Unit Testing framework in Unity besides NUnit. Are you looking for someone that allows you to unit test Coroutines? If so, I think my framework is the only one that fully supports that, that won't require you to do a lot of work to get it ready, at least.

 

I don't really need coroutines for the most part. I'm just looking for a simple test framework for basic class unit tests. However because Unity itself puts its test stuff into asmdefs, I would have to go down a massive hole of making and maintaining many asm defs in order to get all the plugins, code, and other dependencies all finding the code they need.

BTW: I found what might be a bug (Or user error ) and posted on the github. Not sure if you have time to look at it.

 

Looking into it now. I reproduced the issue when I copy-pasted your code. I am struggling to get the new Unity to interface with my Visual Studio, so I cannot debug yet. As soon as I fix that, I will debug your issue.

 

The answer: Q.assert.IsTrue is an IEnumerator. It must be in a "yield return StartCoroutine()" wrapper. Without that logic, it simply will never run the method. It will skip over it. And because of that, no assertions were made at all, and the test passed. Make sure it is "yield return StartCoroutine()", as simply starting a coroutine will not wait for its logic to complete before moving on to the next yield statement - which will cause the test to not function correctly.

I use coroutines and yield returns heavily in my framework. So keep an eye out for that. Also, use U.assert.IsTrue for Unit Tests, instead of "Q.Assert" which is for Automation. There are a couple of example Unit tests in the main Assets > Automation > UnitTests folder, so you don't need to go to the example project to access those examples.

You can also switch between the automation view of tests in your Nexus window to unit test view. It defaults to automation when opening the window, but if you go to the final tab of the window: "Other Settings", and then go down to the "Default Test View", you can switch it to save the Unit Test view of the "Tests View" tab (so that it doesn't revert to the Automation view when you close the Nexus window and re-open).

Feel free to open up the "Example" project on the top level of the repo. It has a demo game with a bunch of demo tests. Sorry for any confusion. I will update the shell of DemoTest.cs to stress review of the example project before using methods.

 

I noticed a bug and checked the fix in. It is a single line change, so you may just want to manually make the change yourself. The bug is that, in Unity 2020 versions, the Test View was only showing one category in category view. The variable _firstPass was being set earlier in the Editor flow than it used to. Not sure what changed. But go to TestManifest.cs under [Assets\Automation\Engine\Xtra\Editor\Swat\SwatWindows\TestManifest.cs] and simply add the line

Code (CSharp):

1. _firstPass = true;

on line 727. Right below:

Code (CSharp):

1. _reget = false;

 

@tsibiski I must say I am very impressed with this framework and eager to use it in my project.
I am unable to access the documentation page
http://trilleonautomation.wiki/

Can you please look into it?
Thanks in advance!!

 

I apologize. I recently added all the wiki content to the github wiki to save money on website hosting. It is now located here: https://github.com/disruptorbeam/trilleon/wiki

I tried updating all references of the old wiki, but I missed some. Will look for them and update now to remove the old links.