Official Automated QA Package [v0.7]

Is the new Input System supported? I'm getting spammed with an error about Input.mousePosition.

 

An error occurred in the "Recorded Test".
After running "Generate Recorded Tests", select "GeneratedTests.dll" and run it. Then the following error appeared in the console.

Code (CSharp):

1. FileNotFoundException: Could not find file "/Users/***/Library/Application Support/DefaultCompany/UnityTestFramework/config.json"
2. System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) (at <fb001e01371b4adca20013e0ac763896>:0)
3. (snip)
4. System.IO.File.ReadAllText (System.String path) (at <fb001e01371b4adca20013e0ac763896>:0)
5. Unity.RecordedTesting.Manager.GetDeviceFarmConfig (Unity.RecordedTesting.DeviceFarmOverrides dfOverrides) (at Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/Manager.cs:31)
6. Unity.RecordedTesting.TestResults.RunStarted (NUnit.Framework.Interfaces.ITest testsToRun) (at Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/TestTools/TestResults.cs:33)
7. UnityEngine.TestRunner.Utils.TestRunCallbackListener+<>c__DisplayClass1_0.<RunStarted>b__0 (UnityEngine.TestRunner.ITestRunCallback callback) (at Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/Utils/TestRunCallbackListener.cs:16)
8. UnityEngine.TestRunner.Utils.TestRunCallbackListener.InvokeAllCallbacks (System.Action`1[T] invoker) (at Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/Utils/TestRunCallbackListener.cs:38)
9. UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr) (at /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/Coroutines.cs:17)

There is no config.json in that directory, but there is config_recording.json and playback_recording.json.

environment:

• macOS Big Sur (11.2.3)
• Unity 2019.4.11f1
• Automated QA 0.2.0-preview.3
• Test Framework 1.1.24

 

Thanks for your response!
I found `CLOUD_TEST_UTF` symbol in Scripting Define Symbols. After removing this, the test will now run.

But, every test failed with the following exception.

Code (CSharp):

1. CanPlayToEnd (3.571s)
2. ---
3. SetUp : System.Reflection.ReflectionTypeLoadException : Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
4. ---
5. --SetUp
6.   at (wrapper managed-to-native) System.Reflection.Assembly.GetTypes(System.Reflection.Assembly,bool)
7.   at System.Reflection.Assembly.GetTypes () [0x00000] in <fb001e01371b4adca20013e0ac763896>:0
8.   at Unity.RecordedTesting.RecordedTesting.GetAllRecordedTests () [0x00023] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:55
9.   at Unity.RecordedTesting.RecordedTesting.GetLocalRecordingFile (System.String testName) [0x00002] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:36
10.   at Unity.RecordedTesting.RecordedTesting.SetupRecordedTest (System.String testName) [0x00008] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:115
11.   at Unity.RecordedTesting.TestTools.RecordedTestSuite+<Setup>d__0.MoveNext () [0x00027] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/TestTools/RecordedTestSuite.cs:23
12.   at UnityEngine.TestTools.BeforeAfterTestCommandBase`1+<ExecuteEnumerable>d__12[T].MoveNext () [0x00185] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/NUnitExtensions/Commands/BeforeAfterTestCommandBase.cs:87

 

Great initiative!

After I click on "Record", I get a bunch of entries like this one on the log:

There are 2 event systems in the scene. Please ensure there is always exactly one event system in the scene
UnityEngine.EventSystems.EventSystem:Update () (at C:/Program Files/Unity/Hub/Editor/2020.3.5f1/Editor/Data/Resources/PackageManager/BuiltInPackages/com.unity.ugui/Runtime/EventSystem/EventSystem.cs:398)

When I click on "Stop", the json file only has mouse inputs logged, keyboard and gamepad are not getting logged (are those not supported yet?)

Thanks!

 

The "There are 2 event systems in the scene" bug was fixed in the most recent release of the package (0.3.0).

The package is not yet recording keyboard/gamepad inputs. Both the old and new input systems need to be handled seamlessly depending on the package customers are using, and keyboard/gamepad inputs will require that. So the handling of those types of inputs is important and is on the horizon.

Right now, clicks and drags are recorded within the Canvas space. However, it will not work yet with GameObjects not on the Canvas (ex: world objects using colliders to perform actions similar to clicks and drags; this functionality is also in the queue).

 

I found the cause.

`System.Reflection.Assembly.GetTypes` seems to throw an exception if the assembly `Microsoft.CodeAnalysis.Scripting`.

In my project, `com.unity.immediate-window` depended on this assembly, so I removed it and was able to run the test!

 

I have two requests.

1. Fail test if object not found in Recorded-testing option

e.g. Add boolean argument in TestPlayToEnd() or Project setting​

2. No need to register in "Scenes in Build" for scene loading

It can be realized with such an implementation. see https://github.com/nowsprinting/Uni...mmit/56ca2d3dbca68e602d2be1aa3b86fe68e629dcec​

 

Hi I also have a question to this toppic. Is it possible to check if a scene was created by a recorded playback? We have our DI set in a way it initializes systems only when our game-scene is starting. Thats why tests fail since a new scene is created for the tests start.

 

And already the next question

Is it possible to create Tests out of Automated Runs? If so how would I go about it? If not I think it would be a very nice feature!

 

Can this be used with the code coverage package?

 

In v0.4.0:

• I'm not want to put AutomatedQASettings.json on "Resources" folder
• Warning mesages in the console. because unnecessary .meta files, follows:
  • RecordedTesting/Editor.meta
  • RecordedTesting/Runtime/TestTools.meta

In v0.5.0

• Warnings in the console
  • There are inconsistent line endings in the 'Assets/AutomatedQA/GeneratedTests/Steps/xxx_Steps.cs' script. Some are Mac OS X (UNIX) and some are Windows.
• Tests fail in batch mode. (same tests are successful in Unity Editor GUI)
  • Message: Timeout value of 180000ms was exceeded
  • see https://github.com/nowsprinting/UnityAutomatedQASandbox/actions/runs/987736845
• I am very happy that the "dynamic wait logic" makes the execution more stable. But, I miss the ease of replacing the JSON file by just recording it again.

 

For the first point, are you able to add it to a git ignore? It needs to be in resources for use in the Cloud and binary builds. If the problem is checking it in, it can be added to git ignore, or a similar feature in other version controls.

For the final issue, you can tell the framework to NOT use dynamic waits by setting the "Use Dynamic Waits" boolean in the Settings editor window. It will then revert to the old way of running tests, and you can continue to re-record failing tests.

 

Thanks. I'm not talking about running tests.

My scenario as follows:
1. Recording tests to json
2. Automated QA | Test Generation
3. Add asserts in test code
4. One day, UI operation and/or transitions will be changed!
5. Re-Recording tests to json
6. Use the test code without changing it

Until v0.4, this was possible.
But, in v0.5, Re-Test Generation is required.

This behavior is acceptable for us programmers, but I think it is cumbersome for test engineers and level designers to use Automated QA.

Given that UI changes are frequent, such as in game tutorials, I hope have preferred to keep the advantage of being able to replace only json.

 

I have one request for the features of Automated QA. Please doesn't overwrite this file, if "Assets/AutomatedQA/GeneratedTests/GeneratedTests.asmdef" already exists when user runs Test Generation. If a user creates test codes in "Assets/AutomatedQA/GeneratedTests", they will add elements to Assembly Definition References in this .asmdef file. But because this .asmdef file is initialized when they creates a new test, they needs to reconfigure this .asmdef file.

 

Thanks for this suggestion!

 

I've been having issues trying to get either version 0.3.1 or 0.5.0 working in our project.

With v0.5.0 specifically, whenever I try to run an Automated Run using a recording added with the RecordedPlaybackAutomatedConfig, I get the following errors and stack traces:

Code (CSharp):

1. BadImageFormatException: VAR 1 (TResult) cannot be expanded in this context with 1 instantiations
2. System.Reflection.MonoMethod.GetBaseMethod () (at <695d1cc93cca45069c528c15c9fdd749>:0)
3. System.MonoCustomAttrs.GetBase (System.Reflection.ICustomAttributeProvider obj) (at <695d1cc93cca45069c528c15c9fdd749>:0)
4. System.MonoCustomAttrs.GetCustomAttributes (System.Reflection.ICustomAttributeProvider obj, System.Type attributeType, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
5. System.Reflection.MonoMethod.GetCustomAttributes (System.Type attributeType, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
6. System.Attribute.GetCustomAttributes (System.Reflection.MemberInfo element, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
7. System.Attribute.GetCustomAttributes (System.Reflection.MemberInfo element) (at <695d1cc93cca45069c528c15c9fdd749>:0)
8. System.Reflection.CustomAttributeExtensions.GetCustomAttributes (System.Reflection.MemberInfo element) (at <695d1cc93cca45069c528c15c9fdd749>:0)
9. Unity.RecordedTesting.RecordedTesting.GetAllRecordedTests () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Automators/RecordedPlayback/RecordedTesting/RecordedTesting.cs:73)
10. Unity.RecordedTesting.RecordedTesting.IsRecordedTest (System.String testName) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Automators/RecordedPlayback/RecordedTesting/RecordedTesting.cs:50)
11. ReportingManager.InitializeDataForNewTest () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:242)
12. ReportingManager.RecordLog (System.String message, System.String stackTrace, UnityEngine.LogType type) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:352)
13. UnityEngine.Application.CallLogCallback (System.String logString, System.String stackTrace, UnityEngine.LogType type, System.Boolean invokedOnMainThread) (at <c7864a0eaeb24b2a999fb177623d54b4>:0)

Code (CSharp):

1. UnityException: List provided to AutomatedQATools.Last() was empty. Cannot invoke Last() on an empty list. Check for list being empty before invoking Last().
2. Unity.AutomatedQA.AutomatedQaTools.Last[T] (System.Collections.Generic.List`1[T] list) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/AutomatedQaTools.cs:146)
3. ReportingManager.FinalizeReport () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:174)
4. ReportingMonitor.OnApplicationQuit () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingMonitor.cs:53)

In v3.0.1 I also get a similar

 BadImageFormatException

whenever I run a generated test from a recording in the Test Runner

Windows 10
Unity 2020.3.7f1
Test Framework 1.1.24
Automated QA 0.5.0-preview.1
Automated QA 0.3.1-preview.2

 

I am having some weird and unorthodox error where if I record step by step using the Record function, it works. However generating the script out to full test breaks.

I think the culprit is that we have MaterialUI, which still uses Unity UI under the hood of the library. However, it seems like it cannot find the gameobject the QA is trying to find (Since it's moving). The JSON file works perfectly, but for whatever reason the full scripted tests (even with changes in deltaTime) it still errors out complaining that it cannot find the button it was target to click or release from.

Can someone help me on what I can do from here? I'm kinda stuck with this Dialog issue. Here's the full test error issue:

Code (CSharp):

1. AQA: Click position recorded relative to spot within the target GameObject "Button - Affirmative" is positioned outside of the camera frustum (is not visible to the camera). Since this is a GameObject in the UI layer, this may mean that the object has not scaled or positioned properly in the current aspect ratio (865w X 487h) and current resolution (1920 X 1080) compared to the recorded aspect ratio (865h X 487w) and recorded resolution (1920 X 1080).
2. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
3. Unity.AutomatedQA.AQALogHandler:LogFormat(LogType, Object, String, Object[]) (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AQALogger.cs:10)
4. UnityEngine.Logger:LogError(String, Object)
5. Unity.AutomatedQA.AQALogger:LogError(Object) (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AQALogger.cs:73)
6. UnityEngine.EventSystems.RecordingInputModule:UpdatePlay(Int32)
7. Unity.AutomatedQA.<Action>d__24:MoveNext() (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AdvancedQuerying/Driver.cs:292)
8. UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
9.  

Win 10 pro
Automated QA preview.2 - 0.6.1
Test Framework - 1.1.29
Unity 2019.4.9f1 - Android configuration

 

Hi there. I actually wrote the logic involving the checks around an object being outside the camera frustum; so hopefully I can help you here.

Is the object still visible on the screen then, despite this error? Can you retrieve the calculated point of click via debug (search for the error message above and put a breakpoint). Is it calculating the position of the object to be outside the bounds of the viewport? It should indeed be outside the expected viewport from a raw x/y position - but if the object is visible to your eyes, something weird must be throwing this calculation off.

Also, is your generated test simply referencing the json playback file, or is each step of the json written into C# (aka, the "Full Test" was generated)? If it's a "Full Test", can you compare the data stored in the associated TouchData object in this test's "Step" file with the raw json object for the step?

 

Hi tsibiski,
Yes the object is still visible on the screen. How do I go about calculating the vector2 position of where it will be clicked?
Currently the UI was set to be outside as "Hidden", but appear out when "activated". It seems that whatever click event calculated, seems to be clicking on the object's original position instead of the new current position. This position appears to be outside of the camera frustum.

To answer your question more precise, I can describe the step-by-step of what I did to reach to the problem:
1) Record, wait until the UI "appears"
2) Re-play, make sure that it doesn't break anything else other than the intended step
3) Tweak deltaTime to speed up the click process (I went into Json file, and manually edit the timeDelta to either 0.001 or 0.5 depends on how long it takes for the UI to appear before I can click next.
4) Re-play it again once more, but with the new timeDelta to ensure nothing else breaks
5) Convert Recordings into Full Test
6) Re-run it again using the Test-Runner -> Breaks.

Perhaps, if it's possible, to click on the UI at "Just-in time" instead of predetermined position of where to click? I can see in the code that it has the object's name and where it's located in the scene hierarchy, so wouldn't it be much safer to obtain the gameobject at that instance of time, and use the transformation for click position?

 

I sent a private conversation to test out a possible fix.

 

I'm getting this after importing into our project. We specifically disabled non 2D physics from the project. Would be great if you would add support to enabled/disabled modules.

Code (CSharp):

1. Library\PackageCache\com.unity.automated-testing@0.6.1-preview.2\Runtime\Core\GameListenerHandler.cs(189,48): error CS1069: The type name 'Collider' could not be found in the namespace 'UnityEngine'. This type has been forwarded to assembly 'UnityEngine.PhysicsModule, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' Enable the built in package 'Physics' in the Package Manager window to fix this error.

 

We will add a fix for this. Meanwhile, that code can be commented out or removed as it is not fully activated yet.

Code (CSharp):

1.                                 if (scriptName == "Collider" || typeof(Collider).IsAssignableFrom(components[co].GetType()))
2.                                 {
3.                                     if (AllActiveAndInteractableGameObjects[x].GetComponent<Collider>().isTrigger)
4.                                     {
5.                                         AddListener(AllActiveAndInteractableGameObjects[x], ActableTypes.Clickable);
6.                                         continue;
7.                                     }
8.                                 }

 

Thanks, I can wait for update. We would like to use that in combination with Unity Cloud Build so not in a hurry.

 

@souranil_sen @dylanscandinaro
Does/Will this work with XR as well?

 

It has not been tailored to XR, and thus is not tested yet for XR apps. But it technically works for any type of game. It is currently best tailored to UI-based games, and testing of UI systems in games. It has limited, but ever increasing, support for other aspects of Unity apps like key-presses. The package is growing fast and the intent is to support everything. But to summarize, the only thing that is explicitly supported at the moment is UI system testing (through any game type).

 

So if I press a button on a VR controller, it doesn't record it for instance?

 

We are actively working to support controller input (coming soon). But if you record the tests on a desktop/laptop, and then play them on VR, they should work using non-controller based inputs. If your VR app is not playable outside of a VR because it only uses controller API's from the old input system, then record and playback will not work. Also, the new input system is not yet supported.

 

Hi, I'm doing some exploratory testing on this package and the features it provides are great. I found and recorded some strange behavior and would like to know more about it. In my test project, if you revisit the scene where the test begins, it will forget earlier inputs.

In my project, I have three scenes: Scene1, Scene2, and Scene3:

• Scene1 has a toggle box and a button that leads to Scene2.
• Scene2 has a slider and two buttons that lead to Scene1 and Scene3 respectively.
• Scene3 has input text and a button that leads back to Scene2.

In this project, if you start your recording in a scene, do stuff, leave the scene, then return to that scene, it will forget what happened earlier.


In the video, my inputs were:

1. (Scene 1) - I toggled the box 8 times and went to Scene 2
2. (Scene 2) - I slid the slider's knob across and went to Scene 3
3. (Scene 3) - I typed "test 1" and went to Scene 2
4. (Scene 2) - I slid the slider's knob across and back and returned to Scene 1
5. (Scene 1) - I missed the toggle, clicked the toggle once, and went to Scene 2
6. (Scene 2) - I did nothing and went to Scene 3
7. (Scene 3) - I typed "test 3" and ended the recording

After, I played the recording. I was expecting it to start from step 1, but it started at step 5.


An interesting warning pops up when you revisit the scene:

 

We will look into this. Thanks for reporting it!

 

New question, is there any way to modify generated test classes so that they don't need the GeneratedTests.asmdef ?

 

Okay, a better question (?), how can the main assembly refer GeneratedTests.asmdef ?

Small context, we can make the project refer to generated test assembly which can be used to create a wrapper runner to run generated tests, right?

but

using GeneratedAutomationTests;

gives this error

error CS0246: The type or namespace name 'GeneratedAutomationTests' could not be found

 

Hi @Daybreaker32. Sorry for the delayed response. That will no longer be an issue with an upcoming release of the AutomatedQA package. Possibly as early as version 8.0. We have some big announcements coming up about the direction the project is taking as a major update is underway that will radically change the underlying test runner logic. This change will completely eliminate many limitations or issues related to the usage of UTF and its inherent dependencies while adding tons of functionality and control around test execution that does not currently exist. We will announce more details sometime in the near future.

 

I found an issue.
It could not find DontDestoryOnLoad Objects.
I am using this module temporarily like this.

Runtime/Core/AdvancedQuerying/ElementQuery.cs

Code (CSharp):

1.   public List<GameObject> GetAllActiveGameObjects()
2.         {
3.             List<GameObject> results = new List<GameObject>();
4.             var scenes = GetOpenScenes();
5.             foreach (var scene in scenes)
6.             {
7.                 results.AddRange(GetChildren(scene.GetRootGameObjects().ToList()));
8.             }
9.  
10.             //I Added it
11.             results.AddRange(GetChildren(Instance.gameObject.scene.GetRootGameObjects().ToList()));
12.             //
13.  
14.             return results.FindAll(x => x.activeInHierarchy && x.activeSelf);
15.         }
16.  

Please check it.

 

This was fixed with the latest release. Please update your package when you get a chance.

 

Hello, I'm getting a couple of issues when moving to the new update. I listed the issues below, and they're recorded in this video. I have also attached the Tests.cs and PageObject.cs files below.

• In a test scene, I have a button that loads another scene if you click it. I make a recording where the only input is clicking this button. If I use this recording to generate a test with "Use Simplified Driver Code" enabled, I'll get these errors when running the test:

Console

Code (CSharp):

1. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
2. #0 GetStacktrace(int)
3. #1 DebugStringToFile(DebugStringToFileData const&)
4. #2 DebugLogHandler_CUSTOM_Internal_Log(LogType, LogOption, ScriptingBackendNativeStringPtrOpaque*, ScriptingBackendNativeObjectPtrOpaque*)
5. #3  (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
6.  

Test Runner

Code (CSharp):

1. CanPlayToEnd (11.447s)
2. ---
3. Unhandled log message: '[Error] AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.'. Use UnityEngine.TestTools.LogAssert.Expect
4. ---
5. #0 GetStacktrace(int)
6. #1 DebugStringToFile(DebugStringToFileData const&)
7. #2 DebugLogHandler_CUSTOM_Internal_Log(LogType, LogOption, ScriptingBackendNativeStringPtrOpaque*, ScriptingBackendNativeObjectPtrOpaque*)
8. #3  (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
9. ---
10. AQA: Using RecordedPlaybackPersistentData - kRecordedPlaybackFilename: playback_recording.json
11. AQA: Load Scene Scene1
12. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
13. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
14. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
15. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
16. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
17. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
18. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
19. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
20. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
21. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
22. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
23. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
24. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
25. AQA: Could not find GameObject provided to driver Click() method, or null object was provided as an argument.  [Query String: Canvas/Btn-ToSceneTwo]

If you disable the "Use Simplified Driver Code" to generate the Steps file instead, there are no errors.

• (0:35-1:03) When generating a test from a recording, GeneratedTests.asmdef resets its Assembly Definition References. If you have additional coroutines/functions in your tests, re-adding those references can be time-consuming.

If I missed anything or can contribute anything else, please let me know and I'll add it!

 

Does this only occur if the button launches into another scene? If you disable the scene load logic, does it still occur? I used your code on my side (made up a similar scenario) and didn't reproduce the issue.

Can you find the error message in the code and put a break point (should be ElementQuery.cs line 188), and then tell me what the variable "querySelector" has as a value?

 

Thanks for the quick reply!

I tried a few things:
- Disabled the scene change code: Same Error (Invalid Query String)
- Tried making a new button, and added the scene change code: Test Succeeded
- Renamed old button from "Btn-ToSceneTwo" to "ToSceneTwo" and added scene change code: Test Succeeded
- Renamed the old button from "ToSceneTwo" to "ToSceneTwo-": Same Error
From this, it looks like a parsing error caused by the hyphen.

The debugger also gave me this:
querySelector: "[name=Btn-ToSceneTwo]"

 

Got it! You found a bug. Sorry for that. We will fix it asap. We appreciate you reporting it and debugging this for us.

 

Hello! I ran into a couple of problems while using the Crawl option in the "Recorded Playback" section (1) and while trying to generate Full Tests (2).

1. Crawler
I have a project with multiple scenes in it. It seems that sometimes the crawler is trying to interact with a button from one of the previous scenes that is not present on the scene it's currently on. This results in the following console message:

Code (CSharp):

1. AQA: Cannot play recorded action: object Continue does not exist or is not viewable on screen. Ensure the object exists and reposition the object inside the screen space.
2.  
3. UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
4. Unity.AutomatedQA.AQALogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[]) (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/AQALogger.cs:10)
5. UnityEngine.Logger:LogError (string,object)
6. Unity.AutomatedQA.AQALogger:LogError (object) (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/AQALogger.cs:73)
7. UnityEngine.EventSystems.RecordingInputModule:UpdatePlay (int)
8. Unity.AutomatedQA.GameCrawler/<PerformAction>d__56:MoveNext () (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/Automators/GameCrawler.cs:307)
9. UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

2. Full Tests

When I try to generate a Full Test from one of the recordings, a lot of errors come up in the generated script. They all seem to be caused by spaces in the scene name (e.g. a scene named "Menu 3D" will cause these errors, "Menu3D" won't).

Error Type 1:
Assets\AutomatedQA\GeneratedTests\PageObjects\Scene_Draggable Panel_PageObject.cs(3,34): error CS1514: { expected

This string was generated for a scene called "Draggable Panel":
public class Scene_Draggable Panel_PageObject { ... }

Error Type 2:
Assets\AutomatedQA\GeneratedTests\Tests_Crawler_Test_1.cs(20,58): error CS1003: Syntax error, ',' expected

This is the string that the error referred to (scene name is "Menu 3D"):
yield return Driver.Perform.Click(Scene_Menu 3D_PageObject.Clickable_Menu_3D_OpenDraggablePanelSceneButton);

Error Type 3:
Assets\AutomatedQA\GeneratedTests\PageObjects\Scene_Layout Groups_PageObject.cs(3,31): error CS0116: A namespace cannot directly contain members such as fields or methods

The string is:
public class Scene_Layout Groups_PageObject { ... }

Error Type 4:
Assets\AutomatedQA\GeneratedTests\PageObjects\Scene_Layout Groups_PageObject.cs(4,5): error CS1022: Type or namespace definition, or end-of-file expected

This error points to the string with the curly brace:
public class Scene_Layout Groups_PageObject
{

Please let me know if there is any additional information I could provide.

 

Thanks for this detailed report. We will fix these issues for the next release.

Here should be a quick fix for the GameCrawler issue. Let me know if it doesn't work for you. This code snippet replaces the existing Refresh methods in GameListenerHandler.cs:

Code (CSharp):

1.        
2.         public static void Refresh()
3.         {
4.             _activeListeners = new List<AutomationListener>();
5.             forceRefresh = true;
6.         }
7.         public void Refresh(Scene scene, LoadSceneMode mod)
8.         {
9.             Refresh();
10.         }
11.  

 

We liked Automated QA.

We found a problem in current version.

In our environment if you try record next scenario:

Click Button_1

Click Button_2

Playback will be:

Click Button_1

Click Button_1

It happens because there two gameobjects has same path
("objectHierarchy": "Canvas/Container").




For our game it happens often, because we have lists of widgets, which are created through code and has different content.

Test project for reproduce: https://yadi.sk/d/jjgMVLKc6Yhxrw

 

Hello again! I've tried to apply the fix you've provided, but another problem has emerged. Every time I alter the code of GameListenerHandler.cs, save it and go back to Unity, it shows a loading bar that says "Compiling C# Scripts", "Reload Script Assemblies" and when it's done, every change I've made is reverted. Is there a way to prevent this?

Additionally, I've discovered a way this error could be reproduced without the scene change. If the interactable object is untoggled while the crawler is still running on the same scene, it can trigger the same error:

Code (CSharp):

1. AQA: Cannot play recorded action: object Newgame does not exist or is not viewable on screen. Ensure the object exists and reposition the object inside the screen space.
2. UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
3. Unity.AutomatedQA.AQALogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[]) (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/AQALogger.cs:10)
4. UnityEngine.Logger:LogError (string,object)
5. Unity.AutomatedQA.AQALogger:LogError (object) (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/AQALogger.cs:73)
6. UnityEngine.EventSystems.RecordingInputModule:UpdatePlay (int)
7. Unity.AutomatedQA.GameCrawler/<PerformAction>d__56:MoveNext () (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/Automators/GameCrawler.cs:307)
8. UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

Not sure if that helps, but here are the specs:
Windows 10
Unity 2020.3.17f1
Automated QA 0.8.0-preview.1

 

The workaround you've linked has really helped, thank you!

Now, I've applied the GameCrawler fix you provided, but it doesn't seem to be working. I've managed to reproduce the same error (e.g. "AQA: Cannot play recorded action: object OpenRenderTextureSceneButton does not exist or is not viewable on screen. Ensure the object exists and reposition the object inside the screen space.") multiple times after the crawler has changed scenes.

As for the toggle question, the element is being toggled into disappearing/being inactive. This error is unaffected by the fix too.

Just in case, here is the project I'm currently using for testing:
https://assetstore.unity.com/packages/essentials/ui-samples-25468

The only change I've made is adding some buttons for scene changing. They are using the following code:

Code (CSharp):

1. public class OpenSceneButton : MonoBehaviour
2. {
3.     public string sceneName;
4.  
5.     public void OpenScene()
6.     {
7.         SceneManager.LoadScene(sceneName);
8.     }
9. }