Search Unity

Unity Automated QA Package [v0.7]

Discussion in 'Testing & Automation' started by dylanscandinaro, Apr 8, 2021.

  1. dylanscandinaro

    dylanscandinaro

    Unity Technologies

    Joined:
    Feb 12, 2020
    Posts:
    8
    The Automated QA package enables users to record and playback touch or drag interactions with the UI of a Unity Project and optionally use recordings to drive Unity Tests - in the editor, on a local iOS or Android device, or on an iOS or Android device managed by Unity in the cloud (email us at AutomatedQA@unity3d.com for access to this feature).



    Key Features
    • Object based recording: record once in the editor and play back on many different devices and aspect ratios
    • Recorded playbacks are stored as time stamped sequences of objects identified by name and tags
    • Composite recorded playbacks: create modular recordings to minimize duplication and automatically keep recorded playbacks up to date when a component is updated
    • Automatic recorded test generation: create Unity Test Framework Tests from recorded playbacks with the click of one button
    • Run tests from your CI pipeline: recorded tests can be run on your build machine, like any other Unity Test, or use the cloud on-device testing feature to run tests on our cloud Android device farm invoked from your build pipeline.


    Recorded Playback Creation -> Unity Test Execution

    Limitations
    • This is an experimental package with volatile features. New versions may include breaking changes.
    • Unity 2019.4 or above required
    • Recording is currently limited to UI elements implemented with Unity UI

    Installation
    1. With your project open in the Unity Editor, open the Package Manager (Window > Package Manager).
    2. Press the plus button (+) in the top left of Package Manager and then select "Add package from git URL...".
    3. Enter com.unity.automated-testing in the text box and then press "Add".
    See our package documentation for getting started instructions and more information. Please email us at AutomatedTesting@unity3d.com with any questions!

    Changelog
    [0.7.0] - 2021-10-27
    Major Features and Improvements

    • Added RecordableInput class that can be used as a drop in replacement for the Input class that works with recorded playback. Currently mouse clicks, touch, keyboard keys, and joystick buttons are supported. Pass-through methods are provided for the remaining functionality for ease of use.
    • Added scriptable object CloudTestDeviceInput that lets a user specify specific devices on which to execute their tests in the cloud.
    • Added support to retrieve HTML reports for cloud device runs
      • Added new report to aggregate display of results for multi-device cloud runs, with links to each individual device's run results in an existing HTML report.
    • Added Heap size performance sampling to reporting charts found in the existing test report.
    • Added Page Objects to test generation tool.
      • Every GameObject that code interacts with has its query string locator stored in a single Page Object class, referenced by any number of tests.
      • Page Object properties are organized in different Page Object classes based on the scene they are located in.
      • When generating tests, the tool will see if an existing Page Object property exists. It will only add a property to a class if no reference exists.
    • All menu options moved to a single "Automated QA Hub" window
    • Moving Cloud Testing behind a feature flag temporarily as we redesign the UX. Please email us at AutomatedQA@unity3d.com for early access.
    Bug Fixes and Minor Changes
    • Added support for custom report data.
    • Fixed error when generating full test from a recording that starts with a number.
    • Added option to generate simpler code from the Test Generator by utilizing the Driver logic's new query string approach to finding GameObjects.
    • Rewrote Recorded Playback editor window in new UIElement logic.
    • Redesigned look of window.
    • Added GameCrawler "Crawl" button to window, which executes a default crawl and generates a recording json file.
    • Added tracking for the object index in the hierarchy to help differentiate objects with the same name
    • Added fix to ensure automation waits for a GameObject animating from off screen to finish moving into the camera frustum before interacting with it.
    • Added enforcement of an en-US culture in CodeGenerator logic.
    • Added a context menu with the ability to generate tests in the main recorded playback window.
    • Added a script for creating and managing service accounts to use in CI workflows.
    • Fixed an issue where spaces in recording json file names were used when generating a class name in the Test Generator.
    • Fixed an issue that causes backslashes to be used in generated code for Staged Runs.
    • Added "TryPerform" to Driver class. If an action cannot be performed, the test will not fail.
    • Added drop downs for builds and jobs in cloud test window
    • Added GameElement "AutoID" - an automatically-generated GUID to uniquely identify GameElements.
    • Adds GameElement script to all interactable GameObjects and assigns a unique GUID automatically.
    • Allows for GameObject to be renamed, moved, etc. without breaking a test that references this ID using Driver.cs.
    • Added menu option to automatically add GameElements to interactable UI Elements in a scene. Menu -> "Automated QA" -> "Add Game Elements To Scene Objects"
    • Renamed AutomatedQAEditorSettings to AutomatedQABuildConfig and AutomatedQARuntimeSettings to AutomatedQASettings.
    • Fixed an issue generating iOS cloud testing builds when there are spaces in the build name.

    Breaking Changes
    • Removed "Full Build" cloud testing support.
    [0.6.1] - 2021-08-12
    Major Features and Improvements

    • New GameElement and ElementQuery classes for reliably finding GameObjects, no matter where they are moved in a scene.
    • Uses Xpath & JQuery-like query string selectors for identifying GameObjects.
    • New Driver class that allows for much simpler ways to write custom code and perform actions on objects (ex: Driver.Perform.Click("#SubmitButton")).
    • Added GameCrawler that allows users to create an AutomatedRun that plays randomly through a game, recording warnings, errors, and notifying users when and where it gets stuck.
    Bug Fixes and Minor Changes
    • Reverting change that removed the ability to generate "Simple Tests", which is a test that points to a recording file.
    • Fixed an issue with temp directories not being properly deleted.
    • Adding FPS tracking to reports. FPS sampled over duration of test execution is displayed in a graph accessible from the html report (FPS Data button).
    • Fixed issue allowing objects off screen to be interacted with.
    • Add support for recording right and middle mouse clicks.
    • Added functionality to allow both recording and playback while editor is in play mode.
    • Both record & playback can be done repeatedly in the same session.
    • Editor does not stop playmode on completion, but both record & playback can still be started outside of playmode.
    • Added support for running cloud tests from the command line using BuildAndRunTests.
    • Fixed issue where tests may timeout during command line batch mode runs.
    • Fixed issue with the InputField text listener not being added right away.
    • Screenshot timing after each playback action completes is now configurable in settings.
    • Exposed "Quit on finish" in Automated Run Config.
    • Added ability to generate a "Full" test from Automated Runs.
    • Added settings for log level
    • Added support for TextMeshPro TMP_InputFields and TMP_Text fields
    • Can record and playback input into TMP_InputFields, and find objects in ElementQuery by TMP_InputField & TMP_Text text content.


    Breaking Changes
    • Several APIs were moved from the test base class to Driver.cs. "Full" generated tests from v0.5.0 will need their PerformAction and RegisterStep method invocations updated.
    • The experimental Composite Recordings window has been removed. Instead, Automated Runs can be used to playback multiple recordings in sequence.

    [0.5.0] - 2021-06-28

    Major Features and Improvements

    • Support for running tests on real iOS devices in the cloud
    • New [CloudTest] attribute can be used to specify any Unity Test Framework test to run on real devices in the cloud
    • Automated QA > Test Generation... allows for step-by-step Unity Test Framework test generation (C# code).
      • Generates code for every step/action taken in a recording.
      • Allows for assertions or additional custom logic between each step.
      • Can select which recordings to generate tests for.
      • Editor Window warns user if about to overwrite custom edits in a recording's test when re-generating test.
    Breaking changes
    • Recording file in RecordedPlaybackAutomator is now an asset reference instead of path string (Note: Only relevant if you're using the experimental Automated Runs feature)
      • Migration: Update the file path in AutomatedRuns using RecordedPlaybackAutomator
    Bug Fixes and Minor Changes
    • Added dynamic wait logic so that differences in load times or animations does not result in automation failing to interact with the target object.
    • Delayed loading of the recorded playback controller to avoid issues with initializing too early.
    [0.4.0] - 2021-05-25
    Major Features and Improvements

    • HTML & XML reports for AutomatedQA runs.
      • XML report is designed for loading tests results into a CI process run.
      • Latest HTML reportcan be opened in editor from the "Recorded Playback" and "Composite Recordings" editor windows after playback.
      • Both currently stored in Application.persistentDataPath. CI/CD must copy report from path (Will be integrated with Cloud services & modified to take raw test data from devices so that explicit file extraction is not necessary).
    • Added settings file for editable values used in Automated Qa package, thus making various Automated QA behaviors customizable (more to be added as time goes on).
      • Added new editor window to edit the settings file (Automated QA > Settings).
      • Multiple config files with varying values can be stored in Assets/AutomatedQA/Resources, and desired settings can be requested from cloud config via file name.
    Bug Fixes and Minor Changes
    • Added placeholder segment file names in Composite Recording window when selecting "Save Segment".
    • Updated VisualFx to prevent visual clutter from a rapid series of events.
    • Updated validation of GameObject that is about to be clicked. Increased performance. Now checks that an object is off screen or under another object that would intercept the click.
    • Updated recorded tests to load an empty scene after execution.
    • Fixed ignoring depth surface warnings when taking screenshots on Mac.
    • Added new Automators: Scene Automator and Text Input Automator.
    • Fixed event timings and scene loading with composite recordings
    [0.3.1] - 2021-05-04
    Major Features and Improvements

    • New AutomatedRun object to link together recordings and custom C# scripts to automate gameplay.
      • Create an AutomatedRun with Create -> Automated QA -> Automated Run
    • New Automator class. Extend this class to create custom automators
      • Extend the AutomatorConfig class for your Automator to expose it in the AutomatedRun inspector.
    Bug Fixes and Minor Changes
    • Fixed an issue where drop events had extra delay
    • Fixed an issue with the Upload Recording window requiring entitlements
    • Removed dependency on com.unity.nuget.newtonsoft-json package
    • Fixed entry scene to work with initialization scenes
    • Moved menu items to top level Automated QA menu
    • Added fix to CloudTestResults.cs that prevents ITestRunCallback from being eagerly stripped by the "ahead of time" compilation in IL2CPP builds.
    • Added logic to cleanup temporary files accumulating from previous recordings and stored in the persistent data path.
    • Removed Linq usage from package as Linq can be a heavy library for mobile game development.
    [0.3.0] - 2021-04-21
    Breaking changes

    • Simplified assembly definitions
      • Migration: Please update asmdefs to reference Unity.AutomatedQA
    • Renamed asset directory to AutomatedQA
      • Migration: Please delete the old AutomatedTesting folder
    Bug Fixes and Minor Changes
    • Package directory restructure
    • Added SettingsManager package and AutomatedQAEditorSettings/AutomatedQARuntimeSettings to wrap package settings
    • Disabled Cloud Testing window for unsupported platforms
    • Fixed an issue with recorded tests using composite recordings
    • Fixed an issue causing two simultaneously active EventSystem components, resulting in a flood of console warnings
    • Fixed an issue where multi-clicking "Record" or "Play" buttons creates multiple RecordedPlaybackController GameObjects
     
    Last edited: Oct 28, 2021
  2. chad-wmgi

    chad-wmgi

    Joined:
    Mar 31, 2020
    Posts:
    8
    Is the new Input System supported? I'm getting spammed with an error about Input.mousePosition.
     
  3. dylanscandinaro

    dylanscandinaro

    Unity Technologies

    Joined:
    Feb 12, 2020
    Posts:
    8
    As long as the elements you'd like to interact with are implemented with Unity UI, then yes with an additional step:

    With your Unity Project open, from the menu bar navigate to Edit > Project Settings > Player > Other Settings > Configuration, then set Active Input Handling to “Both”.
     
    Tamgros likes this.
  4. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    An error occurred in the "Recorded Test".
    After running "Generate Recorded Tests", select "GeneratedTests.dll" and run it. Then the following error appeared in the console.

    Code (CSharp):
    1. FileNotFoundException: Could not find file "/Users/***/Library/Application Support/DefaultCompany/UnityTestFramework/config.json"
    2. System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) (at <fb001e01371b4adca20013e0ac763896>:0)
    3. (snip)
    4. System.IO.File.ReadAllText (System.String path) (at <fb001e01371b4adca20013e0ac763896>:0)
    5. Unity.RecordedTesting.Manager.GetDeviceFarmConfig (Unity.RecordedTesting.DeviceFarmOverrides dfOverrides) (at Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/Manager.cs:31)
    6. Unity.RecordedTesting.TestResults.RunStarted (NUnit.Framework.Interfaces.ITest testsToRun) (at Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/TestTools/TestResults.cs:33)
    7. UnityEngine.TestRunner.Utils.TestRunCallbackListener+<>c__DisplayClass1_0.<RunStarted>b__0 (UnityEngine.TestRunner.ITestRunCallback callback) (at Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/Utils/TestRunCallbackListener.cs:16)
    8. UnityEngine.TestRunner.Utils.TestRunCallbackListener.InvokeAllCallbacks (System.Action`1[T] invoker) (at Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/Utils/TestRunCallbackListener.cs:38)
    9. UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr) (at /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/Coroutines.cs:17)
    There is no config.json in that directory, but there is config_recording.json and playback_recording.json.

    environment:
    • macOS Big Sur (11.2.3)
    • Unity 2019.4.11f1
    • Automated QA 0.2.0-preview.3
    • Test Framework 1.1.24
     
  5. souranil_sen

    souranil_sen

    Unity Technologies

    Joined:
    Mar 19, 2020
    Posts:
    3
    Thanks for the stacktrace, we are investigating this. The SDK shouldn't look for that file while running locally. Can I you please check if there are any scripting defines in "Player Settings" ?
     

    Attached Files:

  6. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    Thanks for your response!
    I found `CLOUD_TEST_UTF` symbol in Scripting Define Symbols. After removing this, the test will now run.

    But, every test failed with the following exception.

    Code (CSharp):
    1. CanPlayToEnd (3.571s)
    2. ---
    3. SetUp : System.Reflection.ReflectionTypeLoadException : Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
    4. ---
    5. --SetUp
    6.   at (wrapper managed-to-native) System.Reflection.Assembly.GetTypes(System.Reflection.Assembly,bool)
    7.   at System.Reflection.Assembly.GetTypes () [0x00000] in <fb001e01371b4adca20013e0ac763896>:0
    8.   at Unity.RecordedTesting.RecordedTesting.GetAllRecordedTests () [0x00023] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:55
    9.   at Unity.RecordedTesting.RecordedTesting.GetLocalRecordingFile (System.String testName) [0x00002] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:36
    10.   at Unity.RecordedTesting.RecordedTesting.SetupRecordedTest (System.String testName) [0x00008] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:115
    11.   at Unity.RecordedTesting.TestTools.RecordedTestSuite+<Setup>d__0.MoveNext () [0x00027] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/TestTools/RecordedTestSuite.cs:23
    12.   at UnityEngine.TestTools.BeforeAfterTestCommandBase`1+<ExecuteEnumerable>d__12[T].MoveNext () [0x00185] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/NUnitExtensions/Commands/BeforeAfterTestCommandBase.cs:87
     
  7. marcelohao

    marcelohao

    Joined:
    May 27, 2014
    Posts:
    1
    Great initiative!

    After I click on "Record", I get a bunch of entries like this one on the log:

    There are 2 event systems in the scene. Please ensure there is always exactly one event system in the scene
    UnityEngine.EventSystems.EventSystem:Update () (at C:/Program Files/Unity/Hub/Editor/2020.3.5f1/Editor/Data/Resources/PackageManager/BuiltInPackages/com.unity.ugui/Runtime/EventSystem/EventSystem.cs:398)

    When I click on "Stop", the json file only has mouse inputs logged, keyboard and gamepad are not getting logged (are those not supported yet?)

    Thanks!
     
  8. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    The "There are 2 event systems in the scene" bug was fixed in the most recent release of the package (0.3.0).

    The package is not yet recording keyboard/gamepad inputs. Both the old and new input systems need to be handled seamlessly depending on the package customers are using, and keyboard/gamepad inputs will require that. So the handling of those types of inputs is important and is on the horizon.

    Right now, clicks and drags are recorded within the Canvas space. However, it will not work yet with GameObjects not on the Canvas (ex: world objects using colliders to perform actions similar to clicks and drags; this functionality is also in the queue).
     
    Last edited: Apr 29, 2021
  9. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    I found the cause.

    `System.Reflection.Assembly.GetTypes` seems to throw an exception if the assembly `Microsoft.CodeAnalysis.Scripting`.

    In my project, `com.unity.immediate-window` depended on this assembly, so I removed it and was able to run the test!
     
    tsibiski likes this.
  10. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    BNS_oyama likes this.
  11. SchmidtDavidSimon

    SchmidtDavidSimon

    Joined:
    Sep 7, 2020
    Posts:
    11
    Hi I also have a question to this toppic. Is it possible to check if a scene was created by a recorded playback? We have our DI set in a way it initializes systems only when our game-scene is starting. Thats why tests fail since a new scene is created for the tests start.
     
  12. SchmidtDavidSimon

    SchmidtDavidSimon

    Joined:
    Sep 7, 2020
    Posts:
    11
    And already the next question :D

    Is it possible to create Tests out of Automated Runs? If so how would I go about it? If not I think it would be a very nice feature!
     
  13. skazmierczak

    skazmierczak

    Joined:
    Feb 28, 2020
    Posts:
    4
    Can this be used with the code coverage package?
     
  14. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    In v0.4.0:
    • I'm not want to put AutomatedQASettings.json on "Resources" folder
    • Warning mesages in the console. because unnecessary .meta files, follows:
      • RecordedTesting/Editor.meta
      • RecordedTesting/Runtime/TestTools.meta

    In v0.5.0
    • Warnings in the console
      • There are inconsistent line endings in the 'Assets/AutomatedQA/GeneratedTests/Steps/xxx_Steps.cs' script. Some are Mac OS X (UNIX) and some are Windows.
    • Tests fail in batch mode. (same tests are successful in Unity Editor GUI)
    • I am very happy that the "dynamic wait logic" makes the execution more stable. But, I miss the ease of replacing the JSON file by just recording it again.
     
    Last edited: Jun 30, 2021
  15. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    For the first point, are you able to add it to a git ignore? It needs to be in resources for use in the Cloud and binary builds. If the problem is checking it in, it can be added to git ignore, or a similar feature in other version controls.

    For the final issue, you can tell the framework to NOT use dynamic waits by setting the "Use Dynamic Waits" boolean in the Settings editor window. It will then revert to the old way of running tests, and you can continue to re-record failing tests.
     
  16. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    Thanks. I'm not talking about running tests.

    My scenario as follows:
    1. Recording tests to json
    2. Automated QA | Test Generation
    3. Add asserts in test code
    4. One day, UI operation and/or transitions will be changed!
    5. Re-Recording tests to json
    6. Use the test code without changing it

    Until v0.4, this was possible.
    But, in v0.5, Re-Test Generation is required.

    This behavior is acceptable for us programmers, but I think it is cumbersome for test engineers and level designers to use Automated QA.

    Given that UI changes are frequent, such as in game tutorials, I hope have preferred to keep the advantage of being able to replace only json.
     
  17. BNS_oyama

    BNS_oyama

    Joined:
    Jun 1, 2020
    Posts:
    1
    I have one request for the features of Automated QA. Please doesn't overwrite this file, if "Assets/AutomatedQA/GeneratedTests/GeneratedTests.asmdef" already exists when user runs Test Generation. If a user creates test codes in "Assets/AutomatedQA/GeneratedTests", they will add elements to Assembly Definition References in this .asmdef file. But because this .asmdef file is initialized when they creates a new test, they needs to reconfigure this .asmdef file.
     
    nowsprinting likes this.
  18. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    Thanks for this suggestion!
     
    BNS_oyama likes this.
  19. bkeeley_unity

    bkeeley_unity

    Joined:
    Jun 14, 2021
    Posts:
    1
    I've been having issues trying to get either version 0.3.1 or 0.5.0 working in our project.

    With v0.5.0 specifically, whenever I try to run an Automated Run using a recording added with the RecordedPlaybackAutomatedConfig, I get the following errors and stack traces:

    Code (CSharp):
    1. BadImageFormatException: VAR 1 (TResult) cannot be expanded in this context with 1 instantiations
    2. System.Reflection.MonoMethod.GetBaseMethod () (at <695d1cc93cca45069c528c15c9fdd749>:0)
    3. System.MonoCustomAttrs.GetBase (System.Reflection.ICustomAttributeProvider obj) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    4. System.MonoCustomAttrs.GetCustomAttributes (System.Reflection.ICustomAttributeProvider obj, System.Type attributeType, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    5. System.Reflection.MonoMethod.GetCustomAttributes (System.Type attributeType, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    6. System.Attribute.GetCustomAttributes (System.Reflection.MemberInfo element, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    7. System.Attribute.GetCustomAttributes (System.Reflection.MemberInfo element) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    8. System.Reflection.CustomAttributeExtensions.GetCustomAttributes (System.Reflection.MemberInfo element) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    9. Unity.RecordedTesting.RecordedTesting.GetAllRecordedTests () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Automators/RecordedPlayback/RecordedTesting/RecordedTesting.cs:73)
    10. Unity.RecordedTesting.RecordedTesting.IsRecordedTest (System.String testName) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Automators/RecordedPlayback/RecordedTesting/RecordedTesting.cs:50)
    11. ReportingManager.InitializeDataForNewTest () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:242)
    12. ReportingManager.RecordLog (System.String message, System.String stackTrace, UnityEngine.LogType type) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:352)
    13. UnityEngine.Application.CallLogCallback (System.String logString, System.String stackTrace, UnityEngine.LogType type, System.Boolean invokedOnMainThread) (at <c7864a0eaeb24b2a999fb177623d54b4>:0)
    Code (CSharp):
    1. UnityException: List provided to AutomatedQATools.Last() was empty. Cannot invoke Last() on an empty list. Check for list being empty before invoking Last().
    2. Unity.AutomatedQA.AutomatedQaTools.Last[T] (System.Collections.Generic.List`1[T] list) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/AutomatedQaTools.cs:146)
    3. ReportingManager.FinalizeReport () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:174)
    4. ReportingMonitor.OnApplicationQuit () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingMonitor.cs:53)
    In v3.0.1 I also get a similar
     BadImageFormatException
    whenever I run a generated test from a recording in the Test Runner

    Windows 10
    Unity 2020.3.7f1
    Test Framework 1.1.24
    Automated QA 0.5.0-preview.1
    Automated QA 0.3.1-preview.2
     
    Last edited: Jul 15, 2021
  20. Regretful123

    Regretful123

    Joined:
    Nov 5, 2014
    Posts:
    9
    I am having some weird and unorthodox error where if I record step by step using the Record function, it works. However generating the script out to full test breaks.

    I think the culprit is that we have MaterialUI, which still uses Unity UI under the hood of the library. However, it seems like it cannot find the gameobject the QA is trying to find (Since it's moving). The JSON file works perfectly, but for whatever reason the full scripted tests (even with changes in deltaTime) it still errors out complaining that it cannot find the button it was target to click or release from.

    Can someone help me on what I can do from here? I'm kinda stuck with this Dialog issue. Here's the full test error issue:

    Code (CSharp):
    1. AQA: Click position recorded relative to spot within the target GameObject "Button - Affirmative" is positioned outside of the camera frustum (is not visible to the camera). Since this is a GameObject in the UI layer, this may mean that the object has not scaled or positioned properly in the current aspect ratio (865w X 487h) and current resolution (1920 X 1080) compared to the recorded aspect ratio (865h X 487w) and recorded resolution (1920 X 1080).
    2. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    3. Unity.AutomatedQA.AQALogHandler:LogFormat(LogType, Object, String, Object[]) (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AQALogger.cs:10)
    4. UnityEngine.Logger:LogError(String, Object)
    5. Unity.AutomatedQA.AQALogger:LogError(Object) (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AQALogger.cs:73)
    6. UnityEngine.EventSystems.RecordingInputModule:UpdatePlay(Int32)
    7. Unity.AutomatedQA.<Action>d__24:MoveNext() (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AdvancedQuerying/Driver.cs:292)
    8. UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
    9.  
    Win 10 pro
    Automated QA preview.2 - 0.6.1
    Test Framework - 1.1.29
    Unity 2019.4.9f1 - Android configuration
     
  21. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    Hi there. I actually wrote the logic involving the checks around an object being outside the camera frustum; so hopefully I can help you here.

    Is the object still visible on the screen then, despite this error? Can you retrieve the calculated point of click via debug (search for the error message above and put a breakpoint). Is it calculating the position of the object to be outside the bounds of the viewport? It should indeed be outside the expected viewport from a raw x/y position - but if the object is visible to your eyes, something weird must be throwing this calculation off.

    Also, is your generated test simply referencing the json playback file, or is each step of the json written into C# (aka, the "Full Test" was generated)? If it's a "Full Test", can you compare the data stored in the associated TouchData object in this test's "Step" file with the raw json object for the step?
     
  22. Regretful123

    Regretful123

    Joined:
    Nov 5, 2014
    Posts:
    9
    Hi tsibiski,
    Yes the object is still visible on the screen. How do I go about calculating the vector2 position of where it will be clicked?
    Currently the UI was set to be outside as "Hidden", but appear out when "activated". It seems that whatever click event calculated, seems to be clicking on the object's original position instead of the new current position. This position appears to be outside of the camera frustum.

    To answer your question more precise, I can describe the step-by-step of what I did to reach to the problem:
    1) Record, wait until the UI "appears"
    2) Re-play, make sure that it doesn't break anything else other than the intended step
    3) Tweak deltaTime to speed up the click process (I went into Json file, and manually edit the timeDelta to either 0.001 or 0.5 depends on how long it takes for the UI to appear before I can click next.
    4) Re-play it again once more, but with the new timeDelta to ensure nothing else breaks
    5) Convert Recordings into Full Test
    6) Re-run it again using the Test-Runner -> Breaks.

    Perhaps, if it's possible, to click on the UI at "Just-in time" instead of predetermined position of where to click? I can see in the code that it has the object's name and where it's located in the scene hierarchy, so wouldn't it be much safer to obtain the gameobject at that instance of time, and use the transformation for click position?
     
  23. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    I sent a private conversation to test out a possible fix.
     
    Regretful123 likes this.
  24. Dasp

    Dasp

    Joined:
    Sep 25, 2012
    Posts:
    9
    I'm getting this after importing into our project. We specifically disabled non 2D physics from the project. Would be great if you would add support to enabled/disabled modules.

    Code (CSharp):
    1. Library\PackageCache\com.unity.automated-testing@0.6.1-preview.2\Runtime\Core\GameListenerHandler.cs(189,48): error CS1069: The type name 'Collider' could not be found in the namespace 'UnityEngine'. This type has been forwarded to assembly 'UnityEngine.PhysicsModule, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' Enable the built in package 'Physics' in the Package Manager window to fix this error.
     
  25. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    We will add a fix for this. Meanwhile, that code can be commented out or removed as it is not fully activated yet.


    Code (CSharp):
    1.                                 if (scriptName == "Collider" || typeof(Collider).IsAssignableFrom(components[co].GetType()))
    2.                                 {
    3.                                     if (AllActiveAndInteractableGameObjects[x].GetComponent<Collider>().isTrigger)
    4.                                     {
    5.                                         AddListener(AllActiveAndInteractableGameObjects[x], ActableTypes.Clickable);
    6.                                         continue;
    7.                                     }
    8.                                 }
     
  26. Dasp

    Dasp

    Joined:
    Sep 25, 2012
    Posts:
    9
    Thanks, I can wait for update. We would like to use that in combination with Unity Cloud Build so not in a hurry.
     
  27. IS_Twyker

    IS_Twyker

    Joined:
    Sep 6, 2021
    Posts:
    9
  28. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    It has not been tailored to XR, and thus is not tested yet for XR apps. But it technically works for any type of game. It is currently best tailored to UI-based games, and testing of UI systems in games. It has limited, but ever increasing, support for other aspects of Unity apps like key-presses. The package is growing fast and the intent is to support everything. But to summarize, the only thing that is explicitly supported at the moment is UI system testing (through any game type).
     
  29. IS_Twyker

    IS_Twyker

    Joined:
    Sep 6, 2021
    Posts:
    9
    So if I press a button on a VR controller, it doesn't record it for instance?
     
  30. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    We are actively working to support controller input (coming soon). But if you record the tests on a desktop/laptop, and then play them on VR, they should work using non-controller based inputs. If your VR app is not playable outside of a VR because it only uses controller API's from the old input system, then record and playback will not work. Also, the new input system is not yet supported.
     
  31. JonPB

    JonPB

    Joined:
    Aug 31, 2021
    Posts:
    3
    Hi, I'm doing some exploratory testing on this package and the features it provides are great. I found and recorded some strange behavior and would like to know more about it. In my test project, if you revisit the scene where the test begins, it will forget earlier inputs.



    In my project, I have three scenes: Scene1, Scene2, and Scene3:
    • Scene1 has a toggle box and a button that leads to Scene2.
    • Scene2 has a slider and two buttons that lead to Scene1 and Scene3 respectively.
    • Scene3 has input text and a button that leads back to Scene2.

    In this project, if you start your recording in a scene, do stuff, leave the scene, then return to that scene, it will forget what happened earlier.


    In the video, my inputs were:
    1. (Scene 1) - I toggled the box 8 times and went to Scene 2
    2. (Scene 2) - I slid the slider's knob across and went to Scene 3
    3. (Scene 3) - I typed "test 1" and went to Scene 2
    4. (Scene 2) - I slid the slider's knob across and back and returned to Scene 1
    5. (Scene 1) - I missed the toggle, clicked the toggle once, and went to Scene 2
    6. (Scene 2) - I did nothing and went to Scene 3
    7. (Scene 3) - I typed "test 3" and ended the recording

    After, I played the recording. I was expecting it to start from step 1, but it started at step 5.


    An interesting warning pops up when you revisit the scene:
     
  32. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    We will look into this. Thanks for reporting it!
     
  33. Daybreaker32

    Daybreaker32

    Joined:
    Jun 11, 2014
    Posts:
    73
    New question, is there any way to modify generated test classes so that they don't need the GeneratedTests.asmdef ?
     
  34. Daybreaker32

    Daybreaker32

    Joined:
    Jun 11, 2014
    Posts:
    73
    Okay, a better question (?), how can the main assembly refer GeneratedTests.asmdef ?

    Small context, we can make the project refer to generated test assembly which can be used to create a wrapper runner to run generated tests, right?

    but
    using GeneratedAutomationTests;
    gives this error
    error CS0246: The type or namespace name 'GeneratedAutomationTests' could not be found
     
  35. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    Hi @Daybreaker32. Sorry for the delayed response. That will no longer be an issue with an upcoming release of the AutomatedQA package. Possibly as early as version 8.0. We have some big announcements coming up about the direction the project is taking as a major update is underway that will radically change the underlying test runner logic. This change will completely eliminate many limitations or issues related to the usage of UTF and its inherent dependencies while adding tons of functionality and control around test execution that does not currently exist. We will announce more details sometime in the near future.
     
    Daybreaker32 likes this.
  36. leet_unity

    leet_unity

    Joined:
    Oct 15, 2021
    Posts:
    10
    I found an issue.
    It could not find DontDestoryOnLoad Objects.
    I am using this module temporarily like this.

    Runtime/Core/AdvancedQuerying/ElementQuery.cs
    Code (CSharp):
    1.   public List<GameObject> GetAllActiveGameObjects()
    2.         {
    3.             List<GameObject> results = new List<GameObject>();
    4.             var scenes = GetOpenScenes();
    5.             foreach (var scene in scenes)
    6.             {
    7.                 results.AddRange(GetChildren(scene.GetRootGameObjects().ToList()));
    8.             }
    9.  
    10.             //I Added it
    11.             results.AddRange(GetChildren(Instance.gameObject.scene.GetRootGameObjects().ToList()));
    12.             //
    13.  
    14.             return results.FindAll(x => x.activeInHierarchy && x.activeSelf);
    15.         }
    16.  
    Please check it.
     
  37. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    This was fixed with the latest release. Please update your package when you get a chance.
     
  38. JonPB

    JonPB

    Joined:
    Aug 31, 2021
    Posts:
    3
    Hello, I'm getting a couple of issues when moving to the new update. I listed the issues below, and they're recorded in this video. I have also attached the Tests.cs and PageObject.cs files below.

    • In a test scene, I have a button that loads another scene if you click it. I make a recording where the only input is clicking this button. If I use this recording to generate a test with "Use Simplified Driver Code" enabled, I'll get these errors when running the test:
    Console
    Code (CSharp):
    1. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    2. #0 GetStacktrace(int)
    3. #1 DebugStringToFile(DebugStringToFileData const&)
    4. #2 DebugLogHandler_CUSTOM_Internal_Log(LogType, LogOption, ScriptingBackendNativeStringPtrOpaque*, ScriptingBackendNativeObjectPtrOpaque*)
    5. #3  (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
    6.  
    Test Runner
    Code (CSharp):
    1. CanPlayToEnd (11.447s)
    2. ---
    3. Unhandled log message: '[Error] AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.'. Use UnityEngine.TestTools.LogAssert.Expect
    4. ---
    5. #0 GetStacktrace(int)
    6. #1 DebugStringToFile(DebugStringToFileData const&)
    7. #2 DebugLogHandler_CUSTOM_Internal_Log(LogType, LogOption, ScriptingBackendNativeStringPtrOpaque*, ScriptingBackendNativeObjectPtrOpaque*)
    8. #3  (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
    9. ---
    10. AQA: Using RecordedPlaybackPersistentData - kRecordedPlaybackFilename: playback_recording.json
    11. AQA: Load Scene Scene1
    12. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    13. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    14. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    15. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    16. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    17. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    18. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    19. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    20. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    21. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    22. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    23. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    24. AQA: Invalid Query String: A starting bracket "[" was encountered, but a closing bracket "]" was not encountered before the expected end of the query.
    25. AQA: Could not find GameObject provided to driver Click() method, or null object was provided as an argument.  [Query String: Canvas/Btn-ToSceneTwo]
    If you disable the "Use Simplified Driver Code" to generate the Steps file instead, there are no errors.

    • (0:35-1:03) When generating a test from a recording, GeneratedTests.asmdef resets its Assembly Definition References. If you have additional coroutines/functions in your tests, re-adding those references can be time-consuming.

    If I missed anything or can contribute anything else, please let me know and I'll add it!
     

    Attached Files:

  39. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    Does this only occur if the button launches into another scene? If you disable the scene load logic, does it still occur? I used your code on my side (made up a similar scenario) and didn't reproduce the issue.

    Can you find the error message in the code and put a break point (should be ElementQuery.cs line 188), and then tell me what the variable "querySelector" has as a value?
     
  40. JonPB

    JonPB

    Joined:
    Aug 31, 2021
    Posts:
    3
    Thanks for the quick reply!

    I tried a few things:
    - Disabled the scene change code: Same Error (Invalid Query String)
    - Tried making a new button, and added the scene change code: Test Succeeded
    - Renamed old button from "Btn-ToSceneTwo" to "ToSceneTwo" and added scene change code: Test Succeeded
    - Renamed the old button from "ToSceneTwo" to "ToSceneTwo-": Same Error
    From this, it looks like a parsing error caused by the hyphen.

    The debugger also gave me this:
    querySelector: "[name=Btn-ToSceneTwo]"
     
  41. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    Got it! You found a bug. Sorry for that. We will fix it asap. We appreciate you reporting it and debugging this for us.
     
    JonPB likes this.
  42. volkov_unity774

    volkov_unity774

    Joined:
    Aug 27, 2021
    Posts:
    2
    Hello! I ran into a couple of problems while using the Crawl option in the "Recorded Playback" section (1) and while trying to generate Full Tests (2).

    1. Crawler
    I have a project with multiple scenes in it. It seems that sometimes the crawler is trying to interact with a button from one of the previous scenes that is not present on the scene it's currently on. This results in the following console message:

    Code (CSharp):
    1. AQA: Cannot play recorded action: object Continue does not exist or is not viewable on screen. Ensure the object exists and reposition the object inside the screen space.
    2.  
    3. UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
    4. Unity.AutomatedQA.AQALogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[]) (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/AQALogger.cs:10)
    5. UnityEngine.Logger:LogError (string,object)
    6. Unity.AutomatedQA.AQALogger:LogError (object) (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/AQALogger.cs:73)
    7. UnityEngine.EventSystems.RecordingInputModule:UpdatePlay (int)
    8. Unity.AutomatedQA.GameCrawler/<PerformAction>d__56:MoveNext () (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/Automators/GameCrawler.cs:307)
    9. UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)
    2. Full Tests

    When I try to generate a Full Test from one of the recordings, a lot of errors come up in the generated script. They all seem to be caused by spaces in the scene name (e.g. a scene named "Menu 3D" will cause these errors, "Menu3D" won't).

    Error Type 1:
    Assets\AutomatedQA\GeneratedTests\PageObjects\Scene_Draggable Panel_PageObject.cs(3,34): error CS1514: { expected

    This string was generated for a scene called "Draggable Panel":
    public class Scene_Draggable Panel_PageObject { ... }

    Error Type 2:
    Assets\AutomatedQA\GeneratedTests\Tests_Crawler_Test_1.cs(20,58): error CS1003: Syntax error, ',' expected

    This is the string that the error referred to (scene name is "Menu 3D"):
    yield return Driver.Perform.Click(Scene_Menu 3D_PageObject.Clickable_Menu_3D_OpenDraggablePanelSceneButton);

    Error Type 3:
    Assets\AutomatedQA\GeneratedTests\PageObjects\Scene_Layout Groups_PageObject.cs(3,31): error CS0116: A namespace cannot directly contain members such as fields or methods

    The string is:
    public class Scene_Layout Groups_PageObject { ... }

    Error Type 4:
    Assets\AutomatedQA\GeneratedTests\PageObjects\Scene_Layout Groups_PageObject.cs(4,5): error CS1022: Type or namespace definition, or end-of-file expected

    This error points to the string with the curly brace:
    public class Scene_Layout Groups_PageObject
    {

    Please let me know if there is any additional information I could provide.
     
    tsibiski likes this.
  43. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    345
    Thanks for this detailed report. We will fix these issues for the next release.

    Here should be a quick fix for the GameCrawler issue. Let me know if it doesn't work for you. This code snippet replaces the existing Refresh methods in GameListenerHandler.cs:


    Code (CSharp):
    1.        
    2.         public static void Refresh()
    3.         {
    4.             _activeListeners = new List<AutomationListener>();
    5.             forceRefresh = true;
    6.         }
    7.         public void Refresh(Scene scene, LoadSceneMode mod)
    8.         {
    9.             Refresh();
    10.         }
    11.  
     
    Last edited: Dec 1, 2021 at 2:30 PM
    volkov_unity774 likes this.
  44. Evgeny-Eliseev

    Evgeny-Eliseev

    Joined:
    Jan 21, 2015
    Posts:
    16
    We liked Automated QA.

    We found a problem in current version.

    In our environment if you try record next scenario:

    Click Button_1
    Click Button_2


    Playback will be:

    Click Button_1
    Click Button_1


    It happens because there two gameobjects has same path
    ("objectHierarchy": "Canvas/Container").

    2021-12-02_16-04-38.png


    For our game it happens often, because we have lists of widgets, which are created through code and has different content.

    Test project for reproduce: https://yadi.sk/d/jjgMVLKc6Yhxrw
     
  45. timsibiski

    timsibiski

    Unity Technologies

    Joined:
    Jun 3, 2021
    Posts:
    1
    Hi,

    The easy fix for this would be to add GameElement scripts to those buttons, and provide unique ID's for each one.
    Code (CSharp):
    1. yield return Driver.Perform.Click("#MyButtonWithUniqueID");
    Let me know if this works for you. I made a note for use to support a hierarchy path that adds an index for identically named objects with the same parent.
     
  46. volkov_unity774

    volkov_unity774

    Joined:
    Aug 27, 2021
    Posts:
    2
    Hello again! I've tried to apply the fix you've provided, but another problem has emerged. Every time I alter the code of GameListenerHandler.cs, save it and go back to Unity, it shows a loading bar that says "Compiling C# Scripts", "Reload Script Assemblies" and when it's done, every change I've made is reverted. Is there a way to prevent this?

    Additionally, I've discovered a way this error could be reproduced without the scene change. If the interactable object is untoggled while the crawler is still running on the same scene, it can trigger the same error:

    Code (CSharp):
    1. AQA: Cannot play recorded action: object Newgame does not exist or is not viewable on screen. Ensure the object exists and reposition the object inside the screen space.
    2. UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
    3. Unity.AutomatedQA.AQALogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[]) (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/AQALogger.cs:10)
    4. UnityEngine.Logger:LogError (string,object)
    5. Unity.AutomatedQA.AQALogger:LogError (object) (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/AQALogger.cs:73)
    6. UnityEngine.EventSystems.RecordingInputModule:UpdatePlay (int)
    7. Unity.AutomatedQA.GameCrawler/<PerformAction>d__56:MoveNext () (at Library/PackageCache/com.unity.automated-testing@0.8.0-preview.1/Runtime/Automators/GameCrawler.cs:307)
    8. UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)
    Not sure if that helps, but here are the specs:
    Windows 10
    Unity 2020.3.17f1
    Automated QA 0.8.0-preview.1
     
unityunity