Search Unity

Unity Automated QA Package [v0.6.1]

Discussion in 'Testing & Automation' started by dylanscandinaro, Apr 8, 2021.

  1. dylanscandinaro

    dylanscandinaro

    Unity Technologies

    Joined:
    Feb 12, 2020
    Posts:
    8
    The Automated QA package enables users to record and playback touch or drag interactions with the UI of a Unity Project and optionally use recordings to drive Unity Tests - in the editor, on a local iOS or Android device, or on an iOS or Android device managed by Unity in the cloud (email us at AutomatedQA@unity3d.com for access to this feature).


    Recorded Playback UI - a module included in the Automated QA package.

    Key Features
    • Object based recording: record once in the editor and play back on many different devices and aspect ratios
    • Recorded playbacks are stored as time stamped sequences of objects identified by name and tags
    • Composite recorded playbacks: create modular recordings to minimize duplication and automatically keep recorded playbacks up to date when a component is updated
    • Automatic recorded test generation: create Unity Test Framework Tests from recorded playbacks with the click of one button
    • Run tests from your CI pipeline: recorded tests can be run on your build machine, like any other Unity Test, or use the cloud on-device testing feature to run tests on our cloud Android device farm invoked from your build pipeline.


    Recorded Playback Creation -> Unity Test Execution

    Limitations
    • This is an experimental package with volatile features. New versions may include breaking changes.
    • Unity 2019.4 or above required
    • Recording is currently limited to UI elements implemented with Unity UI

    Installation
    1. With your project open in the Unity Editor, open the Package Manager (Window > Package Manager).
    2. Press the plus button (+) in the top left of Package Manager and then select "Add package from git URL...".
    3. Enter com.unity.automated-testing in the text box and then press "Add".
    See our package documentation for getting started instructions and more information. Please email us at AutomatedTesting@unity3d.com with any questions!

    Changelog
    [0.6.1] - 2021-08-12
    Major Features and Improvements

    • New GameElement and ElementQuery classes for reliably finding GameObjects, no matter where they are moved in a scene.
    • Uses Xpath & JQuery-like query string selectors for identifying GameObjects.
    • New Driver class that allows for much simpler ways to write custom code and perform actions on objects (ex: Driver.Perform.Click("#SubmitButton")).
    • Added GameCrawler that allows users to create an AutomatedRun that plays randomly through a game, recording warnings, errors, and notifying users when and where it gets stuck.
    Bug Fixes and Minor Changes
    • Reverting change that removed the ability to generate "Simple Tests", which is a test that points to a recording file.
    • Fixed an issue with temp directories not being properly deleted.
    • Adding FPS tracking to reports. FPS sampled over duration of test execution is displayed in a graph accessible from the html report (FPS Data button).
    • Fixed issue allowing objects off screen to be interacted with.
    • Add support for recording right and middle mouse clicks.
    • Added functionality to allow both recording and playback while editor is in play mode.
    • Both record & playback can be done repeatedly in the same session.
    • Editor does not stop playmode on completion, but both record & playback can still be started outside of playmode.
    • Added support for running cloud tests from the command line using BuildAndRunTests.
    • Fixed issue where tests may timeout during command line batch mode runs.
    • Fixed issue with the InputField text listener not being added right away.
    • Screenshot timing after each playback action completes is now configurable in settings.
    • Exposed "Quit on finish" in Automated Run Config.
    • Added ability to generate a "Full" test from Automated Runs.
    • Added settings for log level
    • Added support for TextMeshPro TMP_InputFields and TMP_Text fields
    • Can record and playback input into TMP_InputFields, and find objects in ElementQuery by TMP_InputField & TMP_Text text content.

    Breaking Changes
    • Several APIs were moved from the test base class to Driver.cs. "Full" generated tests from v0.5.0 will need their PerformAction and RegisterStep method invocations updated.
    • The experimental Composite Recordings window has been removed. Instead, Automated Runs can be used to playback multiple recordings in sequence.

    [0.5.0] - 2021-06-28
    Major Features and Improvements
    • Support for running tests on real iOS devices in the cloud
    • New [CloudTest] attribute can be used to specify any Unity Test Framework test to run on real devices in the cloud
    • Automated QA > Test Generation... allows for step-by-step Unity Test Framework test generation (C# code).
      • Generates code for every step/action taken in a recording.
      • Allows for assertions or additional custom logic between each step.
      • Can select which recordings to generate tests for.
      • Editor Window warns user if about to overwrite custom edits in a recording's test when re-generating test.
    Breaking changes
    • Recording file in RecordedPlaybackAutomator is now an asset reference instead of path string (Note: Only relevant if you're using the experimental Automated Runs feature)
      • Migration: Update the file path in AutomatedRuns using RecordedPlaybackAutomator
    Bug Fixes and Minor Changes
    • Added dynamic wait logic so that differences in load times or animations does not result in automation failing to interact with the target object.
    • Delayed loading of the recorded playback controller to avoid issues with initializing too early.
    [0.4.0] - 2021-05-25
    Major Features and Improvements

    • HTML & XML reports for AutomatedQA runs.
      • XML report is designed for loading tests results into a CI process run.
      • Latest HTML reportcan be opened in editor from the "Recorded Playback" and "Composite Recordings" editor windows after playback.
      • Both currently stored in Application.persistentDataPath. CI/CD must copy report from path (Will be integrated with Cloud services & modified to take raw test data from devices so that explicit file extraction is not necessary).
    • Added settings file for editable values used in Automated Qa package, thus making various Automated QA behaviors customizable (more to be added as time goes on).
      • Added new editor window to edit the settings file (Automated QA > Settings).
      • Multiple config files with varying values can be stored in Assets/AutomatedQA/Resources, and desired settings can be requested from cloud config via file name.
    Bug Fixes and Minor Changes
    • Added placeholder segment file names in Composite Recording window when selecting "Save Segment".
    • Updated VisualFx to prevent visual clutter from a rapid series of events.
    • Updated validation of GameObject that is about to be clicked. Increased performance. Now checks that an object is off screen or under another object that would intercept the click.
    • Updated recorded tests to load an empty scene after execution.
    • Fixed ignoring depth surface warnings when taking screenshots on Mac.
    • Added new Automators: Scene Automator and Text Input Automator.
    • Fixed event timings and scene loading with composite recordings
    [0.3.1] - 2021-05-04
    Major Features and Improvements

    • New AutomatedRun object to link together recordings and custom C# scripts to automate gameplay.
      • Create an AutomatedRun with Create -> Automated QA -> Automated Run
    • New Automator class. Extend this class to create custom automators
      • Extend the AutomatorConfig class for your Automator to expose it in the AutomatedRun inspector.
    Bug Fixes and Minor Changes
    • Fixed an issue where drop events had extra delay
    • Fixed an issue with the Upload Recording window requiring entitlements
    • Removed dependency on com.unity.nuget.newtonsoft-json package
    • Fixed entry scene to work with initialization scenes
    • Moved menu items to top level Automated QA menu
    • Added fix to CloudTestResults.cs that prevents ITestRunCallback from being eagerly stripped by the "ahead of time" compilation in IL2CPP builds.
    • Added logic to cleanup temporary files accumulating from previous recordings and stored in the persistent data path.
    • Removed Linq usage from package as Linq can be a heavy library for mobile game development.
    [0.3.0] - 2021-04-21
    Breaking changes

    • Simplified assembly definitions
      • Migration: Please update asmdefs to reference Unity.AutomatedQA
    • Renamed asset directory to AutomatedQA
      • Migration: Please delete the old AutomatedTesting folder
    Bug Fixes and Minor Changes
    • Package directory restructure
    • Added SettingsManager package and AutomatedQAEditorSettings/AutomatedQARuntimeSettings to wrap package settings
    • Disabled Cloud Testing window for unsupported platforms
    • Fixed an issue with recorded tests using composite recordings
    • Fixed an issue causing two simultaneously active EventSystem components, resulting in a flood of console warnings
    • Fixed an issue where multi-clicking "Record" or "Play" buttons creates multiple RecordedPlaybackController GameObjects
     
    Last edited: Aug 12, 2021
  2. chad-wmgi

    chad-wmgi

    Joined:
    Mar 31, 2020
    Posts:
    8
    Is the new Input System supported? I'm getting spammed with an error about Input.mousePosition.
     
  3. dylanscandinaro

    dylanscandinaro

    Unity Technologies

    Joined:
    Feb 12, 2020
    Posts:
    8
    As long as the elements you'd like to interact with are implemented with Unity UI, then yes with an additional step:

    With your Unity Project open, from the menu bar navigate to Edit > Project Settings > Player > Other Settings > Configuration, then set Active Input Handling to “Both”.
     
    Tamgros likes this.
  4. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    An error occurred in the "Recorded Test".
    After running "Generate Recorded Tests", select "GeneratedTests.dll" and run it. Then the following error appeared in the console.

    Code (CSharp):
    1. FileNotFoundException: Could not find file "/Users/***/Library/Application Support/DefaultCompany/UnityTestFramework/config.json"
    2. System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) (at <fb001e01371b4adca20013e0ac763896>:0)
    3. (snip)
    4. System.IO.File.ReadAllText (System.String path) (at <fb001e01371b4adca20013e0ac763896>:0)
    5. Unity.RecordedTesting.Manager.GetDeviceFarmConfig (Unity.RecordedTesting.DeviceFarmOverrides dfOverrides) (at Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/Manager.cs:31)
    6. Unity.RecordedTesting.TestResults.RunStarted (NUnit.Framework.Interfaces.ITest testsToRun) (at Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/TestTools/TestResults.cs:33)
    7. UnityEngine.TestRunner.Utils.TestRunCallbackListener+<>c__DisplayClass1_0.<RunStarted>b__0 (UnityEngine.TestRunner.ITestRunCallback callback) (at Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/Utils/TestRunCallbackListener.cs:16)
    8. UnityEngine.TestRunner.Utils.TestRunCallbackListener.InvokeAllCallbacks (System.Action`1[T] invoker) (at Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/Utils/TestRunCallbackListener.cs:38)
    9. UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr) (at /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/Coroutines.cs:17)
    There is no config.json in that directory, but there is config_recording.json and playback_recording.json.

    environment:
    • macOS Big Sur (11.2.3)
    • Unity 2019.4.11f1
    • Automated QA 0.2.0-preview.3
    • Test Framework 1.1.24
     
  5. souranil_sen

    souranil_sen

    Unity Technologies

    Joined:
    Mar 19, 2020
    Posts:
    3
    Thanks for the stacktrace, we are investigating this. The SDK shouldn't look for that file while running locally. Can I you please check if there are any scripting defines in "Player Settings" ?
     

    Attached Files:

  6. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    Thanks for your response!
    I found `CLOUD_TEST_UTF` symbol in Scripting Define Symbols. After removing this, the test will now run.

    But, every test failed with the following exception.

    Code (CSharp):
    1. CanPlayToEnd (3.571s)
    2. ---
    3. SetUp : System.Reflection.ReflectionTypeLoadException : Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
    4. ---
    5. --SetUp
    6.   at (wrapper managed-to-native) System.Reflection.Assembly.GetTypes(System.Reflection.Assembly,bool)
    7.   at System.Reflection.Assembly.GetTypes () [0x00000] in <fb001e01371b4adca20013e0ac763896>:0
    8.   at Unity.RecordedTesting.RecordedTesting.GetAllRecordedTests () [0x00023] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:55
    9.   at Unity.RecordedTesting.RecordedTesting.GetLocalRecordingFile (System.String testName) [0x00002] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:36
    10.   at Unity.RecordedTesting.RecordedTesting.SetupRecordedTest (System.String testName) [0x00008] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/RecordedTesting.cs:115
    11.   at Unity.RecordedTesting.TestTools.RecordedTestSuite+<Setup>d__0.MoveNext () [0x00027] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.automated-testing@0.2.0-preview.3/RecordedTesting/Runtime/TestTools/RecordedTestSuite.cs:23
    12.   at UnityEngine.TestTools.BeforeAfterTestCommandBase`1+<ExecuteEnumerable>d__12[T].MoveNext () [0x00185] in /Users/****/Documents/UnityTestWorkspace/Library/PackageCache/com.unity.test-framework@1.1.24/UnityEngine.TestRunner/NUnitExtensions/Commands/BeforeAfterTestCommandBase.cs:87
     
  7. marcelohao

    marcelohao

    Joined:
    May 27, 2014
    Posts:
    1
    Great initiative!

    After I click on "Record", I get a bunch of entries like this one on the log:

    There are 2 event systems in the scene. Please ensure there is always exactly one event system in the scene
    UnityEngine.EventSystems.EventSystem:Update () (at C:/Program Files/Unity/Hub/Editor/2020.3.5f1/Editor/Data/Resources/PackageManager/BuiltInPackages/com.unity.ugui/Runtime/EventSystem/EventSystem.cs:398)

    When I click on "Stop", the json file only has mouse inputs logged, keyboard and gamepad are not getting logged (are those not supported yet?)

    Thanks!
     
  8. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    The "There are 2 event systems in the scene" bug was fixed in the most recent release of the package (0.3.0).

    The package is not yet recording keyboard/gamepad inputs. Both the old and new input systems need to be handled seamlessly depending on the package customers are using, and keyboard/gamepad inputs will require that. So the handling of those types of inputs is important and is on the horizon.

    Right now, clicks and drags are recorded within the Canvas space. However, it will not work yet with GameObjects not on the Canvas (ex: world objects using colliders to perform actions similar to clicks and drags; this functionality is also in the queue).
     
    Last edited: Apr 29, 2021
  9. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    I found the cause.

    `System.Reflection.Assembly.GetTypes` seems to throw an exception if the assembly `Microsoft.CodeAnalysis.Scripting`.

    In my project, `com.unity.immediate-window` depended on this assembly, so I removed it and was able to run the test!
     
    tsibiski likes this.
  10. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    BNS_oyama likes this.
  11. SchmidtDavidSimon

    SchmidtDavidSimon

    Joined:
    Sep 7, 2020
    Posts:
    10
    Hi I also have a question to this toppic. Is it possible to check if a scene was created by a recorded playback? We have our DI set in a way it initializes systems only when our game-scene is starting. Thats why tests fail since a new scene is created for the tests start.
     
  12. SchmidtDavidSimon

    SchmidtDavidSimon

    Joined:
    Sep 7, 2020
    Posts:
    10
    And already the next question :D

    Is it possible to create Tests out of Automated Runs? If so how would I go about it? If not I think it would be a very nice feature!
     
  13. skazmierczak

    skazmierczak

    Joined:
    Feb 28, 2020
    Posts:
    4
    Can this be used with the code coverage package?
     
  14. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    In v0.4.0:
    • I'm not want to put AutomatedQASettings.json on "Resources" folder
    • Warning mesages in the console. because unnecessary .meta files, follows:
      • RecordedTesting/Editor.meta
      • RecordedTesting/Runtime/TestTools.meta

    In v0.5.0
    • Warnings in the console
      • There are inconsistent line endings in the 'Assets/AutomatedQA/GeneratedTests/Steps/xxx_Steps.cs' script. Some are Mac OS X (UNIX) and some are Windows.
    • Tests fail in batch mode. (same tests are successful in Unity Editor GUI)
    • I am very happy that the "dynamic wait logic" makes the execution more stable. But, I miss the ease of replacing the JSON file by just recording it again.
     
    Last edited: Jun 30, 2021
  15. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    For the first point, are you able to add it to a git ignore? It needs to be in resources for use in the Cloud and binary builds. If the problem is checking it in, it can be added to git ignore, or a similar feature in other version controls.

    For the final issue, you can tell the framework to NOT use dynamic waits by setting the "Use Dynamic Waits" boolean in the Settings editor window. It will then revert to the old way of running tests, and you can continue to re-record failing tests.
     
  16. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    11
    Thanks. I'm not talking about running tests.

    My scenario as follows:
    1. Recording tests to json
    2. Automated QA | Test Generation
    3. Add asserts in test code
    4. One day, UI operation and/or transitions will be changed!
    5. Re-Recording tests to json
    6. Use the test code without changing it

    Until v0.4, this was possible.
    But, in v0.5, Re-Test Generation is required.

    This behavior is acceptable for us programmers, but I think it is cumbersome for test engineers and level designers to use Automated QA.

    Given that UI changes are frequent, such as in game tutorials, I hope have preferred to keep the advantage of being able to replace only json.
     
  17. BNS_oyama

    BNS_oyama

    Joined:
    Jun 1, 2020
    Posts:
    1
    I have one request for the features of Automated QA. Please doesn't overwrite this file, if "Assets/AutomatedQA/GeneratedTests/GeneratedTests.asmdef" already exists when user runs Test Generation. If a user creates test codes in "Assets/AutomatedQA/GeneratedTests", they will add elements to Assembly Definition References in this .asmdef file. But because this .asmdef file is initialized when they creates a new test, they needs to reconfigure this .asmdef file.
     
    nowsprinting likes this.
  18. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    Thanks for this suggestion!
     
    BNS_oyama likes this.
  19. bkeeley_unity

    bkeeley_unity

    Joined:
    Jun 14, 2021
    Posts:
    1
    I've been having issues trying to get either version 0.3.1 or 0.5.0 working in our project.

    With v0.5.0 specifically, whenever I try to run an Automated Run using a recording added with the RecordedPlaybackAutomatedConfig, I get the following errors and stack traces:

    Code (CSharp):
    1. BadImageFormatException: VAR 1 (TResult) cannot be expanded in this context with 1 instantiations
    2. System.Reflection.MonoMethod.GetBaseMethod () (at <695d1cc93cca45069c528c15c9fdd749>:0)
    3. System.MonoCustomAttrs.GetBase (System.Reflection.ICustomAttributeProvider obj) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    4. System.MonoCustomAttrs.GetCustomAttributes (System.Reflection.ICustomAttributeProvider obj, System.Type attributeType, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    5. System.Reflection.MonoMethod.GetCustomAttributes (System.Type attributeType, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    6. System.Attribute.GetCustomAttributes (System.Reflection.MemberInfo element, System.Boolean inherit) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    7. System.Attribute.GetCustomAttributes (System.Reflection.MemberInfo element) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    8. System.Reflection.CustomAttributeExtensions.GetCustomAttributes (System.Reflection.MemberInfo element) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    9. Unity.RecordedTesting.RecordedTesting.GetAllRecordedTests () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Automators/RecordedPlayback/RecordedTesting/RecordedTesting.cs:73)
    10. Unity.RecordedTesting.RecordedTesting.IsRecordedTest (System.String testName) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Automators/RecordedPlayback/RecordedTesting/RecordedTesting.cs:50)
    11. ReportingManager.InitializeDataForNewTest () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:242)
    12. ReportingManager.RecordLog (System.String message, System.String stackTrace, UnityEngine.LogType type) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:352)
    13. UnityEngine.Application.CallLogCallback (System.String logString, System.String stackTrace, UnityEngine.LogType type, System.Boolean invokedOnMainThread) (at <c7864a0eaeb24b2a999fb177623d54b4>:0)
    Code (CSharp):
    1. UnityException: List provided to AutomatedQATools.Last() was empty. Cannot invoke Last() on an empty list. Check for list being empty before invoking Last().
    2. Unity.AutomatedQA.AutomatedQaTools.Last[T] (System.Collections.Generic.List`1[T] list) (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/AutomatedQaTools.cs:146)
    3. ReportingManager.FinalizeReport () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingManager.cs:174)
    4. ReportingMonitor.OnApplicationQuit () (at Library/PackageCache/com.unity.automated-testing@0.5.0-preview.1/Runtime/Core/Reporting/ReportingMonitor.cs:53)
    In v3.0.1 I also get a similar
     BadImageFormatException
    whenever I run a generated test from a recording in the Test Runner

    Windows 10
    Unity 2020.3.7f1
    Test Framework 1.1.24
    Automated QA 0.5.0-preview.1
    Automated QA 0.3.1-preview.2
     
    Last edited: Jul 15, 2021
  20. Regretful123

    Regretful123

    Joined:
    Nov 5, 2014
    Posts:
    9
    I am having some weird and unorthodox error where if I record step by step using the Record function, it works. However generating the script out to full test breaks.

    I think the culprit is that we have MaterialUI, which still uses Unity UI under the hood of the library. However, it seems like it cannot find the gameobject the QA is trying to find (Since it's moving). The JSON file works perfectly, but for whatever reason the full scripted tests (even with changes in deltaTime) it still errors out complaining that it cannot find the button it was target to click or release from.

    Can someone help me on what I can do from here? I'm kinda stuck with this Dialog issue. Here's the full test error issue:

    Code (CSharp):
    1. AQA: Click position recorded relative to spot within the target GameObject "Button - Affirmative" is positioned outside of the camera frustum (is not visible to the camera). Since this is a GameObject in the UI layer, this may mean that the object has not scaled or positioned properly in the current aspect ratio (865w X 487h) and current resolution (1920 X 1080) compared to the recorded aspect ratio (865h X 487w) and recorded resolution (1920 X 1080).
    2. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    3. Unity.AutomatedQA.AQALogHandler:LogFormat(LogType, Object, String, Object[]) (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AQALogger.cs:10)
    4. UnityEngine.Logger:LogError(String, Object)
    5. Unity.AutomatedQA.AQALogger:LogError(Object) (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AQALogger.cs:73)
    6. UnityEngine.EventSystems.RecordingInputModule:UpdatePlay(Int32)
    7. Unity.AutomatedQA.<Action>d__24:MoveNext() (at Library/PackageCache/com.unity.automated-testing@0.6.1-preview.2/Runtime/Core/AdvancedQuerying/Driver.cs:292)
    8. UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
    9.  
    Win 10 pro
    Automated QA preview.2 - 0.6.1
    Test Framework - 1.1.29
    Unity 2019.4.9f1 - Android configuration
     
  21. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    Hi there. I actually wrote the logic involving the checks around an object being outside the camera frustum; so hopefully I can help you here.

    Is the object still visible on the screen then, despite this error? Can you retrieve the calculated point of click via debug (search for the error message above and put a breakpoint). Is it calculating the position of the object to be outside the bounds of the viewport? It should indeed be outside the expected viewport from a raw x/y position - but if the object is visible to your eyes, something weird must be throwing this calculation off.

    Also, is your generated test simply referencing the json playback file, or is each step of the json written into C# (aka, the "Full Test" was generated)? If it's a "Full Test", can you compare the data stored in the associated TouchData object in this test's "Step" file with the raw json object for the step?
     
  22. Regretful123

    Regretful123

    Joined:
    Nov 5, 2014
    Posts:
    9
    Hi tsibiski,
    Yes the object is still visible on the screen. How do I go about calculating the vector2 position of where it will be clicked?
    Currently the UI was set to be outside as "Hidden", but appear out when "activated". It seems that whatever click event calculated, seems to be clicking on the object's original position instead of the new current position. This position appears to be outside of the camera frustum.

    To answer your question more precise, I can describe the step-by-step of what I did to reach to the problem:
    1) Record, wait until the UI "appears"
    2) Re-play, make sure that it doesn't break anything else other than the intended step
    3) Tweak deltaTime to speed up the click process (I went into Json file, and manually edit the timeDelta to either 0.001 or 0.5 depends on how long it takes for the UI to appear before I can click next.
    4) Re-play it again once more, but with the new timeDelta to ensure nothing else breaks
    5) Convert Recordings into Full Test
    6) Re-run it again using the Test-Runner -> Breaks.

    Perhaps, if it's possible, to click on the UI at "Just-in time" instead of predetermined position of where to click? I can see in the code that it has the object's name and where it's located in the scene hierarchy, so wouldn't it be much safer to obtain the gameobject at that instance of time, and use the transformation for click position?
     
  23. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    I sent a private conversation to test out a possible fix.
     
    Regretful123 likes this.
  24. Dasp

    Dasp

    Joined:
    Sep 25, 2012
    Posts:
    7
    I'm getting this after importing into our project. We specifically disabled non 2D physics from the project. Would be great if you would add support to enabled/disabled modules.

    Code (CSharp):
    1. Library\PackageCache\com.unity.automated-testing@0.6.1-preview.2\Runtime\Core\GameListenerHandler.cs(189,48): error CS1069: The type name 'Collider' could not be found in the namespace 'UnityEngine'. This type has been forwarded to assembly 'UnityEngine.PhysicsModule, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' Enable the built in package 'Physics' in the Package Manager window to fix this error.
     
  25. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    We will add a fix for this. Meanwhile, that code can be commented out or removed as it is not fully activated yet.


    Code (CSharp):
    1.                                 if (scriptName == "Collider" || typeof(Collider).IsAssignableFrom(components[co].GetType()))
    2.                                 {
    3.                                     if (AllActiveAndInteractableGameObjects[x].GetComponent<Collider>().isTrigger)
    4.                                     {
    5.                                         AddListener(AllActiveAndInteractableGameObjects[x], ActableTypes.Clickable);
    6.                                         continue;
    7.                                     }
    8.                                 }
     
  26. Dasp

    Dasp

    Joined:
    Sep 25, 2012
    Posts:
    7
    Thanks, I can wait for update. We would like to use that in combination with Unity Cloud Build so not in a hurry.
     
  27. IS_Twyker

    IS_Twyker

    Joined:
    Sep 6, 2021
    Posts:
    6
  28. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    It has not been tailored to XR, and thus is not tested yet for XR apps. But it technically works for any type of game. It is currently best tailored to UI-based games, and testing of UI systems in games. It has limited, but ever increasing, support for other aspects of Unity apps like key-presses. The package is growing fast and the intent is to support everything. But to summarize, the only thing that is explicitly supported at the moment is UI system testing (through any game type).
     
  29. IS_Twyker

    IS_Twyker

    Joined:
    Sep 6, 2021
    Posts:
    6
    So if I press a button on a VR controller, it doesn't record it for instance?
     
  30. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    We are actively working to support controller input (coming soon). But if you record the tests on a desktop/laptop, and then play them on VR, they should work using non-controller based inputs. If your VR app is not playable outside of a VR because it only uses controller API's from the old input system, then record and playback will not work. Also, the new input system is not yet supported.
     
  31. JonPB

    JonPB

    Joined:
    Aug 31, 2021
    Posts:
    1
    Hi, I'm doing some exploratory testing on this package and the features it provides are great. I found and recorded some strange behavior and would like to know more about it. In my test project, if you revisit the scene where the test begins, it will forget earlier inputs.



    In my project, I have three scenes: Scene1, Scene2, and Scene3:
    • Scene1 has a toggle box and a button that leads to Scene2.
    • Scene2 has a slider and two buttons that lead to Scene1 and Scene3 respectively.
    • Scene3 has input text and a button that leads back to Scene2.

    In this project, if you start your recording in a scene, do stuff, leave the scene, then return to that scene, it will forget what happened earlier.


    In the video, my inputs were:
    1. (Scene 1) - I toggled the box 8 times and went to Scene 2
    2. (Scene 2) - I slid the slider's knob across and went to Scene 3
    3. (Scene 3) - I typed "test 1" and went to Scene 2
    4. (Scene 2) - I slid the slider's knob across and back and returned to Scene 1
    5. (Scene 1) - I missed the toggle, clicked the toggle once, and went to Scene 2
    6. (Scene 2) - I did nothing and went to Scene 3
    7. (Scene 3) - I typed "test 3" and ended the recording

    After, I played the recording. I was expecting it to start from step 1, but it started at step 5.


    An interesting warning pops up when you revisit the scene:
     
  32. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    337
    We will look into this. Thanks for reporting it!
     
  33. Daybreaker32

    Daybreaker32

    Joined:
    Jun 11, 2014
    Posts:
    73
    New question, is there any way to modify generated test classes so that they don't need the GeneratedTests.asmdef ?
     
  34. Daybreaker32

    Daybreaker32

    Joined:
    Jun 11, 2014
    Posts:
    73
    Okay, a better question (?), how can the main assembly refer GeneratedTests.asmdef ?

    Small context, we can make the project refer to generated test assembly which can be used to create a wrapper runner to run generated tests, right?

    but
    using GeneratedAutomationTests;
    gives this error
    error CS0246: The type or namespace name 'GeneratedAutomationTests' could not be found
     
unityunity