Search Unity

  1. We are migrating the Unity Forums to Unity Discussions by the end of July. Read our announcement for more information and let us know if you have any questions.
    Dismiss Notice
  2. Dismiss Notice

Official UTF 1.4

Discussion in 'Testing & Automation' started by Warnecke, Dec 15, 2023.

  1. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    Unity Test Framework (UTF) users,

    We have released v1.4 of UTF for Unity 2023.3 and above. This version includes user experience improvements and bug fixes. The 1.4 version of UTF will be the default for Unity 2023.3 and above.

    Many of the changes have been backported from the UTF 2.0 experiment. It is our intention to backport most of the remaining changes from 2.0 over time, including its bug fixes and performance improvements.

    What’s New
    Update in the interface

    This version includes a revised Test Runner window and several usability improvements:
    • Added a third tab to the Test Runner window, for running in a player explicitly. This makes it easier to run a subset of tests, as well as retaining the test results from the latest player run.
    • Moved the run and action buttons to the bottom of the window, to separate them from the filters.
    • The stack traces are clickable, and will open the relevant file in the external editor.
    • When searching, the parents of the matching tests are expanded, to make it easier to see the context of the matching tests.


    Other notable features added
    Ignore tests based on arguments
    • This version introduces the ParameterizedIgnoreAttribute which allows ignoring tests based on arguments which were passed to the test method of a parameterized test
    API updates
    In this version, we introduced several API changes:
    • Added API for saving results
    • Added API for canceling test runs
    • Added overloads of LogAssert.Expect which allow users to expect a log message without specifying a severity, which will match logged messages of any severity.
    Fixes
    This version includes many bug fixes and performance improvements. For a full list of changes and updates in this version, see the Unity Test Framework package changelog.

    Known Issues
    • Using OneTimeSetup and OneTimeTeardown with async methods will make the call hang.
    If you encounter unexpected issues using UTF 1.4, please file a bug from the bug reporter in the editor.

    Installation steps for versions 2023.2 and older
    • In older versions, the default version of UTF is 1.1.33, however you can still upgrade to 1.4 version by going to the package manager and upgrading it manually, by selecting the Test Framework package and installing the desired version.
    • Alternatively you can manually edit your <project>/Packages/manifest.json file and add “com.unity.test-framework”: “1.4.2” to the dependencies.
     
  2. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    7,019
    Will 1.4 still work with 2021.3?
     
  3. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    Yes. It should work with all Unity versions back to 2019.4.
     
    CodeSmile likes this.
  4. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,356
    Installing this worked! That's great, we've been stuck on 2.0.1-pre.18 for ages, as any switch to a 1.x version would cause our project to not compile due to internal compiler errors until we switched back to 2.0.1-pre.18 and deleted the library.

    Is there any entries for testing on the Roadmap page? I'm still waiting for any work put into making tests that needs to run in a specific scene not be incredibly painful.
     
  5. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    We do currently not have anything on the roadmap, other than backporting the remaining parts of 2.0 into a stable release.
     
  6. CDF

    CDF

    Joined:
    Sep 14, 2013
    Posts:
    1,324
    Coming from v2.8.1-preview

    TestRunnerApi.IsAnyRunActive() and TestRunnerApi.CancelAllTestRuns() have been removed.

    How can we know if any run is active and cancel all runs now?

    Also, RetrieveTestList no longer accepts an ExecutionSettings object?
    How can I retrieve tests for a specific category?
     
    Last edited: Dec 15, 2023
  7. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    The refactoring and improvements of the TestRunnerApi have not yet been backported from 2.0. We do not yet know when that will happen. There is indeed no way of retrieving a test tree with a specified filter.
     
  8. sandolkakos

    sandolkakos

    Joined:
    Jun 3, 2009
    Posts:
    285
    hey @Warnecke, thanks for the constant feedback here.
    I'm using the version "1.4.3", and running async Task methods on Editor Mode will hang most of the time. I have to constantly move the mouse and click on any view of the Unity Editor to get the Tests to get finished.
    Do you know if that is a known issue and if the team is working on a solution for it?

    It is exactly the issue posted by @PhantomGingerGames a while ago here:
    - https://forum.unity.com/threads/unity-test-framework-2-0-ready-for-feedback.1230126/#post-7910305

    Image from the post above:

    TestAsyncUnityTest.gif
     
    Last edited: Feb 10, 2024
  9. sandolkakos

    sandolkakos

    Joined:
    Jun 3, 2009
    Posts:
    285
    Okay, I've made a simple utility to solve that issue until it gets solved on the Unity side.
    I basically added a logic to trigger an infinity loop to execute `EditorApplication.QueuePlayerLoopUpdate()` on every tick whenever the TestRunner starts to run Editor mode tests and then break the loop when the TestRunner tests get finished. Script follows below for anyone experiencing the same issue:

    Code (CSharp):
    1. using System.Threading;
    2. using System.Threading.Tasks;
    3. using UnityEditor;
    4. using UnityEditor.TestTools.TestRunner.Api;
    5. using UnityEngine;
    6.  
    7. namespace TestRunnerUtils
    8. {
    9.     [InitializeOnLoad]
    10.     public static class TestRunnerHangingOnEditorTestsFixer
    11.     {
    12.         private static TestRunnerApi api;
    13.         private static CancellationTokenSource cancellationTokenSource;
    14.  
    15.         static TestRunnerHangingOnEditorTestsFixer()
    16.         {
    17.             if (Application.isBatchMode)
    18.             {
    19.                 // This script should not be used in batch mode.
    20.                 return;
    21.             }
    22.  
    23.             SetupListeners();
    24.         }
    25.  
    26.         private static void SetupListeners()
    27.         {
    28.             Object.DestroyImmediate(api);
    29.             api = ScriptableObject.CreateInstance<TestRunnerApi>();
    30.             api.RegisterCallbacks(new TestRunnerCallbacks());
    31.         }
    32.  
    33.         private class TestRunnerCallbacks : ICallbacks
    34.         {
    35.  
    36.             public void RunStarted(ITestAdaptor testsToRun)
    37.             {
    38.                 Debug.Log($"TestRunner in {testsToRun.TestMode} mode started");
    39.  
    40.                 if (testsToRun.TestMode != TestMode.EditMode)
    41.                 {
    42.                     return;
    43.                 }
    44.  
    45.                 cancellationTokenSource?.Cancel();
    46.                 cancellationTokenSource = new CancellationTokenSource();
    47.                 TriggerEditorUpdateLoop(cancellationTokenSource.Token);
    48.             }
    49.  
    50.             public void RunFinished(ITestResultAdaptor result)
    51.             {
    52.                 Debug.Log("TestRunner finished");
    53.                 cancellationTokenSource?.Cancel();
    54.             }
    55.  
    56.             public void TestStarted(ITestAdaptor test)
    57.             {
    58.                 // Don't need to do anything here.
    59.             }
    60.  
    61.             public void TestFinished(ITestResultAdaptor result)
    62.             {
    63.                 // Don't need to do anything here.
    64.             }
    65.  
    66.             private static void TriggerEditorUpdateLoop(CancellationToken cancellationToken = default)
    67.             {
    68.                 TriggerEditorUpdateLoopAsync(cancellationToken).ConfigureAwait(false);
    69.             }
    70.  
    71.             private static async Task TriggerEditorUpdateLoopAsync(CancellationToken cancellationToken = default)
    72.             {
    73.                 while (cancellationToken.IsCancellationRequested == false)
    74.                 {
    75.                     await Task.Yield();
    76.                     EditorApplication.QueuePlayerLoopUpdate();
    77.                 }
    78.             }
    79.         }
    80.     }
    81. }
     
    glenneroo likes this.
  10. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,137
    @Warnecke One really important feedback I would like mention is please dun backport PlayerHeartBeatTask at 2.0 which is regression that will fail my long running test. I have test needs to take a lot of time to run.
     
  11. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,066
  12. sandolkakos

    sandolkakos

    Joined:
    Jun 3, 2009
    Posts:
    285
    async Task works only with [Test].
    and there is no need to remove [UnityTest] because otherwise everyone would have to refactor their code to change IEnumerator to async Task and that is not good.
    Also, there is no reason to remove the IEnumerator option only because now we can use async Task.
     
  13. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,066
    In my case it is because my tests are async but I was using UniTask.ToCoroutine (before you could run async tests officially) and I want to remove that dependency.
     
  14. sandolkakos

    sandolkakos

    Joined:
    Jun 3, 2009
    Posts:
    285
    In some of my old projects, I was also using a task.ToCoroutine() extension I created, then I started to convert them to use async Task methods with [Test], and it turned out to be an easy refactoring.
    What would be the issue with [UnitTest], in your case, when you start changing your UniTask.ToCoroutine to async Task methods?
     
  15. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,066
    In the section: https://docs.unity3d.com/Packages/com.unity.test-framework@1.4/manual/reference-async-tests.html

    There is no mention of OneTime/Setup OneTime/TearDown attributes or the known issues of the first post of this thread:

    What is the status? It's fixed on any 1.4.x package version (In the changelog there is no mention of it)? Can these attributes be used with "async Task", or not @Warnecke? Thanks.

    Also, if I try to use the [Retry] attribute (I'm using also Test, UnityPlatform and Timeout attributes for what it's worth) with async Task I get the following error:

     
    Last edited: Mar 21, 2024