Official Unity Test Framework 2.0 ready for feedback

OK then I am very sorry, I didn't see what part you have specifically replied to. Thank you for your suggestion

 

Have you tried this package? About Code Coverage | Code Coverage | 1.1.1 (unity3d.com)

 

That's an interesting package, first I've seen it. But code coverage is not the same as test result reporting.

 

Hey, thanks for raising awareness, however yes I am using that.
Code Coverage reporting is great, however this is not unit test reporting (JUnit). Code Coverage reporting I have also transformed to the Cobertura format to allow for Code Coverage Visualization in Merge Requests in Gitlab.

Code (CSharp):

1. dotnet tool install -g dotnet-reportgenerator-globaltool
2. reportgenerator "-reports:coverage.xml" -reporttypes:Cobertura "-targetdir:report"
3. sed -i "s|<UNITY_DIRECTORY>/||g" report/Cobertura.xml

The above will convert the coverage report to cobertura, I'll leave as an exercise for the reader to place in their own "coverage.xml" files that are produced and then using 'sed' to convert file paths so they can be matched up in gitlab.

 

It looks like

TestContext.CurrentContext.Random

from NUnit is null when running tests with Unity Test Framework (because

UnityTestExecutionContext.RandomGenerator

does not get set, I guess?).

Is there any intention to support that in the future, or otherwise is there a recommended approach to getting e.g. random strings for use in test cases?

 

What about an option for a timeout or duration parameter in the Assert methods e.g. Assert.IsTrue(expected, message, Timeout.timeout(1000)) which would wait up to 1 second for 'expected' to be true? It could save a lot of while() loops inside IEnumerator methods waiting for a status to be true before performing the assertion.

AssertJ for Java has this functionality and it's a huge time-saver for component/system tests which may run slower while running on a VM (or in this case via the Editor).

 

What is the type of

expected

here? You can't really do that with plain values, as arguments get evaluated before they are passed to a method. The only timeouts I could find in the link you shared were related to future completion, so if you are working with

Task

s you might want to check out this blog post.

The other option is to use "getter lambdas" (of type

Func<T>

), e.g.

() => expected

. This is how waits are implemented in Responsible, sounds like you might want to check it out.

 

I updated this package, and I'm pleased to report that my test suite worked without a hitch. No need to change my code, and no errors (besides those already caused by my project). Well done!

I would appreciate if the

internal

classes that comprise the test runner itself were made public so I didn't have to screw around with reflection. Ditto for the Editor-exclusive test runner classes. The reason for that is I have a custom test runner for my project, with its own build process.

Also, I have a question about

async

tests. Must they specifically return System.Threading.Task, or can they return any

await

able type? I ask because I'd like to use UniTask so I can take advantage of its Unity-specific optimizations.

 

There's a typo here, should be: "com.unity.test-framework": "2.0.1-exp.1"

Also, little confused on versions. Is 2.0.1-exp.1 the latest? or is 2.0.1-pre.18?



Package manager says 2.0.1-pre.18 is latest, but was updated Jan 24th 2022.
2.0.1-exp.1 was updated April 22, 2022, but is not the latest?

Very confusing.

If I want to continue using Test Framework 2.0 on Unity 2021.3.0. What version should I use?

 

I've noticed a strange (buggy?) behavior with the Timeout attribute. When the test times out the coroutine is not stopped. Instead, the yield instructions merely return without waiting and the coroutine code runs until its end.

Not only is this unexpected, but it also causes code like this to break:

Code (CSharp):

1. while (!myTestSuccessCondition)
2.     yield return null;

If the test times out before the condition is met then the loop becomes an infinite loop that causes the Unity editor to freeze.

 

Can you tell us more about these unforeseen technical challenges? I'm curious.

 

@thewerku I tested the preview package version with Unity Cloud Builds and it blocks the completion of the build, something runs endlessly.

 

Not sure if this addresses the question, but I've had success using an async test function returning System.Threading.Task and then awaiting UniTask values within it.

 

I'm working on an enterprise (meaning B2B, so not a game) project using the MRTK and XR with over 100 scene tests (and a minor amount of unit tests) and have some feedback that may be useful.

Native async/await (TAP) for scene tests
Native async/await support is something I had been waiting for, since we've embraced .NET's TAP along with multi-threading for code that does not need to run on the main thread due to Unity constraints, meaning most non-Unity-specific .NET code such as HttpClient, File IO, and others. Unfortunately, if I understand correctly, this feature does not currently extend to Unity scene tests, but only works with unit tests?

A pain point we currently feel in Unity is that scene tests only support coroutines, and things such as async HTTP requests running in the background during scene tests cannot be awaited, which sometimes needs to happen in between parts of other test logic. A workaround we currently use is an enumerator function allowing us to

yield return WaitForTask(someTask)

- it would be easier if we could

await someTask

instead.

Since scene tests currently need to return IEnumerable, this might require supporting asynchronous generators in C# to allow combining them with async, since LoadScene still uses coroutines, and the reverse of awaiting an enumerator is non-intuitive since they are producers that can theoretically produce infinitely (they solve a different problem, i.e. "how many values would you await").

Automatic timeout of tests
As was already touched on above, when you start working with tasks in tests, you risk things taking too long, potentially infinitely long. (Async) tests should time out after an interval that can be configured, but is also applied by default, to avoid having to configure this everywhere (e.g. what Jest does), and is set to a sane default (e.g. 10 or 15 seconds, not counting the time it takes for Unity to bootstrap them, since that time is often substantial, see below).

Currently we sometimes work around this by using the WaitForTask above combined with a timeout.

Scene test performance
I was personally excited to try 2.x since it mentions several performance improvements, but I think those must have been primarily for unit tests, since our test suite with mostly scene tests performs noticeably worse in 2.x. These aren't exact numbers, but where it took around a minute or two before, it has gone up to sometimes over 10 minutes. It seems to be moving from test to test more slowly than before, perhaps because more state is being restored.

More generally, it would be neat if the performance of bootstrapping scene tests could be improved. As alluded to above, oftentimes the first part of a scene test is to actually load a scene. Most of our scene tests load the same scene, so perhaps being able to configure that scene for Unity through attributes or something might allow Unity to more efficiently prepare/reset/copy/bundle tests for the same scene.

Alternatively or additionally, running (scene and/or unit) tests or test suites fully in parallel should also improve performance, since they - at least in our case - always start from a clean slate (meaning loading a fresh scene). Without this, 15 of the 16 cores in my machine are sitting there unused in most cases. Jest only runs test suites in parallel, for example, but it would be even better if each test could run separately (but even just having putting each test class in a single unit and parallelizing between them could yield a large improvement).

Finally, I've got a feeling this also ties into the story with CI/CD. As mentioned above it's possible to run Unity in containers unofficially (and by the way, it's secretly shifted back from In Progress to Under Consideration on the roadmap), but scene tests are slow on cloud runners such as the ones used by GitLab. For us, this means over 50 seconds of waiting for each scene test to start due to deserialization. Using non-cloud runners improves the situation, but costs extra for us and it's still pretty bad (20 seconds for each scene test with an Azure Debian VM).

I realize CI/CD and containers are not directly related to this package, but in comparison with other standard CI/CD flows in the Node.JS or .NET world, waiting for an hour in CI for some 100 scene tests to run is demotivating, to the point that we've just disabled CI for Unity projects and run tests locally again, losing the benefit of CI and often sneaking in broken code because a developer forgot to run the tests (or "was sure nothing would break", and "didn't want to have to wait 10 minutes again").

Improvements for service containers
Since we're building enterprise applications, we use dependency injection to facilitate inversion of control and often work with non-MonoBehaviour code as well. We currently do this using Zenject (mostly due to not having much choice, as it has to fit into Unity's workflow using MonoBehaviours, which most standard .NET frameworks don't), and it has some convenience functions for working with the container in scenes. It would be nice if there was better support out of the box for dependency injection containers. For example, as you can see from the links, it is for some reason still necessary to clean up "leftovers" from previous tests for Zenject, even though Zenject just works by having a container as component on a Game Object in the scene, and each test reloads the scene and should be running in isolation.

This is also part of a larger problem where, because Unity does not support dependency injection or proper service containers out of the box, code is often static or uses singletons (I'm looking at you, Unity localization package), and it somehow maintains state between scene tests even if the entire scene is reloaded, as the objects are not destroyed in between them (at least this seemed to be the case in 1.x).

(Since Unity does it, 3rd party projects also usually just use singletons and static code, and this type of static code proliferates through the entire Unity ecosystem unless care is taken by the author. Ideally Unity would support proper dependency injection for MonoBehaviours and a service container out of the box, but this is not directly on-topic and I understand this is not necessary for all types of projects.)

 

I second all of this. We are also doing business applications with the same packages/tools.

 

Support for await/async for testing in this release works fine, but non-Unity SetUp function with async/await doesn't seem to be supported/or just doesn't work well !? Ideally, we could use await in UnitySetUp, so that a one-time setup before tests with asynchronous function can be supported. Non-Unity SetUp with this may be another solution.

 

Very excited about being able to write async tests.

Very sad about the Timeout or MaxTime attributes not working.

 

I need to call an async method from OneTimeSetUp, but this makes Unity hang (using version 2020.3.24). For example, this breaks Unity:

Code (CSharp):

1. [OneTimeSetUp]
2. public async Task OneTimeSetupAsync()
3. {
4.     await Task.Delay(1);
5. }

Is this not supported in Unity? Because according to NUnit documentation it is valid.

 

Hello, I am trying to update my test package to 2.0.1-exp.1 but can't. Using Unity 2021.3.8f1.
First I tried the change line approach.

I tried to modify Packages/Test Framework/package.json but nothing happened, the json will restore my changes back.
1) I added the line it directly under

"upmCi": {

    "footprint": "2279493b9acd323ab758f64369634dc0daf0dbac"

  },



"com.unity.test-framework": "2.0.1-exp.1"

2) I replaced "com.unity.test-framework.tests": "1.1.33" to "com.unity.test-framework": "2.0.1-exp.1",
3) replaced "com.unity.test-framework.tests": "1.1.33" to "com.unity.test-framework.tests": "2.0.1-exp.1"
I don't know which line to change, here is my package.json

Code (JavaScript):

1. {
2.   "name": "com.unity.test-framework",
3.   "displayName": "Test Framework",
4.   "version": "1.1.33",
5.   "unity": "2019.2",
6.   "unityRelease": "0a10",
7.   "description": "Test framework for running Edit mode and Play mode tests in Unity.",
8.   "keywords": [
9.     "Test",
10.     "TestFramework"
11.   ],
12.   "category": "Unity Test Framework",
13.   "repository": {
14.     "url": "https://github.com/Unity-Technologies/com.unity.test-framework.git",
15.     "type": "git",
16.     "revision": "34a4d423d64926635eb36d3bb86276179cc186e1"
17.   },
18.   "dependencies": {
19.     "com.unity.ext.nunit": "1.0.6",
20.     "com.unity.modules.imgui": "1.0.0",
21.     "com.unity.modules.jsonserialize": "1.0.0"
22.   },
23.   "relatedPackages": {
24.     "com.unity.test-framework.tests": "1.1.33"
25.   },
26.   "_upm": {
27.     "changelog": "- Fixed an issue where using Assert.Expect with the same string multiple times can lead to incorrect errors in some cases (DSTR-442).\r\n- Improved the logging when using multiple Assert.Expect that the logs appear in another order than expected (DSTR-442).\r\n- Moved the targetPlatform specified when running tests in the TestRunnerApi from the Filter to the ExecutionSettings (DSTR-186).\r\n- Fixed an issue where an inheritance of UnityPlatformAttribute which was not working (ESTT-70).\r\n- Fixed the log of excluded platforms which was not displaying the right information.\r\n- Added filename and linenumber to test finished message (DSTR-505).\r\n- Add the possibility of running tests in a specified order from a test list (DSTR-494)."
28.   },
29.   "upmCi": {
30.     "footprint": "2279493b9acd323ab758f64369634dc0daf0dbac"
31.   }
32. }

Then I tried to find it in package manager.

I enabled the "Enable pre-release packages" in settings
But I can't see the pre-released Test-Framework package, using Unity 2021.3.8f1

 

The instructions might be wrong. I think you have to update

manifest.json

in your project, not

package.json

in one of the package files.

 

It works!!!!!Thank you very much!!!!

 

Isn't using Tasks for doing concurrency in testing a very bad idea, because the Tasks model is inherently designed around being a multi-threaded generator? Existing framework methods that use tasks potentially start new threads and maybe run the continuations on the same thread?
Why is the test runner trying to re-purpose Tasks instead of using another type of awaitable?

In our game Starship Troopers - Terran Command, we are using a custom awaitable to replace coroutines which perfectly fills this purpose. Something like this fork on my github: https://github.com/mhgamework/AwaitableCoroutines.
Credit to https://github.com/tomblind/unity-async-routines because im using a modified version of that repo

 

Tasks are used for asynchronous operations and are heavily used in .Net. Tasks don't have to be concurrent. Yes, other awaitable types work, but it's best for the package to work on the standard, and you can always do this:

Code (CSharp):

1. [Test]
2. public async Task MyTestAsync()
3. {
4.     await MyTestAsyncImpl();
5.  
6.     async MyCustomAwaitable MyTestAsyncImpl()
7.     {
8.         // Test code
9.     }
10. }

Whether a test is safe to run concurrently is up to the test writer to decide. Sometimes tests should run concurrently to speed up the test process.

Also, just because a test is async, doesn't mean the test runner has to run other tests at the same time.

 

Indeed. Also, there are several classes and libraries in .NET and the .NET ecosystem that work with Task by default (e.g. HttpClient, also used by things such as the OpenAPI client generator).

Tasks can cause problems in Unity if you try to call Unity methods off the main thread, such as manipulating transforms, in which case Unity will throw an exception. Unity also doesn't seem to stop running tasks when you exit play mode (can be worked around by using cancellation tokens, which are good practice in TAP anyway).

If you do non-Unity things off the main thread, there is no problem, and you can program asynchronously and run things off the main thread (when desired) to speed things up on certain systems. I don't think Unity coroutines can perfectly fill the Task gap out of the box since they still run on the main thread, which is probably their raison d'être (making code async, but not multithreaded) - in other words, you are still (partially) taxing the main thread with your async code. Put differently, you are running async code without multiple threads, as opposed to async code with multiple threads, which Tasks offers you access to - but doesn't require you to do.

Finally, TAP's

await

syntax is (subjectively) more intuitive to read as it more closely mimics non-asynchronous code than coroutines are. (Arguably, though, you might be able to adapt Unity coroutines to be awaitable and still benefit from it, as the mentioned library does.)

 

Are there any updates to UTF v2.0 ? last update was in January

 

Actually the last update was recently — Nov, 2022.
Checkout 2.0.1-exp.2.

The only thing is that I don't see changelog for experimental versions

 

I''ve just switched to UTF 2.0 and I just wanted to share that I really like that playmode and editmode tests can be run all at once.

What is even nicer (but not documented) it can be done via command line as well, by passing:

-testPlatform editmode -assemblyType "EditorOnly,EditorAndPlatforms"

assemblyType is passed to Enum.Parse, which works fine for comma delimed [Flags] enumeration.

I think it should be reflected in docs, under assemblyType section that "EditorOnly,EditorAndPlatforms" is also valid.
https://docs.unity3d.com/Packages/com.unity.test-framework@2.0/manual/reference-command-line.html

About the [TestScene] attribute, I can add few cents, since I've been using local implementation of it (in modded UTF package) and my main reason to have it was extra "Open corresponding scene" context menu, very handy when test was failing and required checking.

 

@superpig I managed to get standalone tests working in docker by stripping out the test framework and including

nunit.framework.dll

and

nunitlite.dll

, unmarking the assembly as test assembly to be included in the build, and running the tests through NUnitLite via script. I just built the player like normal, then ran the built player with arguments passed to NUnitLite in the script. Unfortunately, that means I can only test regular NUnit tests, and [UnityTest] does not work.

Could the Unity test framework provide similar functionality to just build the test player without running, so that it can then be ran directly without relying on the

PlayerConnection

with the editor? Or an option to just build and run it directly without connecting to the editor?

Code (CSharp):

1.  
2. - name: Build Project
3.   uses: game-ci/unity-builder@v2
4.   env:
5.     UNITY_LICENSE: ${{ secrets.UNITY_LICENSE_2019_X_WINDOWS }}
6.     UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
7.     UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
8.   with:
9.     projectPath: IL2CPP_Tests/IL2CPP_Tests_${{ matrix.unityVersion }}
10.     targetPlatform: StandaloneWindows64
11.  
12. - name: Run Tests
13.   run: |
14.     Start-Process -Wait -FilePath "build\StandaloneWindows64\StandaloneWindows64.exe" -ArgumentList "-batchmode `
15.    --trace=Info `
16.    --labels=All `
17.    --stoponerror `
18.    --result:unity-test-results-il2cpp-${{ matrix.mode }}-${{ matrix.progress }}-${{ matrix.unityVersion }}.xml `
19.    --output=unity-test-results-il2cpp-${{ matrix.mode }}-${{ matrix.progress }}-${{ matrix.unityVersion }}.log"
20.  

Code (CSharp):

1. using UnityEngine;
2. using NUnitLite;
3. using ProtoPromiseTests;
4. using System.Collections.Generic;
5.  
6. public class TestRunner : MonoBehaviour
7. {
8.     void Start()
9.     {
10.         // Command line args include the executable name and unity args, so we need to strip them to only include nunit args.
11.         string[] args = System.Environment.GetCommandLineArgs();
12.         int i = 0;
13.         while (i < args.Length)
14.         {
15.             if (args[i].StartsWith("--"))
16.             {
17.                 break;
18.             }
19.             ++i;
20.         }
21.         List<string> nUnitArgs = new List<string>(args.Length - i);
22.         for (; i < args.Length; ++i)
23.         {
24.             nUnitArgs.Add(args[i]);
25.         }
26.         int quitCode = new AutoRun(typeof(TestHelper).Assembly).Execute(nUnitArgs.ToArray());
27.         Application.Quit(quitCode);
28.     }
29. }

 

Hi, I met the same problem. Is there any way to resolve it?

 

I found a bug about 1.x of test framework, could you please solve this : https://forum.unity.com/threads/tes...ug-log-in-other-thread-when-teardown.1386909/

 

I've comprehensively used 2.0 exp package for a while now — I think a year or so.

Minor
I've found some QoL problems, but I've dealt with them until they are totally naturalized within my workflow. No big deal.
On of them is the popular exception it throws sometimes when both editor and play tests run and it gets crazy about the fixture names. Just a re-run and the problem goes away.
The same with other out-of-sync problems with Rider. You restart them, then they work again.

Critical
Maybe this is the worst breaking change I've found in regard to third-party plugins: runtime generated GameObject has change its name, from "Code-based tests runner" to "test runner". This was sort of a some-days-in-a-row obstacle to us since it breaks Zenject fixtures (which specifically looks for "Code-based tests runner" named object in scene and you as a Zenject user can't change that const).

Mayor
Also I think TDD worfklow is totally messed up with Rider, since play tests from Rider, whether editor and/or play mode, often ends with "Inconclusive" result after some seconds, thus you have to play the fixture twice each time you are TDDing and the loading time makes this unfeasible.
BUT I think this is not an exclusive problem of 2.0, just its worse in the exp package imhe.


Hope this feedback helps! I have plenty more to say but I don't want this report to be endless.

 

RGV's posts reminds me of something that would be very nice to have: the ability to run tests externally from Unity whilst Unity is open.

Currently you can run tests externally already, but it will complain that Unity is already open, I believe, because you can't open the same Unity project multiple times. It would be nice if there was an exception to this for tests (and building, for that matter), so you can at least call the Unity CLI on the project from VSCode or another editor to run tests programmatically more easily without leaving your editor/IDE.

 

I reported an issue about a few Attributes (and Constraints) not working when using async tests.

Attributes: Timeout, MaxTime, Repeat, Retry
Constraints: Throws, Delayed
And Task.Delay

Please vote for this!
https://issuetracker.unity3d.com/is...n-used-with-the-async-test-in-the-test-runner

 

Version 2.0.1-pre.18 has the following error with 2023.1b07 (didnt' try lower versions of Unity):

Library\PackageCache\com.unity.test-framework@2.0.1-pre.18\UnityEditor.TestRunner\TestLaunchers\PlatformSetup\XboxOnePlatformSetup.cs(5,17): error CS0619: 'XboxOneDeployMethod' is obsolete: 'The XDK Xbox One platform was removed in 2021.1'

 

Are there any updates on v2.0 release? from what i can see, last update was released September 2022

 

Have you managed to make it work somehow. Since I do need this functionality. Is there any workaround?

 

Workaround:

Code (CSharp):

1. [UnitySetUp]
2. public IEnumerator Setup()
3. {
4.     Task task = SetupFunc();
5.     while (!task.IsCompleted) yield return null;
6.     task.GetAwaiter().GetResult();
7. }

 

Any update on when 2.x might be released outside of experimental?