Official Unity Test Framework 2.0 ready for feedback

Unity Test Framework v2.0
[updated on May 2nd]

Hi everyone, we have a new experimental version of Unity Test Framework for you to try out in hope of getting feedback regarding the new features. At this point we have no timeline for fully releasing this version, but once we do, we will make sure to keep you updated.

For those of you who are unfamiliar with what Experimental packages are, they are defined as:

Experimental packages are new packages or experimental modifications made to mature packages. Unity does not support Experimental packages because they are in the early stages of development.​

You can read more about them in this article.

How to activate v2.0
If you’re using Unity 2019.4 and beyond, open your package.json and update the com.unity.test-framework line to

“com.unity.test-framework”: “2.0.1-exp.1”

Combine Edit Mode and Play Mode tests
This version removes the previous requirement to keep Edit Mode and Play Mode tests in separate assemblies by introducing the RequiresPlayModeAttribute. An Editor-only test assembly can now include tests that will run in the Editor's Play Mode if given the [RequiresPlayMode] attribute. Likewise, a platform-specific assembly can include Edit Mode tests and exempt them from running in Play Mode if the tests are given the [RequiresPlayMode(False)] attribute.

Async tests
This version introduces support for writing asynchronous tests with the dotnet Task asynchronous programming model. See Async tests. Feedback on this feature is especially welcome.

Ignore tests based on arguments
This version introduces the ParameterizedIgnoreAttribute which allows ignoring tests based on arguments which were passed to the test method of a parameterized test.

Revised Test Runner UI
This version includes a revised Test Runner window with support for combined Edit Mode and Play Mode tests and several usability improvements:

• The former separate Edit Mode and Play Mode tabs are gone and replaced with single options for creating a Test Assembly folder and test script.
• Additional usability improvements make it easier to filter and run tests.
• The Run Selected option is now available when running tests on a player.

​

This version includes many additional bug fixes and improvements. For a full list of changes and updates in this version, see the Unity Test Framework package changelog.

Looking forward to hearing from you
- UTF team

 

This sounds really good! I'll be sure to give it a try soon.

 

Good job!
Do you have a good solution for simulating the advance of time for Tasks in tests?
Like TestScheduler in UniRx for example

 

Will this version be supported in earlier Unity versions as well ?

 

I'm very interested in this as well

 

There's a few things that are annoying in V1, and I'm wondering if they'll be solved in V2:

- The inability to write tests targeting Assembly.CSharp, forcing an assembly-based workflow. Our experience is that assembly-based workflows lead to worse code and slower compile times, so we'd like to avoid it.

- No support for [Timeout]. Right now every test that yields anything else than WaitForSeconds has to include a manual timeout, otherwise tests can hang forever.

- The way runtime tests work. Right now Unity makes a temporary scene asset and loads it when we start tests. After that, we typically load a different scene. This causes a bunch of knock-on issues:
-- If Unity crashes during execution, we get a new scene asset in Assets/
-- Any system that scans for asset changes (like Quick Search and version control plugins) starts doing a bunch of work
-- We're often in the scene that the Unit test is supposed to run in (especially when iterating on fixing a test). Unloading that scene, loading a new scene, and then loading the original scene back is slow, and slows down iteration.
- Ideally all of this would be fixed by two things:
-- the ability to tell Unity which scene(s) to run a test in the context of, and for Unity to not do any scene loading if those are already open
-- when we don't run the tests in the context of a scene, Unity should use an in-memory normal empty scene instead of an asset. Unity is perfectly capable of entering play mode with a clean scene that's not an asset.

 

Timeout issues will be solved in later iterations of v2.0, before the full release of 2022.2.

These two items you mentioned are not addressed in v2.0 at all. While for the latter we discussed a potential solution of enabling users to specify which scene should be loaded for a test, I'd be curious to learn more about your experience regarding assembly-based workflows. How does it lead to worse code and slower compile times? What would be a better way of working for you and your team?

 

I've had that discussion before here, and most of that still holds. Note that we didn't have enter play mode options for that project, so more assemblies meant taking longer to enter play mode even when we didn't change code!

For our current project, we ended up having a single, big asmdef that just covers all of our Assets/ code in order to support tests, and another one for Editor/ stuff that we asmref all over the place. But I'd really love for that to just be Assembly-CSharp and then be able to target that with our test assemblies.

The clean solution that'd fix both the testing problem, and all of the other problems we have around assembly definitions, would be to just allow Assembly-CSharp to be an assembly we could add a reference to from other assemblies. That shouldn't be very hard!


We do like to extract stable, independent parts of our code base out of the general compile/reload flow, but in those cases moving them into a package is a lot cleaner, since that allows versioning and reuse between projects (we use Verdaccio). If something's changed often enough that it's in Assets/, it's changed often enough that we're not saving compile/reload times by segmenting it into it's own assembly. We're just adding to the average case number of assemblies that has to be reloaded by a code change.

 

Oh, and if you have a bunch of plans and ideas, @thewerku, maybe there should be a testing tab on https://unity.com/roadmap/unity-platform/engineering ?

 

Since you're interested @thewerku I agree with @Baste here. I work for a big mobile company and we use modules for every part of the game, and each module has its own Asmdef. (Shop, Dashboard, Each game mode, Social, Rewards, etc).

Since we started using Asmdef for each module, everytime you change any single line of the code, going back to unity takes around 20 seconds which makes the workflow very tedious and hard. It gets 10 times worse when you're developing using TDD. Everytime you change something on the test you have to wait 20 seconds, this makes a simple 2 minute discovery test, something that can take up to 15 minutes.

 

I gave this a spin, and here are my findings:

• The new UI is nice, especially when filtering tests!
• However, the UI does not show pass/fail status while executing, which I find somewhat annoying, as it's nice to start looking at failing tests already while the rest of the tests are executing.
• The first bigger issue I found is that the new Custom NUnit version has Auto Reference enabled. This makes all assemblies look like test assemblies. This is a problem for some of our custom code, SonarCloud analysis, and also the test runner code itself seems to use this criterion to scan assemblies for tests (see bullet below).
• When running all tests in our largest project with lots of assemblies and over 7000 tests, starting up takes extremely long: it took over 2 minutes to get past "Create Bootstrap Scene", and at least another 2 minutes to get past "Prebuild Setup". I'm guessing this is at least partially because all assemblies are treated as test assemblies now (see bullet above). These stages can't even be canceled, making the situation worse.

 

There's a workaround for this called StoppableCoroutines. Trilleon Unit Test & UI Automation framework implements it, and coroutines will never hang/fail. Try implementing it yourself. Here is the specific file. Look at the AutomationMaster.cs file to see how it is used.

Although it may be difficult to implement in an NUnit or UnityTest framework if you can't control the test runner. If you need this functionality, it may make sense just to switch your tests to Trilleon tests, or make your own test runner.

 

Error occurred when run all Edit Mode and Play Mode tests on Test Runner window.

```
ArgumentException: An item with the same key has already been added. Key: RequiresPlayModePerformance.Tests.Editor.EditModeTests.DummyEditModeTest_AlwaysPassed(1)
UnityEditor.TestTools.TestRunner.GUI.TestListGUI.get_ResultsByKey () (at Library/PackageCache/com.unity.test-framework@2.0.1-pre.18/UnityEditor.TestRunner/GUI/Views/TestListGUIBase.cs:56)
(snip)
```

Reproduction Project
https://github.com/nowsprinting/RequiresPlayModePerformance

This error will no longer be reproduced after the following actions:
Causing Assembly-CSharp-Editor assembly recompilation by editing Assets/Editor/RunTestsPerformance.cs.

 

Bugs:
* In the Test Runner window, the IgnoreAttribute and DescriptionAttribute string arguments of tests are not displayed. In the case of ExplicitAttribute, they are displayed reason after “Run All”

Requests:
* I hope to be added string argument (reason) to the ParameterizedIgnoreAttribute
* In the Test Runner window, I hope multi-word search. And if possible, ideographic space (U+3000) should also be recognized as separators.
* I hope `SceneManager.LoadScene` should be able to load scenes that are not in "Scenes in Build" (there is a workaround, but it would be nice to have it implemented in the framework)

 

I'd say "SceneManager.LoadScene` should be able to load scenes that are not in "Scenes in Build" (there is a workaround, but it would be nice to have it implemented in the framework" is quite important

 

The issues with Coroutines and the Unity Test Framework run deeper than just timeouts, I commented on it in this thread. It's totally possible to write properly timing out and cancelable code within the framework, it just requires you to wrap everything asynchronous in something else than a raw coroutine/enumerator.

 

Hey all, thanks for the replies! I'll try to address the points we haven't yet responded to:

@unity_t3DjhpF93gNHLg the only thing that has come up in our discussions was potentially using Time.timeScale - have you looked into it to see if it meets your expectations?

Our plans for this year are very much dependent on v2.0 feedback, so we don't really have a fully fledged roadmap to put on the main page. What we're going to do is collect improvement ideas and then present them to you (again through Forum) to help us prioritise the work - once priorities are clear, we'll factor in the time we need for bugs and maintenance work to plan it against the release deadline. Either way, we might look into actually having a public roadmap once the v2.0 plans are validated with the help of this community.

@sbergen do you mean that the status icons don't show up at all while tests are running? Would you be able to share a screenshot of how it looks like on your end? Feel free to ping me directly via Forum messaging option if you don't want to share such content publicly.

Thanks for reporting @nowsprinting. We logged these bugs on our backlog.

We will take these suggestions into planning. As mentioned above, we'll collect all requests and get back to you to understand the priority and value, that will then help us decide on the work plan for the next quarter.

The last thing I wanted to address is feedback around working within assembly-based workflow. We're taking in what you've shared here and in other threads, and will start an internal discussion about this with the Scripting team, to better understand current approach and what is the best way to alleviate pain for you all. If there will be any work planned, it will be separate from UTF 2.0 release.

To be continued. Appreciate the discussion here so far, keep it coming!

 

In these years using the test framework, this would be my main requests:

• Coroutine version of OneTimeSetUp
  
  • Discussed here: https://forum.unity.com/threads/add-coroutine-version-of-onetimesetup.890092/#post-7842978
  • This is important for loading a test scene just one time
• Support for loading scenes that are not included in the build index
  • Important for loading a test scene that you only need it at testing
• It would be also nice to have serialized fields in the tests. Sometimes you need to reference some project asset in the test and it would be nice to get it by a serialized field.

Thank you

 

This can be done through EditorSceneManager.LoadSceneInPlayMode. We write tests like this:

Code (csharp):

1.  
2. public abstract class TestsForScene {
3.     public abstract string pathToScene { get; }
4.     protected string sceneName => Path.GetFileNameWithoutExtension(pathToScene);
5.  
6. [OneTimeSetUp]
7. public void OneTimeSetup() {
8. #if UNITY_EDITOR
9.         UnityEditor.SceneManagement.EditorSceneManager.LoadSceneInPlayMode(pathToScene, new LoadSceneParameters(LoadSceneMode.Single));
10. #else
11.         SceneManager.LoadScene(sceneName, LoadSceneMode.Single);
12. #endif
13.     }

But having a real way to do this would be much nicer. What if we could:

Code (csharp):

1. [UnityTest(scene="path/to/scene")]

 

Async/await native support would be nice too

https://forum.unity.com/threads/async-await-in-unittests.513857/
https://forum.unity.com/threads/support-for-async-await-in-tests.787853/

 

One of the new features in 2.0 is actually a way to have async tests. Would you be able to verify if what we're proposing satisfies your needs, and if not, what are we missing?

Async tests
This version introduces support for writing asynchronous tests with the dotnet Task asynchronous programming model. See Async tests. Feedback on this feature is especially welcome.​

 

Wow I totally missed that example.

Yes, that's it. Perfect. Thanks!

 

I tried this again, and noticed that this issue only happens the first time you run tests, either after switching from 1.X to 2.X or when restarting the Editor. Seems to be consistently reproducible for me. The icons are visible, they just stay in the "not run" state, until they all turn to their final state all at once. Now that I know it doesn't happen on consecutive runs, it's actually not a big deal, just strange

I'm more curious about Auto Reference being enabled in nunit.framework.dll, and would be interested in seeing if changing that helps with the extremely slow performance I'm seeing in big projects. I had some issues trying to edit that manually in the package source locally, as somehow the Editor didn't want to pick up the change.

 

Feature Request: Native junit format reporting of test cases. This would allow direct integration with other tooling like GitLabs Unit Testing Reporting feature.

 

While I'm thinking about JUnit, it would be also great to have Cobertura format reporting too for Test Coverage Visualization.

Thanks team!

 

Thanks for the suggestion. Could you help me understand what would change for you, if we had those format reporting options available? What problem could it solve?

 

For my use case, it would mean a faster CI pipelines if cobertura reporting was supported natively. As of today, one CI job runs the tests and then in a subsequent job collates all the coverage reports and converts them to cobertura. While this only takes about a minute to run, we runs 100s of these a day and that minute adds up quickly!

The solution right now is simply running the below to generate the report:

dotnet tool install -g dotnet-reportgenerator-globaltool

reportgenerator "-reports:editmode-coverage/coverage.xml;playmode-coverage/coverage.xml" -reporttypes:Cobertura "-targetdir:report"

I raised the feature request for awareness. I was hoping that if one of the underlying libraries that are already being used by the tooling offer this type of reporting. That would make the work (hopefully) a trivial exercise of just exposing existing functionality. Then it becomes much easier to see the value in doing the work. As I write this though, probably feedback more for the test coverage tool than the test framework tool...

Test Coverage visualization is a fantastic tool for developers to help them understand and also give confidence in the features that they are developing (or reviewing) to make sure they continue to be robust and stable in the future as the game progresses (hopefully) into production and then into live operations with continued bug fixes and feature development.


For JUnit reporting, once again I have a solution for this already that uses xslt to transform the nunit reporting into a junit report. JUnit reporting in GitLab helps visualize problems in test cases. If I have 1000s of unit/integration tests and some of them are failing, having a tool that can show and visualize what is failing can save time and improve productivity.

Having this supported out of the box would (I hope) encourage adoption of the tooling offered by Unity and would be a stellar feature to have documented for others to see and therefore not have to come up with custom integration solutions like I had to do.

In summary, the value to me personally is low however by providing it as a feature natively you will save other developers who need to write this level of integration themselves. I would also hope that it results in an increase of adoption of the unity test framework adoption, as it becomes easier for developers to show the value it can bring if it integrates with existing tooling as a proof-of-concept would be trivial to showcase the value to the business.

I hope all this makes sense and apologies for anything that comes across as rambling

 

Are there any plans to support target platform tests in Docker? As a library/package author, currently, not being able to test IL2CPP builds during CI is my biggest pain point, considering how fragile it is with normally-acceptable C# code (virtual generic methods with structs).

 

@superpig Thanks for the response.

Even though it's not officially supported, people have gotten Unity to work for some things in Docker, like making builds and running editor tests. From my own testing trying to get it to work, it seems to me that the problem with target platform tests in Docker is that it can't make a connection between the player and editor, so it just waits for 10 minutes and times out. That seems like it's directly under the purview of this package, not Unity itself. I don't think Unity officially supporting Docker will suddenly make that connection work, but I don't know enough about it to say one way or the other.

In any case, I understand your position, just something to think about.

 

Can we also get a conditional flag to detect when unit tests are running?

 

I found this thread while looking for ways to write test for my asynchronous methods, so of course I excitedly followed the instructions to enable pre-release packages (I can even see the XR Interaction Toolkit tagged as [Pre]). Unfortunately I don't see a Test Framework 2.0 or any option to update the existing Framework. Are there additional steps I'm somehow missing?


Thanks!

 

Who moved my cheese? Why are the buttons located there and not at the top of the window ?

 

Pasting an answer shared somewhere earlier in the thread:

Depending on the Unity version, you will have to add it to the package.json manually, by changing the test-framework line to "com.unity.test-framework": "2.0.1-pre.18"​

@liortal it's a result of decluttering the top part of the package UI. What are your thoughts here?

 

What about copying Visual Studio's Test Explorer UI, using icons instead of text? They're pretty compact.

 

Thank you! I missed that earlier in the thread I guess, appreciate you taking the time to paste if for me!

I did find one strange thing, it seems like the test UI gets hung when doing async tests. It always recovers if I click to another window, then click back onto unity (re-focus on the window essentially). I'm currently using unity v2021.1.19f1. I'll look into updating to 2021.2.12f1 which appears to be the latest version, I'm not sure if there would be any breaking changes for my project. But wanted to note the feedback here in case anyone else sees the same thing and goes searching.

Example gif of what I'm seeing:


Exact code used to test:

Code (CSharp):

1. using NUnit.Framework;
2. using UnityEngine.AddressableAssets;
3. using UnityEngine.ResourceManagement.ResourceLocations;
4. using System.Collections.Generic;
5. using System.Threading.Tasks;
6. using System.Threading;
7.  
8. public class TestAsync
9. {
10.     [Test]
11.     public async Task LoadAddressibleResource()
12.     {
13.         IList<IResourceLocation> landTilesTask = await AsyncLoadLand_GRITTYSOOT();
14.         Assert.Greater(landTilesTask.Count,0, "Loaded some locations");
15.     }
16.  
17.     private async Task<IList<IResourceLocation>> AsyncLoadLand_GRITTYSOOT()
18.     {
19.         IList<IResourceLocation> landTilesTask = await Addressables.LoadResourceLocationsAsync("biomes/GRITTYSOOT/land").Task;
20.         return landTilesTask;
21.     }
22.  
23.     [Test]
24.     public async Task TestGenericAsync()
25.     {      
26.         Assert.IsTrue(await GenericAsync());
27.     }
28.  
29.     private async Task<bool> GenericAsync()
30.     {
31.         await Task.Run(() => Thread.Sleep(1000));
32.         await Task.Run(() => Thread.Sleep(1000));
33.         return true;
34.     }
35.  
36. }
37.  

 

@ProtoTerminator thanks for the suggestion, I'm saving it for future reference. The package UI is not the primary focus of 2.0 changes, but something in need of attention that we're aware of.

@unity_hi5HOPoVLuZIyA thanks for reporting, adding this to our backlog.

 

I got this error:

Library\ScriptAssemblies\ProtoPromiseTests.pdb: Copying the file failed: The process cannot access the file because it is being used by another process.

And I couldn't get rid of it until I re-imported my scripts.

Repro steps:
- Right-click a test and click "Open source code"
- Edit the code and save
- Go back to Unity to recompile
- Observe error

[Edit] This was in Unity 2021.2.7f1 with Test Framework 2.0.1-pre.18

@thewerku Also, I would like to provide some more feedback on the UI. Should I do that here, or where should I do that?

 

Great! I think so attribute style is useful!
Also, it seems that it can be used without conflicting with the RecordedTestAttribute of Automated QA package.

However, I'm worried about testing to transition from SceneA to SceneB.
With the above method, there seems to be no guarantee that Scene2 will be included in the build.
We can avoid it by writing the following.

Code (CSharp):

1. [TestScene("SceneA", true)]
2. [TestScene("SceneB", false)]

 

Thank you for your effort

I also would like to request that, the test should allow [Timeout] attribute in unity, and make it so that every test with [Timeout] would run in another thread, and so it also could be killed separate from unity's main thread

 

Speaking of Timeout, it would be nice if Timeout worked reliably, especially within UnitySetUp and UnityTearDown - I tried the new 2.0.0 and for reasons (faulty code on my end) the Timeout wasn't being respected and some of my tests were getting stuck forever in while() loops. I had to step in with the debugger and force them to end and in some extreme cases kill Unity process via Task Manager.

 

If there was any other way to make unity test not freeze from infinite loop and need to conjure the task manager to kill it all, then I would love to

Currently now I waste my time to run a test case from reactivex and some other dll that have some chance to internally block thread. And unity will unreliably frozen without a way to stop. Seriously it was a weak point of unity that should have some overseer thread that could reset the mainthread and reload domain instead of the need to kill the whole process

But in the meantime, just simple attribute that to specify that some test has tendency to block thread, so need to run in another thread separate from main thread. Timeout is a good indicator

There should be some API like `Application.runOnMainThread` or `EditorApplication.runOnMainThread` to run unity API in other thread and just that

 

There is an API for that:

SynchronizationContext.Current.Post

for asynchronous dispatch, or

SynchronizationContext.Current.Send

for synchronous dispatch. Though you will need to cache the context somewhere to use from a background thread.

 

If you just carefully read my comment and not hurrily argue you would see that I mention "some other dll" "internally block thread" which mean I have no control over it and so the unity mainthread could blocked from any reason and so dispatching anything aren't solve this problem. Unity should just have ability to kill its main thread and do domain reloading is the totally sensible solution and so please don't make any more distraction

 

Okay...? That's a separate issue from asking about an API, though. I had nothing to add about Unity's constraints, so I just responded to the last bit. If they add a new API to run something on the main thread, that won't help with your dll issue.