Support for async / await in tests

Does the Unity test framework support running test methods that are marked as 'async' (e.g: using the await keyword) ?

This test is shown in the test runner (but doesn't properly work since the test exits without waiting for the task to complete):

Code (CSharp):

1. [Test]
2. public async void ThisTestSucceeds()
3. {
4.     await Task.Delay(5500);
5.  
6.     Assert.IsTrue(false);
7. }

 

@Warnecke could you satisfy my curiosity and inform me why is that not a priority? Not being able to unit test async/await paths really limits the scope of what you can test, specially with recent libs.

 

workaround:

Code (CSharp):

1.  public static IEnumerator AsCoroutine (this Task task)
2.         {
3.             while (!task.IsCompleted) yield return null;
4.             // if task is faulted, throws the exception
5.             task.GetAwaiter ().GetResult ();
6.         }

then use IEnumerator-based tests

Code (CSharp):

1. [UnityTest] public IEnumerator Test() {
2.     yield return Run().AsCoroutine();
3.     async Task Run() {
4.         actual test code...
5.     }
6. }

 

Basically the same as M_R suggested, but I am using UniRX.Async where you can go

Code (CSharp):

1. [UnityTest]
2. public IEnumerator AwaitablePlayModeTest()
3. {
4.     yield return Task().ToCoroutine();
5.     async UniTask Task()
6.     {
7.         // test with async/await
8.     }
9. }

I think, you need to wait a frame using

UniTask.Yield()

at the beginning to have the task scheduled at the right time in the Unity game loop. Not sure.
This is a bit more compact than C# Task and more performant, because UniTask is optimized for Unity.

I vote for integrating async/await everywhere in Unity natively!

see https://gametorrahod.com/the-art-of-unirx-async/

 

@Warnecke is there an update on this? Async tests important to me as well.

 

Async Await Support has extension methods that convert a task to an IEnumerator. Here's an example:

Code (CSharp):

1. [UnityTest]
2. public IEnumerator LoadTerrain()
3. {
4.     yield return _().AsIEnumerator();
5.     async Task _()
6.     {
7.         Task loadingOperation = fixture.LoadTerrainAsync();
8.         Assert.AreEqual(GameState.TerrainLoading, fixture.State);
9.         await loadingOperation;
10.         Assert.AreEqual(GameState.TerrainLoaded, fixture.State);
11.         Assert.That(SceneManager.GetActiveScene().path,
12.             Does.EndWith($"{ScenePath}.unity"));
13.     }
14. }

And the LoadTerrainAsync method (which essentially loads a scene asynchronously):

Code (CSharp):

1. public async Task LoadTerrainAsync()
2. {
3.     state = GameState.TerrainLoading;
4.     Debug.Log($"Loading scene '{scenePath}')");
5.     await SceneManager.LoadSceneAsync(scenePath);
6.     state = GameState.TerrainLoaded;
7. }

One advantage of running a test in this way is if `LoadTerrainAsync` method throws an exception, it gets passed up to the test runner as an AgregateException. If the method was a Coroutine, the exception would get swalled and all I know is that the state didn't change as expected but no indicator as to why. Since it's also possible to await methods that return IEnumerator (thanks to extensions in the AsyncAwaitUtil plugin), I think it's in principle possible to only make use of async/await in test code but not use it the production code (should you want that).

 

The coroutine workaround will only work for playmode tests as [UnityTest] is not available for editor tests, I think.

 

[UnityTest] attribute is available for EditMode and PlayMode tests, although with some limitations in the former.

 

It sounds like Unity isn't going to add these any time soon so I've added them myself in my fork of the test framework: https://github.com/8bitforest/com.unity.test-framework

It adds [AsyncTest], [AsyncSetUp], [AsyncTearDown], [AsyncOneTimeSetUp], and [AsyncOneTimeTearDown]
The methods with these attributes will need to return "async Task". "async void" will not work:

Code (CSharp):

1. [AsyncTest]
2. public async Task TestWithDelay()
3. {
4.     Debug.Log("Starting test...");
5.     await Task.Delay(5000);
6.     Debug.Log("Test finished!");
7. }

They are implemented using IEnumerator's in the backend just like the answers above, but this way improves the syntax so they are regular async methods, and should be easy to switch over when/if Unity implements them themselves.

And as a bonus my fork also includes the mysteriously missing [UnityOneTimeSetUp] and [UnityOneTimeTearDown]

 

TestTask:44
m_Context.CurrentResult.RecordException(m_TestTask.Exception!.InnerException); => shouldn't be `?` instead of `!`
Creates a compilation error. Or is it a c#8 trick ?

But then it actually works, very greatful.

One issue I've noticed is that tests doesn't fails when consol has errors. Not sure because wasn't able to reproduce it clearly

 

Another feedback, if the AsyncSetup fails with an exception then it's considered as "Passed" silently.

I have the case while awaiting a task wrapping a RecompileScript that fails

 

The UniTask library is also a way to handle unit testing async:
https://github.com/Cysharp/UniTask#for-unit-testing

 

Yeah I fallback into that one, it's working.

 

Oh heavens, thank you!!!!!

I was going to use Coroutines, but such tests have to 100% completely wait for each one to individually run.. which will just make my tests take forever and freeze the editor unnecessarily.

<3 async / await

 

Wait... your code example (method called TestWithDelay()) isn't working in my edit mode tests.


And no shame, here's my quick and dirty testing code (as used in the screenshot above)

Code (CSharp):

1. using System.Collections;
2. using System.Threading.Tasks;
3. using NUnit.Framework;
4. using UnityEngine;
5. using UnityEngine.TestTools;
6. using UnityEditor;
7.  
8. namespace XXX.Editor.Tests {
9.     public class XXXTests {
10.  
11.         [UnityTest]
12.         public IEnumerator ExampleCoroutine() {
13.             float startTime = (float) EditorApplication.timeSinceStartup;
14.             while (EditorApplication.timeSinceStartup - startTime < 1)
15.                 yield return null;
16.             Debug.Log("!?");
17.  
18.             startTime = (float) EditorApplication.timeSinceStartup;
19.             while (EditorApplication.timeSinceStartup - startTime < 1)
20.                 yield return null;
21.  
22.             startTime = (float) EditorApplication.timeSinceStartup;
23.             while (EditorApplication.timeSinceStartup - startTime < 1)
24.                 yield return null;
25.         }
26.  
27.         [UnityTest]
28.         public IEnumerator ExampleCoroutine2() {
29.             float startTime = (float) EditorApplication.timeSinceStartup;
30.             while (EditorApplication.timeSinceStartup - startTime < 1)
31.                 yield return null;
32.             Debug.Log("!?");
33.  
34.             startTime = (float) EditorApplication.timeSinceStartup;
35.             while (EditorApplication.timeSinceStartup - startTime < 1)
36.                 yield return null;
37.  
38.             startTime = (float) EditorApplication.timeSinceStartup;
39.             while (EditorApplication.timeSinceStartup - startTime < 1)
40.                 yield return null;
41.         }
42.  
43.         [AsyncTest]
44.         public async Task TestWithDelay() {
45.             Debug.Log("Starting test...");
46.             await Task.Delay(5000);
47.             Debug.Log("Test finished!");
48.         }
49.  
50.  
51.         [AsyncTest]
52.         public async Task Example() {
53.             await Task.Delay(1000);
54.             Debug.Log("????");
55.             await Task.Delay(1000);
56.             await Task.Delay(1000);
57.             await Task.CompletedTask;
58.             Assert.Pass("!?");
59.         }
60.  
61.         [AsyncTest]
62.         public async Task NUnit1012SampleTest() {
63.             bool result = await Task.FromResult(true);
64.             Assert.That(result, Is.True);
65.         }
66.  
67.         [AsyncTest]
68.         public async Task<int> TestAdd() {
69.             return await Task.FromResult(2 + 2);
70.         }
71.     }
72. }
73.  

Note the 2 coroutine examples are the same -- I just wanted to verify that they couldn't run "at the same time".

Did you too, convert the Tasks into a "coroutine representation", since it's telling me edit mode tests can only yield return null?
I was thinking that you updated the NUnit Framework version to one that supports async Task.. but I appear to be incorrect.

 

I think you've misunderstood what async/await support in NUnit is intended for. It has nothing to do with parallel test execution, which is a completely different feature: https://docs.nunit.org/articles/nunit/technical-notes/usage/Framework-Parallel-Test-Execution.html

However, that feature uses separate threads, and if you are doing anything that works with Unity APIs, it probably wouldn't work for you anyway, as they mostly require you to work on the Unity main thread. If you are not working with any Unity APIs, and want to put in the extra effort, you could have your code and tests live in standard .NET assemblies, and leverage all the NUnit 3 features available from a separate non-Unity project.

 

Ah...
Right, I forgot about the implications of running on the Unity main thread, since these tests usually interact with UnityEngine/related API.

I guess I see why they didn't prioritize it.. but it's still a stretch to say I still wouldn't want to test async (non-parallel) code in my Unity tests.
I suppose if I wanna test it though, it'll need to be pure .NET/C# projects then.
Thanks for explaining!

 

Oh what I was thinking of was both async tests,

and what I found today:
[Parallelizable] marks unit tests in NUnit as ones that can run at the same time as others.


Since their docs note it requires NUnit 3.0, I assume it won't work with Unity Test Framework / Unity Test Runner though.

 

@ModLunar TestFramework 1.1.24 is using NUnit 3.5 :

From : https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/index.html

Also, my problem is that for whatever reason, my test is taking something like 20 second to execute where in play mode, it's almost instant.

I'm using the IEnumerator work around :

Code (CSharp):

1. Task task = sut.Load();
2. while (!task.IsCompleted)
3.     yield return null;
4. if (task.IsFaulted)
5.     throw task.Exception;

It's so crazy that UTF is not supporting the async task test like NUnit does.

 

NUnit 3.5... A 5 year old version...

"released this on Oct 4, 2016"

https://github.com/nunit/nunit/releases/tag/3.5

 

Omg indeed ! You talked about 3.0+ so I thought 3.5 was fine...

Unity really needs to update this.. we can't test properly

 

My preferred way of running an async method in Unity Test Runner is using a custom SynchronizationContext.
By doing so the same test runs in Unity and in .NET CLR.

In short - when calling its Run() method it sets the SynchronizationContext.Current of the current thread (in Unity the UI thread) and execute the given test function (Func<Task> func). After the test finished the SynchronizationContext.Current is restored to the previous one.
An example a test would look like this:

Code (CSharp):

1. [Test] public static void TestSync () {
2.     SingleThreadSynchronizationContext.Run(async () => {
3.         await Task.Delay(1);
4.     });
5. }

A full explanation and the idea is from a Microsoft Blog.
[Await, SynchronizationContext, and Console Apps | .NET Parallel Programming] https://devblogs.microsoft.com/pfxteam/await-synchronizationcontext-and-console-apps/

The implementation of this fits into 50 lines of code:

Code (CSharp):

1. // [Await, SynchronizationContext, and Console Apps | .NET Parallel Programming] https://devblogs.microsoft.com/pfxteam/await-synchronizationcontext-and-console-apps/
2. public class SingleThreadSynchronizationContext : SynchronizationContext
3. {
4.     private readonly BlockingCollection<ActionPair> queue = new BlockingCollection<ActionPair>();
5.  
6.     public override void Post(SendOrPostCallback callback, object state) {
7.         var action = new ActionPair {callback = callback, state = state};
8.         queue.Add(action);
9.     }
10.  
11.     private void RunOnCurrentThread() {
12.         while (queue.TryTake(out ActionPair action, Timeout.Infinite))
13.             action.callback(action.state);
14.         Console.WriteLine("RunOnCurrentThread exited.");
15.     }
16.  
17.     private void Complete()
18.     {
19.         queue.CompleteAdding();
20.         Console.WriteLine("SingleThreadSynchronizationContext Completed.");
21.     }
22.  
23.     public static void Run(Func<Task> func)
24.     {
25.         var prevCtx = SynchronizationContext.Current;
26.         try
27.         {
28.             var syncCtx = new SingleThreadSynchronizationContext();
29.             SynchronizationContext.SetSynchronizationContext(syncCtx);
30.  
31.             Task funcTask = func();
32.             funcTask.ContinueWith(t => syncCtx.Complete(), TaskScheduler.Default);
33.  
34.             syncCtx.RunOnCurrentThread();
35.  
36.             funcTask.GetAwaiter().GetResult();
37.         }
38.         finally { SynchronizationContext.SetSynchronizationContext(prevCtx); }
39.     }
40. }
41.  
42. internal struct ActionPair {
43.     internal SendOrPostCallback  callback;
44.     internal object              state;
45. }

 

This looks neat!
But to make sure I'm understanding correctly, does what you're describing allow us to make use of parallelizable tests?
Or only tests that are blocking the main thread?

 

The intension of this approach is to execute async methods in synchronous [Test] methods.
As this approach utilize the calling thread for execution [Parallelizable] tests are possible with NUnit. At least I see no reason why not.
I guess support of [Parallelizable] in Unity cannot be implemented as 'Edit Mode' tests are always expected to run on the UI Thread and [Parallelizable] require multi threading.

 

Gotcha, that makes sense now. Thanks for explaining

 

Hi. Team owning the Unity Test Framework package (UTF, also known as Test Runner) has recently made available a pre-release version of v2.0 - you can read the announcement here. One of the new features we added, and are looking to get users' eyes on it, are async tests. We'd appreciate any feedback and comments, if you decide to test it out. Thanks!

 

Thank you very much for your work! It was really needed for me

 

Nice!

 

Is there any progress on that? Does unity support Async Tests?

 

If I recall correctly, the only version I've found to be (the most stable so far) that supports async tests is the following (snippet from my Packages/manifest.json file):

Code (CSharp):

1. "com.unity.test-framework": "2.0.1-exp.2"

With that version, I can write async unit tests (just note they won't run in parallel) like this:

Code (CSharp):

1. [Test]
2. public async Task DoExampleTest() {
3.     //TODO: Your test code here...
4. }

 

Ooh awesome! Thanks for the update, nice to be off such an experimental version now haha