Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Support for async / await in tests

Discussion in 'Testing & Automation' started by liortal, Dec 3, 2019.

  1. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Does the Unity test framework support running test methods that are marked as 'async' (e.g: using the await keyword) ?

    This test is shown in the test runner (but doesn't properly work since the test exits without waiting for the task to complete):
    Code (CSharp):
    1. [Test]
    2. public async void ThisTestSucceeds()
    3. {
    4.     await Task.Delay(5500);
    5.  
    6.     Assert.IsTrue(false);
    7. }
     
    Arkade, Harrishun, morhun_EP and 3 others like this.
  2. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    Currently that is not supported. It might be possible when we upgrade the nunit version, but we do not have a timeframe for that yet.
     
  3. HugoClip

    HugoClip

    Joined:
    Feb 28, 2018
    Posts:
    52
    @Warnecke could you satisfy my curiosity and inform me why is that not a priority? Not being able to unit test async/await paths really limits the scope of what you can test, specially with recent libs.
     
  4. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    561
    workaround:
    Code (CSharp):
    1.  public static IEnumerator AsCoroutine (this Task task)
    2.         {
    3.             while (!task.IsCompleted) yield return null;
    4.             // if task is faulted, throws the exception
    5.             task.GetAwaiter ().GetResult ();
    6.         }
    then use IEnumerator-based tests
    Code (CSharp):
    1. [UnityTest] public IEnumerator Test() {
    2.     yield return Run().AsCoroutine();
    3.     async Task Run() {
    4.         actual test code...
    5.     }
    6. }
     
  5. olejuer

    olejuer

    Joined:
    Dec 1, 2014
    Posts:
    211
    Basically the same as M_R suggested, but I am using UniRX.Async where you can go

    Code (CSharp):
    1. [UnityTest]
    2. public IEnumerator AwaitablePlayModeTest()
    3. {
    4.     yield return Task().ToCoroutine();
    5.     async UniTask Task()
    6.     {
    7.         // test with async/await
    8.     }
    9. }
    I think, you need to wait a frame using
    UniTask.Yield()
    at the beginning to have the task scheduled at the right time in the Unity game loop. Not sure.
    This is a bit more compact than C# Task and more performant, because UniTask is optimized for Unity.

    I vote for integrating async/await everywhere in Unity natively! :)

    see https://gametorrahod.com/the-art-of-unirx-async/
     
    Neiist likes this.
  6. Andrew-Silverblade

    Andrew-Silverblade

    Joined:
    Mar 14, 2015
    Posts:
    2
    @Warnecke is there an update on this? Async tests important to me as well.
     
    Arkade, EirikWahl, DrummerB and 3 others like this.
  7. MajeureX

    MajeureX

    Joined:
    May 4, 2020
    Posts:
    13
    Async Await Support has extension methods that convert a task to an IEnumerator. Here's an example:

    Code (CSharp):
    1. [UnityTest]
    2. public IEnumerator LoadTerrain()
    3. {
    4.     yield return _().AsIEnumerator();
    5.     async Task _()
    6.     {
    7.         Task loadingOperation = fixture.LoadTerrainAsync();
    8.         Assert.AreEqual(GameState.TerrainLoading, fixture.State);
    9.         await loadingOperation;
    10.         Assert.AreEqual(GameState.TerrainLoaded, fixture.State);
    11.         Assert.That(SceneManager.GetActiveScene().path,
    12.             Does.EndWith($"{ScenePath}.unity"));
    13.     }
    14. }
    And the LoadTerrainAsync method (which essentially loads a scene asynchronously):

    Code (CSharp):
    1. public async Task LoadTerrainAsync()
    2. {
    3.     state = GameState.TerrainLoading;
    4.     Debug.Log($"Loading scene '{scenePath}')");
    5.     await SceneManager.LoadSceneAsync(scenePath);
    6.     state = GameState.TerrainLoaded;
    7. }
    One advantage of running a test in this way is if `LoadTerrainAsync` method throws an exception, it gets passed up to the test runner as an AgregateException. If the method was a Coroutine, the exception would get swalled and all I know is that the state didn't change as expected but no indicator as to why. Since it's also possible to await methods that return IEnumerator (thanks to extensions in the AsyncAwaitUtil plugin), I think it's in principle possible to only make use of async/await in test code but not use it the production code (should you want that).
     
    Last edited: Jul 13, 2020
  8. Yacuzo

    Yacuzo

    Joined:
    Sep 22, 2016
    Posts:
    27
    The coroutine workaround will only work for playmode tests as [UnityTest] is not available for editor tests, I think.
     
  9. MajeureX

    MajeureX

    Joined:
    May 4, 2020
    Posts:
    13
    [UnityTest] attribute is available for EditMode and PlayMode tests, although with some limitations in the former.
     
  10. osum4est

    osum4est

    Joined:
    Jan 26, 2015
    Posts:
    13
    It sounds like Unity isn't going to add these any time soon so I've added them myself in my fork of the test framework: https://github.com/8bitforest/com.unity.test-framework

    It adds [AsyncTest], [AsyncSetUp], [AsyncTearDown], [AsyncOneTimeSetUp], and [AsyncOneTimeTearDown]
    The methods with these attributes will need to return "async Task". "async void" will not work:
    Code (CSharp):
    1. [AsyncTest]
    2. public async Task TestWithDelay()
    3. {
    4.     Debug.Log("Starting test...");
    5.     await Task.Delay(5000);
    6.     Debug.Log("Test finished!");
    7. }
    They are implemented using IEnumerator's in the backend just like the answers above, but this way improves the syntax so they are regular async methods, and should be easy to switch over when/if Unity implements them themselves.

    And as a bonus my fork also includes the mysteriously missing [UnityOneTimeSetUp] and [UnityOneTimeTearDown]
     
    Last edited: Feb 15, 2021
  11. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    TestTask:44
    m_Context.CurrentResult.RecordException(m_TestTask.Exception!.InnerException); => shouldn't be `?` instead of `!`
    Creates a compilation error. Or is it a c#8 trick ?

    But then it actually works, very greatful.

    One issue I've noticed is that tests doesn't fails when consol has errors. Not sure because wasn't able to reproduce it clearly
     
    Last edited: Feb 26, 2021
  12. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
    Another feedback, if the AsyncSetup fails with an exception then it's considered as "Passed" silently.

    I have the case while awaiting a task wrapping a RecompileScript that fails
     
  13. pushxtonotdie

    pushxtonotdie

    Joined:
    Oct 21, 2010
    Posts:
    111
  14. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    567
  15. ModLunar

    ModLunar

    Joined:
    Oct 16, 2016
    Posts:
    374
    Oh heavens, thank you!!!!!

    I was going to use Coroutines, but such tests have to 100% completely wait for each one to individually run.. which will just make my tests take forever and freeze the editor unnecessarily.

    <3 async / await
     
  16. ModLunar

    ModLunar

    Joined:
    Oct 16, 2016
    Posts:
    374
    Wait... your code example (method called TestWithDelay()) isn't working in my edit mode tests.
    upload_2021-3-30_13-57-44.png

    And no shame, here's my quick and dirty testing code (as used in the screenshot above)
    Code (CSharp):
    1. using System.Collections;
    2. using System.Threading.Tasks;
    3. using NUnit.Framework;
    4. using UnityEngine;
    5. using UnityEngine.TestTools;
    6. using UnityEditor;
    7.  
    8. namespace XXX.Editor.Tests {
    9.     public class XXXTests {
    10.  
    11.         [UnityTest]
    12.         public IEnumerator ExampleCoroutine() {
    13.             float startTime = (float) EditorApplication.timeSinceStartup;
    14.             while (EditorApplication.timeSinceStartup - startTime < 1)
    15.                 yield return null;
    16.             Debug.Log("!?");
    17.  
    18.             startTime = (float) EditorApplication.timeSinceStartup;
    19.             while (EditorApplication.timeSinceStartup - startTime < 1)
    20.                 yield return null;
    21.  
    22.             startTime = (float) EditorApplication.timeSinceStartup;
    23.             while (EditorApplication.timeSinceStartup - startTime < 1)
    24.                 yield return null;
    25.         }
    26.  
    27.         [UnityTest]
    28.         public IEnumerator ExampleCoroutine2() {
    29.             float startTime = (float) EditorApplication.timeSinceStartup;
    30.             while (EditorApplication.timeSinceStartup - startTime < 1)
    31.                 yield return null;
    32.             Debug.Log("!?");
    33.  
    34.             startTime = (float) EditorApplication.timeSinceStartup;
    35.             while (EditorApplication.timeSinceStartup - startTime < 1)
    36.                 yield return null;
    37.  
    38.             startTime = (float) EditorApplication.timeSinceStartup;
    39.             while (EditorApplication.timeSinceStartup - startTime < 1)
    40.                 yield return null;
    41.         }
    42.  
    43.         [AsyncTest]
    44.         public async Task TestWithDelay() {
    45.             Debug.Log("Starting test...");
    46.             await Task.Delay(5000);
    47.             Debug.Log("Test finished!");
    48.         }
    49.  
    50.  
    51.         [AsyncTest]
    52.         public async Task Example() {
    53.             await Task.Delay(1000);
    54.             Debug.Log("????");
    55.             await Task.Delay(1000);
    56.             await Task.Delay(1000);
    57.             await Task.CompletedTask;
    58.             Assert.Pass("!?");
    59.         }
    60.  
    61.         [AsyncTest]
    62.         public async Task NUnit1012SampleTest() {
    63.             bool result = await Task.FromResult(true);
    64.             Assert.That(result, Is.True);
    65.         }
    66.  
    67.         [AsyncTest]
    68.         public async Task<int> TestAdd() {
    69.             return await Task.FromResult(2 + 2);
    70.         }
    71.     }
    72. }
    73.  
    Note the 2 coroutine examples are the same -- I just wanted to verify that they couldn't run "at the same time".

    Did you too, convert the Tasks into a "coroutine representation", since it's telling me edit mode tests can only yield return null?
    I was thinking that you updated the NUnit Framework version to one that supports async Task.. but I appear to be incorrect.
     
    Last edited: Mar 30, 2021
  17. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    53
    I think you've misunderstood what async/await support in NUnit is intended for. It has nothing to do with parallel test execution, which is a completely different feature: https://docs.nunit.org/articles/nunit/technical-notes/usage/Framework-Parallel-Test-Execution.html

    However, that feature uses separate threads, and if you are doing anything that works with Unity APIs, it probably wouldn't work for you anyway, as they mostly require you to work on the Unity main thread. If you are not working with any Unity APIs, and want to put in the extra effort, you could have your code and tests live in standard .NET assemblies, and leverage all the NUnit 3 features available from a separate non-Unity project.
     
    ModLunar likes this.
  18. ModLunar

    ModLunar

    Joined:
    Oct 16, 2016
    Posts:
    374
    Ah...
    Right, I forgot about the implications of running on the Unity main thread, since these tests usually interact with UnityEngine/related API.

    I guess I see why they didn't prioritize it.. but it's still a stretch to say I still wouldn't want to test async (non-parallel) code in my Unity tests.
    I suppose if I wanna test it though, it'll need to be pure .NET/C# projects then.
    Thanks for explaining!
     
  19. ModLunar

    ModLunar

    Joined:
    Oct 16, 2016
    Posts:
    374
    Oh what I was thinking of was both async tests,

    and what I found today:
    [Parallelizable] marks unit tests in NUnit as ones that can run at the same time as others.
    upload_2021-5-6_2-51-22.png

    Since their docs note it requires NUnit 3.0, I assume it won't work with Unity Test Framework / Unity Test Runner though.
     
  20. MattLT

    MattLT

    Joined:
    Feb 22, 2016
    Posts:
    3
    @ModLunar TestFramework 1.1.24 is using NUnit 3.5 :

    From : https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/index.html

    Also, my problem is that for whatever reason, my test is taking something like 20 second to execute where in play mode, it's almost instant.

    I'm using the IEnumerator work around :

    Code (CSharp):
    1. Task task = sut.Load();
    2. while (!task.IsCompleted)
    3.     yield return null;
    4. if (task.IsFaulted)
    5.     throw task.Exception;
    It's so crazy that UTF is not supporting the async task test like NUnit does.

     
    Last edited: May 14, 2021
    ModLunar likes this.
  21. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,064
    MattLT and ModLunar like this.
  22. MattLT

    MattLT

    Joined:
    Feb 22, 2016
    Posts:
    3
    Omg indeed ! You talked about 3.0+ so I thought 3.5 was fine...

    Unity really needs to update this.. we can't test properly :(
     
    RageAgainstThePixel and ModLunar like this.
  23. friflo

    friflo

    Joined:
    Jan 16, 2016
    Posts:
    11
    My preferred way of running an async method in Unity Test Runner is using a custom SynchronizationContext.
    By doing so the same test runs in Unity and in .NET CLR.

    In short - when calling its Run() method it sets the SynchronizationContext.Current of the current thread (in Unity the UI thread) and execute the given test function (Func<Task> func). After the test finished the SynchronizationContext.Current is restored to the previous one.
    An example a test would look like this:

    Code (CSharp):
    1. [Test] public static void TestSync () {
    2.     SingleThreadSynchronizationContext.Run(async () => {
    3.         await Task.Delay(1);
    4.     });
    5. }
    A full explanation and the idea is from a Microsoft Blog.
    [Await, SynchronizationContext, and Console Apps | .NET Parallel Programming] https://devblogs.microsoft.com/pfxteam/await-synchronizationcontext-and-console-apps/

    The implementation of this fits into 50 lines of code:

    Code (CSharp):
    1. // [Await, SynchronizationContext, and Console Apps | .NET Parallel Programming] https://devblogs.microsoft.com/pfxteam/await-synchronizationcontext-and-console-apps/
    2. public class SingleThreadSynchronizationContext : SynchronizationContext
    3. {
    4.     private readonly BlockingCollection<ActionPair> queue = new BlockingCollection<ActionPair>();
    5.  
    6.     public override void Post(SendOrPostCallback callback, object state) {
    7.         var action = new ActionPair {callback = callback, state = state};
    8.         queue.Add(action);
    9.     }
    10.  
    11.     private void RunOnCurrentThread() {
    12.         while (queue.TryTake(out ActionPair action, Timeout.Infinite))
    13.             action.callback(action.state);
    14.         Console.WriteLine("RunOnCurrentThread exited.");
    15.     }
    16.  
    17.     private void Complete()
    18.     {
    19.         queue.CompleteAdding();
    20.         Console.WriteLine("SingleThreadSynchronizationContext Completed.");
    21.     }
    22.  
    23.     public static void Run(Func<Task> func)
    24.     {
    25.         var prevCtx = SynchronizationContext.Current;
    26.         try
    27.         {
    28.             var syncCtx = new SingleThreadSynchronizationContext();
    29.             SynchronizationContext.SetSynchronizationContext(syncCtx);
    30.  
    31.             Task funcTask = func();
    32.             funcTask.ContinueWith(t => syncCtx.Complete(), TaskScheduler.Default);
    33.  
    34.             syncCtx.RunOnCurrentThread();
    35.  
    36.             funcTask.GetAwaiter().GetResult();
    37.         }
    38.         finally { SynchronizationContext.SetSynchronizationContext(prevCtx); }
    39.     }
    40. }
    41.  
    42. internal struct ActionPair {
    43.     internal SendOrPostCallback  callback;
    44.     internal object              state;
    45. }
     
    Arkade and ModLunar like this.
  24. ModLunar

    ModLunar

    Joined:
    Oct 16, 2016
    Posts:
    374
    This looks neat!
    But to make sure I'm understanding correctly, does what you're describing allow us to make use of parallelizable tests?
    Or only tests that are blocking the main thread?
     
  25. friflo

    friflo

    Joined:
    Jan 16, 2016
    Posts:
    11
    The intension of this approach is to execute async methods in synchronous [Test] methods.
    As this approach utilize the calling thread for execution [Parallelizable] tests are possible with NUnit. At least I see no reason why not.
    I guess support of [Parallelizable] in Unity cannot be implemented as 'Edit Mode' tests are always expected to run on the UI Thread and [Parallelizable] require multi threading.
     
    ModLunar likes this.
  26. ModLunar

    ModLunar

    Joined:
    Oct 16, 2016
    Posts:
    374
    Gotcha, that makes sense now. Thanks for explaining :)
     
  27. thewerku

    thewerku

    Joined:
    Feb 6, 2020
    Posts:
    15
    Hi. Team owning the Unity Test Framework package (UTF, also known as Test Runner) has recently made available a pre-release version of v2.0 - you can read the announcement here. One of the new features we added, and are looking to get users' eyes on it, are async tests. We'd appreciate any feedback and comments, if you decide to test it out. Thanks!
     
    cloghead, Elapotp and Thaina like this.
  28. Karabin

    Karabin

    Joined:
    Apr 26, 2015
    Posts:
    8
    Thank you very much for your work! It was really needed for me
     
  29. Yacuzo

    Yacuzo

    Joined:
    Sep 22, 2016
    Posts:
    27
    Nice!
     
  30. jasursadikov

    jasursadikov

    Joined:
    Feb 25, 2020
    Posts:
    17
    Is there any progress on that? Does unity support Async Tests?
     
  31. ModLunar

    ModLunar

    Joined:
    Oct 16, 2016
    Posts:
    374
    If I recall correctly, the only version I've found to be (the most stable so far) that supports async tests is the following (snippet from my Packages/manifest.json file):

    Code (CSharp):
    1. "com.unity.test-framework": "2.0.1-exp.2"
    With that version, I can write async unit tests (just note they won't run in parallel) like this:

    Code (CSharp):
    1. [Test]
    2. public async Task DoExampleTest() {
    3.     //TODO: Your test code here...
    4. }
     
  32. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    Hey. We have backported the async support implementation to the 1.3.x versions of UTF. You can try it out in e.g. 1.3.4.
     
    leni8ec, ModLunar and bdovaz like this.
  33. ModLunar

    ModLunar

    Joined:
    Oct 16, 2016
    Posts:
    374
    Ooh awesome! Thanks for the update, nice to be off such an experimental version now haha :)