Feature Request Changing the bundled NUnit version

Currently, Unity automatically references a built-in NUnit assembly from the Unity installation folder.
I am not 100% sure which NUnit version is that, but is it going to be possible to drop whatever NUnit version i want so my tests can use that ?

 

The latest version is compatible with .NET Standard 2. I don't know if it still references System.Web in that assembly

https://www.nuget.org/packages/NUnit/

Also about your fork and modifications... Did you suggested a pull request at that time with that valid points so you could use the original assembly still? It will be better for everyone.

 

@ElvisAlistar @superpig I see that in 2019.2 alpha Test Runner it's already a separate package.

I have two questions:
1. Have you considered updating NUnit to the latest version? Now that it is "separated from Unity" (a separate package) will it make it easier for you to update NUnit from time to time and be able to be close to the latest version?
2. Will it ever be possible to test with async/await/Tasks? Please see: https://forum.unity.com/threads/async-await-in-unit-tests.671608/

 

I would also like to express my support for a more recent version of NUnit. Particularly because of the Order attribute for fixtures (instead of just methods) that was introduced in NUnit 3.8. This would be useful to ensure that e.g. the tests defined in SmokeTests are run first. Would also be nice to be able to toggle between different sort modes in the test runner (e.g. by Name, by Order)

 

I would like a newer version too. By now Unity's NUnit version is ancient (I cant even find a version that old on NUnit GitHub which goes 5 years back). I'd also like a .Net 4.5 build of it so I can run async tests. Maybe you can leverage the async support to handle coroutines?
The docs for v2 is also way worse than the v3 docs.

 

+1

I also would want to have async tests.

 

I'd be happy to just have

Assert.Multiple

.

 

+1

 

One of the reason why we desperately need an NUnit update on our side is to support async tests correctly. The embedded NUnit version does not accept test methods that returns Tasks, e.g. this does not run:

Code (CSharp):

1. [Test]
2. public async Task SimpleAsyncTest()
3. {
4.     var value = false;
5.  
6.     await Task.Delay(2000).ContinueWith((t) => {
7.         value = true;
8.     });
9.  
10.     Assert.IsTrue(value);
11. }

And using void is not an option since Nunit will think the test is done at the first await:

Code (CSharp):

1. [Test]
2. public async void SimpleAsyncTest()
3. {
4.     await Task.Delay(2000);
5.  
6.     UnityEngine.Debug.LogWarning("This executes after the test completes, making the test a false positive");
7.     Assert.IsTrue(false);
8. }

If Unity does not have the resources to maintain a custom Nunit version, is there a possibility to integrate another test runner ?

 

I've run into the same issue as remi-sormain-vrdirect mentioned. In Unity 2019.3.6f1 version and Test Framework package 1.1.13 version from March 17, 2020.
The tests with Task return type fails by default.
So I wrote task.Wait() in tests for async methods. It seems to work.

 

I would also prefer async unit tests now that Unity has better support for async features in general.

 

A year has passed since this reply. It seems like bandwidth should be made for this, it's quite painful to have such an old version of NUnit.

 

And another 3 months. Poking here in the hopes of getting a Unity dev's attention on this.

 

I assume you proposed a task var with Task.Run(Action to await for), then .Wait() on this task var and then use .Result().
That's not working to me 'cause it says my action to await for is only capable to be executed from the main thread, so I'm on a deadlock thus I can't test async code in my project.
The transient solution I came with is just return void so all tests pass but they log the exceptions. My decision relies on a proper solution updating NUnit bundled so Task can be returned where on a [Test] method.

 

I will too ask for this to be prioritised, lack of proper async/await support is holding us back from adopting it. Plus, with newer APIs like Addressables supporting the pattern and libraries like UniTask getting better with every update, I believe it would be a huge upgrade for many workflows.

 

+1
I really want this to be prioritised, [UnityTest] and other workarounds bloat tests too much and lead to garbage code

 

Another solution I have dwelt on is use SetUp/Teardown in such a heterodox way so SetUp subscribes to Unity console logs then Teardown spreads any error/exception log the test came up with, throwing a new exception, thus the test will not pass.
It doesn't work because Unity does not raise the log event when logging the test exception — having async void as tests signature, of course.

I tried both events regarding Application.LogCallback type.
https://docs.unity3d.com/ScriptReference/Application.LogCallback.html

May someone finds a way.

 

@superpig @Warnecke @ElvisAlistar I have a couple questions:
1. Is there any chance of this being prioritised in the mid-term?
2. Does this repository reflect the current state of the NUnit package? If not, is the current version source available somewhere?
3. If someone could update to a newer version of NUnit, would you be willing to accept a PR?

 

+1
We need answers ASAP. We have been years waiting and it's slowing us down.

 

Bump

 

Any updates regarding this @superpig ?

Alternatively, is there any user developers can choose their own version of NUnit and add it, instead of the one provided out-of-the-box ?

 

@superpig @Warnecke @ElvisAlistar It would be great if you could share some news about this. Our codebase is currently migrating towards using async/await and Tasks and having no test support is becoming a pain point.

 

While searching about a way to update NUnit version or even use a different solution as XUnit for instance, I just saw some of you guys complaining about not being able to use await Task, so here is how I'm creating my tests which require to await for Tasks.

There are other ways, but that is the one I most like since they could easily be changed when Unity finally gifts us with a new NUnit version.

Code (CSharp):

1. public class MyTests
2. {
3.     [UnityTest]
4.     public IEnumerator MyTestWithAwaitableTask() => TaskUtils.ToCoroutine(async () =>
5.     {
6.         Debug.Log(DateTime.Now);
7.         await Task.Delay(5000);
8.         Debug.Log(DateTime.Now);
9.     });
10. }
11.  
12. public static class TaskUtils
13. {
14.     public static IEnumerator ToCoroutine(Func<Task> taskFactory)
15.     {
16.         return taskFactory().AsCoroutine();
17.     }
18.  
19.     public static IEnumerator AsCoroutine(this Task task)
20.     {
21.         while (!task.IsCompleted)
22.         {
23.             yield return null;
24.         }
25.  
26.         if (task.IsFaulted)
27.         {
28.             throw task.Exception;
29.         }
30.  
31.         task.GetAwaiter().GetResult();
32.     }
33. }

 

Hey guys!
Any updates about ETA for any newer NUnit version for Unity Editor?

 

@remi-sormain-vrdirect for async tests -> just use Unitask it has an UniTask object which converts async method to IEnumerator - we use it for our test constantly (Android + Gitlab CI)

 

or if you don't want to add external dependencies to your project, this example does exactly what you said:

 

Hi everyone, we wanted to share an update regarding NUnit and Unity Test Framework (UTF) package (known also as Test Runner).

Currently UTF uses only a part of the NUnit framework (v3.5.0.0), such as its interfaces and attributes, however, most of the non-public parts of NUnit have been replaced with our own logic which constitutes a substantial part of our code.

We’re aware that there is a feature gap between newer NUnit versions and the one we’re using, and in reality it would be easier for our team to implement support for the missing use cases. That would include backporting any public parts of the NUnit features and implementing the logic on our end.

One of the cases that has been mentioned here is the support for async tests, which we’re addressing in the next release that we just announced today - check this thread for more information.

That being said, we understand there might be other requirements apart from the async support and would like to explore adding adequate support. What are some other features you’re missing in UTF, that are available in newer NUnit versions?

 

@thewerku Thank you for the update! Async testing support sounds great. I will give it a try as soon as I can.

As mentioned here, a coroutine or async version of OneTimeSetUp and OneTimeTearDown, similar to UnitySetUp and UnityTearDown would be nice, for instance, to execute an async Scene loading prior to some tests.