Search Unity

  1. We are looking for feedback on the experimental Unity Safe Mode which is aiming to help you resolve compilation errors faster during project startup.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
Feedback wanted - Unity Safe Mode Unite Now

Feature Request Changing the bundled NUnit version

Discussion in 'Testing & Automation' started by liortal, Mar 1, 2019.

  1. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,422
    Currently, Unity automatically references a built-in NUnit assembly from the Unity installation folder.
    I am not 100% sure which NUnit version is that, but is it going to be possible to drop whatever NUnit version i want so my tests can use that ?
     
    liortal, Mar 1, 2019
    #1

  2. ElvisAlistar

    ElvisAlistar

    Unity Technologies

    Joined:
    Oct 2, 2013
    Posts:
    226
    We are currently moving the Test Runner functionality into a package. The built-in NUnit is going to be a separate package referenced by the Test Runner one. So, in theory, you should be able to swap that one with any other NUnit version, but in practice, things might not work as expected if you do that or might not works at all.

    The reason is that the NUnit version we use contains some modifications compared to the original:
    - it removes System.Web.dll
    - changes some internal NUnit APIs to public, so that we could make our own test runner to use instead of their default one (required for dealing with platforms like WebGL, where multithreading isn't supported)
    - NUnit did not respect equality overloading, so checking for null on UnityEngine.Object and other things like that were added
    - maybe other things I can't remember

    I think the version of NUnit we currently use is 2.6.4 and we want to update to at least NUnit 3
     
    ElvisAlistar, Mar 9, 2019
    #2

  3. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    780
    The latest version is compatible with .NET Standard 2. I don't know if it still references System.Web in that assembly

    https://www.nuget.org/packages/NUnit/

    Also about your fork and modifications... Did you suggested a pull request at that time with that valid points so you could use the original assembly still? It will be better for everyone.
     
    Last edited: Mar 10, 2019
    bdovaz, Mar 10, 2019
    #3

  4. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,291
    @ElvisAlistar We currently ship Unit 3.5-ish.

    The biggest thing we changed in NUnit was restructuring the execution engine to make it possible to write tests as coroutines (i.e. support for the [UnityTest] attribute). I did look at whether there was a nice way to upstream this but couldn't come up with anything that I thought would be appealing to Charlie and the team. Maybe once it's all packaged up, it'll be easier to figure something out.
     
    superpig, Mar 10, 2019
    #4

  5. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    780
    @ElvisAlistar @superpig I see that in 2019.2 alpha Test Runner it's already a separate package.

    I have two questions:
    1. Have you considered updating NUnit to the latest version? Now that it is "separated from Unity" (a separate package) will it make it easier for you to update NUnit from time to time and be able to be close to the latest version?
    2. Will it ever be possible to test with async/await/Tasks? Please see: https://forum.unity.com/threads/async-await-in-unit-tests.671608/
     
    bdovaz, May 5, 2019
    #5

  6. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    60
    Hey.
    Updating nunit is something that we would like to look into when we have the bandwidth for it. However it is currently unknown for us how feasible it will be to do so and how much will break when we upgrade it. We cannot promise anything yet, but it is on our radar.
     
    Warnecke, May 24, 2019
    #6

  7. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    87
    I would also like to express my support for a more recent version of NUnit. Particularly because of the Order attribute for fixtures (instead of just methods) that was introduced in NUnit 3.8. This would be useful to ensure that e.g. the tests defined in SmokeTests are run first. Would also be nice to be able to toggle between different sort modes in the test runner (e.g. by Name, by Order)
     
    DrummerB, Jun 8, 2019
    #7
    liortal and Stormy102 like this.

  8. Yacuzo

    Yacuzo

    Joined:
    Sep 22, 2016
    Posts:
    20
    I would like a newer version too. By now Unity's NUnit version is ancient (I cant even find a version that old on NUnit GitHub which goes 5 years back). I'd also like a .Net 4.5 build of it so I can run async tests. Maybe you can leverage the async support to handle coroutines?
    The docs for v2 is also way worse than the v3 docs.
     
    Yacuzo, Aug 30, 2019
    #8

  9. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    780
    +1

    I also would want to have async tests.
     
    bdovaz, Aug 30, 2019
    #9

  10. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    40
    JesseSTG, Sep 6, 2019
    #10

  11. NHals

    NHals

    Joined:
    Dec 23, 2013
    Posts:
    10
    +1
     
    NHals, Sep 15, 2019
    #11

  12. remi-sormain-vrdirect

    remi-sormain-vrdirect

    Joined:
    Aug 29, 2018
    Posts:
    2
    One of the reason why we desperately need an NUnit update on our side is to support async tests correctly. The embedded NUnit version does not accept test methods that returns Tasks, e.g. this does not run:
    Code (CSharp):
    1. [Test]
    2. public async Task SimpleAsyncTest()
    3. {
    4.     var value = false;
    5.  
    6.     await Task.Delay(2000).ContinueWith((t) => {
    7.         value = true;
    8.     });
    9.  
    10.     Assert.IsTrue(value);
    11. }
    And using void is not an option since Nunit will think the test is done at the first await:
    Code (CSharp):
    1. [Test]
    2. public async void SimpleAsyncTest()
    3. {
    4.     await Task.Delay(2000);
    5.  
    6.     UnityEngine.Debug.LogWarning("This executes after the test completes, making the test a false positive");
    7.     Assert.IsTrue(false);
    8. }
    If Unity does not have the resources to maintain a custom Nunit version, is there a possibility to integrate another test runner ?
     
    Last edited: Oct 24, 2019
    remi-sormain-vrdirect, Oct 11, 2019
    #12
    DrummerB, Soulside and rc_unity97 like this.

  13. Soulside

    Soulside

    Joined:
    Nov 1, 2018
    Posts:
    11
    I've run into the same issue as remi-sormain-vrdirect mentioned. In Unity 2019.3.6f1 version and Test Framework package 1.1.13 version from March 17, 2020.
    The tests with Task return type fails by default.
    So I wrote task.Wait() in tests for async methods. It seems to work.
     
    Soulside, Apr 14, 2020
    #13

  14. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    87
    I would also prefer async unit tests now that Unity has better support for async features in general.
     
    DrummerB, Apr 29, 2020
    #14

  15. jstarrdewar

    jstarrdewar

    Joined:
    Aug 20, 2013
    Posts:
    1
    A year has passed since this reply. It seems like bandwidth should be made for this, it's quite painful to have such an old version of NUnit.
     
    jstarrdewar, May 17, 2020
    #15
unityunity