Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Official Unity Test Framework 2.0 ready for feedback

Discussion in 'Testing & Automation' started by thewerku, Jan 25, 2022.

  1. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    621
    So is there async support or not? I’ve come to find “UnitySetUp”, “UnityTest” and others are a bit limiting (no support for OneTimeSetUp which I want to use, only supports “ValueSource”) - if async just works then I would love to use that, but this makes me think that there’s no proper async support.

    Did you ever find out if the problem was because of Addressables or because of the Test Framework?
     
  2. tday_magicfuel

    tday_magicfuel

    Joined:
    Apr 10, 2018
    Posts:
    21
    @CaseyHofland I first moved all my async dependent tests into a separate asmdef so that I could run them less frequently (they would work but there was a massive delay).

    On a lark I decided to try the latest version of the unit test framework that popped into the package manager--this version was 2.0.1-pre.18 and there is nothing on the changelog a bout it.

    However with this update my async tests run great! With no real extra delay.

    SetUp and Teardown methods via the properties don't seem to be working right unfortunately--though that could certainly be my fault. So for the moment my tests are repeatedly calling setup and teardown methods manually (which does still work).

    Hope that helps.
     
    CaseyHofland and sandolkakos like this.
  3. GainfulSage

    GainfulSage

    Joined:
    Apr 30, 2015
    Posts:
    106
    So the version I'm using is 2.0.1-exp.2 which is labeled as having been released on November 14, 2022. This version is working for me now both in Rider and in Unity. The biggest thing I'm missing is async support for the one-per-class and once-per-test setup and teardown methods - individual test cases work with async but the rest of the methods don't.

    There is a version 2.0.1-pre.18, which Unity is labeling as more recent in the Package Manager, but it's labeled as having been released on January 24, 2022.

    I assume that Unity is looking at the .18 and the .2 when it's parsing the version string. Any chance of another release this year?
     
  4. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    Hello. We have backported most of the async support into the 1.3.x versions of UTF. There are still some parts not supported however, such async OneTimeSetUp and OneTimeTearDown.
     
    CaseyHofland likes this.
  5. GainfulSage

    GainfulSage

    Joined:
    Apr 30, 2015
    Posts:
    106
    @Warnecke thank you for the info!

    So, at this point is there any reason to use/prefer any of the 2.0.1-exp or preview builds? Or is 1.1.33 the latest/greatest/best?

    Right now I see the following versions:

    2.0.1-pre.18, released Jan 24, 2022
    2.0.1-pre.12 released Jan 19, 2022
    2.0.1-exp.2 released Nov 14, 2022
    1.1.33 released July 19, 2022

    So, from this it looks like the 2.0.1-exp.2 released last November is the most recent build. Any chance of a 2023 release?
     
  6. CDF

    CDF

    Joined:
    Sep 14, 2013
    Posts:
    1,324
    This is all so confusing.
     
    bdovaz likes this.
  7. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    621
    [1.3.9] released 2023-08-21
    async was backported to 1.3.x, so I wouldn't use 1.1.33.
    I'm on 1.3.9 myself right now and I have no notable complaints. Do note that not everything with async is supported, read more here:
    https://docs.unity3d.com/Packages/com.unity.test-framework@1.3/manual/reference-async-tests.html
     
    sandolkakos likes this.
  8. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    I do not see that we will see 2.0 as it is now being released. It will be staying in experimental. We do however want to resolve this by backporting fixes and improvements into 1.3 when possible (like we did with async) and hopefully resolve this over time. We cannot make any promises or give any timelines, but it is something we want to get fixed.

    If 2.0 is the version that works best for you, then go ahead and use it for now, but we do recommend 1.3, as it is the supported one.
     
    sandolkakos and CaseyHofland like this.
  9. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    621
    It would be great if this statement could be added to either the 1.3 docs introduction page as a note, or the 2.x introduction page as a warning. Great that new features are being backported, good luck trying to migrate, can't imagine it being easy when every other Unity packages relies on you.
     
    sandolkakos likes this.
  10. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    @Warnecke is v2.0 going to be released or updated any time soon ?
     
  11. krs-two

    krs-two

    Joined:
    Aug 27, 2021
    Posts:
    20
    So 2.0 will not be released, at least not in its current state.
    I only see work being done on 1.3, a version which appears to be released, yet the Package Manager gives no hint of this and forces the use of 1.1 (at least in Unity 2022.3.10).
    Which version am I supposed to use in production scenarios?
     
  12. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    621
    I would strongly recommend using 1.3.9, it's the most up to date version, works, and has (limited) async support.
     
  13. krs-two

    krs-two

    Joined:
    Aug 27, 2021
    Posts:
    20
    Thanks, but what's the official way to install 1.3, since 1.1 is the only available option in the package manager?
     
  14. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    621
    hommfan and RDeluxe like this.
  15. krs-two

    krs-two

    Joined:
    Aug 27, 2021
    Posts:
    20
    Thanks again, I was afraid something like that would be the answer. Doesn't sound very official nor released and it opens up a whole new range of questions about dependencies. Anyhow, guess that is how I'll have to proceed.
     
  16. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,063
  17. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    621
    It’s definitely supported, newer Unity packages already have it as a dependency.
     
  18. Xtro

    Xtro

    Joined:
    Apr 17, 2013
    Posts:
    614
    Why can't I see version 1.3.7 in the package manager of my Unity 2022.2.13 project?

    Why do I have to manually edit the packages file?

    (By the way, 1.3.9 exploded with spam of errors in the console so I used 1.3.7 instead)
     
  19. Xtro

    Xtro

    Joined:
    Apr 17, 2013
    Posts:
    614
    Turns out, TestRunner 1.3.9 works well with Unity 2022.2.13 after doing a full re-import of the entire project. At least for now, it works...
     
    sandolkakos likes this.
  20. RDeluxe

    RDeluxe

    Joined:
    Sep 29, 2013
    Posts:
    117
    Thanks ppl, the whole "I can't see new versions in the UPM" + "why is 2.0 older than 1.3" got me really confused.

    Unity staff ought to do something to clarify this situation
     
  21. callumroseFC

    callumroseFC

    Joined:
    May 10, 2019
    Posts:
    7
    Is work still going ahead on this? Big fan of async tests but appears to be no new releases for over a year
     
  22. sandolkakos

    sandolkakos

    Joined:
    Jun 3, 2009
    Posts:
    285
    Using the version "1.3.9", I was able to run async Task methods normally, but only in Play Mode Tests.
    Editor Mode most of the time will hang.
     
  23. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    We are not going to release any new versions of the 2.0 preview / experiment. The recommended version to use is 1.4 https://forum.unity.com/threads/utf-1-4.1527514/
    We will in general try to backport a lot of the changes and improvements that has been tried out in 2.0. E.g. we have backported the whole async solution when we released 1.3 and with 1.4 we have now backported a lot of the UI improvements and performance fixes.
     
  24. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    621
    Interesting. Why was this decision made? What are the benefits and drawbacks? Will we still get all the improvements introduced in 2.0 or only some?
     
  25. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    93
    The decision to not continue work on 2.0 was done because of the issues validating and releasing it internally. We simply had too many test projects that was working on 1.x, but failing on 2.0 unintendedly. Due to that, we choose the approach of uptaking the 2.0 changes one by one. This ensures that we can validate each individual change against all of our internal test projects. The drawback is that is taking longer time, but the upside, is that the end results is of a higher quality.
    Our intension is to backport the vast majority of changes in 2.0 back into a 1.x version. Some features, such as CustomRunner and CustomBuilder, which was implemented to support DOTS' workflows will not be backported, as they are no longer needed. We will keep you updated, when we know of features or improvements that cannot be backported.
     
    bdovaz and CaseyHofland like this.
  26. eventropy

    eventropy

    Joined:
    Oct 4, 2012
    Posts:
    258
    I tried this with version 1.3.9 but couldn't get it working. The only way I found is to use IEnumerator then yield an Task/Awaitable instead
     
  27. sandolkakos

    sandolkakos

    Joined:
    Jun 3, 2009
    Posts:
    285
  28. TrueJollyRoger

    TrueJollyRoger

    Joined:
    Dec 23, 2017
    Posts:
    9
    >Async tests
    Oh. Bless your kind heart!
     
  29. chin13577

    chin13577

    Joined:
    May 8, 2017
    Posts:
    18
    Greeting.

    I'm using Unity 2022.3.19f1
    and I use "com.unity.test-framework": "1.4.4"
    (Test Framework version 1.4.4)

    and it's work.

    the function need to be
    public async Task .....


       [Test]
    public async Task TestGenericAsync()
    {
    Assert.IsTrue(await GenericAsync());
    }

    private async UniTask<bool> GenericAsync()
    {
    Debug.Log("test 1");
    await UniTask.Delay(1000);
    Debug.Log("test 2");
    return true;
    }


    PS. I'm using UniTask and it still work :D
     
    sandolkakos likes this.