Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Official Unity Test Framework 2.0 ready for feedback

Discussion in 'Testing & Automation' started by thewerku, Jan 25, 2022.

  1. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    1,226
    OK then I am very sorry, I didn't see what part you have specifically replied to. Thank you for your suggestion
     
  2. Yacuzo

    Yacuzo

    Joined:
    Sep 22, 2016
    Posts:
    27
  3. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    586
  4. WilderBurn

    WilderBurn

    Joined:
    Aug 19, 2021
    Posts:
    5
    Hey, thanks for raising awareness, however yes I am using that.
    Code Coverage reporting is great, however this is not unit test reporting (JUnit). Code Coverage reporting I have also transformed to the Cobertura format to allow for Code Coverage Visualization in Merge Requests in Gitlab.

    Code (CSharp):
    1. dotnet tool install -g dotnet-reportgenerator-globaltool
    2. reportgenerator "-reports:coverage.xml" -reporttypes:Cobertura "-targetdir:report"
    3. sed -i "s|<UNITY_DIRECTORY>/||g" report/Cobertura.xml
    The above will convert the coverage report to cobertura, I'll leave as an exercise for the reader to place in their own "coverage.xml" files that are produced and then using 'sed' to convert file paths so they can be matched up in gitlab.
     
  5. HamledLLC

    HamledLLC

    Joined:
    Oct 13, 2021
    Posts:
    3
    It looks like
    TestContext.CurrentContext.Random
    from NUnit is null when running tests with Unity Test Framework (because
    UnityTestExecutionContext.RandomGenerator
    does not get set, I guess?).

    Is there any intention to support that in the future, or otherwise is there a recommended approach to getting e.g. random strings for use in test cases?
     
  6. glenneroo

    glenneroo

    Joined:
    Oct 27, 2016
    Posts:
    233
    What about an option for a timeout or duration parameter in the Assert methods e.g. Assert.IsTrue(expected, message, Timeout.timeout(1000)) which would wait up to 1 second for 'expected' to be true? It could save a lot of while() loops inside IEnumerator methods waiting for a status to be true before performing the assertion.

    AssertJ for Java has this functionality and it's a huge time-saver for component/system tests which may run slower while running on a VM (or in this case via the Editor).
     
    Thaina likes this.
  7. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    53
    What is the type of
    expected
    here? You can't really do that with plain values, as arguments get evaluated before they are passed to a method. The only timeouts I could find in the link you shared were related to future completion, so if you are working with
    Task
    s you might want to check out this blog post.

    The other option is to use "getter lambdas" (of type
    Func<T>
    ), e.g.
    () => expected
    . This is how waits are implemented in Responsible, sounds like you might want to check it out.
     
    glenneroo likes this.
  8. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    236
    I updated this package, and I'm pleased to report that my test suite worked without a hitch. No need to change my code, and no errors (besides those already caused by my project). Well done!

    I would appreciate if the
    internal
    classes that comprise the test runner itself were made public so I didn't have to screw around with reflection. Ditto for the Editor-exclusive test runner classes. The reason for that is I have a custom test runner for my project, with its own build process.

    Also, I have a question about
    async
    tests. Must they specifically return System.Threading.Task, or can they return any
    await
    able type? I ask because I'd like to use UniTask so I can take advantage of its Unity-specific optimizations.
     
    Last edited: Apr 10, 2022
  9. thewerku

    thewerku

    Joined:
    Feb 6, 2020
    Posts:
    15
    Hi everyone, thank you for your continued feedback. I wanted to share an update on where we’re at with the 2.0 version of the package.

    At this time, we recognise that we won’t be able to fully release the UTF 2.0 version with Unity 2022.2 later this year due to unforeseen technical challenges that we recently discovered.

    That being said, we still want you to have access to the updated version and so we decided to release a new experimental version of the package. This way we can continue to learn about your experiences and understand what should be improved.

    Please note that at this point we don’t have any new timeline for the full release. Once we do, we will make sure to keep you updated.

    For those of you who are unfamiliar with what Experimental packages are, they are defined as:
    Experimental packages are new packages or experimental modifications made to mature packages. Unity does not support Experimental packages because they are in the early stages of development.
    You can read more about them in this article.

    How to enable the experimental version of UTF 2.0 ?
    If you’re using Unity 2019.4 and beyond, open your package.json and update the com.unity.test-framework line to
    “com.unity.test-framework”: “2.0.1-exp.1”


    We’re leaving this thread open to discuss any new feedback. Once again, thank you for trying out the new features and engaging with us here.
     
    Last edited: May 6, 2022
  10. CDF

    CDF

    Joined:
    Sep 14, 2013
    Posts:
    1,324
    There's a typo here, should be: "com.unity.test-framework": "2.0.1-exp.1"

    Also, little confused on versions. Is 2.0.1-exp.1 the latest? or is 2.0.1-pre.18?

    test-framework.png

    Package manager says 2.0.1-pre.18 is latest, but was updated Jan 24th 2022.
    2.0.1-exp.1 was updated April 22, 2022, but is not the latest?

    Very confusing.

    If I want to continue using Test Framework 2.0 on Unity 2021.3.0. What version should I use?
     
    GainfulSage and Thaina like this.
  11. thewerku

    thewerku

    Joined:
    Feb 6, 2020
    Posts:
    15
    Thanks for pointing the typo (fixed now) and for raising this question. The answer is - if you want to continue using the new features, please use 2.0.1-exp.1 and please keep in mind that by definition there's no support and maintenance for experimental packages, which means we are not actively working on it or fixing bugs there.
     
    CDF likes this.
  12. booferei

    booferei

    Joined:
    Sep 29, 2016
    Posts:
    21
    I've noticed a strange (buggy?) behavior with the Timeout attribute. When the test times out the coroutine is not stopped. Instead, the yield instructions merely return without waiting and the coroutine code runs until its end.

    Not only is this unexpected, but it also causes code like this to break:
    Code (CSharp):
    1. while (!myTestSuccessCondition)
    2.     yield return null;
    If the test times out before the condition is met then the loop becomes an infinite loop that causes the Unity editor to freeze.
     
    glenneroo likes this.
  13. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    236
    Can you tell us more about these unforeseen technical challenges? I'm curious.
     
  14. pohype

    pohype

    Joined:
    Apr 12, 2019
    Posts:
    72
    @thewerku I tested the preview package version with Unity Cloud Builds and it blocks the completion of the build, something runs endlessly.
     
  15. thewerku

    thewerku

    Joined:
    Feb 6, 2020
    Posts:
    15
    @booferei Thanks for sharing. We know there are issues with the [Timeout] attribute in general, so I'm adding your example for the team.

    @JesseSTG It's mainly connected to the processes we have in place for testing and releasing our own package code. UTF is such an intrinsic part of testing - also for our internal code - that it makes it difficult to release breaking changes (e.g. API changes) without having to spend a lot of time mitigating breakage risks with other teams and their products. We want to make sure we're able to bring you new technology needed for testing, and at the same time ensure the other parts of the engine work without disruption.

    @poprev-3d Are you able to provide any details or repro steps, so that I can log this issue in our system?
     
  16. HamledLLC

    HamledLLC

    Joined:
    Oct 13, 2021
    Posts:
    3
    Not sure if this addresses the question, but I've had success using an async test function returning System.Threading.Task and then awaiting UniTask values within it.
     
    JesseSTG likes this.
  17. NoTuxNoBux

    NoTuxNoBux

    Joined:
    Oct 2, 2020
    Posts:
    35
    I'm working on an enterprise (meaning B2B, so not a game) project using the MRTK and XR with over 100 scene tests (and a minor amount of unit tests) and have some feedback that may be useful.

    Native async/await (TAP) for scene tests
    Native async/await support is something I had been waiting for, since we've embraced .NET's TAP along with multi-threading for code that does not need to run on the main thread due to Unity constraints, meaning most non-Unity-specific .NET code such as HttpClient, File IO, and others. Unfortunately, if I understand correctly, this feature does not currently extend to Unity scene tests, but only works with unit tests?

    A pain point we currently feel in Unity is that scene tests only support coroutines, and things such as async HTTP requests running in the background during scene tests cannot be awaited, which sometimes needs to happen in between parts of other test logic. A workaround we currently use is an enumerator function allowing us to
    yield return WaitForTask(someTask)
    - it would be easier if we could
    await someTask
    instead.

    Since scene tests currently need to return IEnumerable, this might require supporting asynchronous generators in C# to allow combining them with async, since LoadScene still uses coroutines, and the reverse of awaiting an enumerator is non-intuitive since they are producers that can theoretically produce infinitely (they solve a different problem, i.e. "how many values would you await").

    Automatic timeout of tests
    As was already touched on above, when you start working with tasks in tests, you risk things taking too long, potentially infinitely long. (Async) tests should time out after an interval that can be configured, but is also applied by default, to avoid having to configure this everywhere (e.g. what Jest does), and is set to a sane default (e.g. 10 or 15 seconds, not counting the time it takes for Unity to bootstrap them, since that time is often substantial, see below).

    Currently we sometimes work around this by using the WaitForTask above combined with a timeout.

    Scene test performance
    I was personally excited to try 2.x since it mentions several performance improvements, but I think those must have been primarily for unit tests, since our test suite with mostly scene tests performs noticeably worse in 2.x. These aren't exact numbers, but where it took around a minute or two before, it has gone up to sometimes over 10 minutes. It seems to be moving from test to test more slowly than before, perhaps because more state is being restored.

    More generally, it would be neat if the performance of bootstrapping scene tests could be improved. As alluded to above, oftentimes the first part of a scene test is to actually load a scene. Most of our scene tests load the same scene, so perhaps being able to configure that scene for Unity through attributes or something might allow Unity to more efficiently prepare/reset/copy/bundle tests for the same scene.

    Alternatively or additionally, running (scene and/or unit) tests or test suites fully in parallel should also improve performance, since they - at least in our case - always start from a clean slate (meaning loading a fresh scene). Without this, 15 of the 16 cores in my machine are sitting there unused in most cases. Jest only runs test suites in parallel, for example, but it would be even better if each test could run separately (but even just having putting each test class in a single unit and parallelizing between them could yield a large improvement).

    Finally, I've got a feeling this also ties into the story with CI/CD. As mentioned above it's possible to run Unity in containers unofficially (and by the way, it's secretly shifted back from In Progress to Under Consideration on the roadmap), but scene tests are slow on cloud runners such as the ones used by GitLab. For us, this means over 50 seconds of waiting for each scene test to start due to deserialization. Using non-cloud runners improves the situation, but costs extra for us and it's still pretty bad (20 seconds for each scene test with an Azure Debian VM).

    I realize CI/CD and containers are not directly related to this package, but in comparison with other standard CI/CD flows in the Node.JS or .NET world, waiting for an hour in CI for some 100 scene tests to run is demotivating, to the point that we've just disabled CI for Unity projects and run tests locally again, losing the benefit of CI and often sneaking in broken code because a developer forgot to run the tests (or "was sure nothing would break", and "didn't want to have to wait 10 minutes again").

    Improvements for service containers
    Since we're building enterprise applications, we use dependency injection to facilitate inversion of control and often work with non-MonoBehaviour code as well. We currently do this using Zenject (mostly due to not having much choice, as it has to fit into Unity's workflow using MonoBehaviours, which most standard .NET frameworks don't), and it has some convenience functions for working with the container in scenes. It would be nice if there was better support out of the box for dependency injection containers. For example, as you can see from the links, it is for some reason still necessary to clean up "leftovers" from previous tests for Zenject, even though Zenject just works by having a container as component on a Game Object in the scene, and each test reloads the scene and should be running in isolation.

    This is also part of a larger problem where, because Unity does not support dependency injection or proper service containers out of the box, code is often static or uses singletons (I'm looking at you, Unity localization package), and it somehow maintains state between scene tests even if the entire scene is reloaded, as the objects are not destroyed in between them (at least this seemed to be the case in 1.x).

    (Since Unity does it, 3rd party projects also usually just use singletons and static code, and this type of static code proliferates through the entire Unity ecosystem unless care is taken by the author. Ideally Unity would support proper dependency injection for MonoBehaviours and a service container out of the box, but this is not directly on-topic and I understand this is not necessary for all types of projects.)
     
    Last edited: Jun 9, 2022
    nehvaleem, Elapotp, Arkade and 7 others like this.
  18. Yacuzo

    Yacuzo

    Joined:
    Sep 22, 2016
    Posts:
    27
    I second all of this. We are also doing business applications with the same packages/tools.
     
  19. Simon186

    Simon186

    Joined:
    Jul 14, 2022
    Posts:
    1
    Support for await/async for testing in this release works fine, but non-Unity SetUp function with async/await doesn't seem to be supported/or just doesn't work well !? Ideally, we could use await in UnitySetUp, so that a one-time setup before tests with asynchronous function can be supported. Non-Unity SetUp with this may be another solution.
     
  20. JackAlta

    JackAlta

    Joined:
    Mar 31, 2022
    Posts:
    2
    Very excited about being able to write async tests.

    Very sad about the Timeout or MaxTime attributes not working.
     
  21. sramosubm

    sramosubm

    Joined:
    Nov 20, 2018
    Posts:
    1
    I need to call an async method from OneTimeSetUp, but this makes Unity hang (using version 2020.3.24). For example, this breaks Unity:

    Code (CSharp):
    1. [OneTimeSetUp]
    2. public async Task OneTimeSetupAsync()
    3. {
    4.     await Task.Delay(1);
    5. }
    Is this not supported in Unity? Because according to NUnit documentation it is valid.
    upload_2022-8-23_15-54-20.png
     
    GainfulSage likes this.
  22. ted05051997

    ted05051997

    Joined:
    May 7, 2020
    Posts:
    4
    Hello, I am trying to update my test package to 2.0.1-exp.1 but can't. Using Unity 2021.3.8f1.
    First I tried the change line approach.

    I tried to modify Packages/Test Framework/package.json but nothing happened, the json will restore my changes back.
    1) I added the line it directly under
    "upmCi": {
    "footprint": "2279493b9acd323ab758f64369634dc0daf0dbac"
    },

    "com.unity.test-framework": "2.0.1-exp.1"



    2) I replaced "com.unity.test-framework.tests": "1.1.33" to "com.unity.test-framework": "2.0.1-exp.1",
    3) replaced "com.unity.test-framework.tests": "1.1.33" to "com.unity.test-framework.tests": "2.0.1-exp.1"
    I don't know which line to change, here is my package.json
    Code (JavaScript):
    1. {
    2.   "name": "com.unity.test-framework",
    3.   "displayName": "Test Framework",
    4.   "version": "1.1.33",
    5.   "unity": "2019.2",
    6.   "unityRelease": "0a10",
    7.   "description": "Test framework for running Edit mode and Play mode tests in Unity.",
    8.   "keywords": [
    9.     "Test",
    10.     "TestFramework"
    11.   ],
    12.   "category": "Unity Test Framework",
    13.   "repository": {
    14.     "url": "https://github.com/Unity-Technologies/com.unity.test-framework.git",
    15.     "type": "git",
    16.     "revision": "34a4d423d64926635eb36d3bb86276179cc186e1"
    17.   },
    18.   "dependencies": {
    19.     "com.unity.ext.nunit": "1.0.6",
    20.     "com.unity.modules.imgui": "1.0.0",
    21.     "com.unity.modules.jsonserialize": "1.0.0"
    22.   },
    23.   "relatedPackages": {
    24.     "com.unity.test-framework.tests": "1.1.33"
    25.   },
    26.   "_upm": {
    27.     "changelog": "- Fixed an issue where using Assert.Expect with the same string multiple times can lead to incorrect errors in some cases (DSTR-442).\r\n- Improved the logging when using multiple Assert.Expect that the logs appear in another order than expected (DSTR-442).\r\n- Moved the targetPlatform specified when running tests in the TestRunnerApi from the Filter to the ExecutionSettings (DSTR-186).\r\n- Fixed an issue where an inheritance of UnityPlatformAttribute which was not working (ESTT-70).\r\n- Fixed the log of excluded platforms which was not displaying the right information.\r\n- Added filename and linenumber to test finished message (DSTR-505).\r\n- Add the possibility of running tests in a specified order from a test list (DSTR-494)."
    28.   },
    29.   "upmCi": {
    30.     "footprint": "2279493b9acd323ab758f64369634dc0daf0dbac"
    31.   }
    32. }
    Then I tried to find it in package manager.

    I enabled the "Enable pre-release packages" in settings
    But I can't see the pre-released Test-Framework package, using Unity 2021.3.8f1
    upload_2022-8-26_11-27-14.png
     
    Last edited: Aug 26, 2022
  23. NoTuxNoBux

    NoTuxNoBux

    Joined:
    Oct 2, 2020
    Posts:
    35
    The instructions might be wrong. I think you have to update
    manifest.json
    in your project, not
    package.json
    in one of the package files.
     
    ZenUnity and ted05051997 like this.
  24. ted05051997

    ted05051997

    Joined:
    May 7, 2020
    Posts:
    4
    It works!!!!!Thank you very much!!!!;)
     
    NoTuxNoBux likes this.
  25. mhgamework

    mhgamework

    Joined:
    Dec 13, 2014
    Posts:
    14
    Isn't using Tasks for doing concurrency in testing a very bad idea, because the Tasks model is inherently designed around being a multi-threaded generator? Existing framework methods that use tasks potentially start new threads and maybe run the continuations on the same thread?
    Why is the test runner trying to re-purpose Tasks instead of using another type of awaitable?

    In our game Starship Troopers - Terran Command, we are using a custom awaitable to replace coroutines which perfectly fills this purpose. Something like this fork on my github: https://github.com/mhgamework/AwaitableCoroutines.
    Credit to https://github.com/tomblind/unity-async-routines because im using a modified version of that repo
     
  26. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    586
    Tasks are used for asynchronous operations and are heavily used in .Net. Tasks don't have to be concurrent. Yes, other awaitable types work, but it's best for the package to work on the standard, and you can always do this:

    Code (CSharp):
    1. [Test]
    2. public async Task MyTestAsync()
    3. {
    4.     await MyTestAsyncImpl();
    5.  
    6.     async MyCustomAwaitable MyTestAsyncImpl()
    7.     {
    8.         // Test code
    9.     }
    10. }
    Whether a test is safe to run concurrently is up to the test writer to decide. Sometimes tests should run concurrently to speed up the test process.

    Also, just because a test is async, doesn't mean the test runner has to run other tests at the same time.
     
    Last edited: Oct 18, 2022
  27. NoTuxNoBux

    NoTuxNoBux

    Joined:
    Oct 2, 2020
    Posts:
    35
    Indeed. Also, there are several classes and libraries in .NET and the .NET ecosystem that work with Task by default (e.g. HttpClient, also used by things such as the OpenAPI client generator).

    Tasks can cause problems in Unity if you try to call Unity methods off the main thread, such as manipulating transforms, in which case Unity will throw an exception. Unity also doesn't seem to stop running tasks when you exit play mode (can be worked around by using cancellation tokens, which are good practice in TAP anyway).

    If you do non-Unity things off the main thread, there is no problem, and you can program asynchronously and run things off the main thread (when desired) to speed things up on certain systems. I don't think Unity coroutines can perfectly fill the Task gap out of the box since they still run on the main thread, which is probably their raison d'être (making code async, but not multithreaded) - in other words, you are still (partially) taxing the main thread with your async code. Put differently, you are running async code without multiple threads, as opposed to async code with multiple threads, which Tasks offers you access to - but doesn't require you to do.

    Finally, TAP's
    await
    syntax is (subjectively) more intuitive to read as it more closely mimics non-asynchronous code than coroutines are. (Arguably, though, you might be able to adapt Unity coroutines to be awaitable and still benefit from it, as the mentioned library does.)
     
  28. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Are there any updates to UTF v2.0 ? last update was in January
     
    sandolkakos, Dasp and bdovaz like this.
  29. Elapotp

    Elapotp

    Joined:
    May 14, 2014
    Posts:
    98
    Actually the last update was recently — Nov, 2022.
    Checkout 2.0.1-exp.2.

    The only thing is that I don't see changelog for experimental versions
     
    sandolkakos likes this.
  30. R2-RT

    R2-RT

    Joined:
    May 8, 2019
    Posts:
    38
    I''ve just switched to UTF 2.0 and I just wanted to share that I really like that playmode and editmode tests can be run all at once.

    What is even nicer (but not documented) it can be done via command line as well, by passing:

    -testPlatform editmode -assemblyType "EditorOnly,EditorAndPlatforms"

    assemblyType is passed to Enum.Parse, which works fine for comma delimed [Flags] enumeration.

    I think it should be reflected in docs, under assemblyType section that "EditorOnly,EditorAndPlatforms" is also valid.
    https://docs.unity3d.com/Packages/com.unity.test-framework@2.0/manual/reference-command-line.html

    About the [TestScene] attribute, I can add few cents, since I've been using local implementation of it (in modded UTF package) and my main reason to have it was extra "Open corresponding scene" context menu, very handy when test was failing and required checking.

    upload_2022-11-22_16-53-21.png
     
  31. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    586
    @superpig I managed to get standalone tests working in docker by stripping out the test framework and including
    nunit.framework.dll
    and
    nunitlite.dll
    , unmarking the assembly as test assembly to be included in the build, and running the tests through NUnitLite via script. I just built the player like normal, then ran the built player with arguments passed to NUnitLite in the script. Unfortunately, that means I can only test regular NUnit tests, and [UnityTest] does not work.

    Could the Unity test framework provide similar functionality to just build the test player without running, so that it can then be ran directly without relying on the
    PlayerConnection
    with the editor? Or an option to just build and run it directly without connecting to the editor?


    Code (CSharp):
    1.  
    2. - name: Build Project
    3.   uses: game-ci/unity-builder@v2
    4.   env:
    5.     UNITY_LICENSE: ${{ secrets.UNITY_LICENSE_2019_X_WINDOWS }}
    6.     UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
    7.     UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
    8.   with:
    9.     projectPath: IL2CPP_Tests/IL2CPP_Tests_${{ matrix.unityVersion }}
    10.     targetPlatform: StandaloneWindows64
    11.  
    12. - name: Run Tests
    13.   run: |
    14.     Start-Process -Wait -FilePath "build\StandaloneWindows64\StandaloneWindows64.exe" -ArgumentList "-batchmode `
    15.    --trace=Info `
    16.    --labels=All `
    17.    --stoponerror `
    18.    --result:unity-test-results-il2cpp-${{ matrix.mode }}-${{ matrix.progress }}-${{ matrix.unityVersion }}.xml `
    19.    --output=unity-test-results-il2cpp-${{ matrix.mode }}-${{ matrix.progress }}-${{ matrix.unityVersion }}.log"
    20.  

    Code (CSharp):
    1. using UnityEngine;
    2. using NUnitLite;
    3. using ProtoPromiseTests;
    4. using System.Collections.Generic;
    5.  
    6. public class TestRunner : MonoBehaviour
    7. {
    8.     void Start()
    9.     {
    10.         // Command line args include the executable name and unity args, so we need to strip them to only include nunit args.
    11.         string[] args = System.Environment.GetCommandLineArgs();
    12.         int i = 0;
    13.         while (i < args.Length)
    14.         {
    15.             if (args[i].StartsWith("--"))
    16.             {
    17.                 break;
    18.             }
    19.             ++i;
    20.         }
    21.         List<string> nUnitArgs = new List<string>(args.Length - i);
    22.         for (; i < args.Length; ++i)
    23.         {
    24.             nUnitArgs.Add(args[i]);
    25.         }
    26.         int quitCode = new AutoRun(typeof(TestHelper).Assembly).Execute(nUnitArgs.ToArray());
    27.         Application.Quit(quitCode);
    28.     }
    29. }
     
    Last edited: Dec 16, 2022
  32. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,663
    ProtoTerminator likes this.
  33. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    555
    Hi, I met the same problem. Is there any way to resolve it?
     
  34. tianzhizhi

    tianzhizhi

    Joined:
    Jul 21, 2021
    Posts:
    3
    sandolkakos likes this.
  35. RGV

    RGV

    Joined:
    Jan 25, 2016
    Posts:
    48
    I've comprehensively used 2.0 exp package for a while now — I think a year or so.

    Minor
    I've found some QoL problems, but I've dealt with them until they are totally naturalized within my workflow. No big deal.
    On of them is the popular exception it throws sometimes when both editor and play tests run and it gets crazy about the fixture names. Just a re-run and the problem goes away.
    The same with other out-of-sync problems with Rider. You restart them, then they work again.

    Critical
    Maybe this is the worst breaking change I've found in regard to third-party plugins: runtime generated GameObject has change its name, from "Code-based tests runner" to "test runner". This was sort of a some-days-in-a-row obstacle to us since it breaks Zenject fixtures (which specifically looks for "Code-based tests runner" named object in scene and you as a Zenject user can't change that const).

    Mayor
    Also I think TDD worfklow is totally messed up with Rider, since play tests from Rider, whether editor and/or play mode, often ends with "Inconclusive" result after some seconds, thus you have to play the fixture twice each time you are TDDing and the loading time makes this unfeasible.
    BUT I think this is not an exclusive problem of 2.0, just its worse in the exp package imhe.


    Hope this feedback helps! I have plenty more to say but I don't want this report to be endless. :)
     
  36. NoTuxNoBux

    NoTuxNoBux

    Joined:
    Oct 2, 2020
    Posts:
    35
    RGV's posts reminds me of something that would be very nice to have: the ability to run tests externally from Unity whilst Unity is open.

    Currently you can run tests externally already, but it will complain that Unity is already open, I believe, because you can't open the same Unity project multiple times. It would be nice if there was an exception to this for tests (and building, for that matter), so you can at least call the Unity CLI on the project from VSCode or another editor to run tests programmatically more easily without leaving your editor/IDE.
     
    RGV, nowsprinting and sandolkakos like this.
  37. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    19
    RGV and sandolkakos like this.
  38. MiTschMR

    MiTschMR

    Joined:
    Aug 28, 2018
    Posts:
    523
    Version 2.0.1-pre.18 has the following error with 2023.1b07 (didnt' try lower versions of Unity):


    Library\PackageCache\com.unity.test-framework@2.0.1-pre.18\UnityEditor.TestRunner\TestLaunchers\PlatformSetup\XboxOnePlatformSetup.cs(5,17): error CS0619: 'XboxOneDeployMethod' is obsolete: 'The XDK Xbox One platform was removed in 2021.1'
     
  39. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Are there any updates on v2.0 release? from what i can see, last update was released September 2022
     
    CaseyHofland, NoTuxNoBux and bdovaz like this.
  40. jasursadikov

    jasursadikov

    Joined:
    Feb 25, 2020
    Posts:
    17
    Have you managed to make it work somehow. Since I do need this functionality. Is there any workaround?
     
  41. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    586
    Workaround:

    Code (CSharp):
    1. [UnitySetUp]
    2. public IEnumerator Setup()
    3. {
    4.     Task task = SetupFunc();
    5.     while (!task.IsCompleted) yield return null;
    6.     task.GetAwaiter().GetResult();
    7. }
     
    Last edited: May 15, 2023
    liortal likes this.
  42. GainfulSage

    GainfulSage

    Joined:
    Apr 30, 2015
    Posts:
    106
    Any update on when 2.x might be released outside of experimental?
     
    CDF and bdovaz like this.
  43. GainfulSage

    GainfulSage

    Joined:
    Apr 30, 2015
    Posts:
    106
    So, I posted asking about 2.x updates a few weeks ago, nobody posted anything in response.

    Can someone at Unity do a proof of life post that the test framework is still being worked on...? I know there were posts on some of the other tools/etc after the last layoff...?
     
  44. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,663
    Yes, the test framework is still being worked on.
     
  45. tday_magicfuel

    tday_magicfuel

    Joined:
    Apr 10, 2018
    Posts:
    21
    The async testing support is pretty critical when writing more modern C# code--not to mention code that interacts with other unity packages (such as addressables!).

    Glad to hear it's still being worked on--is there any sort of timeline on when the package will leave the experimental stage?
     
    nehvaleem, RGV, sandolkakos and 2 others like this.
  46. RGV

    RGV

    Joined:
    Jan 25, 2016
    Posts:
    48
    We're properly using async Task as test signature for months, even out of the experimental — which we downgraded for, due to some compatibilitiy problems.
    upload_2023-6-16_8-56-18.png

    Hope it helps!
     
    sandolkakos likes this.
  47. tday_magicfuel

    tday_magicfuel

    Joined:
    Apr 10, 2018
    Posts:
    21
    Hey RGV, thanks for the tip. I did see that the async support was backported to 1.3.x (however the package manager in the 2022 LTS installs 1.1.x !).

    I've been trying the async tests with different versions of the UTF by manually editing the package manifest. The tests always hang indefinitely unless I'm using the experimental package.

    The tests are essentially just waiting for the async handle on an addressables load.

    With the experimental package the tests do run and conduct the tests properly. Though the time seems to be wildly inconsistent. Same test will sometimes take merely a second or two, others have occupied as much as two minutes before finally completing.

    Code (CSharp):
    1.        
    2.         [Test]
    3.         [Timeout(10000)]
    4.         public async Task TestLoad()
    5.         {
    6.             RulesLoader testLoader = new RulesLoader();
    7.             await testLoader.LoadRules();
    8.             Assert.IsTrue(testLoader.LoadedRulesCount > 0);
    9.             List<object> testRulesClass = testLoader.LoadedRulesData[RulesTestClass.Key];
    10.             Assert.IsTrue(testRulesClass.Count > 0);
    11.             RulesTestClass testClassObject = testRulesClass[0] as RulesTestClass;
    12.             Assert.IsNotNull(testClassObject);
    13.             Assert.AreEqual(15, testClassObject.TestNumber);
    14.             testLoader.ReleaseRules();
    15.         }
    16.  
    The load rules function for reference as well:

    Code (CSharp):
    1.  
    2. public async Task LoadRules()
    3. {
    4.     this.CreateAttributeMap();
    5.     this.LoadedRulesData = new Dictionary<string, List<object>>();
    6.  
    7.     this.loadHandle = Addressables.LoadAssetsAsync<TextAsset>
    8.         (RulesLoader.RulesJsonAddressableKeys,
    9.         this.ProcessRulesTextAsset,
    10.         Addressables.MergeMode.Union);
    11.  
    12.     await this.loadHandle.Task;
    13. }
    14.  
    This has been happening with simpler async methods I've tried (that essentially just wait a small out of time) as well. It doesn't seem to be addressables related. The timeout attribute on TestLoad also doesn't seem to function properly at all.

    In the experimental package any SetUp and Teardown Test fixture methods (one time or otherwise) also cause the tests to hang indefinitely so I've been repeating what would otherwise be handled in Setup at the start of each of these tests.

    Fortunately I can separate the async tests into a separate assembly so I don't always roll the dice on how long things take but it would be good to be able to rely on consistent performance and testing here.
     
  48. daniilkorotin

    daniilkorotin

    Joined:
    Oct 8, 2017
    Posts:
    5
    Same here with 2023.1.0f1... Is there a workaround or a planned release of the test framework to address the issue?
     
  49. MiTschMR

    MiTschMR

    Joined:
    Aug 28, 2018
    Posts:
    523
    I haven‘t found anything except to downgrade to pre.2. That version works.
     
  50. CaseyHofland

    CaseyHofland

    Joined:
    Mar 18, 2016
    Posts:
    621
    I'm currently adding the test framework to my skillset: my question is should I learn how to use 2.0.1-exp and hope it updates the coming year, or learn a possibly soon-to-be-deprecated 1.3.8? Are the differences small enough to justify getting into 1.3.8 now?

    While we're on the subject, is there a release plan for version 2 yet?
     
    GainfulSage likes this.