Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Official Unity Test Framework 2.0 ready for feedback

Discussion in 'Testing & Automation' started by thewerku, Jan 25, 2022.

  1. thewerku

    thewerku

    Unity Technologies

    Joined:
    Feb 6, 2020
    Posts:
    15
    Unity Test Framework v2.0
    [updated on May 2nd]

    Hi everyone, we have a new experimental version of Unity Test Framework for you to try out in hope of getting feedback regarding the new features. At this point we have no timeline for fully releasing this version, but once we do, we will make sure to keep you updated.

    For those of you who are unfamiliar with what Experimental packages are, they are defined as:
    Experimental packages are new packages or experimental modifications made to mature packages. Unity does not support Experimental packages because they are in the early stages of development.

    You can read more about them in this article.

    How to activate v2.0
    If you’re using Unity 2019.4 and beyond, open your package.json and update the com.unity.test-framework line to
    “com.unity.test-framework”: “2.0.1-exp.1”


    Combine Edit Mode and Play Mode tests
    This version removes the previous requirement to keep Edit Mode and Play Mode tests in separate assemblies by introducing the RequiresPlayModeAttribute. An Editor-only test assembly can now include tests that will run in the Editor's Play Mode if given the [RequiresPlayMode] attribute. Likewise, a platform-specific assembly can include Edit Mode tests and exempt them from running in Play Mode if the tests are given the [RequiresPlayMode(False)] attribute.

    Async tests
    This version introduces support for writing asynchronous tests with the dotnet Task asynchronous programming model. See Async tests. Feedback on this feature is especially welcome.

    Ignore tests based on arguments
    This version introduces the ParameterizedIgnoreAttribute which allows ignoring tests based on arguments which were passed to the test method of a parameterized test.

    Revised Test Runner UI
    This version includes a revised Test Runner window with support for combined Edit Mode and Play Mode tests and several usability improvements:

    This version includes many additional bug fixes and improvements. For a full list of changes and updates in this version, see the Unity Test Framework package changelog.

    Looking forward to hearing from you

    - UTF team
     
    Last edited: May 2, 2022
  2. Jonas-Neuston

    Jonas-Neuston

    Joined:
    Jun 10, 2017
    Posts:
    69
    This sounds really good! I'll be sure to give it a try soon.
     
  3. unity_t3DjhpF93gNHLg

    unity_t3DjhpF93gNHLg

    Joined:
    May 12, 2019
    Posts:
    2
    Good job!
    Do you have a good solution for simulating the advance of time for Tasks in tests?
    Like TestScheduler in UniRx for example
     
  4. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,532
    Will this version be supported in earlier Unity versions as well ?
     
  5. JoNax97

    JoNax97

    Joined:
    Feb 4, 2016
    Posts:
    597
    I'm very interested in this as well
     
  6. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    75
    Yes, it is available all the way back to 2019.4. Depending on the Unity version, you will have to add it to the package.json manually, by changing the test-framework line to
    "com.unity.test-framework": "2.0.1-pre.18"
     
    bdovaz likes this.
  7. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,959
    There's a few things that are annoying in V1, and I'm wondering if they'll be solved in V2:

    - The inability to write tests targeting Assembly.CSharp, forcing an assembly-based workflow. Our experience is that assembly-based workflows lead to worse code and slower compile times, so we'd like to avoid it.

    - No support for [Timeout]. Right now every test that yields anything else than WaitForSeconds has to include a manual timeout, otherwise tests can hang forever.

    - The way runtime tests work. Right now Unity makes a temporary scene asset and loads it when we start tests. After that, we typically load a different scene. This causes a bunch of knock-on issues:
    -- If Unity crashes during execution, we get a new scene asset in Assets/
    -- Any system that scans for asset changes (like Quick Search and version control plugins) starts doing a bunch of work
    -- We're often in the scene that the Unit test is supposed to run in (especially when iterating on fixing a test). Unloading that scene, loading a new scene, and then loading the original scene back is slow, and slows down iteration.
    - Ideally all of this would be fixed by two things:
    -- the ability to tell Unity which scene(s) to run a test in the context of, and for Unity to not do any scene loading if those are already open
    -- when we don't run the tests in the context of a scene, Unity should use an in-memory normal empty scene instead of an asset. Unity is perfectly capable of entering play mode with a clean scene that's not an asset.
     
  8. thewerku

    thewerku

    Unity Technologies

    Joined:
    Feb 6, 2020
    Posts:
    15
    Timeout issues will be solved in later iterations of v2.0, before the full release of 2022.2.

    These two items you mentioned are not addressed in v2.0 at all. While for the latter we discussed a potential solution of enabling users to specify which scene should be loaded for a test, I'd be curious to learn more about your experience regarding assembly-based workflows. How does it lead to worse code and slower compile times? What would be a better way of working for you and your team?
     
  9. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,959
    I've had that discussion before here, and most of that still holds. Note that we didn't have enter play mode options for that project, so more assemblies meant taking longer to enter play mode even when we didn't change code!

    For our current project, we ended up having a single, big asmdef that just covers all of our Assets/ code in order to support tests, and another one for Editor/ stuff that we asmref all over the place. But I'd really love for that to just be Assembly-CSharp and then be able to target that with our test assemblies.

    The clean solution that'd fix both the testing problem, and all of the other problems we have around assembly definitions, would be to just allow Assembly-CSharp to be an assembly we could add a reference to from other assemblies. That shouldn't be very hard!


    We do like to extract stable, independent parts of our code base out of the general compile/reload flow, but in those cases moving them into a package is a lot cleaner, since that allows versioning and reuse between projects (we use Verdaccio). If something's changed often enough that it's in Assets/, it's changed often enough that we're not saving compile/reload times by segmenting it into it's own assembly. We're just adding to the average case number of assemblies that has to be reloaded by a code change.
     
    Arkade likes this.
  10. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,959
    JoNax97 likes this.
  11. unity_eXyvNWWMEwFWvQ

    unity_eXyvNWWMEwFWvQ

    Joined:
    Jan 31, 2022
    Posts:
    1
    Since you're interested @thewerku I agree with @Baste here. I work for a big mobile company and we use modules for every part of the game, and each module has its own Asmdef. (Shop, Dashboard, Each game mode, Social, Rewards, etc).

    Since we started using Asmdef for each module, everytime you change any single line of the code, going back to unity takes around 20 seconds which makes the workflow very tedious and hard. It gets 10 times worse when you're developing using TDD. Everytime you change something on the test you have to wait 20 seconds, this makes a simple 2 minute discovery test, something that can take up to 15 minutes.
     
    Arkade, JoNax97 and glenneroo like this.
  12. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    41
    I gave this a spin, and here are my findings:
    • The new UI is nice, especially when filtering tests!
    • However, the UI does not show pass/fail status while executing, which I find somewhat annoying, as it's nice to start looking at failing tests already while the rest of the tests are executing.
    • The first bigger issue I found is that the new Custom NUnit version has Auto Reference enabled. This makes all assemblies look like test assemblies. This is a problem for some of our custom code, SonarCloud analysis, and also the test runner code itself seems to use this criterion to scan assemblies for tests (see bullet below).
    • When running all tests in our largest project with lots of assemblies and over 7000 tests, starting up takes extremely long: it took over 2 minutes to get past "Create Bootstrap Scene", and at least another 2 minutes to get past "Prebuild Setup". I'm guessing this is at least partially because all assemblies are treated as test assemblies now (see bullet above). These stages can't even be canceled, making the situation worse.
     
    Arkade, glenneroo and JoNax97 like this.
  13. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    347
    There's a workaround for this called StoppableCoroutines. Trilleon Unit Test & UI Automation framework implements it, and coroutines will never hang/fail. Try implementing it yourself. Here is the specific file. Look at the AutomationMaster.cs file to see how it is used.

    Although it may be difficult to implement in an NUnit or UnityTest framework if you can't control the test runner. If you need this functionality, it may make sense just to switch your tests to Trilleon tests, or make your own test runner.
     
    Arkade likes this.
  14. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    14
    Error occurred when run all Edit Mode and Play Mode tests on Test Runner window.

    ```
    ArgumentException: An item with the same key has already been added. Key: RequiresPlayModePerformance.Tests.Editor.EditModeTests.DummyEditModeTest_AlwaysPassed(1)
    UnityEditor.TestTools.TestRunner.GUI.TestListGUI.get_ResultsByKey () (at Library/PackageCache/com.unity.test-framework@2.0.1-pre.18/UnityEditor.TestRunner/GUI/Views/TestListGUIBase.cs:56)
    (snip)
    ```

    Reproduction Project
    https://github.com/nowsprinting/RequiresPlayModePerformance

    This error will no longer be reproduced after the following actions:
    Causing Assembly-CSharp-Editor assembly recompilation by editing Assets/Editor/RunTestsPerformance.cs.
     
  15. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    14
    Bugs:
    * In the Test Runner window, the IgnoreAttribute and DescriptionAttribute string arguments of tests are not displayed. In the case of ExplicitAttribute, they are displayed reason after “Run All”

    Requests:
    * I hope to be added string argument (reason) to the ParameterizedIgnoreAttribute
    * In the Test Runner window, I hope multi-word search. And if possible, ideographic space (U+3000) should also be recognized as separators.
    * I hope `SceneManager.LoadScene` should be able to load scenes that are not in "Scenes in Build" (there is a workaround, but it would be nice to have it implemented in the framework)
     
    sandolkakos and Maeslezo like this.
  16. Maeslezo

    Maeslezo

    Joined:
    Jun 16, 2015
    Posts:
    255
    I'd say "SceneManager.LoadScene` should be able to load scenes that are not in "Scenes in Build" (there is a workaround, but it would be nice to have it implemented in the framework" is quite important
     
    glenneroo and sandolkakos like this.
  17. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    41
    The issues with Coroutines and the Unity Test Framework run deeper than just timeouts, I commented on it in this thread. It's totally possible to write properly timing out and cancelable code within the framework, it just requires you to wrap everything asynchronous in something else than a raw coroutine/enumerator.
     
    Elapotp likes this.
  18. thewerku

    thewerku

    Unity Technologies

    Joined:
    Feb 6, 2020
    Posts:
    15
    Hey all, thanks for the replies! I'll try to address the points we haven't yet responded to:

    @unity_t3DjhpF93gNHLg the only thing that has come up in our discussions was potentially using Time.timeScale - have you looked into it to see if it meets your expectations?

    Our plans for this year are very much dependent on v2.0 feedback, so we don't really have a fully fledged roadmap to put on the main page. What we're going to do is collect improvement ideas and then present them to you (again through Forum) to help us prioritise the work - once priorities are clear, we'll factor in the time we need for bugs and maintenance work to plan it against the release deadline. Either way, we might look into actually having a public roadmap once the v2.0 plans are validated with the help of this community.

    @sbergen do you mean that the status icons don't show up at all while tests are running? Would you be able to share a screenshot of how it looks like on your end? Feel free to ping me directly via Forum messaging option if you don't want to share such content publicly.

    Thanks for reporting @nowsprinting. We logged these bugs on our backlog.

    We will take these suggestions into planning. As mentioned above, we'll collect all requests and get back to you to understand the priority and value, that will then help us decide on the work plan for the next quarter.

    The last thing I wanted to address is feedback around working within assembly-based workflow. We're taking in what you've shared here and in other threads, and will start an internal discussion about this with the Scripting team, to better understand current approach and what is the best way to alleviate pain for you all. If there will be any work planned, it will be separate from UTF 2.0 release.

    To be continued. Appreciate the discussion here so far, keep it coming!
     
    Elapotp, nowsprinting, Arkade and 2 others like this.
  19. Maeslezo

    Maeslezo

    Joined:
    Jun 16, 2015
    Posts:
    255
    In these years using the test framework, this would be my main requests:
    • Coroutine version of OneTimeSetUp
    • Support for loading scenes that are not included in the build index
      • Important for loading a test scene that you only need it at testing
    • It would be also nice to have serialized fields in the tests. Sometimes you need to reference some project asset in the test and it would be nice to get it by a serialized field.
    Thank you
     
    Arkade and glenneroo like this.
  20. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,959
    This can be done through EditorSceneManager.LoadSceneInPlayMode. We write tests like this:

    Code (csharp):
    1.  
    2. public abstract class TestsForScene {
    3.     public abstract string pathToScene { get; }
    4.     protected string sceneName => Path.GetFileNameWithoutExtension(pathToScene);
    5.  
    6. [OneTimeSetUp]
    7. public void OneTimeSetup() {
    8. #if UNITY_EDITOR
    9.         UnityEditor.SceneManagement.EditorSceneManager.LoadSceneInPlayMode(pathToScene, new LoadSceneParameters(LoadSceneMode.Single));
    10. #else
    11.         SceneManager.LoadScene(sceneName, LoadSceneMode.Single);
    12. #endif
    13.     }
    But having a real way to do this would be much nicer. What if we could:

    Code (csharp):
    1. [UnityTest(scene="path/to/scene")]
     
  21. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    75
    We are indeed considering an attribute like [TestScene("Scene/path", reloadBeforeTest (bool) )], which would then also ensure that the tests are included in the player build. We are tracking this as a feature request, but we do not have any concrete plans yet.
     
  22. Maeslezo

    Maeslezo

    Joined:
    Jun 16, 2015
    Posts:
    255
  23. thewerku

    thewerku

    Unity Technologies

    Joined:
    Feb 6, 2020
    Posts:
    15
    One of the new features in 2.0 is actually a way to have async tests. Would you be able to verify if what we're proposing satisfies your needs, and if not, what are we missing?

    Async tests
    This version introduces support for writing asynchronous tests with the dotnet Task asynchronous programming model. See Async tests. Feedback on this feature is especially welcome.​
     
    Xarbrough, Thaina and sandolkakos like this.
  24. Maeslezo

    Maeslezo

    Joined:
    Jun 16, 2015
    Posts:
    255
    Wow I totally missed that example.

    Yes, that's it. Perfect. Thanks!
     
    Last edited: Feb 9, 2022
    thewerku likes this.
  25. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    41
    I tried this again, and noticed that this issue only happens the first time you run tests, either after switching from 1.X to 2.X or when restarting the Editor. Seems to be consistently reproducible for me. The icons are visible, they just stay in the "not run" state, until they all turn to their final state all at once. Now that I know it doesn't happen on consecutive runs, it's actually not a big deal, just strange :)

    I'm more curious about Auto Reference being enabled in nunit.framework.dll, and would be interested in seeing if changing that helps with the extremely slow performance I'm seeing in big projects. I had some issues trying to edit that manually in the package source locally, as somehow the Editor didn't want to pick up the change.
     
  26. unity_90E9C1A7C49BD2E0B1A8

    unity_90E9C1A7C49BD2E0B1A8

    Joined:
    Aug 19, 2021
    Posts:
    5
    Feature Request: Native junit format reporting of test cases. This would allow direct integration with other tooling like GitLabs Unit Testing Reporting feature.
     
  27. unity_90E9C1A7C49BD2E0B1A8

    unity_90E9C1A7C49BD2E0B1A8

    Joined:
    Aug 19, 2021
    Posts:
    5
    While I'm thinking about JUnit, it would be also great to have Cobertura format reporting too for Test Coverage Visualization.

    Thanks team!
     
    NoTuxNoBux likes this.
  28. thewerku

    thewerku

    Unity Technologies

    Joined:
    Feb 6, 2020
    Posts:
    15
    Thanks for the suggestion. Could you help me understand what would change for you, if we had those format reporting options available? What problem could it solve?
     
  29. unity_90E9C1A7C49BD2E0B1A8

    unity_90E9C1A7C49BD2E0B1A8

    Joined:
    Aug 19, 2021
    Posts:
    5
    For my use case, it would mean a faster CI pipelines if cobertura reporting was supported natively. As of today, one CI job runs the tests and then in a subsequent job collates all the coverage reports and converts them to cobertura. While this only takes about a minute to run, we runs 100s of these a day and that minute adds up quickly!

    The solution right now is simply running the below to generate the report:

    dotnet tool install -g dotnet-reportgenerator-globaltool
    reportgenerator "-reports:editmode-coverage/coverage.xml;playmode-coverage/coverage.xml" -reporttypes:Cobertura "-targetdir:report"


    I raised the feature request for awareness. I was hoping that if one of the underlying libraries that are already being used by the tooling offer this type of reporting. That would make the work (hopefully) a trivial exercise of just exposing existing functionality. Then it becomes much easier to see the value in doing the work. As I write this though, probably feedback more for the test coverage tool than the test framework tool...

    Test Coverage visualization is a fantastic tool for developers to help them understand and also give confidence in the features that they are developing (or reviewing) to make sure they continue to be robust and stable in the future as the game progresses (hopefully) into production and then into live operations with continued bug fixes and feature development.


    For JUnit reporting, once again I have a solution for this already that uses xslt to transform the nunit reporting into a junit report. JUnit reporting in GitLab helps visualize problems in test cases. If I have 1000s of unit/integration tests and some of them are failing, having a tool that can show and visualize what is failing can save time and improve productivity.

    Having this supported out of the box would (I hope) encourage adoption of the tooling offered by Unity and would be a stellar feature to have documented for others to see and therefore not have to come up with custom integration solutions like I had to do.

    In summary, the value to me personally is low however by providing it as a feature natively you will save other developers who need to write this level of integration themselves. I would also hope that it results in an increase of adoption of the unity test framework adoption, as it becomes easier for developers to show the value it can bring if it integrates with existing tooling as a proof-of-concept would be trivial to showcase the value to the business.

    I hope all this makes sense and apologies for anything that comes across as rambling :D
     
    Arkade likes this.
  30. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    538
    Are there any plans to support target platform tests in Docker? As a library/package author, currently, not being able to test IL2CPP builds during CI is my biggest pain point, considering how fragile it is with normally-acceptable C# code (virtual generic methods with structs).
     
  31. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,559
    I’m not sure that this is really a test framework thing, because currently Unity does not officially support running inside Docker containers - I am not surprised that some things may not be working, but the test framework team cannot prioritise looking at unsupported scenarios like this.

    However, as you can see from the Platforms roadmap, official support for running in containers is 'In Progress' - it's being worked on right now. So there's a good chance that whatever problems you're running into will be solved by the team who are working on that. After they deliver the official support, if you're still seeing a problem, then you can file a bug and we'll fix it :)
     
    ProtoTerminator likes this.
  32. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    538
    @superpig Thanks for the response.

    Even though it's not officially supported, people have gotten Unity to work for some things in Docker, like making builds and running editor tests. From my own testing trying to get it to work, it seems to me that the problem with target platform tests in Docker is that it can't make a connection between the player and editor, so it just waits for 10 minutes and times out. That seems like it's directly under the purview of this package, not Unity itself. I don't think Unity officially supporting Docker will suddenly make that connection work, but I don't know enough about it to say one way or the other.

    In any case, I understand your position, just something to think about.
     
  33. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,559
    In fact, the Test Framework uses a 'standard' Unity channel called the 'PlayerConnection' for communicating between the Editor and Player - so the odds are high that it actually will get fixed as part of the official Docker support. The Test Framework is not the only system that uses it - it could be interesting for you to see if you are able to connect the Profiler to a player running in Docker as well, because that also uses the PlayerConnection system.
     
    ProtoTerminator likes this.
  34. glenneroo

    glenneroo

    Joined:
    Oct 27, 2016
    Posts:
    204
  35. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,559
    If you mean in the Editor, I don't think we want to do that, because it would force us to recompile all your code when starting a test run. If you only need it in the Player, though, you can already set that up using ITestPlayerBuildModifier - the BuildOptions object that you can modify there includes the ability to supply extra scripting defines.
     
    Arkade and glenneroo like this.
  36. PhantomGingerGames

    PhantomGingerGames

    Joined:
    Jun 23, 2019
    Posts:
    4
    I found this thread while looking for ways to write test for my asynchronous methods, so of course I excitedly followed the instructions to enable pre-release packages (I can even see the XR Interaction Toolkit tagged as [Pre]). Unfortunately I don't see a Test Framework 2.0 or any option to update the existing Framework. Are there additional steps I'm somehow missing?
    upload_2022-2-16_21-3-39.png
    upload_2022-2-16_21-3-20.png
    Thanks!
     
  37. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,532
    Who moved my cheese? Why are the buttons located there and not at the top of the window ?

    upload_2022-2-17_13-10-13.png
     
  38. thewerku

    thewerku

    Unity Technologies

    Joined:
    Feb 6, 2020
    Posts:
    15
    Pasting an answer shared somewhere earlier in the thread:
    Depending on the Unity version, you will have to add it to the package.json manually, by changing the test-framework line to "com.unity.test-framework": "2.0.1-pre.18"​

    @liortal it's a result of decluttering the top part of the package UI. What are your thoughts here?
     
    PhantomGingerGames likes this.
  39. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    538
    What about copying Visual Studio's Test Explorer UI, using icons instead of text? They're pretty compact.

    upload_2022-2-18_13-54-23.png
     
    JoNax97 likes this.
  40. PhantomGingerGames

    PhantomGingerGames

    Joined:
    Jun 23, 2019
    Posts:
    4
    Thank you! I missed that earlier in the thread I guess, appreciate you taking the time to paste if for me!

    I did find one strange thing, it seems like the test UI gets hung when doing async tests. It always recovers if I click to another window, then click back onto unity (re-focus on the window essentially). I'm currently using unity v2021.1.19f1. I'll look into updating to 2021.2.12f1 which appears to be the latest version, I'm not sure if there would be any breaking changes for my project. But wanted to note the feedback here in case anyone else sees the same thing and goes searching.

    Example gif of what I'm seeing:
    TestAsyncUnityTest.gif

    Exact code used to test:

    Code (CSharp):
    1. using NUnit.Framework;
    2. using UnityEngine.AddressableAssets;
    3. using UnityEngine.ResourceManagement.ResourceLocations;
    4. using System.Collections.Generic;
    5. using System.Threading.Tasks;
    6. using System.Threading;
    7.  
    8. public class TestAsync
    9. {
    10.     [Test]
    11.     public async Task LoadAddressibleResource()
    12.     {
    13.         IList<IResourceLocation> landTilesTask = await AsyncLoadLand_GRITTYSOOT();
    14.         Assert.Greater(landTilesTask.Count,0, "Loaded some locations");
    15.     }
    16.  
    17.     private async Task<IList<IResourceLocation>> AsyncLoadLand_GRITTYSOOT()
    18.     {
    19.         IList<IResourceLocation> landTilesTask = await Addressables.LoadResourceLocationsAsync("biomes/GRITTYSOOT/land").Task;
    20.         return landTilesTask;
    21.     }
    22.  
    23.     [Test]
    24.     public async Task TestGenericAsync()
    25.     {      
    26.         Assert.IsTrue(await GenericAsync());
    27.     }
    28.  
    29.     private async Task<bool> GenericAsync()
    30.     {
    31.         await Task.Run(() => Thread.Sleep(1000));
    32.         await Task.Run(() => Thread.Sleep(1000));
    33.         return true;
    34.     }
    35.  
    36. }
    37.  
     
    Last edited: Feb 20, 2022
    Inter-Illusion likes this.
  41. thewerku

    thewerku

    Unity Technologies

    Joined:
    Feb 6, 2020
    Posts:
    15
    @ProtoTerminator thanks for the suggestion, I'm saving it for future reference. The package UI is not the primary focus of 2.0 changes, but something in need of attention that we're aware of.

    @unity_hi5HOPoVLuZIyA thanks for reporting, adding this to our backlog.
     
  42. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    538
    I got this error:

    Library\ScriptAssemblies\ProtoPromiseTests.pdb: Copying the file failed: The process cannot access the file because it is being used by another process.

    And I couldn't get rid of it until I re-imported my scripts.

    Repro steps:
    - Right-click a test and click "Open source code"
    - Edit the code and save
    - Go back to Unity to recompile
    - Observe error

    [Edit] This was in Unity 2021.2.7f1 with Test Framework 2.0.1-pre.18

    @thewerku Also, I would like to provide some more feedback on the UI. Should I do that here, or where should I do that?
     
  43. nowsprinting

    nowsprinting

    Joined:
    Nov 3, 2014
    Posts:
    14
    Great! I think so attribute style is useful!
    Also, it seems that it can be used without conflicting with the RecordedTestAttribute of Automated QA package.

    However, I'm worried about testing to transition from SceneA to SceneB.
    With the above method, there seems to be no guarantee that Scene2 will be included in the build.
    We can avoid it by writing the following.
    Code (CSharp):
    1. [TestScene("SceneA", true)]
    2. [TestScene("SceneB", false)]
     
  44. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    754
    Thank you for your effort

    I also would like to request that, the test should allow [Timeout] attribute in unity, and make it so that every test with [Timeout] would run in another thread, and so it also could be killed separate from unity's main thread
     
  45. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,559
    You wouldn't be able to call the vast majority of Unity's APIs from the other thread. Would it still have any value to you like that?
     
  46. glenneroo

    glenneroo

    Joined:
    Oct 27, 2016
    Posts:
    204
    Speaking of Timeout, it would be nice if Timeout worked reliably, especially within UnitySetUp and UnityTearDown - I tried the new 2.0.0 and for reasons (faulty code on my end) the Timeout wasn't being respected and some of my tests were getting stuck forever in while() loops. I had to step in with the debugger and force them to end and in some extreme cases kill Unity process via Task Manager.
     
    booferei likes this.
  47. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    754
    If there was any other way to make unity test not freeze from infinite loop and need to conjure the task manager to kill it all, then I would love to

    Currently now I waste my time to run a test case from reactivex and some other dll that have some chance to internally block thread. And unity will unreliably frozen without a way to stop. Seriously it was a weak point of unity that should have some overseer thread that could reset the mainthread and reload domain instead of the need to kill the whole process

    But in the meantime, just simple attribute that to specify that some test has tendency to block thread, so need to run in another thread separate from main thread. Timeout is a good indicator

    There should be some API like `Application.runOnMainThread` or `EditorApplication.runOnMainThread` to run unity API in other thread and just that
     
  48. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    538
    There is an API for that:
    SynchronizationContext.Current.Post
    for asynchronous dispatch, or
    SynchronizationContext.Current.Send
    for synchronous dispatch. Though you will need to cache the context somewhere to use from a background thread.
     
  49. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    754
    If you just carefully read my comment and not hurrily argue you would see that I mention "some other dll" "internally block thread" which mean I have no control over it and so the unity mainthread could blocked from any reason and so dispatching anything aren't solve this problem. Unity should just have ability to kill its main thread and do domain reloading is the totally sensible solution and so please don't make any more distraction
     
  50. ProtoTerminator

    ProtoTerminator

    Joined:
    Nov 19, 2013
    Posts:
    538
    Okay...? That's a separate issue from asking about an API, though. I had nothing to add about Unity's constraints, so I just responded to the last bit. If they add a new API to run something on the main thread, that won't help with your dll issue.