Search Unity

Feedback Add coroutine version of OneTimeSetUp

Discussion in 'Testing & Automation' started by DrummerB, May 14, 2020.

  1. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    You already provide the UnityTest and UnitySetUp/Teardown attributes which work with coroutines. It would be nice to also have a way to run a one-time setup (once per test class) that works with coroutines, i.e. this would be a good place to load a test scene, then have a number of test methods do different tests in the same scene.
     
    artoonie, JesseSTG, bobbaluba and 6 others like this.
  2. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,506
    Isn't it possible to achieve this by loading the scene in the TestFixture's constructor? (it doesn't have to be a coroutine).
     
  3. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    Probably. The unloading only works asynchronously though. The synchronous method is deprecated.

    And there might be other use cases for a UnityOneTimeSetUp attribute.
     
    liortal likes this.
  4. Maeslezo

    Maeslezo

    Joined:
    Jun 16, 2015
    Posts:
    209
    Precisely, I was searching this option for a bunch of tests that have to be checked when the complete flow has been finished
     
  5. sbethge

    sbethge

    Joined:
    Mar 20, 2019
    Posts:
    16
    I'm also trying to wrap my head around this at the moment. How is it supposed to work to
    1. Load a scene or two until they have finished loading
    2. Run a test with those scenes
    3. Run another test with those same scenes
    4. Finish, unload etc.
    Using [UnitySetUp] I can use yield to wait until the scene is loaded which is actually necessary for both synchronous and LoadSceneAsync. But then it will be run before every test.
    Using a TestFixtureSource as I understand it would not even allow loading the scene synchronously because it won't be loaded before the next frame, it is semi-asnychronuous (https://docs.unity3d.com/ScriptReference/SceneManagement.SceneManager.LoadScene.html). It seems crucial to me to have [UnityOneTimeSetUp] where all scene setup can be done for a test suite.
    A workaround I have now found is to simply check if the scenes have loaded already in the SetUp call. Not very clean though.
    Also I haven't figured out how to determine that all tests in the class have run and I can unload all scenes and GameObjects from DontDestroyOnLoad in [TearDown] since it will be necessary if I wan't to have another test suite that should start fresh.
     
    glenneroo likes this.
  6. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    I'm wondering if a more complete framework for loading and unloading test scenes would be beneficial, or if everyone's requirements are too different to find a good foundation. I imagine setting up separate test scenes where you want to run isolated tests would be a quite common use case.
     
  7. Maeslezo

    Maeslezo

    Joined:
    Jun 16, 2015
    Posts:
    209
    In my opinion that's the way to go. The Test Framework has improved so much, but there are still some flaws.

    Loading test scenes for me is one of them. If you want to load a test scene you have to add it to BuildSettings and then remove it.
    Maybe something like TestSceneManager.LoadAsync
     
    SolidAlloy likes this.
  8. MajeureX

    MajeureX

    Joined:
    May 4, 2020
    Posts:
    13
    Yes, please can the Unity development team look into this. I'm trying to figure out what the 'best practice' way of loading scenes for testing. Ideally I'd like to do this once, but it's not clear how this can be done with an async `OneTimeSetup` method.
     
    DrummerB likes this.
  9. Inter-Illusion

    Inter-Illusion

    Joined:
    Jan 5, 2014
    Posts:
    543
    I just needed the same thing, and while Unity adds a proper way, I hacked it in.
    To make it work, just replace the two attached files inside the Library\PackageCache\com.unity.test-framework@1.1.14 folder:

    The relevant parts are:
    Defining the attributes in UnitySetupAttribute.cs
    Code (csharp):
    1.  
    2. [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
    3. public class UnityOneTimeSetUpAttribute : NUnitAttribute
    4. {
    5. }
    6.  
    7. [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
    8. public class UnityOneTimeTearDownAttribute : NUnitAttribute
    9. {
    10. }
    11.  
    Variables to hold the UnityOneSetup and UnityOneTimeTearDown in CompositeWorkITem.cs (line 24)
    Code (csharp):
    1.  
    2. MethodInfo _unityOneTimeSetup, _unityOneTimeTearDown;
    3.  
    Capturing the methods (line 167)
    Code (csharp):
    1.  
    2. if (_suite.TypeInfo != null)
    3. {
    4.     _unityOneTimeSetup = Reflect.GetMethodsWithAttribute(_suite.TypeInfo.Type, typeof(UnityOneTimeSetUpAttribute), true)
    5.                                 .FirstOrDefault(x => x.ReturnType == typeof(IEnumerator));
    6.  
    7.     _unityOneTimeTearDown = Reflect.GetMethodsWithAttribute(_suite.TypeInfo.Type, typeof(UnityOneTimeTearDownAttribute), true)
    8.                                    .FirstOrDefault(x => x.ReturnType == typeof(IEnumerator));
    9. }
    10.  

    Running them before and after the tests (line 72)
    Code (csharp):
    1.  
    2. PerformOneTimeSetUp();
    3. if (_unityOneTimeSetup!=null)
    4. {
    5.     yield return Reflect.InvokeMethod(_unityOneTimeSetup, Context.TestObject);
    6. }
    7.  
    line (103)
    Code (csharp):
    1.  
    2. if (_unityOneTimeTearDown!=null)
    3. {
    4.     yield return Reflect.InvokeMethod(_unityOneTimeTearDown, Context.TestObject);
    5. }
    6.  
    7. PerformOneTimeTearDown();
    8.  
    Its a hack, but should work for most cases.

    BTW, to avoid having Unity override the changes everytime unity starts, just copy the package from the library folder to the Packages folder.

    Hope that helps
     

    Attached Files:

    Last edited: Jul 4, 2020
    glenneroo, akuno, Maeslezo and 2 others like this.
  10. MajeureX

    MajeureX

    Joined:
    May 4, 2020
    Posts:
    13
    Thanks for this Inter-Illusion. I've added your changes to the git repository https://github.com/DanStevens/com.unity.test-framework.git. To use with Unity, all one has to do is go to Window > Package Manager, click the '+', select Add package from git URL and enter the repository URL above.
     
    Last edited: Jul 6, 2020
    glenneroo, akuno and Maeslezo like this.
  11. akuno

    akuno

    Joined:
    Dec 14, 2015
    Posts:
    42
    Is it working on 2019.4?
    It compiles, but doesnt work for me. This code doesnt seem to execute anything:

    Code (CSharp):
    1.  
    2.         [UnityOneTimeSetUp]
    3.         public void OneTimeSetup () {
    4.             Debug.Log("UnityeOneTimeSetUp");
    5.         }
    6.  

    @superpig
    We really need this as default in the Test Tools.
    Here the use case: client and servers. The server start is a bit slow to do every [UnitySetup], and it cant be done via [OneTimeSetup] because many of it's dependencies are initialized on Start(), so it needs to wait a yield return null to work.
    Also, clients need to reconnect for every test, and the server also needs to wait for those connections every single test. This is too slow and a pain to synchronize. A [UnityOneTimeSetUp] solves this.
     
    Last edited: Oct 9, 2020
  12. pwdstaraster

    pwdstaraster

    Joined:
    Nov 11, 2020
    Posts:
    3
    I have not been able to get @Inter-Illusion's solution to work with Unity's Test Framework 1.1.14 or 1.1.18. Please bear in mind that I am a Unity and C# noob. I put UnitySetupAttributes.cs in \Library\PackageCache\com.unity.test-framework@1.1.1x\UnityEngine.TestRunner\NUnitExtensions\Attributes and CompositeWorkItem.cs in \Library\PackageCache\com.unity.test-framework@1.1.1x\UnityEngine.TestRunner\NUnitExtensions\Runner. When I use UnityOneTimeSetup and UnityOneTimeTearDown, I get compilation errors. I want to use the official Unity test framework, not danstevens. Can someone help me figure out why this isn't working for me?

    Thanks!
     
  13. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    193
    I, too, would like this feature. It would greatly simplify my test suite.

    My test suite consists of a bunch of scenarios, followed by a bunch of Asserts to ensure the world is in my desired state. I could run my scenario to completion in a hypothetical [UnityOneTimeSetup], then run the actual Asserts in some trivial test methods.
     
  14. Meatloaf4

    Meatloaf4

    Joined:
    Jul 30, 2013
    Posts:
    107
    Would also really love to see this. I completely agree with @akuno. Here's a concrete example of what he described that I would really love to be able to handle out of the box.


    Code (CSharp):
    1.  
    2.     /// <summary>
    3.     /// Contains tests for our <see cref="PlayFabClientAPI"/>
    4.     /// </summary>
    5.     public class PlayFabClientAPITests
    6.     {
    7.         [OneTimeSetUp]
    8.         public IEnumerator SetupTest()
    9.         {
    10.             var playFabAuthorizationService = new PlayFabAuthorizationService();
    11.             var unityTestUser = new PlayFabAuthorizationService.AuthorizedUser("UnitTester","UnitTester");
    12.             var signInTask = playFabAuthorizationService.SignInAsync(unityTestUser);
    13.             yield return signInTask.AsIEnumerator();
    14.         }
    15.      
    16.         [UnityTest]
    17.         [TestCase(7496062027744821083,"Betty", ExpectedResult = null)]
    18.         [TestCase(1312412027744821083,"Todd", ExpectedResult = null)]
    19.         [TestCase(5326062027744124354,"Miami", ExpectedResult = null)]
    20.         public IEnumerator GetCatalogItem_LevelDataId_PlayFabCatalogItemFound(long levelDataId, string levelName)
    21.         {
    22.             //Get CatalogItem from server & verify not null...requires a sign in before PlayFabClientAPI.GetCatalogItems can be used.
    23.         }
    24.  
     
    Last edited: Nov 26, 2020
    akuno likes this.
  15. osum4est

    osum4est

    Joined:
    Jan 26, 2015
    Posts:
    13
    I can't believe this hasn't been added yet. I need this for my project as well. Surely it can't be that hard for Unity to add this to the official package.

    EDIT: I have created my own fork here: https://github.com/8bitforest/com.unity.test-framework I have confirmed this works on 2020.2.1f1, and is based off the latest version of the official package. (@akuno @pwdstaraster a little late, but might help you guys)

    EDIT 2: I also added [AsyncTest] [AsyncSetUp/TearDown] and [AsyncOneTimeSetUp/TearDown] attributes that can be added to methods returning "async Task"!
     
    Last edited: Feb 15, 2021
    glenneroo, floky, RGV and 3 others like this.
  16. KurtGokhan

    KurtGokhan

    Joined:
    Jan 16, 2013
    Posts:
    32
    I also need this feature. Is anybody from Unity team aware of this feature request?
     
  17. Maeslezo

    Maeslezo

    Joined:
    Jun 16, 2015
    Posts:
    209
    I don't think so. It would be good to know if this feature is at least in their roadmap
     
  18. davidrochin

    davidrochin

    Joined:
    Dec 17, 2015
    Posts:
    65
  19. MarkusFiredrill

    MarkusFiredrill

    Joined:
    Jan 11, 2021
    Posts:
    1
  20. glenneroo

    glenneroo

    Joined:
    Oct 27, 2016
    Posts:
    70
    Should we keep adding +1 in this thread until Unity sees it?
     
  21. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    72
    I can confirm that the feature is indeed on our roadmap. We do not have any eta of when it will come yet.
     
  22. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    193
    Mission complete, boys and girls.
     
    RGV, DrummerB and Maeslezo like this.
  23. nostalgicbear

    nostalgicbear

    Joined:
    Mar 21, 2013
    Posts:
    97
    This should just become Unitys tag line at this stage. At least you all had time to create a lovely new logo though. It's what we have all been crying out for.
     
  24. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    193
    Please be nice, that decision was almost certainly above the pay grade of the people working on this library.
     
    Last edited: Oct 7, 2021
    DrummerB likes this.
unityunity