Feedback Add coroutine version of OneTimeSetUp

You already provide the UnityTest and UnitySetUp/Teardown attributes which work with coroutines. It would be nice to also have a way to run a one-time setup (once per test class) that works with coroutines, i.e. this would be a good place to load a test scene, then have a number of test methods do different tests in the same scene.

 

Isn't it possible to achieve this by loading the scene in the TestFixture's constructor? (it doesn't have to be a coroutine).

 

Probably. The unloading only works asynchronously though. The synchronous method is deprecated.

And there might be other use cases for a UnityOneTimeSetUp attribute.

 

Precisely, I was searching this option for a bunch of tests that have to be checked when the complete flow has been finished

 

I'm also trying to wrap my head around this at the moment. How is it supposed to work to

1. Load a scene or two until they have finished loading
2. Run a test with those scenes
3. Run another test with those same scenes
4. Finish, unload etc.

Using [UnitySetUp] I can use yield to wait until the scene is loaded which is actually necessary for both synchronous and LoadSceneAsync. But then it will be run before every test.
Using a TestFixtureSource as I understand it would not even allow loading the scene synchronously because it won't be loaded before the next frame, it is semi-asnychronuous (https://docs.unity3d.com/ScriptReference/SceneManagement.SceneManager.LoadScene.html). It seems crucial to me to have [UnityOneTimeSetUp] where all scene setup can be done for a test suite.
A workaround I have now found is to simply check if the scenes have loaded already in the SetUp call. Not very clean though.
Also I haven't figured out how to determine that all tests in the class have run and I can unload all scenes and GameObjects from DontDestroyOnLoad in [TearDown] since it will be necessary if I wan't to have another test suite that should start fresh.

 

I'm wondering if a more complete framework for loading and unloading test scenes would be beneficial, or if everyone's requirements are too different to find a good foundation. I imagine setting up separate test scenes where you want to run isolated tests would be a quite common use case.

 

In my opinion that's the way to go. The Test Framework has improved so much, but there are still some flaws.

Loading test scenes for me is one of them. If you want to load a test scene you have to add it to BuildSettings and then remove it.
Maybe something like TestSceneManager.LoadAsync

 

Yes, please can the Unity development team look into this. I'm trying to figure out what the 'best practice' way of loading scenes for testing. Ideally I'd like to do this once, but it's not clear how this can be done with an async `OneTimeSetup` method.

 

I just needed the same thing, and while Unity adds a proper way, I hacked it in.
To make it work, just replace the two attached files inside the Library\PackageCache\com.unity.test-framework@1.1.14 folder:

The relevant parts are:

Spoiler

Defining the attributes in UnitySetupAttribute.cs

Code (csharp):

1.  
2. [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
3. public class UnityOneTimeSetUpAttribute : NUnitAttribute
4. {
5. }
6.  
7. [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
8. public class UnityOneTimeTearDownAttribute : NUnitAttribute
9. {
10. }
11.  

Variables to hold the UnityOneSetup and UnityOneTimeTearDown in CompositeWorkITem.cs (line 24)

Code (csharp):

1.  
2. MethodInfo _unityOneTimeSetup, _unityOneTimeTearDown;
3.  

Capturing the methods (line 167)

Code (csharp):

1.  
2. if (_suite.TypeInfo != null)
3. {
4.     _unityOneTimeSetup = Reflect.GetMethodsWithAttribute(_suite.TypeInfo.Type, typeof(UnityOneTimeSetUpAttribute), true)
5.                                 .FirstOrDefault(x => x.ReturnType == typeof(IEnumerator));
6.  
7.     _unityOneTimeTearDown = Reflect.GetMethodsWithAttribute(_suite.TypeInfo.Type, typeof(UnityOneTimeTearDownAttribute), true)
8.                                    .FirstOrDefault(x => x.ReturnType == typeof(IEnumerator));
9. }
10.  

Running them before and after the tests (line 72)

Code (csharp):

1.  
2. PerformOneTimeSetUp();
3. if (_unityOneTimeSetup!=null)
4. {
5.     yield return Reflect.InvokeMethod(_unityOneTimeSetup, Context.TestObject);
6. }
7.  

line (103)

Code (csharp):

1.  
2. if (_unityOneTimeTearDown!=null)
3. {
4.     yield return Reflect.InvokeMethod(_unityOneTimeTearDown, Context.TestObject);
5. }
6.  
7. PerformOneTimeTearDown();
8.  

Its a hack, but should work for most cases.

BTW, to avoid having Unity override the changes everytime unity starts, just copy the package from the library folder to the Packages folder.

Hope that helps

 

Thanks for this Inter-Illusion. I've added your changes to the git repository https://github.com/DanStevens/com.unity.test-framework.git. To use with Unity, all one has to do is go to Window > Package Manager, click the '+', select Add package from git URL and enter the repository URL above.

 

Is it working on 2019.4?
It compiles, but doesnt work for me. This code doesnt seem to execute anything:

Code (CSharp):

1.  
2.         [UnityOneTimeSetUp]
3.         public void OneTimeSetup () {
4.             Debug.Log("UnityeOneTimeSetUp");
5.         }
6.  

@superpig
We really need this as default in the Test Tools.
Here the use case: client and servers. The server start is a bit slow to do every [UnitySetup], and it cant be done via [OneTimeSetup] because many of it's dependencies are initialized on Start(), so it needs to wait a yield return null to work.
Also, clients need to reconnect for every test, and the server also needs to wait for those connections every single test. This is too slow and a pain to synchronize. A [UnityOneTimeSetUp] solves this.

 

I have not been able to get @Inter-Illusion's solution to work with Unity's Test Framework 1.1.14 or 1.1.18. Please bear in mind that I am a Unity and C# noob. I put UnitySetupAttributes.cs in \Library\PackageCache\com.unity.test-framework@1.1.1x\UnityEngine.TestRunner\NUnitExtensions\Attributes and CompositeWorkItem.cs in \Library\PackageCache\com.unity.test-framework@1.1.1x\UnityEngine.TestRunner\NUnitExtensions\Runner. When I use UnityOneTimeSetup and UnityOneTimeTearDown, I get compilation errors. I want to use the official Unity test framework, not danstevens. Can someone help me figure out why this isn't working for me?

Thanks!

 

I, too, would like this feature. It would greatly simplify my test suite.

My test suite consists of a bunch of scenarios, followed by a bunch of Asserts to ensure the world is in my desired state. I could run my scenario to completion in a hypothetical [UnityOneTimeSetup], then run the actual Asserts in some trivial test methods.

 

Would also really love to see this. I completely agree with @akuno. Here's a concrete example of what he described that I would really love to be able to handle out of the box.

Code (CSharp):

1.  
2.     /// <summary>
3.     /// Contains tests for our <see cref="PlayFabClientAPI"/>
4.     /// </summary>
5.     public class PlayFabClientAPITests
6.     {
7.         [OneTimeSetUp]
8.         public IEnumerator SetupTest()
9.         {
10.             var playFabAuthorizationService = new PlayFabAuthorizationService();
11.             var unityTestUser = new PlayFabAuthorizationService.AuthorizedUser("UnitTester","UnitTester");
12.             var signInTask = playFabAuthorizationService.SignInAsync(unityTestUser);
13.             yield return signInTask.AsIEnumerator();
14.         }
15.      
16.         [UnityTest]
17.         [TestCase(7496062027744821083,"Betty", ExpectedResult = null)]
18.         [TestCase(1312412027744821083,"Todd", ExpectedResult = null)]
19.         [TestCase(5326062027744124354,"Miami", ExpectedResult = null)]
20.         public IEnumerator GetCatalogItem_LevelDataId_PlayFabCatalogItemFound(long levelDataId, string levelName)
21.         {
22.             //Get CatalogItem from server & verify not null...requires a sign in before PlayFabClientAPI.GetCatalogItems can be used.
23.         }
24.  

 

I can't believe this hasn't been added yet. I need this for my project as well. Surely it can't be that hard for Unity to add this to the official package.

EDIT: I have created my own fork here: https://github.com/8bitforest/com.unity.test-framework I have confirmed this works on 2020.2.1f1, and is based off the latest version of the official package. (@akuno @pwdstaraster a little late, but might help you guys)

EDIT 2: I also added [AsyncTest] [AsyncSetUp/TearDown] and [AsyncOneTimeSetUp/TearDown] attributes that can be added to methods returning "async Task"!

 

I also need this feature. Is anybody from Unity team aware of this feature request?

 

I don't think so. It would be good to know if this feature is at least in their roadmap

 

+1

 

+1

 

Should we keep adding +1 in this thread until Unity sees it?

 

Mission complete, boys and girls.

 

This should just become Unitys tag line at this stage. At least you all had time to create a lovely new logo though. It's what we have all been crying out for.

 

Please be nice, that decision was almost certainly above the pay grade of the people working on this library.

 

I am currently using this approach. There may be some downsides but it's working in my case.

Code (CSharp):

1.         private bool initialized = false;
2.         [UnitySetUp]
3.         public IEnumerator UnitySetup()
4.         {
5.             if (!initialized)
6.             {
7.                 initialized = true;
8.                 SceneManager.LoadScene(0, LoadSceneMode.Additive);
9.                 yield return null;
10.  
11.                 // Do some Stuff with Scene Objects
12.             }
13.         }
14.  
15.         [OneTimeTearDown]
16.         public static void OneTimeTearDown()
17.         {
18.             SceneManager.UnloadSceneAsync(0);
19.         }

 

+1 - That's also how I ended up doing it.

 

Hi everyone. I was pointed to this thread by @DrummerB here and wanted to drop in to share that the Unity Test Framework team has been working on a new pre-release version in the past months (read the announcement here) and that we are exploring what other product gaps we have. This new version doesn't address the requirement you are all discussing here, but we had it on our radar for quite some time. Before promising any delivery dates, we will first assess technical effort needed and will keep you updated on the plan.

 

It would be incredibly intuitive to add it, as it fits the schema perfectly and Googling what I thought would exist brought me here at the very top. For NUnit-related dev, I hope this is prioritized

Before I ended up here, I tried to type [UnityOneTimeSetup] (as it just makes sense) and was surprised that this doesn't exist.

 

A year later, any news?

 

@thewerku come on, it should be super simple to do this.
This is my workaround, looks bad/convoluted.

Code (CSharp):

1.  
2.         bool firstSetupCompleted = false;
3.  
4.         [UnitySetUp]
5.         public IEnumerator InitialSetup() {
6.             if (!firstSetupCompleted) {
7.                 firstSetupCompleted = true;
8.                 Debug.Log($"ONE TIME SETUP");
9.          
10.                 //DO STUFF
11.  
12.                 yield return new WaitForEndOfFrame();
13.             };
14.             Debug.Log($"ALL TIME SETUP");
15.         }

Edit: Just realized it's the exact same workaround posted earlier by @Flownity. Different roads, same destination