Starting tests from a Dll in a Standalone application

Hello,

I'm currently trying to use Unity Test Framework v1.1.33 to run some tests from a Dll (that references nunit.framework.dll) from a Standalone executable, because I want to use this exe in an automated test without worrying if the machine running it has Unity installed or not.

I can see the tests from the Dll in the Editor and run them in Play Mode from the Test Runner window, and everything works.

My issues come when I build the Standalone app and try to run it. I have a script on a GameObject in the scene called TestController that starts the tests programatically and also registers some callbacks to log information about the tests.

When I run this exe, I get a runtime error saying that:

The referenced script (TestController) on this Behaviour is missing!
The referenced script on this Behaviour (Game Object 'TestController') is missing!

Even though the script is not missing, the class inside it has the same name as the file (and the GameObject) as I've seen that's a common cause for this type of issue.

I also tried to split the Build and Run process for Standalone tests, documented here, in hope that I can use the executable built by this step, but this comes with this problem:

When I press Run All Tests (StandaloneWindows64) in the Editor, suddenly I get errors that no type from the TestRunnerApi can be found.

E.g.
Assets\BuildModifier.cs(19,49): error CS0246: The type or namespace name 'BuildPlayerOptions' could not be found (are you missing a using directive or an assembly reference?)
Assets\CallbacksHandler.cs(22,29): error CS0246: The type or namespace name 'ITestResultAdaptor' could not be found (are you missing a using directive or an assembly reference?)

Has anyone managed to achieve something similar to what I'm trying to do or stumbled upon issues like these?

I basically just want to have an .exe that I can run and it starts the tests automatically, without depending on the Editor or Unity at all.

Thanks in advance!

 

Thank you for the clarification, I've stopped trying to start them programatically and indeed I can build a scene that starts the tests, however splitting the Build and Run doesn't seem to work, and neither does registerring callbacks.

I have two scripts, BuildModifier.cs and ResultSerializer.cs that look like this.

Code (CSharp):

1. using System;
2. using System.IO;
3. using System.Linq;
4. using UnityEditor;
5. using UnityEditor.TestTools;
6. using UnityEngine;
7. using UnityEngine.TestTools;
8.  
9. [assembly: TestPlayerBuildModifier(typeof(HeadlessPlayModeSetup))]
10. public class HeadlessPlayModeSetup : ITestPlayerBuildModifier
11. {
12.     public BuildPlayerOptions ModifyOptions(BuildPlayerOptions playerOptions)
13.     {
14.         // Do not launch the player after the build completes.
15.         playerOptions.options &= ~BuildOptions.AutoRunPlayer;
16.  
17.         // Set the headlessBuildLocation to the output directory you desire. It does not need to be inside the project.
18.         var headlessBuildLocation = Path.GetFullPath(Path.Combine(Application.dataPath, ".//..//PlayModeTestPlayer"));
19.         var fileName = Path.GetFileName(playerOptions.locationPathName);
20.         if (!string.IsNullOrEmpty(fileName))
21.         {
22.             headlessBuildLocation = Path.Combine(headlessBuildLocation, fileName);
23.         }
24.         playerOptions.locationPathName = headlessBuildLocation;
25.  
26.         return playerOptions;
27.     }
28. }

Code (CSharp):

1. using NUnit.Framework.Interfaces;
2. using System.Collections;
3. using System.Collections.Generic;
4. using System.IO;
5. using System.Xml;
6. using UnityEngine;
7. using UnityEngine.TestRunner;
8.  
9. [assembly:TestRunCallback(typeof(ResultSerializer))]
10.  
11. public class ResultSerializer : ITestRunCallback
12. {
13.     public void RunStarted(ITest testsToRun)
14.     {
15.         Debug.Log($"Run started, the tests to run are: ");
16.  
17.         if (testsToRun.HasChildren)
18.         {
19.             foreach (var child in testsToRun.Tests)
20.             {
21.                 Debug.Log($"{child.Name}");
22.             }
23.         }
24.     }
25.  
26.     public void RunFinished(ITestResult testResults)
27.     {
28.         Debug.Log($"Run finished with result: {testResults}");
29.  
30.         var path = Path.Combine(Application.persistentDataPath, "testresults.xml");
31.  
32.         using (var xw = XmlWriter.Create(path, new XmlWriterSettings { Indent = true }))
33.             testResults.ToXml(true).WriteTo(xw);
34.  
35.         Debug.Log($"***\n\nTEST RESULTS WRITTEN TO\n\n\t{path}In\n***");
36.         Application.Quit(testResults.FailCount > 0 ? 1 : 0);
37.     }
38.  
39.     public void TestStarted(ITest test)
40.     {
41.         Debug.Log($"Test {test.Name} started. ");
42.     }
43.  
44.     public void TestFinished(ITestResult result)
45.     {
46.         if (!result.Test.IsSuite)
47.         {
48.             Debug.Log($"Test finished {result.Test.Name} {result.ResultState}.");
49.             result.ToXml(true);
50.         }
51.     }
52. }
53.  

But when I click Run All Tests (StandaloneWindows64), the build folder generated is still in a temp directory, instead of where I tell it to be, and I see no logs or result files from the callbacks, it's like these scripts are ignored.

I have the Tests Assembly Definition Asset enabled on all Platforms in the Inspector (including Windows 64-bit), but weirdly enough, if I uncheck the "Any Platform" checkbox and manually check the platforms instead, it looks like splitting the Build and Run works (I can see that the build options are set correctly, e.g.: the locationPathName and the options.AutoRunPlayer = False), but the build fails because of the old errors:

Assets\BuildModifier.cs(19,49): error CS0246: The type or namespace name 'BuildPlayerOptions' could not be found (are you missing a using directive or an assembly reference?)
Assets\CallbacksHandler.cs(22,29): error CS0246: The type or namespace name 'ITestResultAdaptor' could not be found (are you missing a using directive or an assembly reference?)

Edit: I'm using Unity 2021.3.6f1 if it helps.

 

Update:

Turns out I shouldn't include the BuildModifier.cs code in the Standalone application, so I've disabled the code on anything that is not the Editor and now I can split the Build and Run and also have callbacks!

This can be closed!