GitLab CI and PlayMode tests?

Hello.
Apparently you can't run playmode tests in batchmode, as batchmode does not support coroutines and async code. However, running Unity in regular mode (non batchmode) via GitLab CI seems to not spawn any test result files.

How do you folks manage to run tests in playmode on your GitLab CI configuration?

 

Hey. Thanks for the answer. To be honest I can't find the source but I'm pretty sure that I've read that on a ticket about unit tests in Unity.
Ok so, you confirm that play mode tests are supposed to work even in batch mode?

 

Thanks. Another question @superpig (my last question I'd say): do you use it with GitLabCI or know devs who do it?
I just need confirmation that my GitLabCI + play mode tests is possible and that the issue I get comes from a mistake I did and not an incompatibility of the two systems.

 

@laurentvictorino We use GitLab CI with Playmode tests. We're running a self-hosted GitLab Runner on Windows, executing a PS script. Works mostly fine. Recently we ran into issues about Unity not being able to create an OpenGL context, because GitLab Runner was being installed as a Windows Service, which can cause these kinds of issues. For some reason this only seems to happen on some systems or GPU drivers but not all. For now GitLab Runner is just registered as a startup item instead of a service.

 

@DrummerB Hey thank you for your answer. I'm looking that way now. I can't really set gitlab-runner as a startup item as it's on a building machine that is not operated - no real log in or session here -. But it gave me an idea and a good lead I guess.
May I ask you if you could share the command line you use in the `.gitlab-ci.yml` configuration file to start a Unity Build?
Thanks for your help.

 

Sure. Sorry for the late response, I was on holidays. I'm posting the relevant parts of the GitLab test setup. I was also thinking of writing it all up into a blog post at some point. It takes a bit of tinkering, but you have full control and it works quite well.

This is our current GitLab CI config file.

Code (yaml):

1. stages:
2.   - Verify
3.   - Build
4.   - Test
5.  
6. .Job:
7.   tags:
8.     - self-hosted, windows, powershell, unity
9.   variables:
10.     GIT_CLEAN_FLAGS: -ffdx -e Library/
11.   before_script:
12.     - BuildScripts/before_script.ps1
13.  
14. .Verify:
15.   extends: .Job
16.   stage: Verify
17.  
18. Verify Commit:
19.   extends: .Verify
20.   script:
21.     - BuildScripts/verify_lfs.ps1
22.  
23. .Build:
24.   extends: .Job
25.   stage: Build
26.   rules:
27.     - if: '$CI_MERGE_REQUEST_ID == null && $CI_COMMIT_MESSAGE !~ /[skip build]]/'
28.   script:
29.     - BuildScripts/build.ps1
30.   artifacts:
31.     name: "${env:CI_JOB_NAME}-${env:CI_COMMIT_SHORT_SHA}"
32.     expire_in: 3 days
33.     paths:
34.       - "./Builds"
35.  
36. Redacted-Win64:
37.   extends: .Build
38.   variables:
39.     BUILD_NAME: Redacted
40.     BUILD_TARGET: StandaloneWindows64
41.  
42. Redacted-Linux64:
43.   extends: .Build
44.   variables:
45.     BUILD_NAME: Redacted
46.     BUILD_TARGET: StandaloneLinux64
47.  
48. .Test:
49.   extends: .Job
50.   stage: Test
51.   dependencies: []
52.   script:
53.     - BuildScripts/build.ps1
54.   artifacts:
55.     when: on_failure
56.     reports:
57.       junit: TestResults/JUnit*.xml
58.       cobertura: CodeCoverage/*-cobertura/Cobertura.xml
59.     paths:
60.       - TestResults/
61.  
62. Editor Tests:
63.   extends: .Test
64.  
65. Runtime Tests:
66.   extends: .Test
67.  

And the relevant part of the build script:

Code (powershell):

1. $BUILD_PATH = Join-Path $PROJECT_PATH -ChildPath "Builds/$BUILD_TARGET/"
2. $TEST_PATH  = Join-Path $PROJECT_PATH -ChildPath "TestResults/"
3. $NUNIT_PATH = Join-Path $TEST_PATH -ChildPath "NUnit.xml"
4. $JUNIT_PATH = Join-Path $TEST_PATH -ChildPath "JUnit.xml"
5. $LOG_PATH   = Join-Path $PROJECT_PATH -ChildPath "Logs/Unity.log"
6. New-Item -Path $LOG_PATH -ItemType "file" -Force | Out-Null
7.  
8. # Prepare the arguments to pass to Unity.
9. $UNITY_ARGS = @()
10. $UNITY_ARGS += "-projectPath", """$PROJECT_PATH"""
11. $UNITY_ARGS += "-batchmode"
12. $UNITY_ARGS += "-logFile", "-" # This makes Unity print the logs to stdout, which we forward to the console (for GitLab and PowerShell) and a log file
13. $UNITY_ARGS += "-buildTarget", """$BUILD_TARGET"""
14. $UNITY_ARGS += "-buildPath", """$BUILD_PATH"""
15. $UNITY_ARGS += "-buildName", """$BUILD_NAME"""
16.  
17. if ($UNITY_CACHE_SERVER) {
18.   $UNITY_ARGS += "-CacheServerIPAddress", "$UNITY_CACHE_SERVER"
19. }
20.  
21. if ($CI_JOB_STAGE -eq "Build") {
22.   $UNITY_ARGS += "-executeMethod", "Redacted.Editor.BuildScript.PerformBuild"
23.   $UNITY_ARGS += "-quit"
24. }
25.  
26. if ($CI_JOB_NAME -eq "Editor Tests") {
27.   $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunEditorTests"
28.   $UNITY_ARGS += "-enableCodeCoverage"
29.   $UNITY_ARGS += "-coverageOptions", "assemblyFilters:+Redacted"
30. }
31.  
32. if ($CI_JOB_NAME -eq "Runtime Tests") {
33.   $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunRuntimeTests"
34. }
35.  
36. if ($CI_JOB_NAME -eq "Standalone Tests") {
37.   $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunStandaloneTests"
38. }
39.  
40. if (!(Test-Path $UNITY_EXE)) {
41.   "Did not find Unity $UNITY_VERSION."
42.   exit 1
43. }
44.  
45. # Run Unity
46. & $UNITY_EXE $UNITY_ARGS | Tee-Object -FilePath $LOG_PATH
47. $UNITY_EXIT_CODE = $LastExitCode
48.  
49. "--------------------------------------------------------------------------------"
50. "Finished with exit code: $UNITY_EXIT_CODE"
51. exit $UNITY_EXIT_CODE
52.  

The build method:

Code (CSharp):

1.  static void PerformBuild()
2.         {
3.             Console.WriteLine("Performing build");
4.  
5.             // Use the build target specified on the command line, or default to the active build target otherwise.
6.             string buildTargetArg = GetArgument("buildTarget");
7.             if (string.IsNullOrWhiteSpace(buildTargetArg) || !Enum.TryParse(buildTargetArg, out BuildTarget buildTarget))
8.             {
9.                 buildTarget = EditorUserBuildSettings.activeBuildTarget;
10.             }
11.  
12.             // If we do not support the selected build target (e.g. corresponding editor module is missing), abort.
13.             if (!BuildPipeline.IsBuildTargetSupported(BuildTargetGroup.Standalone, buildTarget))
14.             {
15.                 Console.WriteLine($"Build target {buildTarget.ToString()} not supported. Install the module from Unity Hub.");
16.                 if (Application.isBatchMode)
17.                 {
18.                     EditorApplication.Exit(1);
19.                 }
20.                 return;
21.             }
22.  
23.             string buildName = GetArgument("buildName") ?? Application.productName;
24.             string buildPath = GetArgument("buildPath") ?? Path.Combine("./Builds", buildTarget.ToString());
25.             string fullPath = Path.Combine(buildPath, buildName);
26.  
27.             if (buildTarget == BuildTarget.StandaloneWindows64 || buildTarget == BuildTarget.StandaloneWindows)
28.             {
29.                 fullPath += ".exe";
30.             }
31.  
32.             var options = new BuildPlayerOptions
33.             {
34.                 scenes = (from scene in EditorBuildSettings.scenes where scene.enabled select scene.path).ToArray(),
35.                 locationPathName = fullPath,
36.                 target = buildTarget,
37.                 options = Application.isBatchMode ? BuildOptions.None : BuildOptions.ShowBuiltPlayer
38.             };
39.  
40.             var report = BuildPipeline.BuildPlayer(options);
41.             LogBuildReport(report);
42.  
43.             if (Application.isBatchMode)
44.                 EditorApplication.Exit(report.summary.result == BuildResult.Succeeded ? 0 : 1);
45.         }

And the test runner:

Code (CSharp):

1. public class TestRunner : IPrebuildSetup, IPostBuildCleanup, ITestRunCallback
2.     {
3.         #if UNITY_EDITOR
4.         [MenuItem("Test/Run Editor Tests")]
5.         static void RunEditorTests()
6.         {
7.             Console.WriteLine("Running editor tests");
8.  
9.             var filter = new Filter
10.             {
11.                 testMode = TestMode.EditMode,
12.                 groupNames = new[] {@"^Redacted\.Tests\.Editor\."}
13.             };
14.             ScriptableObject.CreateInstance<TestRunnerApi>().Execute(new ExecutionSettings(filter));
15.         }
16.  
17.         [MenuItem("Test/Run Runtime Tests")]
18.         static void RunRuntimeTests()
19.         {
20.             Console.WriteLine("Running runtime tests");
21.  
22.             var testRunner = ScriptableObject.CreateInstance<TestRunnerApi>();
23.             var filter = new Filter
24.             {
25.                 testMode = TestMode.PlayMode,
26.                 groupNames = new[] {@"^Redacted\.Tests\.Runtime\."}
27.             };
28.             testRunner.Execute(new ExecutionSettings(filter));
29.         }
30.  
31.         [MenuItem("Test/Run Standalone Tests")]
32.         static void RunStandaloneTests()
33.         {
34.             Console.WriteLine("Running standalone tests");
35.  
36.             var testRunner = ScriptableObject.CreateInstance<TestRunnerApi>();
37.             var filter = new Filter
38.             {
39.                 testMode = TestMode.PlayMode,
40.                 groupNames = new[] {@"^Redacted\.Tests\.Runtime\."},
41.                 targetPlatform = EditorUserBuildSettings.activeBuildTarget
42.             };
43.             testRunner.Execute(new ExecutionSettings(filter));
44.         }
45.         #endif
46.  
47.         public void RunStarted(ITest tests)
48.         {
49.             Debug.Log("TestManager.RunStarted");
50.         }
51.  
52.         public void RunFinished(ITestResult testResults)
53.         {
54.             Debug.Log("TestManager.RunFinished");
55.  
56.             ResultsWriter.WriteResults(testResults);
57.             CoverageReportGenerator.Generate();
58.  
59.             if (Application.isBatchMode)
60.             {
61.                 int returnValue = 1;
62.                 switch (testResults.ResultState.Status)
63.                 {
64.                     case NUnit.Framework.Interfaces.TestStatus.Inconclusive:
65.                         returnValue = 2;
66.                         break;
67.                     case NUnit.Framework.Interfaces.TestStatus.Skipped:
68.                         returnValue = 0;
69.                         break;
70.                     case NUnit.Framework.Interfaces.TestStatus.Passed:
71.                         returnValue = 0;
72.                         break;
73.                     case NUnit.Framework.Interfaces.TestStatus.Failed:
74.                         returnValue = 1;
75.                         break;
76.                 }
77.                 Console.WriteLine($"Test Result: {testResults.ResultState}");
78.                 #if UNITY_EDITOR
79.                 EditorApplication.Exit(returnValue);
80.                 #endif
81.             }
82.         }
83.     }

 

Hey @DrummerB thanks for your reply (I hope holidays were good!).
Thanks for sharing all the good stuff. I've been looking at it, and there is only one thing I'm not sure to get: What does GetArgument() is supposed to do? Where does it come from?
Thanks.

 

@laurentvictorino It just reads the passed command line arguments.

Code (CSharp):

1.  
2.         static string GetArgument(string name)
3.         {
4.             var args = Environment.GetCommandLineArgs();
5.             for (int i = 0; i < args.Length; i++)
6.             {
7.                 if (args[i].Contains(name))
8.                 {
9.                     return args[i + 1];
10.                 }
11.             }
12.  
13.             return null;
14.         }
15.  

Should probably just parse the arguments once into a hash table and then use that, but this works ok.

 

@DrummerB apologies for posting on such an old thread, but it was the only one that I could find that was relevant, so I thought it might be useful for future readers.

What you've done looks great! Got most of it working, but would love to get the coverage integrated with GitLab's tracking and highlighting in MRs for which you need cobertura formatted output, which it looks like you have.

From what I can tell, the default Unity coverage output is in a custom format. Did you write a custom tool to that reformatting, or is that an option in Unity that I'm missing?

Thanks!

 

Unity's coverage package uses the third-party ReportGenerator project to generate its coverage output. So if your project includes the coverage package, you have access to the ReportGenerator and can use it to output any format you want. Not the most elegant solution, but this is what we do:

Code (CSharp):

1. using System;
2. using System.IO;
3. using System.Linq;
4. using System.Xml;
5. using Palmmedia.ReportGenerator.Core;
6. using Palmmedia.ReportGenerator.Core.CodeAnalysis;
7. using UnityEditor;
8. using UnityEngine;
9.  
10. /// <summary>
11. /// This is used by the TestRunner after a test run was finished to convert the generated
12. /// OpenCover formatted code coverage results into other report formats.
13. /// The CodeCoverage package from Unity currently (v0.3-preview) only supports generating
14. /// html reports. GitLab however requires Cobertura format, which we generate here.
15. /// </summary>
16. public static class CoverageReportGenerator
17. {
18.     public static bool Generate()
19.     {
20.         // For config details, see https://github.com/danielpalme/ReportGenerator#usage--command-line-parameters
21.         var projectPath = Directory.GetCurrentDirectory();
22.         var coveragePath = Path.Combine(projectPath, "CodeCoverage");
23.         var reportFilePatterns = new[] {Path.Combine(coveragePath, "**/TestCoverageResults_????.xml")};
24.         var targetDirectory = coveragePath;
25.         var sourceDirectories = new string[]{};
26.         string historyDirectory = null;
27.         var reportTypes = new[]{"Cobertura", "TextSummary", "Html"};
28.         var plugins = new string[]{};
29.         var assemblyFilters = new[]{"+*"};
30.         var classFilters = new[]{"+*"};
31.         var fileFilters = new[]{"+*"};
32.  
33.         string verbosityLevel = null;
34.         string tag = null;
35.  
36.         var config = new ReportConfiguration(
37.             reportFilePatterns,
38.             targetDirectory,
39.             sourceDirectories,
40.             historyDirectory,
41.             reportTypes,
42.             plugins,
43.             assemblyFilters,
44.             classFilters,
45.             fileFilters,
46.             verbosityLevel,
47.             tag);
48.  
49.         var settings = new Settings();
50.         var thresholds = new RiskHotspotsAnalysisThresholds();
51.         string coberturaPath = Path.Combine(targetDirectory, "Cobertura.xml");
52.  
53.         if (!Application.isBatchMode)
54.             EditorUtility.DisplayProgressBar("Code Coverage", "Generating Coverage Report", 0.4f);
55.         else
56.             Debug.Log("Generating Cobertura Report");
57.  
58.         bool success;
59.         try
60.         {
61.             var generator = new Generator();
62.             success = generator.GenerateReport(config, settings, thresholds);
63.  
64.             if (!File.Exists(coberturaPath))
65.             {
66.                 success = false;
67.                 throw new FileNotFoundException(coberturaPath);
68.             }
69.  
70.             // Replace absolute file paths in the cobertura XML with relative paths (to the project root).
71.             // This is required for GitLab to be able to parse the results.
72.             var document = new XmlDocument();
73.             document.Load(coberturaPath);
74.             const string xPath = "/coverage/packages/package/classes/class/@filename";
75.             var fileNameAttributes = document.DocumentElement?.SelectNodes(xPath);
76.             if (fileNameAttributes != null)
77.             {
78.                 foreach (XmlAttribute attribute in fileNameAttributes)
79.                 {
80.                     // Remove project path prefix, Path.GetRelativePath() requires .NET Standard 2.1
81.                     if (attribute.Value.StartsWith(projectPath)) {
82.                         attribute.Value = attribute.Value.Substring(projectPath.Length + 1).Replace("\\", "/");
83.                     }
84.                 }
85.             }
86.             document.Save(coberturaPath);
87.         }
88.         finally
89.         {
90.             EditorUtility.ClearProgressBar();
91.         }
92.  
93.         if (success)
94.             Debug.Log($"Cobertura Code Coverage Report was generated in {targetDirectory}");
95.         else
96.             Debug.LogError("Failed to generate Code Coverage Report.");
97.  
98.         // Log the coverage summary to the console.
99.         var summaryPath = Path.Combine(coveragePath, "Summary.txt");
100.         if (File.Exists(summaryPath))
101.         {
102.             var summary = string.Join("\n", File.ReadLines(summaryPath).Take(11));
103.             if (Application.isBatchMode)
104.                 Console.WriteLine(summary);
105.             else
106.                 Debug.Log(summary);
107.         }
108.  
109.         return success;
110.     }
111. }

Code (CSharp):

1. public class TestRunner
2. {
3.     [UsedImplicitly] // by the build script
4.     static void GenerateCodeCoverageReport()
5.     {
6.         Console.WriteLine("Generating code coverage report.");
7.         bool success = CoverageReportGenerator.Generate();
8.  
9.         #if UNITY_EDITOR
10.         EditorApplication.Exit(success ? 0 : 1);
11.         #endif
12.     }
13. }

The TestRunner.GenerateCodeCoverageReport method is executed using the Unity CLI in a separate GitLab job after all test jobs have been completed.

Code (csharp):

1. Code Coverage:
2.   stage: Report
3.   tags:
4.     - self-hosted, windows, powershell, unity
5.   variables:
6.     GIT_CLEAN_FLAGS: -ffdx -e Library/
7.   before_script:
8.     - BuildScripts/before_script.ps1
9.   script:
10.     - BuildScripts/build.ps1
11.   dependencies:
12.     - Editor Tests
13.     - Runtime Tests
14.   rules:
15.     - if: '$CI_MERGE_REQUEST_ID == null && $CI_COMMIT_MESSAGE !~ /[skip test]]/'
16.       when: always
17.   coverage: '/^ *Line coverage: (\d+\.\d+%)$/'
18.   artifacts:
19.     reports:
20.       cobertura: CodeCoverage/Cobertura.xml
21.     paths:
22.       - CodeCoverage/