Search Unity

GitLab CI and PlayMode tests?

Discussion in 'Testing & Automation' started by laurentvictorino, Jul 30, 2020.

  1. laurentvictorino

    laurentvictorino

    Joined:
    Mar 2, 2017
    Posts:
    7
    Hello.
    Apparently you can't run playmode tests in batchmode, as batchmode does not support coroutines and async code. However, running Unity in regular mode (non batchmode) via GitLab CI seems to not spawn any test result files.

    How do you folks manage to run tests in playmode on your GitLab CI configuration?
     
    Last edited: Jul 30, 2020
  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
  3. laurentvictorino

    laurentvictorino

    Joined:
    Mar 2, 2017
    Posts:
    7
    Hey. Thanks for the answer. To be honest I can't find the source but I'm pretty sure that I've read that on a ticket about unit tests in Unity.
    Ok so, you confirm that play mode tests are supposed to work even in batch mode?
     
  4. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    Yes, absolutely. We use them extensively when testing Unity itself.
     
    Juxta and karl_jones like this.
  5. laurentvictorino

    laurentvictorino

    Joined:
    Mar 2, 2017
    Posts:
    7
    Thanks. Another question @superpig (my last question I'd say): do you use it with GitLabCI or know devs who do it?
    I just need confirmation that my GitLabCI + play mode tests is possible and that the issue I get comes from a mistake I did and not an incompatibility of the two systems.
     
  6. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    @laurentvictorino We use GitLab CI with Playmode tests. We're running a self-hosted GitLab Runner on Windows, executing a PS script. Works mostly fine. Recently we ran into issues about Unity not being able to create an OpenGL context, because GitLab Runner was being installed as a Windows Service, which can cause these kinds of issues. For some reason this only seems to happen on some systems or GPU drivers but not all. For now GitLab Runner is just registered as a startup item instead of a service.
     
  7. laurentvictorino

    laurentvictorino

    Joined:
    Mar 2, 2017
    Posts:
    7
    @DrummerB Hey thank you for your answer. I'm looking that way now. I can't really set gitlab-runner as a startup item as it's on a building machine that is not operated - no real log in or session here -. But it gave me an idea and a good lead I guess.
    May I ask you if you could share the command line you use in the `.gitlab-ci.yml` configuration file to start a Unity Build?
    Thanks for your help.
     
  8. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    Sure. Sorry for the late response, I was on holidays. I'm posting the relevant parts of the GitLab test setup. I was also thinking of writing it all up into a blog post at some point. It takes a bit of tinkering, but you have full control and it works quite well.

    This is our current GitLab CI config file.

    Code (yaml):
    1. stages:
    2.   - Verify
    3.   - Build
    4.   - Test
    5.  
    6. .Job:
    7.   tags:
    8.     - self-hosted, windows, powershell, unity
    9.   variables:
    10.     GIT_CLEAN_FLAGS: -ffdx -e Library/
    11.   before_script:
    12.     - BuildScripts/before_script.ps1
    13.  
    14. .Verify:
    15.   extends: .Job
    16.   stage: Verify
    17.  
    18. Verify Commit:
    19.   extends: .Verify
    20.   script:
    21.     - BuildScripts/verify_lfs.ps1
    22.  
    23. .Build:
    24.   extends: .Job
    25.   stage: Build
    26.   rules:
    27.     - if: '$CI_MERGE_REQUEST_ID == null && $CI_COMMIT_MESSAGE !~ /[skip build]]/'
    28.   script:
    29.     - BuildScripts/build.ps1
    30.   artifacts:
    31.     name: "${env:CI_JOB_NAME}-${env:CI_COMMIT_SHORT_SHA}"
    32.     expire_in: 3 days
    33.     paths:
    34.       - "./Builds"
    35.  
    36. Redacted-Win64:
    37.   extends: .Build
    38.   variables:
    39.     BUILD_NAME: Redacted
    40.     BUILD_TARGET: StandaloneWindows64
    41.  
    42. Redacted-Linux64:
    43.   extends: .Build
    44.   variables:
    45.     BUILD_NAME: Redacted
    46.     BUILD_TARGET: StandaloneLinux64
    47.  
    48. .Test:
    49.   extends: .Job
    50.   stage: Test
    51.   dependencies: []
    52.   script:
    53.     - BuildScripts/build.ps1
    54.   artifacts:
    55.     when: on_failure
    56.     reports:
    57.       junit: TestResults/JUnit*.xml
    58.       cobertura: CodeCoverage/*-cobertura/Cobertura.xml
    59.     paths:
    60.       - TestResults/
    61.  
    62. Editor Tests:
    63.   extends: .Test
    64.  
    65. Runtime Tests:
    66.   extends: .Test
    67.  
    And the relevant part of the build script:

    Code (powershell):
    1. $BUILD_PATH = Join-Path $PROJECT_PATH -ChildPath "Builds/$BUILD_TARGET/"
    2. $TEST_PATH  = Join-Path $PROJECT_PATH -ChildPath "TestResults/"
    3. $NUNIT_PATH = Join-Path $TEST_PATH -ChildPath "NUnit.xml"
    4. $JUNIT_PATH = Join-Path $TEST_PATH -ChildPath "JUnit.xml"
    5. $LOG_PATH   = Join-Path $PROJECT_PATH -ChildPath "Logs/Unity.log"
    6. New-Item -Path $LOG_PATH -ItemType "file" -Force | Out-Null
    7.  
    8. # Prepare the arguments to pass to Unity.
    9. $UNITY_ARGS = @()
    10. $UNITY_ARGS += "-projectPath", """$PROJECT_PATH"""
    11. $UNITY_ARGS += "-batchmode"
    12. $UNITY_ARGS += "-logFile", "-" # This makes Unity print the logs to stdout, which we forward to the console (for GitLab and PowerShell) and a log file
    13. $UNITY_ARGS += "-buildTarget", """$BUILD_TARGET"""
    14. $UNITY_ARGS += "-buildPath", """$BUILD_PATH"""
    15. $UNITY_ARGS += "-buildName", """$BUILD_NAME"""
    16.  
    17. if ($UNITY_CACHE_SERVER) {
    18.   $UNITY_ARGS += "-CacheServerIPAddress", "$UNITY_CACHE_SERVER"
    19. }
    20.  
    21. if ($CI_JOB_STAGE -eq "Build") {
    22.   $UNITY_ARGS += "-executeMethod", "Redacted.Editor.BuildScript.PerformBuild"
    23.   $UNITY_ARGS += "-quit"
    24. }
    25.  
    26. if ($CI_JOB_NAME -eq "Editor Tests") {
    27.   $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunEditorTests"
    28.   $UNITY_ARGS += "-enableCodeCoverage"
    29.   $UNITY_ARGS += "-coverageOptions", "assemblyFilters:+Redacted"
    30. }
    31.  
    32. if ($CI_JOB_NAME -eq "Runtime Tests") {
    33.   $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunRuntimeTests"
    34. }
    35.  
    36. if ($CI_JOB_NAME -eq "Standalone Tests") {
    37.   $UNITY_ARGS += "-executeMethod", "Redacted.Tests.TestRunner.RunStandaloneTests"
    38. }
    39.  
    40. if (!(Test-Path $UNITY_EXE)) {
    41.   "Did not find Unity $UNITY_VERSION."
    42.   exit 1
    43. }
    44.  
    45. # Run Unity
    46. & $UNITY_EXE $UNITY_ARGS | Tee-Object -FilePath $LOG_PATH
    47. $UNITY_EXIT_CODE = $LastExitCode
    48.  
    49. "--------------------------------------------------------------------------------"
    50. "Finished with exit code: $UNITY_EXIT_CODE"
    51. exit $UNITY_EXIT_CODE
    52.  
    The build method:

    Code (CSharp):
    1.  static void PerformBuild()
    2.         {
    3.             Console.WriteLine("Performing build");
    4.  
    5.             // Use the build target specified on the command line, or default to the active build target otherwise.
    6.             string buildTargetArg = GetArgument("buildTarget");
    7.             if (string.IsNullOrWhiteSpace(buildTargetArg) || !Enum.TryParse(buildTargetArg, out BuildTarget buildTarget))
    8.             {
    9.                 buildTarget = EditorUserBuildSettings.activeBuildTarget;
    10.             }
    11.  
    12.             // If we do not support the selected build target (e.g. corresponding editor module is missing), abort.
    13.             if (!BuildPipeline.IsBuildTargetSupported(BuildTargetGroup.Standalone, buildTarget))
    14.             {
    15.                 Console.WriteLine($"Build target {buildTarget.ToString()} not supported. Install the module from Unity Hub.");
    16.                 if (Application.isBatchMode)
    17.                 {
    18.                     EditorApplication.Exit(1);
    19.                 }
    20.                 return;
    21.             }
    22.  
    23.             string buildName = GetArgument("buildName") ?? Application.productName;
    24.             string buildPath = GetArgument("buildPath") ?? Path.Combine("./Builds", buildTarget.ToString());
    25.             string fullPath = Path.Combine(buildPath, buildName);
    26.  
    27.             if (buildTarget == BuildTarget.StandaloneWindows64 || buildTarget == BuildTarget.StandaloneWindows)
    28.             {
    29.                 fullPath += ".exe";
    30.             }
    31.  
    32.             var options = new BuildPlayerOptions
    33.             {
    34.                 scenes = (from scene in EditorBuildSettings.scenes where scene.enabled select scene.path).ToArray(),
    35.                 locationPathName = fullPath,
    36.                 target = buildTarget,
    37.                 options = Application.isBatchMode ? BuildOptions.None : BuildOptions.ShowBuiltPlayer
    38.             };
    39.  
    40.             var report = BuildPipeline.BuildPlayer(options);
    41.             LogBuildReport(report);
    42.  
    43.             if (Application.isBatchMode)
    44.                 EditorApplication.Exit(report.summary.result == BuildResult.Succeeded ? 0 : 1);
    45.         }
    And the test runner:

    Code (CSharp):
    1. public class TestRunner : IPrebuildSetup, IPostBuildCleanup, ITestRunCallback
    2.     {
    3.         #if UNITY_EDITOR
    4.         [MenuItem("Test/Run Editor Tests")]
    5.         static void RunEditorTests()
    6.         {
    7.             Console.WriteLine("Running editor tests");
    8.  
    9.             var filter = new Filter
    10.             {
    11.                 testMode = TestMode.EditMode,
    12.                 groupNames = new[] {@"^Redacted\.Tests\.Editor\."}
    13.             };
    14.             ScriptableObject.CreateInstance<TestRunnerApi>().Execute(new ExecutionSettings(filter));
    15.         }
    16.  
    17.         [MenuItem("Test/Run Runtime Tests")]
    18.         static void RunRuntimeTests()
    19.         {
    20.             Console.WriteLine("Running runtime tests");
    21.  
    22.             var testRunner = ScriptableObject.CreateInstance<TestRunnerApi>();
    23.             var filter = new Filter
    24.             {
    25.                 testMode = TestMode.PlayMode,
    26.                 groupNames = new[] {@"^Redacted\.Tests\.Runtime\."}
    27.             };
    28.             testRunner.Execute(new ExecutionSettings(filter));
    29.         }
    30.  
    31.         [MenuItem("Test/Run Standalone Tests")]
    32.         static void RunStandaloneTests()
    33.         {
    34.             Console.WriteLine("Running standalone tests");
    35.  
    36.             var testRunner = ScriptableObject.CreateInstance<TestRunnerApi>();
    37.             var filter = new Filter
    38.             {
    39.                 testMode = TestMode.PlayMode,
    40.                 groupNames = new[] {@"^Redacted\.Tests\.Runtime\."},
    41.                 targetPlatform = EditorUserBuildSettings.activeBuildTarget
    42.             };
    43.             testRunner.Execute(new ExecutionSettings(filter));
    44.         }
    45.         #endif
    46.  
    47.         public void RunStarted(ITest tests)
    48.         {
    49.             Debug.Log("TestManager.RunStarted");
    50.         }
    51.  
    52.         public void RunFinished(ITestResult testResults)
    53.         {
    54.             Debug.Log("TestManager.RunFinished");
    55.  
    56.             ResultsWriter.WriteResults(testResults);
    57.             CoverageReportGenerator.Generate();
    58.  
    59.             if (Application.isBatchMode)
    60.             {
    61.                 int returnValue = 1;
    62.                 switch (testResults.ResultState.Status)
    63.                 {
    64.                     case NUnit.Framework.Interfaces.TestStatus.Inconclusive:
    65.                         returnValue = 2;
    66.                         break;
    67.                     case NUnit.Framework.Interfaces.TestStatus.Skipped:
    68.                         returnValue = 0;
    69.                         break;
    70.                     case NUnit.Framework.Interfaces.TestStatus.Passed:
    71.                         returnValue = 0;
    72.                         break;
    73.                     case NUnit.Framework.Interfaces.TestStatus.Failed:
    74.                         returnValue = 1;
    75.                         break;
    76.                 }
    77.                 Console.WriteLine($"Test Result: {testResults.ResultState}");
    78.                 #if UNITY_EDITOR
    79.                 EditorApplication.Exit(returnValue);
    80.                 #endif
    81.             }
    82.         }
    83.     }
     
    Noxalus likes this.
  9. laurentvictorino

    laurentvictorino

    Joined:
    Mar 2, 2017
    Posts:
    7
    Hey @DrummerB thanks for your reply (I hope holidays were good!).
    Thanks for sharing all the good stuff. I've been looking at it, and there is only one thing I'm not sure to get: What does GetArgument() is supposed to do? Where does it come from?
    Thanks.
     
  10. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    @laurentvictorino It just reads the passed command line arguments.

    Code (CSharp):
    1.  
    2.         static string GetArgument(string name)
    3.         {
    4.             var args = Environment.GetCommandLineArgs();
    5.             for (int i = 0; i < args.Length; i++)
    6.             {
    7.                 if (args[i].Contains(name))
    8.                 {
    9.                     return args[i + 1];
    10.                 }
    11.             }
    12.  
    13.             return null;
    14.         }
    15.  
    Should probably just parse the arguments once into a hash table and then use that, but this works ok.
     
    Last edited: Aug 12, 2020
  11. caspar-five

    caspar-five

    Joined:
    Feb 7, 2019
    Posts:
    1
    @DrummerB apologies for posting on such an old thread, but it was the only one that I could find that was relevant, so I thought it might be useful for future readers.

    What you've done looks great! Got most of it working, but would love to get the coverage integrated with GitLab's tracking and highlighting in MRs for which you need cobertura formatted output, which it looks like you have.

    From what I can tell, the default Unity coverage output is in a custom format. Did you write a custom tool to that reformatting, or is that an option in Unity that I'm missing?

    Thanks!
     
  12. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    127
    Unity's coverage package uses the third-party ReportGenerator project to generate its coverage output. So if your project includes the coverage package, you have access to the ReportGenerator and can use it to output any format you want. Not the most elegant solution, but this is what we do:

    Code (CSharp):
    1. using System;
    2. using System.IO;
    3. using System.Linq;
    4. using System.Xml;
    5. using Palmmedia.ReportGenerator.Core;
    6. using Palmmedia.ReportGenerator.Core.CodeAnalysis;
    7. using UnityEditor;
    8. using UnityEngine;
    9.  
    10. /// <summary>
    11. /// This is used by the TestRunner after a test run was finished to convert the generated
    12. /// OpenCover formatted code coverage results into other report formats.
    13. /// The CodeCoverage package from Unity currently (v0.3-preview) only supports generating
    14. /// html reports. GitLab however requires Cobertura format, which we generate here.
    15. /// </summary>
    16. public static class CoverageReportGenerator
    17. {
    18.     public static bool Generate()
    19.     {
    20.         // For config details, see https://github.com/danielpalme/ReportGenerator#usage--command-line-parameters
    21.         var projectPath = Directory.GetCurrentDirectory();
    22.         var coveragePath = Path.Combine(projectPath, "CodeCoverage");
    23.         var reportFilePatterns = new[] {Path.Combine(coveragePath, "**/TestCoverageResults_????.xml")};
    24.         var targetDirectory = coveragePath;
    25.         var sourceDirectories = new string[]{};
    26.         string historyDirectory = null;
    27.         var reportTypes = new[]{"Cobertura", "TextSummary", "Html"};
    28.         var plugins = new string[]{};
    29.         var assemblyFilters = new[]{"+*"};
    30.         var classFilters = new[]{"+*"};
    31.         var fileFilters = new[]{"+*"};
    32.  
    33.         string verbosityLevel = null;
    34.         string tag = null;
    35.  
    36.         var config = new ReportConfiguration(
    37.             reportFilePatterns,
    38.             targetDirectory,
    39.             sourceDirectories,
    40.             historyDirectory,
    41.             reportTypes,
    42.             plugins,
    43.             assemblyFilters,
    44.             classFilters,
    45.             fileFilters,
    46.             verbosityLevel,
    47.             tag);
    48.  
    49.         var settings = new Settings();
    50.         var thresholds = new RiskHotspotsAnalysisThresholds();
    51.         string coberturaPath = Path.Combine(targetDirectory, "Cobertura.xml");
    52.  
    53.         if (!Application.isBatchMode)
    54.             EditorUtility.DisplayProgressBar("Code Coverage", "Generating Coverage Report", 0.4f);
    55.         else
    56.             Debug.Log("Generating Cobertura Report");
    57.  
    58.         bool success;
    59.         try
    60.         {
    61.             var generator = new Generator();
    62.             success = generator.GenerateReport(config, settings, thresholds);
    63.  
    64.             if (!File.Exists(coberturaPath))
    65.             {
    66.                 success = false;
    67.                 throw new FileNotFoundException(coberturaPath);
    68.             }
    69.  
    70.             // Replace absolute file paths in the cobertura XML with relative paths (to the project root).
    71.             // This is required for GitLab to be able to parse the results.
    72.             var document = new XmlDocument();
    73.             document.Load(coberturaPath);
    74.             const string xPath = "/coverage/packages/package/classes/class/@filename";
    75.             var fileNameAttributes = document.DocumentElement?.SelectNodes(xPath);
    76.             if (fileNameAttributes != null)
    77.             {
    78.                 foreach (XmlAttribute attribute in fileNameAttributes)
    79.                 {
    80.                     // Remove project path prefix, Path.GetRelativePath() requires .NET Standard 2.1
    81.                     if (attribute.Value.StartsWith(projectPath)) {
    82.                         attribute.Value = attribute.Value.Substring(projectPath.Length + 1).Replace("\\", "/");
    83.                     }
    84.                 }
    85.             }
    86.             document.Save(coberturaPath);
    87.         }
    88.         finally
    89.         {
    90.             EditorUtility.ClearProgressBar();
    91.         }
    92.  
    93.         if (success)
    94.             Debug.Log($"Cobertura Code Coverage Report was generated in {targetDirectory}");
    95.         else
    96.             Debug.LogError("Failed to generate Code Coverage Report.");
    97.  
    98.         // Log the coverage summary to the console.
    99.         var summaryPath = Path.Combine(coveragePath, "Summary.txt");
    100.         if (File.Exists(summaryPath))
    101.         {
    102.             var summary = string.Join("\n", File.ReadLines(summaryPath).Take(11));
    103.             if (Application.isBatchMode)
    104.                 Console.WriteLine(summary);
    105.             else
    106.                 Debug.Log(summary);
    107.         }
    108.  
    109.         return success;
    110.     }
    111. }
    Code (CSharp):
    1. public class TestRunner
    2. {
    3.     [UsedImplicitly] // by the build script
    4.     static void GenerateCodeCoverageReport()
    5.     {
    6.         Console.WriteLine("Generating code coverage report.");
    7.         bool success = CoverageReportGenerator.Generate();
    8.  
    9.         #if UNITY_EDITOR
    10.         EditorApplication.Exit(success ? 0 : 1);
    11.         #endif
    12.     }
    13. }
    The TestRunner.GenerateCodeCoverageReport method is executed using the Unity CLI in a separate GitLab job after all test jobs have been completed.

    Code (csharp):
    1. Code Coverage:
    2.   stage: Report
    3.   tags:
    4.     - self-hosted, windows, powershell, unity
    5.   variables:
    6.     GIT_CLEAN_FLAGS: -ffdx -e Library/
    7.   before_script:
    8.     - BuildScripts/before_script.ps1
    9.   script:
    10.     - BuildScripts/build.ps1
    11.   dependencies:
    12.     - Editor Tests
    13.     - Runtime Tests
    14.   rules:
    15.     - if: '$CI_MERGE_REQUEST_ID == null && $CI_COMMIT_MESSAGE !~ /[skip test]]/'
    16.       when: always
    17.   coverage: '/^ *Line coverage: (\d+\.\d+%)$/'
    18.   artifacts:
    19.     reports:
    20.       cobertura: CodeCoverage/Cobertura.xml
    21.     paths:
    22.       - CodeCoverage/
     
    caspar-five likes this.
unityunity