Creating New Unity Tests Programatically

Here's our testing scenario. We have developed tools for our content creators to make as many levels in our game as they want. We'd like a way to generate soak tests for each level.
Launch the level in isolation -> run through a generic set of commands -> end the level

Our tests get deployed on our CLI and a device farm.

Our current approach is to have an engineer create a new tests for every level like this:

Code (CSharp):

1. [UnityTest]
2. public IEnumerator Level1()
3. {  
4.     yield return LevelTest(TEST_SCENE_PATHS[0]);
5. }
6.  
7. [UnityTest]
8. public IEnumerator Level2()
9. {  
10.     yield return LevelTest(TEST_SCENE_PATHS[1]);
11. }
12.  
13. [UnityTest]
14. public IEnumerator Level3()
15. {
16.     yield return LevelTest(TEST_SCENE_PATHS[2]);
17. }

This is not ideal for a lot of reasons. An engineer needs to be available to create a new test. When designers create a new level they need an engineer to create a test for that level. When designers remove/move a level the tests can break, etc...

What is the optimal workflow here? It would be great if there was an API where tests could be generated programmatically. I could provide a list of levels and create a test for each one.

 

I recommend using the TestCaseSource attribute on a test and then return a list of objects that can be used to load the scenes from the source. I would also try to create this list dynamically on the fly instead of hardcoding the paths.

Here's an example:

Code (CSharp):

1. namespace Nementic.SecretProject.Tests
2. {
3.     using System.Collections;
4.     using System.Collections.Generic;
5.     using NUnit.Framework;
6.     using UnityEditor;
7.     using UnityEditor.SceneManagement;
8.     using UnityEngine;
9.     using UnityEngine.SceneManagement;
10.  
11.     [NementicTest]
12.     public class RoomValidator
13.     {
14.         /// <summary>
15.         /// Maps ItemID to GameObject name.
16.         /// </summary>
17.         private Dictionary<string, List<string>> itemGameObjects;
18.  
19.         [OneTimeSetUp]
20.         public void Setup()
21.         {
22.             itemGameObjects = new Dictionary<string, List<string>>(6);
23.         }
24.  
25.         [OneTimeTearDown]
26.         public void TearDown()
27.         {
28.             itemGameObjects = null;
29.         }
30.  
31.         [Test]
32.         [Category("SceneValidation")]
33.         [TestCaseSource(nameof(LoadScenes))]
34.         public void ValidateScene(AssetInfo assetInfo)
35.         {
36.             Scene scene;
37.             try
38.             {
39.                 scene = EditorSceneManager.OpenScene(assetInfo.Path);
40.             }
41.             catch
42.             {
43.                 // This can only happen if the TestCaseSource has retrieved
44.                 // a scene path from the AssetDatabase, but then the scene
45.                 // was deleted from disk without the project refreshing.
46.                 scene = new Scene();
47.                 Assert.Inconclusive("Failed to load scene: " + assetInfo.Path);
48.             }
49.  
50.             ValidateScene(scene);
51.         }
52.  
53.         private void ValidateScene(Scene scene)
54.         {
55.             itemGameObjects.Clear();
56.  
57.             GameObject[] roots = scene.GetRootGameObjects();
58.             foreach (GameObject root in roots)
59.                 ValidateGameObject(root);
60.  
61.             foreach (var kvp in itemGameObjects)
62.             {
63.                 if (kvp.Value.Count == 1)
64.                     continue;
65.  
66.                 Assert.Fail(
67.                     $"Duplicate item found. ID: {kvp.Key}, GOs:\n{string.Join("\n", kvp.Value)}");
68.             }
69.         }
70.  
71.         private void ValidateGameObject(GameObject go)
72.         {
73.             if (go.IsItem())
74.             {
75.                 string itemID = go.name;
76.                 string path = go.scene.name + ":" + go.HierarchyPath();
77.  
78.                 if (itemGameObjects.TryGetValue(itemID, out List<string> gameObjects))
79.                     gameObjects.Add(path);
80.                 else
81.                     itemGameObjects.Add(itemID, new List<string> { path });
82.  
83.                 Assert.AreEqual("Item", go.tag, "Item tag missing: " + itemID);
84.                 Assert.IsTrue(go.TryGetComponent(out Collider2D _), message: "Item collider missing: " + itemID);
85.             }
86.             else
87.             {
88.                 for (int i = 0; i < go.transform.childCount; i++)
89.                     ValidateGameObject(go.transform.GetChild(i).gameObject);
90.             }
91.         }
92.  
93.         public static IEnumerable LoadScenes()
94.         {
95.             string[] guids = AssetDatabase.FindAssets("Room_ t:scene", new[] { "Assets" });
96.             for (int i = 0; i < guids.Length; i++)
97.                 yield return new AssetInfo(guids[i]);
98.         }
99.     }
100. }

The AssetInfo class is needed to shorten the display name of the tests while still persisting the original path to load from:

Code (CSharp):

1. namespace Nementic.SecretProject.Tests
2. {
3.     using UnityEditor;
4.  
5.     public readonly struct AssetInfo
6.     {
7.         public readonly string Path;
8.  
9.         private readonly string displayName;
10.  
11.         public AssetInfo(string guid)
12.         {
13.             this.Path = AssetDatabase.GUIDToAssetPath(guid);
14.             this.displayName = TrimForDisplay(Path);
15.         }
16.  
17.         private static string TrimForDisplay(string path)
18.         {
19.             path = path.Replace("Assets/", string.Empty);
20.             return path.Replace(System.IO.Path.GetExtension(path), string.Empty);
21.         }
22.  
23.         public override string ToString()
24.         {
25.             return displayName;
26.         }
27.     }
28. }

 

Thanks guys I didn't know about those attributes.

I went with the ValueSource attribute.

Code (CSharp):

1. static Levels[] GetLevels()
2. {
3.     GameConfig gameConfig = GetGameConfig();
4.     if (vtgCase == null)
5.     {
6.         return new Levels[] { };
7.     }
8.     return gameConfig.levels.Where(x => x.enablePlayModeTest).ToArray();
9. }
10.  
11. [UnityTest]
12. public IEnumerator LaunchChapter([ValueSource(nameof(GetLevels))] Level level)
13. {
14.     ...
15. }

One thing that is kind of weird was getting access to the game configuration in the playmode tests. I ended up moving the game config to the resources folder as a playmode build step.

Code (CSharp):

1. public class SetupPlayModeTests : UnityEditor.TestTools.ITestPlayerBuildModifier
2.     {
3.         const string RESOURCES_FOLDER = "Resources";
4.         public UnityEditor.BuildPlayerOptions ModifyOptions(UnityEditor.BuildPlayerOptions playerOptions)
5.         {
6.             // Find the main VTG Case and move it to the Resources folder
7.             string[] guids = UnityEditor.AssetDatabase.FindAssets($"t:{typeof(GameConfig)}");
8.             if (guids.Length > 0)
9.             {
10.                 string path = UnityEditor.AssetDatabase.GUIDToAssetPath(guids[0]);
11.                 string resourcesGuid = UnityEditor.AssetDatabase.CreateFolder(Path.GetDirectoryName(path), RESOURCES_FOLDER);
12.                 UnityEditor.AssetDatabase.MoveAsset(path, Path.Combine(UnityEditor.AssetDatabase.GUIDToAssetPath(resourcesGuid), Path.GetFileName(path)));
13.             }
14.  
15.             return playerOptions;
16.         }
17.     }

Not sure if there's a better way of doing that.