Question How to use automated testing in Unity in a productive way? Tips or best practices?

I'm a proponent of automated testing, but I'm having lots of troubles making it work for my small studio (10 developers). My personal understanding of testing is this:

• Manual testing by developers and QA will always be needed
• Unit tests and TDD workflows can help write code more quickly and robust since it forces the developer to think through more than the most obvious use-cases
• Integration tests should help to solidify contracts between systems and nail down invariants/assumptions or even reproduce complex bug cases

I've also did a lot of research, e.g. I read "The Art of Unit Testing", Bob Martin books and multiple online blogs about testing in general.

However, I seem to completely fail to introduce and make productive use of automated testing at my company. I'm using the Unity test runner with edit and play mode tests, but no other infrastructure beyond that. Although I try to make it work I keep running into problems:

• Coworkers disapprove of automated testing and say it would make them slower and doesn't make the game better
• My unit tests only cover classes or use-cases for which they are easy to implement such as math or decoupled logic. Although these tests went well when writing them, I have no measurement whether writing them made me faster or slower. In the end, the tests never failed again in production, so they didn't find any bugs although of course there were issues with other code using these classes.
• My integration tests seem like the hackiest piece of code because they are coupled to all of the implementation details of our game (setup player, press ready, wait for countdown, spawn abilities, wait for effects, call into UI to submit, etc). This is rather difficult to get right and then it has to change a lot when the game changes. With these tests I was mostly hoping to automatically reproduce bug cases which our QA has found with manual testing, but I never managed to reproduce the same kind of bugs humans could.

I'm still searching for a way to make testing work for me... so does anyone have tips or resources that helped them with automated testing?

I'd be especially interested in seeing real-world examples of tested code. So far, I only found fake examples or code from non-game software.

 

We integrated Unit & Integration tests pretty early on in our process. A few months into production we had them running (which was roughly 2 years ago) and most of those tests are still the same. We're a smaller studio (4 people), but I don't think it would anything should be different for you.

Yes, you'll always be manually testing. There's unfortunately no real way around it, thought you could automate things pretty far by introducing AI/Bots into the process. We haven't gone that far, but I did consider it at some point (by recording input and having a test play through the first 15-ish minutes). There's a brief post from the Thalos Principle over here that used an AI over here: https://blog.us.playstation.com/201...-on-ps4-designing-ai-to-test-a-game-about-ai/ and there's one from the Retro City Rampage dev here:

I personally don't believe it'll speed up actual writing of code, but it definitely saves on QA & debugging in the long run.

Yes it makes them slower. But how do you currently ensure that your entire game is playable from start to finish when any of you makes a change? For our previous game we had to manually play through 2 hours of gameplay after every change. Nowadays we automatically push builds to Steam after every commit and very rarely have something broken on there. This is while in full production, building new features & adding content to existing features. You can't test against that unless you have a full-time QA person/team.

Yes, that's absolutely correct, but I think you're looking at the effect of that in the wrong way. It's not about making sure every tiny method works fine, a lot of them are simple enough to assume that you can't mess it up. It's about making sure the slightly more difficult ones work fine at all times. It's about quantity at that point, if that same test is being ran every day (or in our case, every commit) you can be pretty damn sure that your game is at the very least playable.

Yes, integration tests are a pain in the ass and I would not recommend you writing them too early on. Maintenance on them takes forever. That said, having them running after you've set things in stone is the best thing ever. You can now be sure that any change to the result of that test is unintended and needs to be looked at. We have about 20-ish integration tests running (and 140+ unit tests).

Now for some examples of stuff that we do:

Asset validation! This is pretty huge for us, as we can now be sure that the assets that we add to the game are set up correctly (components/tag/layer/variables/materials/shader). We also use a lot of spreadsheets for our data, and we validate all of that to make sure there's no bogus data, forgotten fields, etc. These tests are in my opinion the easiest to write with the highest gain, as it goes over things that have to be set up manually, and people make mistakes. Often. I'd recommend that you'd pitch these as the first ones you'll write, as you can do them yourself in a few hours.

Integration tests. I mentioned already we have about 20 of em, and they test the truly core features of our game. In our case that's placing furniture, placing items, highlighting things as you walk past them, that kind of stuff. Pretty small features on their own, but vital to the game. We also do some tests on the UI to make sure they're populated as intended.

Testing other game features. I guess this lives somewhere between a true unit test and a integration test. There's some specific features that require player input to trigger, in our case like placing a wall. Instead of simulating the player input I just assume that the player input was valid and that there's no bugs in that. I then run all the code after that and make sure the output is correct.

That said, I don't think this would be very useful if you aren't automating it properly. Set up a build server and automate the tests. We run them after every commit (perks of having a small team), but you could also run them every x hours, or run them overnight if it takes too long (or split them up). If you have any questions, don't hesitate to ask.

 

Thank you for sharing!

How do you check this in code? UI is a visual thing right, or do you only check if the UI state (string getter/setter e.g.) returns the correct value?

Are you working on multiple branches? If so, do tests run on the branches or before merging or only after committing to main?

 

We simply call a function (let's say, open up the inventory), which runs some code that should populate a UI element based on the actual player inventory. At the end we just check if the correct amount of GameObjects have been spawned. If it looks good is not important for that test. You could however, use an image assert for another test that checks if it looks as intended, but I guess that kind of test would break often as you develop the game.

We don't, we use Perforce and use Shelves instead of Branches for that purpose. If your goal is the stability of the final product, run them on the main branch after committing. You could however write some tests that check for things (like the validation stuff) before the merge happens if you want to catch it sooner, but I think you're overcomplicating things with that.

 

@superpig Thank, this is helpful advice!

No, it's all explorative testing, simply playing the game or exercising specific tasks. Either we as developers check systems by playing and we usually have external QA hired by our publisher to look for issues. Which means that we often happily write code for weeks or months without any known issues or bug fixing, but then towards the end of a project all of the bug reports come in and the schedule becomes even tighter. I'd like to shift this workload more towards the front of development by introducing automated testing which we can to mid development ourselves to know if systems are regressing over time, etc.

I think I want to explore using NSubstitute and read up on the NUnit Theory attribute next.

 

QA has a minimal documentation about the rules and general features of the game, but details are left for them to judge from a players standpoint aka "This feels broken". I realize that this approach has issues, but so far it does work for finding a lot of bugs. The bigger problem for us is that this only happens at the very end of development. But maybe this would be a way to introduce more testing to my company, by starting with more formal test cases and then only in the second step automate these when it becomes obvious that they are repetitive.

 

I'm gonna be using this when I'm explaining automated testing to students. I completely agree with this, the boxes that these tests are put in (traditionally) are generally not that useful. I'm guessing that -technically- 95% of the tests that we use would be classified as an integration test if you use the classic definitions. But in the real world it doesn't matter: I need stuff tested automatically so I don't have to bother to think about it and I don't really care what it's called.

@Xarbrough I guess this kinda depends on where you are in your development stage, but having more structured QA sessions are generally a lot more useful rather than "just do whatever for a few hours". We tend to set somewhat hard goals such as "Can the player boot up the game and make it to level 9 without issues?". In our game this means that you'll have to use pretty much every mechanic in our game.

While on the topic of QA, I'll share my favorite feature that we use to improve quality / find bugs AFTER a build has shipped:



Whenever an error happens, we capture it and use a Discord webhook to post it to a (private) channel on our Discord (along with the save file and a low-res screenshot). This allows us to find errors in real-time. We generally have about 20 or so people playing at all times, so it's a good overview for finding critical errors really quickly.

 

@Aceria_ I just found this thread and I thought your little Discord notifier was a pretty cool setup.

Would you by any chance consider open-sourcing it as a little utility/tool for us to plug into our own game/server? That'd be super nice

 

@neshius108

It really isn't all that spectacular (which is why it's one of my favorite tools), here's all the code that's necessary. I've had to strip out some stuff to post it publicly and it obviously references some of my code, but it's simple enough to modify for your own needs.

Code (CSharp):

1. using UnityEngine;
2. using System.IO;
3. using UnityEngine.Networking;
4.  
5. //To use, add these to any monobehaviour:
6. //OnEnable: Application.logMessageReceived += CustomLogOpenSource .HandleLog;
7. //OnDisable: Application.logMessageReceived -= CustomLogOpenSource .HandleLog;
8.  
9. public static class CustomLogOpenSource {
10.     private static string lastMsg = "";
11.  
12.     public static void HandleLog(string msg, string stack, LogType type) {
13.         if (!Application.isEditor) {
14.             if (type == LogType.Assert || type == LogType.Error || type == LogType.Exception) {
15.                 //don't spam duplicate messages, ignore some useless ones
16.                 if (lastMsg != msg && !msg.Contains("Releasing render texture") && !msg.Contains("Failed to read input report") && !msg.Contains("FMOD failed to initialize the output") && !msg.Contains("Failed to create device file") && !msg.Contains("peeked BinaryEntryType") && !msg.Contains("Failed to get preparsed data")) {   //F*** this S***
17.                     lastMsg = msg;
18.                     UnityWebRequest www = UnityWebRequest.Post("PUT_YOUR_WEBHOOK_URL_HERE", GetFormData(msg, stack, type));
19.                     www.SendWebRequest();
20.                 }
21.             }
22.         }
23.     }
24.  
25.     private static WWWForm GetFormData(string msg, string stack, LogType type) {
26.         string content = "";
27.         WWWForm formData = new WWWForm();
28.  
29.         //append a build version
30.         if (Debug.isDebugBuild) {   //useful if you're testing this out in the editor, so you can differentiate between messages easily
31.             content = "\n----------------------\n**DEBUG BUILD** ";
32.         }
33.         else if (App.version != null) {
34.             content = "**" + App.version + "** ";
35.         }
36.         else {    //this was a fallback for very old versions
37.             content = "\n----------------------\n**UNKNOWN VERSION** ";
38.         }
39.  
40.         //set up the actual report
41.         content += "\n\n**Message**: " + msg;
42.         content += "\n\n**Callstack**: " + stack;
43.         formData.AddField("content", content);
44.  
45.         //Add a save file if it's available
46.         byte[] bytearr = null;
47.         if (File.Exists(SaveManager.PATH + App.SaveManager.GetSaveName() + ".shop")) {
48.             bytearr = File.ReadAllBytes(SaveManager.PATH + App.SaveManager.GetSaveName() + ".shop");
49.             formData.AddBinaryData("file2", bytearr, "discord_error.shop");
50.         }
51.  
52.         //optional, but very useful for seeing UI errors
53.         if (App.GameCamera != null) {
54.             formData.AddBinaryData("screenshot", App.GameCamera.RenderImageWithUI(640, 360).EncodeToPNG(), "discord_error_screenshot.png");
55.         }
56.  
57.         return formData;
58.     }
59. }

 

@Aceria_ lovely! Yeah, it's nice and simple

Thanks a lot!

 

@Aceria_ : I made a gist with some modifications and cleanup:

https://gist.github.com/Nesh108/22b3f6e350fe2fd6a483e014210d5215

Changes:
- Fixed some errors and other references to your code
- Added screenshot through Unity
- Switched to list for filters
- Changed logic for checking duplicates (keeping track of the stack, as opposed to only the last message worked better for me)
- Added Debug/Release flag and Platform info

 

@neshius108

Good stuff, just a small point on the screenshot part: That doesn't work with more complicated camera setups. In our case using ScreenCapture.CaptureScreenshotAsTexture() did not allow us to capture the UI (which is truly what we were using the screenshots for).

We solved that with the help of this post: https://www.tangledrealitystudios.c...apture-overlay-canvas-ui-in-screenshot-unity/