Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Question How to use automated testing in Unity in a productive way? Tips or best practices?

Discussion in 'Testing & Automation' started by Xarbrough, Jan 22, 2020.

  1. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    I'm a proponent of automated testing, but I'm having lots of troubles making it work for my small studio (10 developers). My personal understanding of testing is this:

    • Manual testing by developers and QA will always be needed
    • Unit tests and TDD workflows can help write code more quickly and robust since it forces the developer to think through more than the most obvious use-cases
    • Integration tests should help to solidify contracts between systems and nail down invariants/assumptions or even reproduce complex bug cases
    I've also did a lot of research, e.g. I read "The Art of Unit Testing", Bob Martin books and multiple online blogs about testing in general.

    However, I seem to completely fail to introduce and make productive use of automated testing at my company. I'm using the Unity test runner with edit and play mode tests, but no other infrastructure beyond that. Although I try to make it work I keep running into problems:
    • Coworkers disapprove of automated testing and say it would make them slower and doesn't make the game better
    • My unit tests only cover classes or use-cases for which they are easy to implement such as math or decoupled logic. Although these tests went well when writing them, I have no measurement whether writing them made me faster or slower. In the end, the tests never failed again in production, so they didn't find any bugs although of course there were issues with other code using these classes.
    • My integration tests seem like the hackiest piece of code because they are coupled to all of the implementation details of our game (setup player, press ready, wait for countdown, spawn abilities, wait for effects, call into UI to submit, etc). This is rather difficult to get right and then it has to change a lot when the game changes. With these tests I was mostly hoping to automatically reproduce bug cases which our QA has found with manual testing, but I never managed to reproduce the same kind of bugs humans could.
    I'm still searching for a way to make testing work for me... so does anyone have tips or resources that helped them with automated testing?

    I'd be especially interested in seeing real-world examples of tested code. So far, I only found fake examples or code from non-game software.
     
    atuanylp, crekri, sonnyb and 2 others like this.
  2. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    81
    We integrated Unit & Integration tests pretty early on in our process. A few months into production we had them running (which was roughly 2 years ago) and most of those tests are still the same. We're a smaller studio (4 people), but I don't think it would anything should be different for you.

    Yes, you'll always be manually testing. There's unfortunately no real way around it, thought you could automate things pretty far by introducing AI/Bots into the process. We haven't gone that far, but I did consider it at some point (by recording input and having a test play through the first 15-ish minutes). There's a brief post from the Thalos Principle over here that used an AI over here: https://blog.us.playstation.com/201...-on-ps4-designing-ai-to-test-a-game-about-ai/ and there's one from the Retro City Rampage dev here:


    I personally don't believe it'll speed up actual writing of code, but it definitely saves on QA & debugging in the long run.

    Yes it makes them slower. But how do you currently ensure that your entire game is playable from start to finish when any of you makes a change? For our previous game we had to manually play through 2 hours of gameplay after every change. Nowadays we automatically push builds to Steam after every commit and very rarely have something broken on there. This is while in full production, building new features & adding content to existing features. You can't test against that unless you have a full-time QA person/team.

    Yes, that's absolutely correct, but I think you're looking at the effect of that in the wrong way. It's not about making sure every tiny method works fine, a lot of them are simple enough to assume that you can't mess it up. It's about making sure the slightly more difficult ones work fine at all times. It's about quantity at that point, if that same test is being ran every day (or in our case, every commit) you can be pretty damn sure that your game is at the very least playable.

    Yes, integration tests are a pain in the ass and I would not recommend you writing them too early on. Maintenance on them takes forever. That said, having them running after you've set things in stone is the best thing ever. You can now be sure that any change to the result of that test is unintended and needs to be looked at. We have about 20-ish integration tests running (and 140+ unit tests).

    Now for some examples of stuff that we do:

    Asset validation! This is pretty huge for us, as we can now be sure that the assets that we add to the game are set up correctly (components/tag/layer/variables/materials/shader). We also use a lot of spreadsheets for our data, and we validate all of that to make sure there's no bogus data, forgotten fields, etc. These tests are in my opinion the easiest to write with the highest gain, as it goes over things that have to be set up manually, and people make mistakes. Often. I'd recommend that you'd pitch these as the first ones you'll write, as you can do them yourself in a few hours.

    Integration tests. I mentioned already we have about 20 of em, and they test the truly core features of our game. In our case that's placing furniture, placing items, highlighting things as you walk past them, that kind of stuff. Pretty small features on their own, but vital to the game. We also do some tests on the UI to make sure they're populated as intended.

    Testing other game features. I guess this lives somewhere between a true unit test and a integration test. There's some specific features that require player input to trigger, in our case like placing a wall. Instead of simulating the player input I just assume that the player input was valid and that there's no bugs in that. I then run all the code after that and make sure the output is correct.

    That said, I don't think this would be very useful if you aren't automating it properly. Set up a build server and automate the tests. We run them after every commit (perks of having a small team), but you could also run them every x hours, or run them overnight if it takes too long (or split them up). If you have any questions, don't hesitate to ask.
     
    sonnyb, matkoniecz, liortal and 2 others like this.
  3. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    Thank you for sharing!

    How do you check this in code? UI is a visual thing right, or do you only check if the UI state (string getter/setter e.g.) returns the correct value?

    Are you working on multiple branches? If so, do tests run on the branches or before merging or only after committing to main?
     
  4. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    81
    We simply call a function (let's say, open up the inventory), which runs some code that should populate a UI element based on the actual player inventory. At the end we just check if the correct amount of GameObjects have been spawned. If it looks good is not important for that test. You could however, use an image assert for another test that checks if it looks as intended, but I guess that kind of test would break often as you develop the game.


    We don't, we use Perforce and use Shelves instead of Branches for that purpose. If your goal is the stability of the final product, run them on the main branch after committing. You could however write some tests that check for things (like the validation stuff) before the merge happens if you want to catch it sooner, but I think you're overcomplicating things with that.
     
    Xarbrough likes this.
  5. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,667
    @Aceria_'s advice is great.

    I'll add:
    • You want your tests to be at a level of abstraction which means they do not have to change often. For example, people who try input-recording-and-playback tests (which are testing 'if the player does this specific sequence of inputs, do they win the level?') often find that the tests just become a constant maintenance nightmare, because every time a designer moves an object, the recording has to be updated. In practice, what those people are usually trying to test is actually 'is this level solvable?' which is much more abstract - if much harder to write a test for without inadvertently constraining things more than intended.
    • Terms like 'unit test' and 'integration test' are best thought of descriptively, not prescriptively. They're like design patterns - they're a way of talking about the tests that you have, not a menu from which you must make a choice. The closer your test is to exercising a single unit in isolation, the more 'unit test-y' it is, and the closer your test is to exercising multiple systems in concert, the more 'integration test-y' it is - but conforming to those descriptions of tests is not important compared to 'is this test actually valuable?'
    • It can be helpful to think of your test suite as an 'automated bug-check tool.' I think this is way easier to see in the content validation case: it's very common for developers to write tools for the designers that e.g. check that every enemy in the level has actually been placed on the navmesh, is not partially embedded in a wall, etc. The check is common because the mistake is common. In the same way I think you get the most value out of code tests when they're addressing the kinds of mistakes that are common - for example, failing to check for null - and ideally doing that _without_ relying on the programmer to remember to test that case, because if they remember to test the case, they'll remember to fix the implementation in the first place. This is where things like property-based testing and the NUnit [Theory] attribute can be more powerful than example-based testing.
    • The integration tests will indeed have quite broad knowledge of game systems, but they can also be a nice example of how your code benefits from being testable: if it is possible to 'puppeteer' your game from an integration test, then it should also be possible to use the exact same systems to drive the game for tutorials and set pieces; being able to put the game into a particular state at the start of a test has a lot in common with loading a saved game; etc. Look for synergies there.
    • Are your coworkers involved in designing manual tests for their features? Are your manual tests even 'designed' at all - with formal test plans etc?
     
    sonnyb, Aceria_ and Xarbrough like this.
  6. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    @superpig Thank, this is helpful advice!

    No, it's all explorative testing, simply playing the game or exercising specific tasks. Either we as developers check systems by playing and we usually have external QA hired by our publisher to look for issues. Which means that we often happily write code for weeks or months without any known issues or bug fixing, but then towards the end of a project all of the bug reports come in and the schedule becomes even tighter. I'd like to shift this workload more towards the front of development by introducing automated testing which we can to mid development ourselves to know if systems are regressing over time, etc.

    I think I want to explore using NSubstitute and read up on the NUnit Theory attribute next. :)
     
  7. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,667
    I think that is a big part of your problem then - it sounds like you do not have a culture of doing any kind of methodical, repeatable testing (whether automated or manual).

    When the publisher hires external QA, how do you brief them? How do they know what they should test, and how do they know which behaviours are bugs vs by design?
     
  8. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    QA has a minimal documentation about the rules and general features of the game, but details are left for them to judge from a players standpoint aka "This feels broken". I realize that this approach has issues, but so far it does work for finding a lot of bugs. The bigger problem for us is that this only happens at the very end of development. But maybe this would be a way to introduce more testing to my company, by starting with more formal test cases and then only in the second step automate these when it becomes obvious that they are repetitive.
     
  9. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    81
    I'm gonna be using this when I'm explaining automated testing to students. I completely agree with this, the boxes that these tests are put in (traditionally) are generally not that useful. I'm guessing that -technically- 95% of the tests that we use would be classified as an integration test if you use the classic definitions. But in the real world it doesn't matter: I need stuff tested automatically so I don't have to bother to think about it and I don't really care what it's called.

    @Xarbrough I guess this kinda depends on where you are in your development stage, but having more structured QA sessions are generally a lot more useful rather than "just do whatever for a few hours". We tend to set somewhat hard goals such as "Can the player boot up the game and make it to level 9 without issues?". In our game this means that you'll have to use pretty much every mechanic in our game.

    While on the topic of QA, I'll share my favorite feature that we use to improve quality / find bugs AFTER a build has shipped:



    Whenever an error happens, we capture it and use a Discord webhook to post it to a (private) channel on our Discord (along with the save file and a low-res screenshot). This allows us to find errors in real-time. We generally have about 20 or so people playing at all times, so it's a good overview for finding critical errors really quickly.
     
  10. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,667
    Yeah, I think that is quite an effective strategy.

    From my own experience - I went from 'exploratory' testing to test automation by realising: my exploratory testing was basically looking for the same things each time. I'd make some changes, then:
    • Launch the game
    • Start a new game
    • Watch through the opening cutscene
    • Run around in the first area and check that enemies were still spawning, would still attack me, and that I could still shoot them, and that they'd die when I did
    Once I realised those were basically the things I was looking for each time, it inspired me to then go and create specific tests for 'can I spawn an enemy', 'if I spawn an enemy with a target then does it destroy the target within a few seconds', things like that.
     
    Xarbrough likes this.
  11. neshius108

    neshius108

    Joined:
    Nov 19, 2015
    Posts:
    110
    @Aceria_ I just found this thread and I thought your little Discord notifier was a pretty cool setup.

    Would you by any chance consider open-sourcing it as a little utility/tool for us to plug into our own game/server? That'd be super nice :D
     
    webbertakken likes this.
  12. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    81
    @neshius108

    It really isn't all that spectacular (which is why it's one of my favorite tools), here's all the code that's necessary. I've had to strip out some stuff to post it publicly and it obviously references some of my code, but it's simple enough to modify for your own needs.

    Code (CSharp):
    1. using UnityEngine;
    2. using System.IO;
    3. using UnityEngine.Networking;
    4.  
    5. //To use, add these to any monobehaviour:
    6. //OnEnable: Application.logMessageReceived += CustomLogOpenSource .HandleLog;
    7. //OnDisable: Application.logMessageReceived -= CustomLogOpenSource .HandleLog;
    8.  
    9. public static class CustomLogOpenSource {
    10.     private static string lastMsg = "";
    11.  
    12.     public static void HandleLog(string msg, string stack, LogType type) {
    13.         if (!Application.isEditor) {
    14.             if (type == LogType.Assert || type == LogType.Error || type == LogType.Exception) {
    15.                 //don't spam duplicate messages, ignore some useless ones
    16.                 if (lastMsg != msg && !msg.Contains("Releasing render texture") && !msg.Contains("Failed to read input report") && !msg.Contains("FMOD failed to initialize the output") && !msg.Contains("Failed to create device file") && !msg.Contains("peeked BinaryEntryType") && !msg.Contains("Failed to get preparsed data")) {   //F*** this S***
    17.                     lastMsg = msg;
    18.                     UnityWebRequest www = UnityWebRequest.Post("PUT_YOUR_WEBHOOK_URL_HERE", GetFormData(msg, stack, type));
    19.                     www.SendWebRequest();
    20.                 }
    21.             }
    22.         }
    23.     }
    24.  
    25.     private static WWWForm GetFormData(string msg, string stack, LogType type) {
    26.         string content = "";
    27.         WWWForm formData = new WWWForm();
    28.  
    29.         //append a build version
    30.         if (Debug.isDebugBuild) {   //useful if you're testing this out in the editor, so you can differentiate between messages easily
    31.             content = "\n----------------------\n**DEBUG BUILD** ";
    32.         }
    33.         else if (App.version != null) {
    34.             content = "**" + App.version + "** ";
    35.         }
    36.         else {    //this was a fallback for very old versions
    37.             content = "\n----------------------\n**UNKNOWN VERSION** ";
    38.         }
    39.  
    40.         //set up the actual report
    41.         content += "\n\n**Message**: " + msg;
    42.         content += "\n\n**Callstack**: " + stack;
    43.         formData.AddField("content", content);
    44.  
    45.         //Add a save file if it's available
    46.         byte[] bytearr = null;
    47.         if (File.Exists(SaveManager.PATH + App.SaveManager.GetSaveName() + ".shop")) {
    48.             bytearr = File.ReadAllBytes(SaveManager.PATH + App.SaveManager.GetSaveName() + ".shop");
    49.             formData.AddBinaryData("file2", bytearr, "discord_error.shop");
    50.         }
    51.  
    52.         //optional, but very useful for seeing UI errors
    53.         if (App.GameCamera != null) {
    54.             formData.AddBinaryData("screenshot", App.GameCamera.RenderImageWithUI(640, 360).EncodeToPNG(), "discord_error_screenshot.png");
    55.         }
    56.  
    57.         return formData;
    58.     }
    59. }
     
  13. neshius108

    neshius108

    Joined:
    Nov 19, 2015
    Posts:
    110
    @Aceria_ lovely! Yeah, it's nice and simple :D

    Thanks a lot!
     
  14. neshius108

    neshius108

    Joined:
    Nov 19, 2015
    Posts:
    110
    @Aceria_ : I made a gist with some modifications and cleanup:

    https://gist.github.com/Nesh108/22b3f6e350fe2fd6a483e014210d5215

    Changes:
    - Fixed some errors and other references to your code
    - Added screenshot through Unity
    - Switched to list for filters
    - Changed logic for checking duplicates (keeping track of the stack, as opposed to only the last message worked better for me)
    - Added Debug/Release flag and Platform info
     
    Whatever560 and thienhaflash like this.
  15. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    81
    Whatever560 and thienhaflash like this.