Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Do you write unit tests when developing games?

Discussion in 'Scripting' started by wayneforce, Aug 1, 2022.

?

Do you write unit tests when developing games?

Poll closed Oct 1, 2022.
  1. I write unit tests

    33.3%
  2. I do not write unit test

    66.7%
  1. wayneforce

    wayneforce

    Joined:
    Feb 12, 2014
    Posts:
    41
    A simple question to know, how many actually practice writing unit tests in the awesome Unity game engine :)
     
  2. Gladyon

    Gladyon

    Joined:
    Sep 10, 2015
    Posts:
    389
    I am writing a few unit tests, but only for very 'standard' features such as a custom circular array.
    All the gameplay part has no unit test, only scenarios which are played to automatically do actions, and I check manually if the result is OK or not.
    In the end, my unit tests will probably cover less than 1% of the code.


    The thing is, unit tests are only useful when you have a 'perfect' specification of your software.
    In the industry, when you're working on the gear system of an airliner for example, you write a complete specification detailing every little detail of the software, every possible (and impossible) situation is taken into account.
    What every function does is written down for every possible combination of its inputs, even impossible values (for example, you write a function which will only be called with positive values, yet it must still work with negative values, you cannot just throw an exception and disable the breaking system...).

    With such a specification, it is very easy (but very very very long...) to write unit tests for all possible situations, and so ensuring that your software works exactly as the specification says.
    So it is very useful and even cost efficient as you can avoid regressions very easily.

    Unfortunately, with video games it is not very common (to say the least) to have a completely detailed specification of all the behaviour of the game, how it is broken down in functions, what are all the possible inputs for the functions and what are all the expected outputs for all the possible inputs of all the functions.
    Such a documentation would require to write hundred of pages of documentation prior writing the very first line of code when creating a simple Tetris game.


    So, I think that unit tests for video games are a mistake.

    I have seen people writing unit tests for video games, but their unit tests only test a tiny fraction of the possible combination of the inputs of a function.
    The result is that they write their function, then their unit tests (note that in the industry the person writing the unit tests must NOT be the one who wrote the code or the documentation) they just test what they think the function should do, and mostly in the normal situation, then they go over other features.
    The problem is that their function is full of bugs because they only tested a tiny part of it and declared it OK without even testing other possibilities.


    In the end, I only use unit tests when I know perfectly what a function must do, in all the cases, and that's mostly the tools functions.
    All the gameplay is tested by hand, possibly with some sort of replays, and I check visually that nothing is wrong or out of place.
    Not perfect, but the alternative would be to think the whole game in all its details prior writing the first line of code, and I don't think it can work for a video game, maybe for a Sudoku, but even there it wouldn't be that easy.
     
    GuirieSanchez and mopthrow like this.
  3. wayneforce

    wayneforce

    Joined:
    Feb 12, 2014
    Posts:
    41
    I personally think of testing the core game logic that has nothing to do with the mono behaviour classes / physics.

    It’s good to know how things are done in the game industry. I would never imagine testing every possible physics scenario.
     
  4. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    38,514
    Sometimes we write some unit tests for things like engine code, puzzle generators, score advancement, just numeric logic and progression calculations, etc.

    However, writing meaningful tests really does take a meaningful nonzero amount of time.

    For UI and user interaction stuff the value of a unit test drops to near zero.

    The reason is that 99% of the bugs we get are stuff such as this one:

    On iPhoneX in landscape mode, if you dismiss the bonus window before collecting coins, the collect coins button becomes non-responsive because it is covered by the invisible dismiss-bonus-window particle system.


    Good luck unit testing that. Those bugs form the bulk of our debugging time.
     
  5. Gladyon

    Gladyon

    Joined:
    Sep 10, 2015
    Posts:
    389
    If you can define exactly how it works then it will help ensure you do not lose behaviours, which will reduce the regressions noticeably.
    Unfortunately, most of the time bugs occur in situation we haven't anticipated, so these situation do not have unit tests, so it won't be a magic bullet.
    If you have the will to write the unit tests, the resulting quality will be higher.

    But you can also write non-unit tests.
    A unit test typically test only one method, or a small set of methods, by stubbing all its external calls.
    But you can choose to do some tests on the your whole game automatically.
    For example if your game is a pac-man, you can replace the AI of the ghosts by scripts, and replace the control of the player by a script.
    Then you can move the ghosts and the pac-man with scripts, so you know what will happen.
    So you can test that a ghost will eat the pac-man, that the pac-man will eat the dots, etc.
    You can check by counting the dots, compare to pre-recorded screenshots, etc.

    So a test of the whole game can be automated, the main disadvantage is that if the result of the test is not OK, you do not know where the bug is, you have to look for it.
    With a unit test testing only one method, when the result is NOK then you know where the bug is, or at least you're very close to it.


    Note that it depends of the level of security required for the software.
    For the braking gear of an airliner that's the highest level of security possible, so every situation has to be anticipated, like the temperature of the tire being -900 degrees kelvin or +1000000 degrees.
    In that case the system must continue to work at at least 120% of the required capacity, with 2 errors you are allowed to drop to 'only' 100% capacity.

    When working on less secure things, you just do the level of test you feel is good.
    Having one person writing the code and another writing the tests will reduce drastically the number of bugs because the tester will not have the same thinking about the feature, so he will test things the dev hasn't anticipated. And the dev will code carefully things the tester hasn't anticipated but that's no problem as the coder tried this situation and was satisfied of the result.



    Another thing about unit tests, they do have one additional advantage, they can be used to avoid having twice the same bug.
    When a team is working on a software for years, with continuous additional features, you may break the same function several times over the years.
    The idea is that each time a bug is reported, you create a unit test to reproduce it, you check that the unit test is reporting the bug, then you fix the bug, and you execute the unit test again to ensure it reports the bug as fixed.
    That way, over the years you'll cover more and more of the software, for a limited cost because when a bug is to be fixed it has to be reproduced anyway, and writing a unit test doesn't really take longer than reproducing the bug 'by hand'.


    That's the reason why I have unit tests for my tools.
    I create my tools (my custom circular array for example), then I use it.
    At some point I find a bug, so I fix it, then another, then another, then I'm upset and I add unit tests for these 3 bugs.
    Then I find another bug and I'm so frustrated that I add 2-3 dozen unit tests to cover the circular array, and I fix half a dozen more bugs doing that...
    In the end, I should have written the unit tests first, that would have caused much less frustration...
     
  6. Gladyon

    Gladyon

    Joined:
    Sep 10, 2015
    Posts:
    389
    So true, I'll add that the amount of time, while being non-zero, does contain a lot of zeroes.

    I can believe you, UI is a nightmare to design, a nightmare to code, a nightmare to test, and a nightmare to debug.
    Why am I working on video games?!?!
     
  7. lordofduct

    lordofduct

    Joined:
    Oct 3, 2011
    Posts:
    8,513
    In the enterprise world, all the time.

    In video game world... honestly... not that often. I'm with @Kurt-Dekker, I may write them for some core elements of the game. But not so much from there on out.
     
    wayneforce likes this.