Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

What about unit testing?

Discussion in 'Open Projects' started by indexhever, Oct 2, 2020.

  1. indexhever

    indexhever

    Joined:
    Mar 30, 2012
    Posts:
    6
    Hey guys,

    I love unit testing because it can help develop better code, serves as code documentation, and also helps to add change without breaking other systems.

    Were you already thinking about using it? I opened the project and didn't see any test case yet.

    Using it in this project would also help people who are beginning in game development to better separate code in specific responsibilities.
     
    MehO and hemanthvanam like this.
  2. MileyUnity

    MileyUnity

    Joined:
    Jan 3, 2020
    Posts:
    92
    I think that adding unit tests could be very valuable, right now I don’t think there’s much to test yet but as we start to progress it would be very useful to keep up with Unit tests and setup a rule of “don’t break the tests with your PR” at the very least. It would be nice to also enforce all new features to have unit tests added to them but I’m not quite sure how well that would work as I’ve not yet written any unit tests for a game.

    @cirocontinisio what do you think about unit tests?
     
    indexhever likes this.
  3. Sangemdoko

    Sangemdoko

    Joined:
    Dec 15, 2013
    Posts:
    220
    With so many programmers writing things on their own and fixing bugs, etc... I feel like having Unit tests will really help us from keeping any from breaking something by mistake.

    It's also a going to save time as people could test things on their machine before asking for a branch merge.

    Having testable code also usually helps keeping code simple and modular.

    Any thoughts on the subject?
     
    MehO, dm67x and indexhever like this.
  4. MileyUnity

    MileyUnity

    Joined:
    Jan 3, 2020
    Posts:
    92
    @Sangemdoko I merged your thread in with this one since they're talking about the same topic :)
     
  5. Sangemdoko

    Sangemdoko

    Joined:
    Dec 15, 2013
    Posts:
    220
    Thank you,

    For those who are unfamiliar with unit testing in Unity (it's not always straight forward with Monobehaviours) here's a video from Jason Weimann:


    It's a bit long but I think it explains things quite clearly
     
    indexhever likes this.
  6. Erenquin

    Erenquin

    Joined:
    Apr 9, 2018
    Posts:
    164
    I was about to start a thread on unit test and found out this one.
    Few months ago I started a project and implemented a lot of unit test.
    But in the end I thought I was doing too much.

    I'm very interested in seeing, in the case of a game, what we should test and more specifically when it starts to be "too much", that is when the cost of writing the test is bigger than what the test brings as value.

    I'm familiar with "standard application" unit testing, but more with "backend stuff", with no UI, no displayed objects, no physics, things like that.
     
  7. Sangemdoko

    Sangemdoko

    Joined:
    Dec 15, 2013
    Posts:
    220
    I think its rare to write too many unit tests. It really depends on the amount of people working on a project and for how long.

    I think the best approach here is to use this project to get some experience and learn from others. So writing too many tests won't be a problem, it'll show some people the benefits of testing and they might start doing it in their own project.

    I think a good place to start would be to test the existing features.
    So for example loading the test scene with the player and start mocking some input to make it move.

    @MileyUnity Is it ok to import the Nsubstitue dll in the project for easy mocks of interfaces?

    To be fair I haven't opened the project yet, I'll do so sometime this weekend hopefully
     
    indexhever likes this.
  8. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,638
    We need to be careful with testing at such an early stage of the project. Lots of things in are flux - writing tests for things that are intentionally only temporary will just slow things down by making people repeatedly need to fix the tests. (For example, creating input recording/replay tests based on the current CharacterController test scene would be a bad idea).
     
    LoveraSantiago and kirbygc00 like this.
  9. indexhever

    indexhever

    Joined:
    Mar 30, 2012
    Posts:
    6
    Actually, TDD is about testing at the beginning. The problem is the contrary, testing after code is made is more difficult. We should not be afraid of change when developing by test. The changes should actually be easier because we know where the code is breaking if a test breaks.
     
    lchaia, kirbygc00, MehO and 2 others like this.
  10. samuelmiller

    samuelmiller

    Joined:
    Aug 15, 2020
    Posts:
    8
    I would like to help with Unit Testing. I am relatively new to Unity and to unit testing. I will check out the resources and follow to see where I can help.
     
  11. shuttle127

    shuttle127

    Joined:
    Oct 1, 2020
    Posts:
    183
    I was curious about test driven development too, sounds like the challenge here is there are still too many moving parts to have enough requirements to generate relevant tests first.
     
    luisquid likes this.
  12. MUGIK

    MUGIK

    Joined:
    Jul 2, 2015
    Posts:
    471
    In my humble opinion, this project is too small to take benefits from Unit testing.
    Also, making game code "testable" will increase its overall complexity, which is bad for dudes that just starting using Unity.
     
    kcastagnini likes this.
  13. shuttle127

    shuttle127

    Joined:
    Oct 1, 2020
    Posts:
    183
    While I agree a vertical slice may not benefit much from unit tests, this is also a learning project. Even if it adds complexity, if the community has the bandwidth to get it done, teaching new developers good programming practices like this is still valuable.
     
  14. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,638
    I'm extremely familiar with TDD, but thank you :)

    As @shuttle127 says, we do not have clear requirements at this point - we are effectively still in the 'pre-production' phase of the project, where the basic functionality is in a state of flux.

    A test like "Can the player jump?" might be OK to write right now, because it's very unlikely that we are going to remove jumping as a feature.

    However, a test like "When the player jumps, do they reach a height of 2 meters?" is a bad idea to write right now, because that's the sort of detail that is very likely to change. It will probably be a good test to have later, once we start locking down the level design, because changing the jump height at that point might affect the navigability of the levels in ways we don't want.
     
    indexhever and MUGIK like this.
  15. indexhever

    indexhever

    Joined:
    Mar 30, 2012
    Posts:
    6
    I see... so we are still figuring out what the game will be and can be totally different in a few days.
     
  16. cirocontinisio

    cirocontinisio

    Joined:
    Jun 20, 2016
    Posts:
    884
    Exactly. While we set some things in stone to give the project a direction, we also wanted to give you all the ability to steer it a bit in unforeseen ways.
    (although I wouldn't say it's going to be completely different)

    Never forget that this thing is going to last months, and at the moment we're merely 72 hours after launch :)
     
  17. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    1,084
    I'd like to recommend something different that I'm currently using in my project - Test Scenes.

    Idea is very simple, every test scene has only one (or few systems) to be tested. This will make sure that game code is properly decoupled and could be reused in different projects (most common issue is missing singleton :)).

    Examples (scene names):
    MovementTest
    InventoryTest
    EnemySpawnerTest
    GameSettingsUITest
    etc

    This will allow everyone to work on their PRs in separate scenes, prefabs will be used to carryover changes from these scenes to the game.
     
    jwinn and Neonage like this.
  18. ironCookie

    ironCookie

    Joined:
    Sep 3, 2014
    Posts:
    10
    Hi everyone,

    I would also like to see Unit tests in the long run and I would prefer Test Driven Development.
    As already mentioned, the project complexity will increase and might overwhelm developers at the beginner level.
    But nobody knows, neither the total number of contributors nor the amount of developers, who will frequently apply some changes. I assume, that later pull requests will have long review cycles when no test cases are available.

    Moreover, I would like to make a suggestion.
    We could start a second project (code only!), use custom NuGet packages containing Unity DLLs and use Github Actions or some other build pipeline for automatic validations. The only problem might be, that the project structure will differ from the typical Unity style.

    I could describe the steps, how those NuGet packages must be structured. But I do not know, if we are allowed to distribute Unity DLLs within a public repository. Moreover, the prototype for the second part (setting up the validation with Github Actions) will require at least one week. The final product, which should be understandable and usable for most developers, will require much more time. So my last question is, is there anyone who is thrilled in this topic? Even if one does not contribute to the main project! There might be a lot to learn for people, who want to apply for a job in the QA (quality assurance) or integration team of some company in the future.

    Keep in mind though: The development is an ongoing process and not every developer will write Unit tests by themself or stick to further guidelines, which were developed by some guys of the community. Last but not least, it is not certain that we can keep up with the development progress.
     
  19. ironCookie

    ironCookie

    Joined:
    Sep 3, 2014
    Posts:
    10
    In my second post, i just want to reply to @Kamyker
    I do like your idea and I do something similar in other projects, too. But I presume, that your recommendation would violate the Contribution guidelines, as discussed here: https://forum.unity.com/threads/lets-stick-to-the-contribution-guidelines.984449/
     
  20. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    1,084
    Sounds like pretty bad guidelines. There's difference between messy prototyping scene/scripts and properly made test scene. Is Unity expecting to everything work perfectly out of the box? Testing changes by playing game itself will eat up a lot of time. Not to mention everything will be heavily coupled.

    A month ago I was pulling out movement from FPS Microgame to my own project. It was a mess because of lack of Assembly Definitions and test scenes to force decoupling. In the end had to pull all scripts and remove not needed ones as that was easier to do then refactoring one class.
     
    Neonage likes this.
  21. Megatank58

    Megatank58

    Joined:
    Jul 20, 2020
    Posts:
    44
    I wouldn't suggest you to push the actual scenes that your using for testing recently (quite recently) the InputTest scene was removed because it was obviously against the guidelines another reason i think Unity doesn't allows this is because it would be overall too messy to handle the test scenes it self considering everyone makes their own test scenes (and there are more than 400 Forks on GitHub of the repository itself).
     
  22. ironCookie

    ironCookie

    Joined:
    Sep 3, 2014
    Posts:
    10

    Yes, indeed.
    I think the organizers are fully aware of this problem. But right now, the restrictions prevent the project from expanding too quickly and overwhelming developers at the beginner level. The main idea is, to let everyone contribute, which means we will have to discuss and develop a lot of workflows in the background, while few others may implement some assigned features.

    Maybe someone from the organizers will let us develop such a workflow idea in the next development cycle! Well, we will see.
     
    alexchesser likes this.
  23. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    In defense of testing early, if the code is written with a dependency injection pattern and making use of interfaces, it actually makes it a lot easier to change things over time. Adding unit tests early forces developers to write testable code from the start, which by design becomes more modular.

    To some extent, I agree that new and junior developers may find the added complexity of interfaces and inversion-of-control a challenge at first, I also think their value becomes very clear as the project progresses. When you finally get the opportunity to leverage the strengths of the patterns, they more than pay for the effort they required.
     
    ironCookie and indexhever like this.
  24. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    So - I've gone and gotten a demo / proof of concept of an initial testing framework set up

    https://github.com/UnityTechnologies/open-project-1/pull/88

    There's a PR with some thoughts on direction available for discussion.

    As mentioned in the PR - I'd actually like to add MOCK and actually demonstrate the creation of a fully featured test using mocks of something medium-simple like the InputReader

    I think having working sample code of actual unit tests will help demystify them and diffuse some of the fear and trepidation surrounding the discussion of "when to introduce testing" (hint: immediately :) ).

    edits for tagging keen parties:
    @Sangemdoko, @indexhever, @ironCookie, @MileyUnity

    edit: 10/19/2020

    I've made a bunch of improvements to naming in terms of the assemblies, additionally moving scripts and input into 2 assemblies instead of on in the project root should mean the recompile time on edits to /scripts/ will be a lot faster.

    I've really been hammering my head against the wall trying to write tests for InputReader - but there seems to be a challenge found within the InputSystem itself. I have an open forum thread where I *fingers crossed* can get in direct contact with the owners of the input system to see if they know a way where input tests can be mocked. (maybe there's a way to create a mock controller!)

    Anyways, this pull request contains working tests and NOW also has the ability to MOCK interfaces. When custom code starts coming in where tests are appropriate, the detail work of initial is at least done, and individuals just have to worry about their specific tests.
     
    Last edited: Oct 19, 2020
    indexhever likes this.
  25. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    really great point @ironCookie - I'm glad I saw this (event though I've already done the work in the PR)

    In looking at a license audit of the dependencies my work added
    So in all cases, the redistribution of nuget packages is technically fine, but it comes down to the methods required by Unity as an entity.
     
  26. kirbygc00

    kirbygc00

    Joined:
    Apr 4, 2016
    Posts:
    50
    Hi All,

    I think its a good idea to at least get some example of what a test is into the project (even if we hold off for some time before writing more). I saw that @alexchesser was having some trouble unit testing the input reader. I had previous experience with unit testing the input system so I took a crack at it. Seems to be working fine.

    I set up a second PR for this because I did not include any external libraries (which I see Alex has). I think it's a good idea to get some mocking library in the project, but I didn't want to get the PR declined because of that... So I made a new branch without any external dependencies.

    Would love some input on what ya'll think... the PR is here: (https://github.com/UnityTechnologies/open-project-1/pull/144)
     
  27. alexchesser

    alexchesser

    Joined:
    Sep 15, 2017
    Posts:
    147
    That's awesome! THanks Kirbygc00!
     
    kirbygc00 and Smurjo like this.
  28. krupps

    krupps

    Joined:
    Oct 17, 2017
    Posts:
    159
    Thats one of the benefits of TDD. If you write the tests before you create the object/GameObject, you are documenting the behaviour of the system, demonstrating to other developers how to use the code, and to your point; you won't write needless code because you will figure that out while writing the test.

    Everyone has their opinion, but I've been doing TDD for 10 years now and it saves developers from writing code they dont need and the tests reveals the flaws in your design before actually implementing it.
     
    Saucyminator and BrettHuang like this.