Test-driven development

Hi,

I want to pursue a project in Unity3D, one that will probably require some bespoke scripting.

Having picked up test-driven development (write test, run test, see test fail, know test works, write code, see test pass, rinse repeat) I can not think of doing anything any other way.

Has anybody attempted test-driven development for Unity scripts? How would I go about this? I can simply develop all my scripts as libraries independently, but what about testing them as part of Unity3D itself?

Thanks

 

Hum, I wonder if it would be possible to extend the Unity editor to have some sort of testing framework in it. Like, click on a script, then click on "Test" to run it or "Edit test" to edit the test...

I write tests a lot at work too, so I'd be interested in something like this (though I certainly think sometimes people take testing too far). Might have to look into it, when I get some spare time

As for your original question, I don't think Unity provides any testing frameworks. There are frameworks for .NET but I don't know if they're usable with Unity (unlikely), so you're probably going to have to write separate test scripts yourself.

Unless you extend Unity yourself

 

I know about, but don't have any experience with test driven development. Therefore I'm not really the right person to help you. However, I guess that this unit testing framework I found on the unify wiki may be of interest to you:
http://www.unifycommunity.com/wiki/index.php?title=UUnit

I never used it, so I can't even tell you if it works or not .

 

That's pretty cool, thanks for the link. Something to check out later

 

Firstly, as for 'taking testing too far' - I don't know how that's possible. You can test wrongly - badly written tests are horrid to modify and extend, and difficult to work with - but too far? Never.

I like to have at least 10 to 15 times more test code than actual code, and that's if I cut corners.

Result? My code has become more stable and error-free, and I never get the same bug appear twice as a result of refactoring or bug-fixing, which often happens otherwise.

UUnit seems to be exactly what I'm looking for, thanks. Also, I didn't know about that Unify wiki - double thanks!

Regards,
A Y Q

 

the problem is that unit testing only guarantees that code that can be tested works correctly.

what about stuff that can not be tested or what about things (which in games are common) that require "experiencing it"?

If you create libraries, test driven development naturally saves you a lot of headache.

 

I was working on a game in Ruby. I tried to test everything. But as you said, I couldn't.

However, lots of things I could test.

e.g. when a player presses the action button, and the 'action' collision region has a 'chest' object colliding with it - what should happen? What messages should the 'chest' object receive?

If the player object has an impulse of XX applied to it, and it's current velocity vector is vv, after f fixed-timestep frames pass, what should happen?

Of course, when I was playing around with 'look and feel', I didn't use any test-first - it was all adjusting variables. But I knew if I set a variable, it'd work the way I wanted it to!

The trick was to use lots of stubbing and mocking to take away dependence on factors you can't control in an automated test. So you didn't get complete code coverage, but it was waaay better than nothing, and at least gave a guarantee of some of the game mechanics.

That said, I was using Ruby, whose super-dynamic-ness lets me do all the awesome stubbing and mocking easily. But Unity has Boo and JS right?

 

During my career, I have never seen any test process which I would call unit testing in computer games. I think it only makes sense on system level, when you are writing renderer, physics engine, audio engine, script interpreter... all the hard stuff Unity team has already done. Behaviour scripts written for Unity engine tend to be so interdependent that pure and exhaustive unit testing is hard if not impossible. Fortunately that does not really matter. What you want to achieve by testing is to catch as many faults as possible for smallest possible cost or effort. You will need to do a little tradeoff here.

What works best for me is a combination of below
- test scenes for testing all game mechanics with set of interactive test cases. Yes, you need to test all of these manually but trust me, it is much simpler than trying to determine failure conditions (i.e. Is jump animation OK?). Test cases are rather high level and do not need further examination as long as everything works
- list of tests performed on test cases - everything I should attempt to do while trying test scene (i.e. open the door, try to pick up weapon...) It is important to be in written form, so you won't leave out anything.
- whenever new bug, uncovered by test cases occurs, test case is constructed for its occurrence. One would be surprised how once solved bug tends to re-occur
- automated load and initialization of all available levels. I also do first update of scene to catch possible errors.
- build test - lightweight test whether everything is OK in built game (outside editor)

Unlike unit testing, this methodology has slightly lower probability of catching bugs (remember, there is no testing methodology that guaranties all bugs trapped), yet still 90% of problems is caught. And still, nothing but thorough manual testing (preferably by people different from authors) will ensure bug-free game before shipping

 

Hi, we wanted to do this here at eye3ware so we made TestStar. I made a <5min overview of how TestStar works.

we successfully use it in an "agile" scrum process where all of our behaviour is tested.

~Ethan

 

Mmm I agree with Marek. I think testing is actually quite a creative art. Its all well and good writing a unit test for some programs, but absurd to do so with others (like the "Is jump animation OK?"). I guess you find what works best for you and use that.

I wouldnt be too anal writing unit tests for scripts as I feel these are lightweight. I think its more important to test the gameplay, if it looks right it is right. Certainly if you were writing an engine you may want to write a couple unit tests.

I personally lolz @ TDD, scrum, and agile etc etc I think they are just more buzz words, but if it makes you happy and you get good results doing them then why not?

 

This is honestly how I program. Unless you are perfect, you are going to test... and test some more. Your even going to find ways to test code that you can't see except for very specific things happen. I always encourage testing. If a plane doesn't fly like you want it, test it. If a car doesn't drive like you want it, test it, tweak it and perfect it.

 

We are developing a drawing app with U3d - and most bugs are really hard to reproduce. We started to record some drawing sessions (on video, not within the program) to see what people are doing when defects are occurring.

Test plans are also written in very easy language - like draw "a ship" or "the sun" - add some waves, do this, do that.


Some tools/services i fell over:

FoneMonkey (http://www.gorillalogic.com/fonemonkey)

lets you create automated tests for your iOS apps by giving you the ability to easily record, edit, and add verification steps to scripts you create by simply interracting with your app in the iOS simulator or device. Scripts can be saved as Objective-C or JavaScript-based tests that can be extended as necessary to automate even the most complex testing scenarios!


Test Flight (http://blog.testflightapp.com/)

iOS beta testing on the fly



Pay4Bugs (http://www.pay4bugs.com/)

Pay4Bugs is used by software developers, companies, and even bloggers who want to have their software or website tested by a worldwide network of testers with no minimum cost and little risk.



let´s keep this topic alive,
mat

 

First, test driven development is very cool when done right.

That said, test driven development in games beyond the core libraries or core systems is almost never done, and for reasonably good reasons. But let me clarify that, often test driven development is not done for all the wrong reasons. Velocity and tuning are dragged way down with creating unit tests, and there are a few areas where you just cannot write a quantifiable test for the situation. I have yet to work on any game project for any company or client where test driven development was given anything other than short shrift. It's talked about a lot, but when your back is against the wall, every programmers first thought is "Gotta code, gotta get this thing out the door or we don't get paid by the publisher" This aversion is mostly to do with the culture of game development and the Han Shot First approach to coding - "You guys start coding, I'll go find out what it is they want." but smart programmers have also realised that not everything needs testing and sometimes, you're just testing for the sake of it.

I do believe that many programmers use writing unit tests as a form of procrastination: "Look Ma! I'm writing unit tests! See? I'm being productive!" and you have to be very aware that you are not falling in to that very easy trap: "I don't feel like working productively today, I know, let's write some unit tests for testing some obscure corner of my code that has been bug free for the past three years."

Test driven development is a powerful tool in your software development tool box, don't let it become your only tool.

 

Totally agree with that.

Prototyping helps us alot to get some user/customer/artist feedback about important framework requirements.

Also this book recently helped a lot to understand a bit more about software industry and testing- http://www.amazon.de/Agile-Testing-Practical-Addison-Wesley-Signature/dp/0321534468

 

Unity makes TDD extremely difficult.

The problem is the tight coupling of scripts to UnityEngine.dll*, which is not testable; if code depends on UnityEngine directly I can't give it a mock implementation under my control, ergo it is not testable.

None of the Unity unit testing frameworks I've seen help with this, they just replicate an XUnit GUI inside the Unity editor.

Our solution is a framework that mocks UnityEngine; instead of depending on UnityEngine you depend on the framework, which looks like UnityEngine. When unit testing, which is done outside of Unity entirely, your code gets mock dependencies. When running for real, it gets dependencies that use the actual UnityEngine API.

See this post for an introduction. We need contributors.

On the more general note of games and TDD, unless you mock UnityEngine you're likely to end up in a world of pain. We certainly did, with complex, brittle tests that took a lot of maintenance. Now we have simple robust tests and simple robust code.

*Yes, you can factor out as much of your logic as possible into testable chunks, but you'll end up mocking Unity on a small scale.

 

It is interesting, but I find it uncomfortable that it changes the fundamental way you write Unity components - essentially you're not programming to Unity's API any more, you're programming to the Testable API.

Isn't it practical to make it work without such sweeping changes to the standard Unity code? The fact that components don't have constructors should help a great deal, as Unity already expects to initialise them by serialization, which is compatible with testing. I would have thought you'd get a lot of mileage from mocking MonoBehaviour and allowing tests to poke components in various ways, including calling Update() and so on.

Inter-object interactions are trickier, but in theory shouldn't come up in unit testing - one of the objects should be a mock. But you could still test pairs of objects interacting if you really wanted to.

Similarly, testing combinations of components on one GameObject also shouldn't be seen as a problem, because you should be testing them in isolation. So GetComponent<> should always return a mock of some kind.

 

Certainly, it changes the type you're deriving from, but the spirit and 'way' you structure your code is the same. You write a Testable component just like you would a MonoBehaviour, featuring Update(), etc.

I don't follow, I don't see how serialization is relevant to testing.

I'm not sure I fully understand what you mean; one cannot change the base class of a type at runtime. If you derive your component from MonoBehaviour you can't substitute your own base class implementation, ie a mock, when running a test. This is why inheritance sucks.

The point of the framework is to make this possible; testable components can be exercised in total isolation, going through Update(), OnDestroy() etc.

Testable components are intrinsically tested in isolation; we only ever create the minimum dependencies that we have to in order to construct them.

 

Uniject is now on Github https://github.com/banderous/Uniject.git