Search Unity

  1. Unity 2019.1 is now released.
    Dismiss Notice

Unity Unity Test Tools

Discussion in 'Testing & Automation' started by Tomek-Paszek, Dec 18, 2013.

  1. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
  2. raphaelbaldi

    raphaelbaldi

    Joined:
    Mar 4, 2008
    Posts:
    23
    I'll try to achieve the effect we want taking your approach as a starting point. Thanks.
     
  3. alexlopashev

    alexlopashev

    Joined:
    Mar 3, 2014
    Posts:
    7
    Hi all,
    I wonder if it's possible to contribute some code to Unity Test Tools.
    During IntegrationTest feature research I've realized that test result depends only on Debug.Log output lines so some incorrect behaviour is possible, e.g. two simultaneous invocations {Fail, Pass} can pass test.

    I've prepared as I think a useful fix:
    1. Added "Proceed After Fail" flag for Test Component, which makes test run due timeout and collect all fail messages;
    2. Added two additional internal states for ITestComponent: isFailed and isFinished, to remove string-dependent test resolving and for correct work with "Proceed After Fail" flag.

    Is there public repository of Unity Test Tools? I found https://github.com/Unity-Technologies/UnityTestTools , but it's out of date, and unofficial one (out-of-dated too) - https://github.com/haysclark/UnityTestTools .
     
  4. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Hi PsixROs,

    There is no public repository with the tools. The one you found (https://github.com/Unity-Technologies/UnityTestTools) has code for our custom nUnit and nSubstitude builds. We didn't publish the tools in any repo because we constantly improve them and there are quite a lot of changes in the code.

    Nevertheless, I reproduced the issue you found. The behaviour is indeed incorrect. In this case however, there should probably be an extra warning for the user that informs about calling passing and failing method in the same frame. We will give it a second look, thanks!

    About your proposed changes, I'm not sure I get the idea behind "Proceed After Fail". Could write few more words about it?

    Tomek
     
  5. alexlopashev

    alexlopashev

    Joined:
    Mar 3, 2014
    Posts:
    7
    Hi Tomek,

    "Proceed After Fail" flag enables collecting all test's fails due timeout, not only the first one. Of course first fail may produce fail chain (if first fail implies that something important is broken), but sometimes it's really useful, e.g. when you want to check several isolated behaviours in one test, so in this case we'll get information about all interesting things for us by single run, so main idea of this function is to reduce test run count where we can.

    Also I implemented "succeed on timeout" feature locally, which I recommend you to implement too (in this case test checks situation "pass if no one reported the opposite").
     
  6. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    As I can see what you are trying to achieve, I don't think that would fit as a feature for the test runner. It sounds like something you would use to debug a failing test while the test runner itself doesn't care about. Maybe you can try running the test just by pressing "Play" instead of running it from the test runner. It won't stop the execution and you will have all messages in the log.

    Another thing is the design of your test. "When you want to check several isolated behaviours in one test" is not a good practice in writing tests. Maybe you should redesign it? The new version of the tools allow you 'nest' the test object which allow you to have semi-shared code for tests.
    Feel free to write more about what you are trying to test. Maybe we can figure something out together!

    I assume you expect the behaviour to be similar to unit tests (that succeed by default).
    In case of a unit test, the bounds of the test are clear. The start point and the end point are defined by the test method.
    For integration tests the case is a bit different. You don't have any explicit entry and exit points. As an entry point you can, of course, consider the moment when the test's GameObject is being enabled. You can 'catch' that moment in Awake method. The exit point, however, is not defined. It could happen in the same frame or in some time in the future. And at this point, it's up to the developer to define it.
    The reason I find having "succeed on timeout" a bad idea is because such approach can hide defects and give you false-negative result. You can't really say in this case that your testing code (the one where you perform all the checks for failures) got executed at all.
     
  7. alexlopashev

    alexlopashev

    Joined:
    Mar 3, 2014
    Posts:
    7
    Ability to test different independent situations within one single test (of course you must be sure that test cases are not interfering with each other) is not so bad and not always means bad design - it's a way to improve performance. You can say that performance in test is the last thing to carry about, and you'll be right. But if you have dozens of heavy long tests then reducing overall testing time gives benefits to your team. Now I don't know will this function be as useful for all users as it is for my team.

    Actually you're right, absence of fails can be caused by broken code. Maybe this can be fixed by introducing additional mechanics "proceed after pass" similar to prevously discussed "proceed after fail" but with slight difference - 'pass' signals is simply ignored but 'fail' signal will fail test immediately. With this signal processing 'pass on timeout" starts to make sence.
     
  8. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    I have to disagree that is not bad to have a long, multipurpose tests. And here is why:
    - The improvement in performance is happening only if you need a long setup step. You can, however, share the setup between different tests while still having separate test objects.
    - Having one long test makes it's hard to debug. Once a check from the end of the test fails, you need to go through all the test to reproduce every time.
    - It's hard to make the check independent form each other and completely eliminate situations where some tests failed because the previous ones did.
    - It's harder to maintain such tests, and it's less scalable.
    - Performance can be important in some cases - I agree. Some tests suites you want to run before every commit, some are run over the night. With the latter you don't care about the performance as much as with the former.
    - I'm not saying having high level, long test is completely bad but they shouldn't be the core of your test suits. For reference you can search for the automation pyramid.

    My guess is that you are trying to automate manual test scenario or some sort of acceptance test. To sum up, this framework wasn't designed for the approach you are aiming for. It has the benefit, however, of having public code so you can change modify for your needs (as you did :)). Unless there is a global need (see Unit Feedback) I don't think this would benefit greater amount of users.

    You can very easily and simply set a passing call after period of time - see the CallTesting component. I'm not fond of such approach tough.
     
  9. alexlopashev

    alexlopashev

    Joined:
    Mar 3, 2014
    Posts:
    7
    Yes, I agree that other users are probably not interested in this feature.

    In this case we'll have to syncronize timeout of test and "after seconds", but this solution is acceptable though.

    Tomek, thank you very much for your detailed answers.
     
  10. Nezabyte

    Nezabyte

    Joined:
    Aug 21, 2012
    Posts:
    110
    Besides the tutorial video, any recommendations on unit testing books/guides? Also, any plans for more examples? The unit test examples in the pack were pretty generic unlike the integration tests.
     
  11. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @PsixROs
    You are very welcome! I'm happy that my answer was useful.

    @Nezabyte
    Yes, we are planning to create tutorials and patterns for using the tools. It's hard for me to say when this is going to happen though. You can find tons of materials about unit testing since it's not a concept specific to Unity. The book depends on the level you are so hard to recommend a specific one but I can tell what my favorite authors are. Uncle Bob (Robert Cecil Martin) has a series of video tutorials about clean code (cleancoders.com). He also wrote books about that. It's not only about unit tests but about having clean, well-designed code which is very beneficial when writing good unit tests. Another author is called Roy Osherove. Besides coding, he also focuses on team leadership. He wrote a book called The Art of Unit Testing which I can recommend. One more book I've heard lots of good things about is called How Google Tests Software (by J. Whittaker, J. Arbon, J. Carollo). It's more generic but a really good read.
     
  12. BrianHallTurbine

    BrianHallTurbine

    Joined:
    Apr 10, 2014
    Posts:
    1
    Hey folks,

    I made a couple changes to my version of the Testing Tools that I think would be useful for folks:

    1. The unit test names are not escaped in the XML, so if you name a unit test "5 < 3" you will get invalid XML in the test results.

    In UnityTestTools/Common/ResultWriter/XmlResultWriter.cs:WriteOpeningElement
    Old:
    Code (csharp):
    1.  
    2. 80: foreach (var attribute in attributes)
    3. 81: {
    4. 82:   resultWriter.AppendFormat (" {0}=\"{1}\"", attribute.Key, attribute.Value);
    5. 83: }
    6.  
    New:
    Code (csharp):
    1.  
    2. 80: foreach (var attribute in attributes)
    3. 81: {
    4. 82:   resultWriter.AppendFormat (" {0}=\"{1}\"", attribute.Key,  System.Security.SecurityElement.Escape(attribute.Value));
    5. 83: }
    6.  
    2. Unity doesn't return any kind of useful return value if the tests fail and batch mode is requested:

    In UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:StartTestRun
    Old:
    Code (csharp):
    1.  
    2. 476: if (UnityEditorInternal.InternalEditorUtility.inBatchMode)
    3. 477:   EditorApplication.Exit(0);
    4.  
    New:
    Code (csharp):
    1.  
    2. 476: if (UnityEditorInternal.InternalEditorUtility.inBatchMode)
    3. 477: {
    4. 478:   int errorCount = 0;
    5. 479:   foreach (UnitTestResult result in testList)
    6. 480:   {
    7. 481:     if (result.IsError || result.IsFailure || result.IsInconclusive)
    8. 482:     {
    9. 483:       errorCount++;
    10. 484:     }
    11. 485:   }
    12. 486:   System.Console.WriteLine ("***UNIT TESTS: Error Count: " + errorCount);
    13. 487:   EditorApplication.Exit(errorCount);
    14. 488: }
    15.  
    Returning the number of errors allows automation environments to see that the tests failed without parsing the results xml.

    Brian
     
  13. Norsfell

    Norsfell

    Joined:
    Oct 30, 2013
    Posts:
    2
    Very nice tool !

    Is there a way to order the integration tests execution in the integration test editor without renaming all tests ?

    Thank you
     
  14. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @Brian
    Good suggestions, thanks!

    @
    Norsfell
    There is no way to order them. The order of your tests shouldn't matter. Maybe grouping them would help? What are you trying to achieve?
     
  15. Norsfell

    Norsfell

    Joined:
    Oct 30, 2013
    Posts:
    2
    Thank you for the response.

    I'm trying to have dependency between tests.

    For example I have 3 tests :

    - One to init a map
    - One to create blocks (needs an initialized map)
    - One to construct a building on the blocks (needs blocks)

    The first test init a map and check if all is ok. The second create blocks on this map and the third constrcut a building on this block.

    If the tests are not properly sorted a test can fail. we don't use Unit Tests to do them but normally Integrations Tests should be able to do this.
     
  16. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Having dependacies between tests is a bad idea. It leads to several issues like maintenance problems, instability and it makes it hard to debug. You should rather try not to have dependacies :)
    This looks like initialization steps. Maybe try using nested structure with object that perform the setup you need. For example like that:
    $Untitled.png
     
  17. Nezabyte

    Nezabyte

    Joined:
    Aug 21, 2012
    Posts:
    110
    Thanks for the recommendations. I've started writing unit tests and reading Art of Unit Testing - good stuff!
     
  18. Nezabyte

    Nezabyte

    Joined:
    Aug 21, 2012
    Posts:
    110
    Question - if I use Random.seed = 1 in a Test function, will this affect other test functions that do something with the random class? Not sure how the lifetime works for this.

    Edit: Also, I noticed that when I change the code for a test and try re-running it in the unit test runner, the progress bar finishes, and it still looks like the test failed. However, if I wait a few more seconds, or try running the test again, it seems to finally update the status in the unit test runner.
     
    Last edited: May 5, 2014
  19. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    yes, the framework does not reset any variables.
    It could be the scene cleanup (undo) working in the background. Is it also happening when you run the tests on a new scene or with no undo work to do?
     
  20. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    I've just tried Unity Test Tools for the first time. Definitely nice to have this framework available.

    I have some feedback based on my first half day of testing:

    • code in DLL's can be tested, but the tests themselves cannot be in a DLL -- this can be fixed with a small tweak to NUnitTestEngine.GetAssemblies -- I changed it to load any assembly with the word "Test" in its name, but I'm sure there's a cleaner solution
    • unit tests marked [Explicit] are handled correctly, but it's hard to tell from the unit test runner, since the test status is never cleared. I think test status should be cleared whenever script compilation occurs, when tests are run, and via an explicit context menu in the test runner window.
    • double-clicking on a test attempts to open the code, but it fails if you have an external editor configured via the "$(File)" syntax (in my case, Notepad++). This looks like a bug in InternalEditorUtility.OpenFileAtLineExternal.
    • would be nice if selecting a test in the unit test runner would ping that source file in the Project panel

    I also wanted to mention that MonoDevelop already comes with the NUnit framework (and Mono.Cecil) -- I wonder if you could simply use this version (which is already installed with Unity) and avoid some of the overhead in the Test Tools package?

    Thanks for the package, I look forward to using this for real.
     
  21. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    It's not the first time someone brings up this issue. I guess we will need to address it at some point but this would require to improve the UI accordingly to split the tests between assemblies they come from.
    Having historical results (from previous runs) is good but it might be a good idea to mark them somehow as well.
    Could use please submit a bug via the bug reporter with repro steps?
    Yes, we are aware of that. At this point, since the package is released as an Asset Store package and we use custom builds for nUnit, it has to be like that :(
     
  22. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Tests are already split in the UI by namespace, so there is no need to do anything special for splitting by assembly. I would suggest that the right way to decide which assemblies should be included is either to scan assemblies for tests (which might be slow) or apply a new [TestAssembly] assembly attribute to mark the assembly (e.g. via AssemblyInfo.cs).

    Perhaps if the historical results were cleared when code is recompiled? Or add a toolbar button to clear the results.

    I just submitted a bug with a test project demonstrating the InternalEditorUtility.OpenFileAtLineExternal bug. (http://fogbugz.unity3d.com/default.asp?606440_kd5had2lu2sllvqs)

    Fair enough, I figured that might be the case.
     
    Last edited: May 6, 2014
  23. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    We keep our project folder as clean as possible, which includes putting almost all code into DLL's. I was able to put the Unity Test Tools package into two DLL's (one runtime, one Editor-only), plus the additional support DLL's for nunit and nsubstitute. The benefits of doing this are cleaner Assets folder structure, and no need to recompile the test tools every time one of my own scripts changes.

    For unit testing, the only change I had to make was to use embedded resources for the Unit Test Runner icons, rather than loose files loaded from the Assets folder.

    For integration testing, I have run afoul of a Unity bug with how MonoBehaviour and ScriptableObject types are identified in DLL's. There are a couple of issues here, but the one that causes trouble for Unity Test Tools is that a ScriptableObject implemented in a DLL, which derives from a generic base class implemented in the same DLL, is not recognized by Unity as a ScriptableObject, and CreateInstance will fail for this type. This means that BoolComparer and all the other test actions that derive from ComparerBaseGeneric do not work -- the symptom is a continuous spew of null reference exceptions from the AssertionComponentEditor.

    A workaround is to change the ActionBase class hierarchy to not use any generic types. The proper solution is to fix Unity's assembly loader to correctly recognize all forms of inheritance.

    I have previously submitted a bug report about the derived classes in DLL's issue here.

    Thanks!
     
  24. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    I started to write a unit test to validate some assets we have. The first thing I did was to check that referenced game objects had been attached. Imagine my surprise to find that Assert.IsNotNull always succeeds on a GameObject property of a component, even if the reference has not been set. I expect this is to do with operator overloads in the GameObject class.

    A workaround is to instead Assert.IsTrue, which the operator bool overload allows to work. This works -- but is awkward to read and hard to remember to do.

    Bug report: http://fogbugz.unity3d.com/default.asp?606464_pm876pqs0r3jj8ba
     
  25. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Thanks yoyo for quality feedback!

    While I can't help you with the first issue (but others will give it a look!) the second problem is most likely cause by the implementation of Assert in nUnit. To compare GameObject with null you need to use == operator (which is overridden). The implementation of Assert is probably using Equals method which simply will return incorrect results.

    Try "Assert.That(component.Reference == null)" which should work and is nice to read imo :)

    When I get a chance to look the nUnit's source code, I will know for sure what's wrong.

    Tomek
     
  26. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Yeah, Assert.That(obj != null) is definitely a better-looking workaround, just need to get over my instinct to write "Assert.Is[Not]Null".

    I've got my local version of Unity Test Tools itself working as a DLL, and for loading tests from DLL's I check whether the DLL has an NUnit.Description assembly attribute, which was a handy way to designate which assemblies should be scanned for tests. Any chance you would consider open-sourcing the package via github or similar? If so I would be happy to contribute merge requests back to your branch.
     
  27. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    The Assert.That is designed to be readable even with complicated assertions. Ideally, you would write Assert.That(obj, Is.Not.Null) which reads "assert that obj is not null ;)
    I looked into the issue and it was indeed a bug. It's fixed already :).
    We considered it but at this very moment we are not capable of doing that :(
     
  28. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Thanks Tomek for the update. By the bug that's "fixed already" do you mean the Assert.IsNull issue, or the inheritance across DLL boundaries issue? If the latter, that's great news -- fixed in 4.5 I guess?

    As for code contributions, I've made the following changes to my local copy of the Unity Test Tools package, let me know if any of this is of interest and I can send you the updated source code:
    • package converted to two DLL's, one for Editor, one for Engine (compiled with Visual Studio)
    • icons stored as embedded resources inside the Editor DLL
    • use of NUnit.Framework.Description attribute to identify which assemblies should be scanned for tests
    • make text area below unit test result tree scrollable, so that large messages can be displayed effectively (I use this for compound tests which run a number of tests then do a final Assert when finished indicating all sub-test results)
     
  29. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    I meant asserting GameObjects for null. Someone will give it a look at the other issue but afaik we only officially support MonoDevelop as IDE :(
    The info text area has already been made scrollable and resizable :)
     
  30. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Nothing I've done is specific to Visual Studio -- you could also use MonoDevelop to create multiple assemblies, and you would have the same inheritance issues across assembly boundaries. You can also put embedded resources in an assembly with MonoDevelop, so putting code and icons together in a DLL is equally possible with MonoDevelop as Visual Studio.

    Good to hear about the scrollable / resizable text area :)
     
  31. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    The problem is with conditional compilation. The API slightly changed across 4.x version and if we want to keep it backwards compatible, we need to ship source.

    The new version is coming out soon. If anyone is interested in beta testing please PM me :)
     
  32. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,174
  33. ChewyWaffles

    ChewyWaffles

    Joined:
    Jul 3, 2013
    Posts:
    5
    Thanks, but can I ask a question? How did you "new" a class derived from MonoBehavior (i.e. HealthComponent)? I thought that wasn't possible.
     
  34. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Technically it is possible but you should never do that. Bad things can happen.
     
  35. ChewyWaffles

    ChewyWaffles

    Joined:
    Jul 3, 2013
    Posts:
    5
    His example has him new-ing a MonoBehavior-derived class in an NUnit test. What is the proper way to instantiate classed derived from MonoBehavior?
     
  36. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,174
    Perhaps my example was not perfect. In this particular case there's really no justification for using a MonoBehaviour. It can be done using aggregation:

    public class Health : MonoBehaviour
    {
    private HealthManager healthMgr; // this is the class that will be unit tested
    }

    If the MonoBehaviour class cannot be instantiated directly (e.g: there's something terribly wrong with the example i gave and it should never ever be done), then in my opinion it should not be used for unit testing.

    For unit testing though, this should not have such a huge impact, as your code and logic should be decoupled and not dependant on other classes anyway, so MonoBehaviour gives little benefits anyway (i only used it since it's well known by Unity users).
     
  37. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    As liortal wrote, there shouldn't be a need for instantiating MBs. You can extract all the logic you want to test to a separate class and test that class. I will be writing a blogpost about that soon.

    PS. The only proper way of creating instances of MonoBehaviours is to use GameObject.AddComponent method (which obviously requires you to have a GameObject instance)
     
  38. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,174
    Having all those objects just for the sake of a unit test defeats the purpose of testing at the "unit" level in my opinion.
     
  39. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    You mean GameObjects? I never said that's the way to test MonoBehaviours (or the logic behind them). I wanted to point out that the only 'legal' way of instantiating MBs is by attaching them to GOs which doesn't really fit for unit tests, as you said.
     
  40. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,174
    I Know. My answer was more directed towards @ChewyWaffles that was asking how to create MB for testing. I wanted to stress out the point that the best way to instantiate those for testing is to not instantiate them or any other "unity" object at all :)
     
  41. Arkade

    Arkade

    Joined:
    Oct 11, 2012
    Posts:
    558
    Hi, test aficionados and TDD'ers

    3 things:
    1. An integration testing helper
    2. A couple of Comparers
    3. A collaboration/open-source query
    (1)
    I've mostly written a tiny tool to help integration testing that works with the marvellous Unity Test Tools. It eases writing integration tests by allowing invoking methods, pausing and enabling AssertionComponents.

    Here's the in-progress documentation and here's the GUI:

    View attachment 102019

    Useful? Already provided by something else? (he said, preparing the "doh!")

    If it has some merit, I thought to release as a free Unity Asset, perhaps open-sourced. Reasonable? (figuring Asset Store presence is most like a package manager, highlighting update availability). Noting the presence of @Tomek in this thread, I'd be happy for it to be integrated into UTT if that's an option.

    (2)
    Also I've written a couple of Comparers that seem handy:
    • MaterialComparer does the obvious
    • FloatDirectionComparer asserts a float always moves in a defined direction (up/down) with optional leeway.
    I intended to package them with (1) but start to feel is creeping towards something larger so...

    (3)
    I'm sure I'm not the only one making extensions for UTT (I'd thought they'd probably be posted here but do please redirect me if I've missed them elsewhere). This isn't something I think any TDD advocate would believe should cost (we're building on open source and raising the level of our craft(s) -- pun intended). Is there an open source repository we could all put our stuff in? (If not, I'll set one up on... GitHub = fair?)

    All thoughts welcome!
    Rupert
     
  42. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Hi Rupert,

    Great job with extending the tools! I'm sure lot's of users will find it very useful.

    About the collaboration/public repository. We are aware of the 'problem' and still discussing what would be the best way to approach it. Sorry for not having an easy way to contribute to the Tools, but I promise we'll find a solution as soon as it's possible.

    Tomek
     
  43. Arkade

    Arkade

    Joined:
    Oct 11, 2012
    Posts:
    558
    Hi, @Tomek (and all)

    Cool, I'll proceed forward with the package then.

    I was offline yesterday but pondered the collaboration thing. I wondered whether we should start with somewhere we can all put both (1) extensions and (2) patches to official UTT. I'll comment in reverse order:

    2. UTT patches

    It seems several people are creating fixes and mods for the Unity Test Tools and offering them. While duplicating the UTT package in GitHub is a way (and probably the most community-friendly way?), I'm getting the feeling Unity would prefer not \[yet\] for their own reasons. (right?) As such, old school patches might be simplest (perhaps "patch -u" form). If they cannot yet be accepted by Unity, perhaps we can gather them together to (a) ease distribution, (b) reduce duplication of effort, (c) allow providing updates and (d) have somewhere for Unity to gather them from if/when it becomes possible to accept them. Obviously, all toes are sacred so I really hope I'm not treading on anyone's!

    1. Extension package

    If my original assertion (re. posting in AssetStore is most convenient to users), we want to keep our extension work in a form that expedites this. Obviously we'll also want versionning. While posting them individually to the wiki would suffice for the latter, not so much the former.

    As such, I'll tidyup my little package and create a GitHub repository with roughly the following structure...

    • Unity Project root, perhaps named "Community Testing Tools Extensions"
      Containing all the usual project stuff including "Assets/TestingExt/" (A bit shorter than "CommunityTestingToolsExtension" ?)
    • Patches
    • Docs
      .../Extensions
      .../Patches
      .../img (for screenshots, etc.)
    • README.md

    README.md can include links to documentation. We'll expect people to write at least something for anything they offer.

    Re. control, I'll create and push to GitHub but don't want to be gatekeeper, bottlenecking or blocking contributions or builds. Hopefully some others will volunteer and we can have more than 1 owner of the top repo -- perhaps a GitHub "Organization". We can run the repo with usual Git clone and offer "Pull Requests".

    Right, that's probably more than enough spam from me. Feedback on this and everything else most welcome... ideally before I push ;-)
    Thanks, Rupert.
     
  44. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    It is possible to create a new GameObject, add components to it, call some methods on those components, and then DestroyImmediate the object. This is sometimes sufficient for unit tests. But Tomek is correct that if you want a long-running test that goes through the normal Awake / Start / Update cycle then you are better off using integration tests.
     
  45. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @Rupert
    Thanks for the initiative! As I wrote before, we do have a plan to do something about it but we don't know how to approach this yet. A public repo would be one thing to do but it will require a gatekeeper, as you said.

    @yoyo
    Correct :)

    PS
    A new version of the tools just got into the Asset Store. There will be a blogpost with the new features soon but in the meantime you can check the changelog. Enjoy :)

    Tomek
     
  46. AdamV.SG

    AdamV.SG

    Joined:
    Sep 20, 2013
    Posts:
    3
    Hi! I'm happy this is evolving so fast.

    However, I would like to have the unit tests outside of the Editor folder.
    First - I like to keep my test files clearly visible and close to the tested code... and the Editor folders don't say anything like "Look here for the unit tests!". (Also, it's totally Unity specific and I would like to use this framework even on some general libraries, where I don't have and don't want to have anything Unity specific.)
    Second - It's useful to actually build the tests for a target platform and also to run them there. And that's impossible if they are editor only. (Yes, if things would be ideal, the code would run the same everywhere... but they are not. ;))

    What's your point of view regarding this?
    Thanks, Adam
     
  47. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Hi AdamV.SG

    I believe it was mentioned somewhere earlier in this thread. The reason you'd want to put your tests in the Editor folder is not to have them included in the the production code. Unfortunately there is no project manager in Unity that would allow to define "Test" folders as folders with tests and tell the compiler to exclude them in the builds.

    The reason that unit tests are not runnable on the platforms is because unit tests are meant for testing different things. You can use integration tests in order to test code that you think might work differently on different platforms. As I could imagine some cases where such solution could be useful, we simply didn't do anything towards implementing it so far.

    Cheers, Tomek
     
  48. kromenak

    kromenak

    Joined:
    Feb 9, 2011
    Posts:
    236
    FWIW, I found this page pretty helpful in understanding the difference between unit and integration tests: http://www.typemock.com/unit-tests-integration-tests. Going by their definitions, it makes sense that unit tests are only runnable in the editor, since unit tests shouldn't test things that change from platform to platform.

    So far, I'm finding this to be a great tool! I did have a few bits of feedback for you:

    1) I may be the only person in the world who ran into this issue, but I ended up having two folders in my project called UnityTestTools, which broke the icons in my editor. After a bit of debugging, I fixed it by renaming one of my UnityTestTools folders. However it might be best, in Icons.cs, instead of just using the first directory with that name, maybe also check that the icons folder exists at that location with Directory.Exists or something similar. To demonstrate, I attached a modified Icons.cs.

    2) For usage in something like Jenkins, I wanted to establish a pattern for Integration Tests such that instead of passing a list of test scenes to the command line, I could just find all scenes ending with "Tests" and pass those to the Integration Test Runner through code. However, it seems that RunIntegrationTests() internally is checking command line arguments.

    It would be better for my purposes if I could just run the Integration Tests by passing a list of scenes to the test runner instead of always relying on the command line. I noticed I can easily do this by making BuildAndRun and RunInEditor public functions, but it might be good functionality to expose in the API in future releases. I attached a modified Batch.cs that demonstrates what I'm talking about.

    Anyway great work thus far - excited to see where this goes in the future!
     

    Attached Files:

  49. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Hi kromenak,

    It's very good suggestions! I'll try to address those issues with next release.

    Thanks! Tomek
     
  50. Arkade

    Arkade

    Joined:
    Oct 11, 2012
    Posts:
    558
    Hey, guys
    Well, it's been a while of silence but...

    Please find attached an initial version of the extension I mentioned in #91 for you to try.
    I've presumptuously named the package CUTTE -- CommUnity Test Tools Extensions ;-) (one can but hope)

    I'll be posting it to GitHub here shortly and later to the Asset Store ... (if I can figure out the process!) (actually, I might submit the Asset Store version first since it will help me decide directory layout to keep the git history clean.)

    All feedback welcome
    HTH, Rupert.
     

    Attached Files: