Search Unity

Unity Unity Test Tools

Discussion in 'Testing & Automation' started by Tomek-Paszek, Dec 18, 2013.

  1. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    Unity test tools are open source. You can fix this issue yourself (Unity 5.3 obsoletes the use of this type of scene management, you should use the SceneManager class for that).

    You could also post your fixes as a pull request to the official Unity test tools repository on bitbucket so that it will be merged into the package in a later version.
     
    goldbug and holliebuckets like this.
  2. goldbug

    goldbug

    Joined:
    Oct 12, 2011
    Posts:
    679
    Tx, I didn't realize it was open source, it seems they already did it. It's just a matter of waiting for the next release now.
     
    doggan likes this.
  3. kromenak

    kromenak

    Joined:
    Feb 9, 2011
    Posts:
    236
    Not sure if this is an issue or not, but we recently upgraded to the latest Unity Test Tools in a project using 5.2.2, and we got a bunch of compiler errors related to the use of the SceneManager. Makes sense, since SceneManager wasn't introduced until 5.3, I think?

    However, judging from the asset store description stating that Unity Test Tools works with 4.0, 4.2.2, 5.0.0, 5.1.0, etc - it was kind of surprising that references to SceneManager weren't behind some script defines or something. It seems like you'd have to do that to maintain backward compatibility? There's nothing on the asset store page stating that it requires a certain version of Unity.
     
  4. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @kromenak latest UTT version only supports 5.3. Try downloading the package from Unity 5.2 and it should give you a compatible version.
     
  5. dimitroff

    dimitroff

    Joined:
    Apr 3, 2013
    Posts:
    103
    Anyone able to run Integration tests in batchmode using Unity 5.3.4 and OSX El Capitan?

    For some reason, first I am getting an exception in NetworkResultsReceiver.cs. Then after adding a temp fix, I don't get any results from the tests. Unity just quits, without processing my test scene.
     
  6. kevinsbk

    kevinsbk

    Joined:
    Apr 23, 2016
    Posts:
    2
    Does anyone know how to run each test on a new scene in Unity 5.3.4? The option doesn't exist in the editor test runner. I saw in some threads that the tests are being run on a new scene by default. But it doesn't seem to be the case. I tried running a collection of tests where only the first test starts with opening a test scene using " EditorSceneManager.OpenScene(___) ". The expected behavior is that the rest of the tests should fail because no scenes were open. However, I noticed the same scene opened in the first test is being used for the rest of the test cases. Is there any way to ensure that all tests are run on a clean, unpolluted scene?
     
  7. Alex-Chouls

    Alex-Chouls

    Joined:
    Mar 24, 2009
    Posts:
    2,294
    Running into these errors when testing on platforms (e.g., osx universal build)

    Screen Shot 2016-05-08 at 9.34.11 PM.png

    Build settings has a single test scene.

    Testing locally, but the goal is to use Unity Cloud Build (even if I have to manually run each build to see results). But currently not even getting results in standalone builds.
     
  8. Alex-Chouls

    Alex-Chouls

    Joined:
    Mar 24, 2009
    Posts:
    2,294
    I only get the error in 5.3. Works fine in 4.6 and 5.0.
    Also get a null ref error with Run Current Scene in 5.3 only:
    NullReferenceException: Object reference not set to an instance of an object
    UnityTest.IntegrationTests.PlatformRunner.BuildAndRunInPlayer (.PlatformRunnerConfiguration configuration) (at Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/Editor/PlatformRunner/PlatformRunner.cs:77)
    UnityTest.IntegrationTests.PlatformRunner.BuildAndRunCurrentScene () (at Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/Editor/PlatformRunner/PlatformRunner.cs:37)

    Guess I'll poke around the code and try to figure it out...
     
  9. nresonbrown

    nresonbrown

    Joined:
    May 14, 2016
    Posts:
    5
    It's been a long time since this conversation, but maybe you can help here: is there any way to send more than just the results xml file back to the computer that invoked the tests? I'd love to be able to save some screenshots (for comparison purposes) back from an Android, iOS, or even WebGL build to that same results folder. Right now I don't see how to get access to that directory or save anything there without some rather serious work arounds.
     
  10. vololdyy

    vololdyy

    Joined:
    Jun 1, 2016
    Posts:
    1
    Hi,

    I have 2 scenes. The first is a launcher and the second game. How can I use the tests in the game if they are created in the launcher? Now the problem is that the tests do not experience load. Of course I used DontDestroenLoad (this).

    Thank you.
     
    Last edited: Jun 1, 2016
  11. DavieTag

    DavieTag

    Joined:
    Mar 23, 2015
    Posts:
    10
    Hi,

    I'm having a small problem with Integration test runner. If I try to test anything which has [RequireComponent] on it, I get an error when RebuildTestList() tries to Destroy the required component. it complains about dependencies. Is this a known issue? Is there a way around it? if I could check for dependencies before DestroyImmediate() I could just loop twice and it'd work fine!

    Thanks
     
  12. DavieTag

    DavieTag

    Joined:
    Mar 23, 2015
    Posts:
    10
    I've worked around this by manually destroying components in order of dependencies, but there are some circular dependencies (yes, that may be bad design) that can not be worked around in this way. I still would appreciate some answers, though.
     
  13. Garm

    Garm

    Joined:
    Jan 22, 2013
    Posts:
    5
    I'd really love to see such feature. I've tried to run some basic tests using that principle and concept seem to be working. I had to create tests from the code though, assertionComponent won't hold the reference to the gameobject from another scene in runtime. I've yet to figure why.
     
    Last edited: Aug 9, 2016
    Arkade likes this.
  14. clintonb

    clintonb

    Joined:
    Dec 14, 2013
    Posts:
    15
    Do I need Unity Pro to get Integration Test Results back onto my computer?

    I'm using Unity 5.3.1p2 and have tried the latest Test Tools from the asset store, and the latest ones from the 'stable' branch on bitbucket. (I recognize that that may be designed for Unity 5.4). When I run Integration Tests on my computer, which has a Pro license, to an Android device, they run and report back. When I do it on a build slave, which does not have a Pro license, the Integration Test Monitor window waits for a connection from the device and never gets one [which makes it rather difficult to use in a continuous integration system :) ]

    Is this the expected behaviour? I thought as a Unity 5 there wasn't any difference with Sockets between Unity Free and Unity Pro.

    Addendum: I tried with Unity 5.4.0p4 and get the same result.
     
    Last edited: Sep 1, 2016
  15. BugKiller

    BugKiller

    Joined:
    May 8, 2014
    Posts:
    22
    Hello,

    does someone know how to integrate/connect Unity Test Tools with SikuliX (http://sikulix.com/) to execute some GUI tests on Unity 3D games?

    I already implemented lots of unit and integration tests with the Test Tools but now I want to combine integration tests from the Unity plugin and functional (UI) tests from Sikuli.

    Example: Run the game and achieve a new high score in the end of the run and check if the new "high score popup menu" was shown to the user.
    I know how to implement one integration test to run the game and achieve a new high score in the end of the run and I know how to run the game manually and run one Sikuli test to check if the new "high score popup menu" was shown to the player, but I have no idea how can I connect both tools. As I am able to execute all the unit and integration tests battery with one click on Unity Test Tools, I would also like to connect my Sikuli tests to that automated test process and run all the unit, integration and UI tests with the minimum manual effort.

    Thanks for any suggestions.
    João
     
    Arkade likes this.
  16. Arkade

    Arkade

    Joined:
    Oct 11, 2012
    Posts:
    564
    Thanks, João -- SikuliX sounds fascinating and really interesting approach for testing Unity UI / game results (I guess it could even image recognise non UI stuff!)

    I'll be interested to hear how you get it working! When you do, do you think you might post a tutorial on it all so others might use it please?
    Thanks!
    Rupert
     
  17. BugKiller

    BugKiller

    Joined:
    May 8, 2014
    Posts:
    22
    @Arkade As soon as I am able to connect both tools and get some results I will share here my solution with everyone.

    I'm having another problem that someone might know how to fix.

    When running the Editor Tests Runner from the command line (Android):

    to execute the Unit tests everything is fine on Android and iOS and I get the results from the tests.

    However when I run the Integration Test Runner on Android:

    I get this error:

    and then after a while this one:

    I've searched a lot in the forums and other websites and couldn't find anything to solve this. Does anyone know how to fix this?

    Thanks.
    João
     
  18. naomijub

    naomijub

    Joined:
    Aug 1, 2016
    Posts:
    1
    So where I work we use Monogame Framework as a solution to our agile necessities (Also, it greatly integrates with jenkins and Go CI). Recently we found out about Unity Test Tools (UTT) and decided to test it in a new prototype game. Our programming practices include test driven development (and BDD) and many XP tools. However, we could not adapt to the UTT due to the fact that it is almost impossible to write test before coding, or before having a prefab ready to develop. We think the test runner should support more features and components test. Another thing we think is useful is to keep C# version updated and allow integration with other frameworks like XNA. As an example I can quote the hard time we had developing a car race game with unity 5.4, which made us quit the projeto due to poor testability and even poorer test first code later capabilities.
    Hope to see improvement with UTT, it would make our life a lot easier.
     
  19. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,941
    As long as it's not possible to mock MonoBehaviour messages, I don't think it's possible to unit test MonoBehaviours in any meaningful way. Which, of course, rules out TDD. Having UTT able to simulate an Update or a Collision would make it actually possible to write unit tests. As of now, supporting tests would require adding a layer between MonoBehaviours and logic, which is painful.

    It is possible to test some of the messages through reflection, but that's way too brittle to maintain.

    I did have some luck using TDD-like principles with the integration testing framework for when developing enemies. Things like:
    - place an enemy prefab in front of something the enemy should attack
    - wait a second
    - check that the target has taken damage

    Doing tests first for those kinds of things works. It would be a lot better if integration tests could be run with a locked framerate (afaik they can't), and you have to make sure that you're able to eliminate randomness.

    The update for the C# version is under development. There's beta builds available.
     
  20. createthiscom

    createthiscom

    Joined:
    Dec 19, 2016
    Posts:
    67
    Usually it's the job of the testing framework to manage the state of the assertions and pass/fail the test. What's the recommended way to manage a long list of assertions so we can determine if the test passes or fails?

    Thinking about it, I guess we can use Exceptions to avoid having to track the state. It's not ideal, but it does the job.

    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEngine.Assertions;
    3. using System.Collections;
    4. [IntegrationTest.DynamicTest("some-scene")]
    5. [IntegrationTest.Timeout(5)]
    6.  
    7. public class SomeIntegrationTest: MonoBehaviour {
    8.     private class IntegrationTestFailure : System.Exception {
    9.     }
    10.     private void Assert(bool value) {
    11.         if (!value) throw (new IntegrationTestFailure());
    12.     }
    13.     private void Start() {
    14.         Assert(true == true);
    15.         Assert(false);
    16.         IntegrationTest.Pass(gameObject);
    17.     }
    18. }
    19.  
    In the above example, the test fails on the second Assert. We get a line number in the console output, so it's serviceable, I guess.
     
    Last edited: Feb 9, 2017
  21. Creative-Pudding

    Creative-Pudding

    Joined:
    Nov 7, 2014
    Posts:
    46
    Sorry, but this is just a hack! I have several problems for your test. While it would be cool, it's just not production ready. What if you have a build script to do the job (CI build?), what if you want the game to be tested and built (as production build server would do) and what if I want my game not to be running at the background?
    Should I change the game design and application behaviour because of a test? Seems to be a no go for me!
     
  22. EibisDeveloper

    EibisDeveloper

    Joined:
    Mar 16, 2017
    Posts:
    1
    Hi Everyone! I am trying to use the UTT but I am having some problem! I already tried to search online as I normally do but I didn't found anything either in the documentation or in forums : /

    So I have to ask here and hope someone will answer.

    I have my game in a Main scene, and my Test scene named TestMain.
    I load the scene from inside my unique Test and then I take the references to what I need to make the test and everything runs fine in the editor.

    When I try to use the Platform Runner to see the test on an Android device I get an error because the only scene loaded on the device is my current test scene (but the test is actually run).

    I tried to add the scene in the Platform Runner but this way it's seen as a Test Scene and it runs stand-alone (and not with my test scene).

    I tried to run the two scenes together like a normal game but this way the test is not being kicked off, I think, because again my Main game runs without any test being kicked off.

    I tried to do these experiments putting the TestScene before the Main in my build order but no difference.

    So there is a way to run a TestScene that depends on a NON - test scene on a device?
     
  23. Nigey

    Nigey

    Joined:
    Sep 29, 2013
    Posts:
    1,115
    Hi Guys,

    I see the Unity Test Tools is now depricated on BitBucket. Is there a timeline for implementing the Unity Test Tools and integration test runner into a Unity beta? CI will always be a wanted feature :).

    Cheers!

    EDIT: I've found that Unity 5.6 has both edit and play mode tests! Yippee! My problem is solved :).
     
    Last edited: May 17, 2017
    superpig likes this.
  24. Aquiris

    Aquiris

    Joined:
    Jan 19, 2016
    Posts:
    1
    Hey guys, is the test result file generated by Unity (we're using 5.6.1f1) suposed to comply with Nunit definition? We are trying to get Jenkins to export results using the Nunit plugin but we're getting the following error:
    Code (csharp):
    1.  'Publish NUnit test result' failed: Could not read the XSL XML file.
    As far as I can tell, the file is missing the test-run container (it starts with a test-suite container instead, which seems to be not the way other Nunit integrated tools handle it).
     
  25. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    It's an oversight from our side.
    For now I can onlt recommnend you to postprocess the result file before uploading it to Jenkins.
    If someone could confirm that wrapping the result file in <test-suite> is sufficies for Jenkins to parse it would make it easier for us to come with a fix.
     
  26. Talsidor

    Talsidor

    Joined:
    Aug 7, 2012
    Posts:
    22
    Has anyone else used the Test Tools in 2017.1?
    We upgraded a project from 5.5.0x3 to 2017.1.0x2 and now when attempting to run our integration tests we get this error:
    Which points to line 13 in this file:
    Code (CSharp):
    1. using System;
    2. using System.Reflection;
    3. using UnityEditor;
    4.  
    5. namespace UnityTest
    6. {
    7.     public static class GuiHelper
    8.     {
    9.         public static bool GetConsoleErrorPause()
    10.         {
    11.             Assembly assembly = Assembly.GetAssembly(typeof(SceneView));
    12.             Type type = assembly.GetType("UnityEditorInternal.LogEntries");
    13.             PropertyInfo method = type.GetProperty("consoleFlags");
    14.             var result = (int)method.GetValue(new object(), new object[] { });
    15.             return (result & (1 << 2)) != 0;
    16.         }
    17.  
    18.         public static void SetConsoleErrorPause(bool b)
    19.         {
    20.             Assembly assembly = Assembly.GetAssembly(typeof(SceneView));
    21.             Type type = assembly.GetType("UnityEditorInternal.LogEntries");
    22.             MethodInfo method = type.GetMethod("SetConsoleFlag");
    23.             method.Invoke(new object(), new object[] { 1 << 2, b });
    24.         }
    25.     }
    26. }
    Any help would be very appreciated. :D
     
  27. SwathiJayaraman

    SwathiJayaraman

    Joined:
    May 26, 2017
    Posts:
    2
    Hi,

    I am new to unity and unity test tools..I would like to know :
    1. How to run the tests on hololens emulator from unity
    2. If I can run the same tests on Visual studio, after building the unity project as a 3D holographic application?
     
  28. coman_games

    coman_games

    Joined:
    Dec 12, 2015
    Posts:
    7
    Hello, I found and fixed the problem it's really where simple.
    So all this code is actually is doing is change pause on play to false if it set to true.
    It's look like unity changed the core architeture and moved LogEntries from UnityEditorInternal into UnityEditor namespace.
    What can be seen in assembly info
    upload_2017-8-10_8-3-49.png
    So the fix is changing
    upload_2017-8-10_8-7-20.png
    into
    upload_2017-8-10_8-7-53.png
    The finaly code will be
    Code (CSharp):
    1. using System;
    2. using System.Reflection;
    3. using UnityEditor;
    4.  
    5. namespace UnityTest
    6. {
    7.     public static class GuiHelper
    8.     {
    9.         public static bool GetConsoleErrorPause()
    10.         {
    11.             Assembly assembly = Assembly.GetAssembly(typeof(SceneView));
    12.             Type type = assembly.GetType("UnityEditor.LogEntries");
    13.             PropertyInfo method = type.GetProperty("consoleFlags");
    14.             var result = (int)method.GetValue(new object(), new object[] { });
    15.             return (result & (1 << 2)) != 0;
    16.         }
    17.  
    18.         public static void SetConsoleErrorPause(bool b)
    19.         {
    20.             Assembly assembly = Assembly.GetAssembly(typeof(SceneView));
    21.             Type type = assembly.GetType("UnityEditor.LogEntries");
    22.             MethodInfo method = type.GetMethod("SetConsoleFlag");
    23.             method.Invoke(new object(), new object[] { 1 << 2, b });
    24.         }
    25.     }
    26. }
    27.  
     
    Fesener, sapanda and Talsidor like this.
  29. Talsidor

    Talsidor

    Joined:
    Aug 7, 2012
    Posts:
    22
  30. jreed-curious

    jreed-curious

    Joined:
    Jul 24, 2017
    Posts:
    10
    I can confirm that if you post-process the test results XML output file by wrapping the document in a <test-run>...</test-run> tag then the NUnit plugin for Jenkins will correctly parse the file and publish the results.

    It's important to note that it should be a <test-run> tag, NOT a <test-suite> tag. You can find the complete documentation for the correct XML format on their github page (click here).

    Most of the attributes for the <test-run> are the same as the <test-suite> and can be copied directly from the top-most <test-suite> tag with the following exceptions:
    • id: is always 2
    • engine-version: I don't know what NUnit test engine is being used by Unity but I don't think this matters at all. So I guess put whatever you want in there?
    • clr-version: I don'w know what runtime version is being used by Unity but I don't think this matters at all. So I guess put whatever you want in there?
    Is there a timeline for a fix on this?
     
  31. jreed-curious

    jreed-curious

    Joined:
    Jul 24, 2017
    Posts:
    10
    In case anyone is wondering how I implemented the post-processing:
    1. Install XMLStarlet on the Jenkins server: brew install xmlstarlet
    2. Commit XSLT file for transforming Unity test results file into proper NUnit format
    3. Commit shell script for using XMLStarlet to transform Unity test results file into NUnit format
    4. Add build step to your Jenkins Unity-Test job to execute the shell script (you may have to export PATH=$PATH:/usr/local/bin before calling script, depending on how PATH is configured in Jenkins)
    You can view the XSLT file and shell script here.
     
    Last edited: Aug 19, 2017
  32. Dlala

    Dlala

    Joined:
    Dec 6, 2016
    Posts:
    1
    Hi jreed,

    I was wondering what version of Unity and NUnit you are using? After implementing your workaround we are still getting "Content not allowed in prolog" and conversion errors?

    The script seems to have generated the new xml fine with the required parameters.

    Thanks
     
  33. diego_mp

    diego_mp

    Joined:
    Jun 12, 2017
    Posts:
    23
    Hi guys,
    does someone know how to run an existent scene for automated test? @liortal any idea?

    Let's say I have a menu with that open several scenes, i would like to check automatically if i can open all of them, opening the main scene and providing the navigation input.
     
    Last edited: Nov 15, 2017
  34. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    @diego_mp as far as i know, unity test tools is now deprecated. you should use the built-in test runner (e.g: Play mode tests) to achieve that.
     
  35. Creative-Pudding

    Creative-Pudding

    Joined:
    Nov 7, 2014
    Posts:
    46
    current version 2017 p2 does not do any test if runs from script (command line) and results always Succeed. Test system was never stable, never was actually reliable, but it's just another thing getting worst in time :(
    It still works when runs from editor -as before-, but what if you have a build scripts or build server?