Search Unity

Unity Unity Test Tools

Discussion in 'Testing & Automation' started by Tomek-Paszek, Dec 18, 2013.

  1. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,941
    The latest versions of Test Tools are not up to date with the editor - there's some obsolete warnings, and one obsolete error.

    If I import the tools, Unity wants to "upgrade the project", which means that it'll spend an hour scanning all files for upgradeable errors. Yay.
     
  2. Alex-Chouls

    Alex-Chouls

    Joined:
    Mar 24, 2009
    Posts:
    2,294
    Hi, I've been using Test Tools and find it quite useful. However the lack of support for WSA and WP8 leaves a big hole in our automated testing. Particularly since these platforms seem to have many special rules/unity quirks... What are the plans for WSA/WP8 support?
     
  3. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,157
    This should be fixed as of version 1.5.4.
     
  4. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @Alex Chouls I'm aware of this problem but it's not so straightforward to fix it.
    We want to focus at the moment on native integration of the tools with the editor. We hope this will allow proper support of the tools on all of the platforms and better UX. I can't promise any dates yet unfortunately.
     
  5. Alex-Chouls

    Alex-Chouls

    Joined:
    Mar 24, 2009
    Posts:
    2,294
    Sounds like a good plan - looking forward to it! :)
     
  6. asset mania

    asset mania

    Joined:
    Dec 17, 2013
    Posts:
    3
    I run with jenkins and use these command, but no xml output
    please any help

    /Applications/Unity/Unity.app/Contents/MacOS/Unity -quit -projectPath "${WORKSPACE}" -logFile unity_unittests_results.log -executeMethod UnityTest.Batch.RunUnitTests -resultFilePath="${WORKSPACE}/unity_unittests_results.xml"
     
  7. Alex-Chouls

    Alex-Chouls

    Joined:
    Mar 24, 2009
    Posts:
    2,294
    Are there known imitations with WebGL integration tests?

    Using a simple AssertionComponent testing gameObject.activeSelf
    I get an InvalidPathException: Invalid path part gameObject

    Same tests work on other platforms.
     
  8. TheJimz

    TheJimz

    Joined:
    Jun 26, 2015
    Posts:
    42
    In the latest Unity version (5.1.1f1) it appears the -testscenes argument causes an overwrite of the scene selection in EditorBuildSettings.asset, overwriting our normal scene selection setup. This was not occuring on 5.0.x builds and the -testscenes remained temporary for that run.

    Any work around? This is causing issues in our build process as we run tests first, then a regular build and the regular builds ends up with the test levels.
     
  9. KyleHatch85

    KyleHatch85

    Joined:
    Dec 13, 2011
    Posts:
    99
    Hi, i was wondering if their is any way to call this function UnityTest.Batch.RunUnitTests from code. I have another pre build script where we set up various configs for our project and want to call the unit tests running from there, but it keeps telling me Batch does not exisit, i assume it's because it's a partial class and can't be called?

    Any ideas? can you call multiple functions from execute method?
     
  10. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @KyleHatch85 there should be no problem with that. Maybe your scripts are not in the Editor folder?
     
  11. KyleHatch85

    KyleHatch85

    Joined:
    Dec 13, 2011
    Posts:
    99
    I'm a step further it will compile (thanks for that, editor folder...i was silly) but our builder (teamcity) returns with exit code 1, which is a polite way of it tell me the unity build crashed.

    To make sure it wasn't a teamcity error, i wrote an autobuilder to run within the editor. Simulate the same process.

    Code (CSharp):
    1. /// <summary>
    2. /// Auto Builder
    3. ///
    4. /// Editor: Kyle Hatch
    5. ///
    6. /// Description: For Building Debug builds of the product
    7. /// </summary>
    8.  
    9. using UnityEditor;
    10. using UnityEngine;
    11. using UnityTest;
    12. using System.Collections;
    13.  
    14. public static class AutoBuilder
    15. {
    16.     private const string FILENAME_EXT = "/Base.exe";
    17.  
    18.     private static string[] m_Levels = new string[]
    19.         {"Assets/Main Project/Scenes/FrontEnd.unity" };
    20.  
    21.     [MenuItem("Tools/Windows Build With Higher Quality Settings &B")]
    22.     public static void SingleBuild ()
    23.     {
    24.         BuildProject(false);
    25.     }
    26.  
    27.     private static void BuildProject(bool bDev)
    28.     {
    29.         System.DateTime startTime = System.DateTime.Now;
    30.  
    31.         //Set Configs
    32.         PreBuildScript();
    33.  
    34.         string sPath = EditorUtility.SaveFolderPanel("Select a folder", Application.dataPath + "/../Build", "");
    35.  
    36.         if(null != sPath && string.Empty != sPath)
    37.         {
    38.             Utils.Code.ClearFolder(sPath, ".gitignore", ".lnk");
    39.  
    40.             Texture2D[] icon = PlayerSettings.GetIconsForTargetGroup(BuildTargetGroup.Standalone);
    41.             if(null != icon)
    42.             {
    43.                 icon[0] = Resources.Load("Icon") as Texture2D;
    44.             }
    45.             PlayerSettings.productName = "Basic Principle Switching Simulator";
    46.             PlayerSettings.SetIconsForTargetGroup(BuildTargetGroup.Standalone, icon);
    47.  
    48.             BuildOptions Options = BuildOptions.None;
    49.             if(true == bDev)
    50.             {
    51.                 Options =  BuildOptions.Development | BuildOptions.ConnectWithProfiler | BuildOptions.AllowDebugging;
    52.             }
    53.  
    54.             BuildPipeline.BuildPlayer(m_Levels, sPath + FILENAME_EXT, BuildTarget.StandaloneWindows, Options);
    55.         }
    56.  
    57.         System.DateTime endTime = System.DateTime.Now;
    58.         System.TimeSpan span = endTime.Subtract (startTime);
    59.      
    60.         EditorUtility.DisplayDialog("Build Complete", "Build Completed at: " + endTime + " Build Time: " + span, "OK");
    61.     }
    62.  
    63.     public static void PreBuildScript()
    64.     {
    65.         //Run Tests
    66.         Batch.RunUnitTests();
    67.  
    68. //Other things i want pre build
    69. //DoSomeChanges();
    70.     }
    The editor crashes when i run this script. i comment the line Batch.RunUnitTests() it builds fine. I've attached the Editor log from the crash.

    Ideas welcome.
     

    Attached Files:

  12. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    It doesn't seem like it crashed from the log. The test runner will close the editor after the run has finished when running in batch mode.
     
  13. dittytwo

    dittytwo

    Joined:
    Apr 30, 2015
    Posts:
    9
    I am using Unity 5.1 and unity test tools for integrations tests
    I am trying to simulate key pressing so that we can test the GUI responds correctly (dependant on screen size, desktop to wall size screens)
    I can't seem to find anyway to simulate a key press I obviously could hook into the code and run the procedure that the key press would instantiate, however this is not what I want to do for code maintainability. I have tried looking at using system.windows.forms.sendkeys (on active window), I have looked at using the dlIImport methodology to import user or win32.dll to hook in at the Kernel level (again not something I really want to do both of these are not available in the current version of .net with Unity.
    for example we have a private c# method in a game object that calls a screen shot and then places it in a specific folder under the users area. I would love just to be able to press "." (for example) and then wait to see fi we get a folder and then a screen shot. again for a system generated word document it goes into a documentation folder.

    Is there anyway to simulate a user key press/ mouse click within the unity test tools.

    cheers
    D2
     
    Last edited: Aug 17, 2015
  14. iivo_k

    iivo_k

    Joined:
    Jan 28, 2013
    Posts:
    312
    Excluding the test tools from builds by copying the folder away and back in the build script is very error prone, quite often copying the test stuff back just fails for some reason and then you have to manually sort the mess.

    Is there some better way to exclude the test tools resources from builds?
     
  15. weigun

    weigun

    Joined:
    Jan 13, 2014
    Posts:
    1
    hi all,
    when i import the package,some error occurred:
    unitytesttool the imported type mono.cecil.cil.sequencepoint is defined multiple times

    empty project is ok!

    thanks for help
     
  16. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
  17. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    I am trying to play around with integration tests (for the first time, yes... the shame).

    One thing i've noticed is the DynamicTest attribute. I am not sure how to use it though. I've placed it over a MonoBehaviour class but it doesn't seem to be picked up by the test runner.

    Also, it looks like it needs to run in conjunction with a scene. Tried that as well but still no go.

    Is there any option to drive an integration test strictly from code? (e.g: the test runner will create a new temp scene and i will create all objects in code and run the assertions on whatever i want).
     
  18. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @liortal You need to specify the name of the scene in the DynamicTestAttribute and the work on this scene. The dynamic tests will be picked up automatically.
    You still need that scene (even if it's empty) to kickstart the test run.
     
  19. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    What does this workflow save me? why should i use it instead of the older one (e.g: create a scene, add objects, etc).
     
  20. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    You can keep your tests (almost) fully code-based. For some people it might be easier to maintain (i.e. to have a file diff). It's not replacement (or a 'new' way) for the 'visual' way of creating tests, it's an alternative.
     
    liortal likes this.
  21. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    @Tomek Paszek I actually tried to use it but was unable to run the test...

    I have a MonoBehaviour that i marked with that attribute.
    For the tests i created a generic, empty scene (passed that name to the attribute). Since all objects will be created in code, i don't need anything in that scene, so i assume i can share the same (empty) scene between all my dynamic tests that are driven by code.

    This test was not picked up by the runner though... I could not see it in the list (only when that empty scene was the loaded), but at that point i was still unable to run the test.

    I wonder if i am not getting it, or if the way i attempted to use it was the correct way ....
     
  22. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    You need the TestRunner object to be on the scene. It's created along with the first integration tests you create from the runner (not code). Maybe you're missing that? I think such case is not clarified in the docs.
     
  23. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    I dont have any other integration test...so i guess i won't have the test runner object anywhere...
     
  24. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    So create one, and then remove it and then save the scene only with the TestRunner object. Not the best workflow but never had time to improve it :)
     
    liortal likes this.
  25. IndyNarf

    IndyNarf

    Joined:
    Sep 8, 2015
    Posts:
    1
    Same problem here with unity 5.2 on visual studio community 2015.

    Edit: I was able to fix the problem, unity 5.2 uses Mono.cecil 0.9.6 and unity test tools 0.9.4, to make it work replace Mono.cecil and Mono.cecil.Mdb dll in yourProject\Assets\UnityTestTools\UnitTesting\Editor\NUnit\Libs by the ones in C:\Program Files\Unity\Editor\Data\Managed
     
    Last edited: Sep 8, 2015
  26. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    I continued to develop one of my projects throughout the whole Unity 5.2 beta period and regularly updated. The last version I used was Unity 5.2.0f2 and I had no issues. Now with Unity 5.2.0f3, I am also getting the SequencePoint issue that was mentioned a few times. I don't understand why it only appeared in Unit 5.2.0f3 and not in previous versions.

    Case 725779
     
  27. KyleHatch85

    KyleHatch85

    Joined:
    Dec 13, 2011
    Posts:
    99
    Ditto to Dantus, You can comment out the function and the call to continue work.
     
  28. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
  29. boinged

    boinged

    Joined:
    May 8, 2013
    Posts:
    16
    There is a bug with version 1.5.6 in TestComponent.CreateTest on line 266. The use of Undo causes projects that contain this library to not build as Undo is unavailable outside the editor.

    Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/TestComponent.cs(266,25): error CS0103: The name `Undo' does not exist in the current context

    I was running this on Unity 4.6.8p3 but don't think that should matter. A workaround is to wrap the offending line with #IF UNITY_EDITOR
     
  30. nigelw

    nigelw

    Joined:
    Sep 29, 2015
    Posts:
    1
    Is there a way to run a single test from the command line on Android?

    This will run the whole suite:

    adb shell am start -a android.intent.action.MAIN -n com.unity.IntegrationTest/com.unity3d.player.UnityPlayerActivity

    But I need to debug a single test in the suite. Are there any parameters I can provide to this command to run a specific test? Or is there anything I can add to the com.unity.IntegrationTest.xml file to specify a single test run?

    Let me know if there is a more appropriate place to ask this question.
     
  31. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    Sorry if this was posted before: I just tested with the latest version of UTT from the asset store (1.5.6) on Unity 5.2.1:
    upload_2015-10-1_18-47-6.png

    I suppose the "Passing tests" should pass, right ?
     
  32. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Sorry for no replying for almost a month. I've been traveling and on vacations. I'll be looking at all the issues soon.
    @nigelw currently there is no way to run a single test on a platform
    @liortal that's right :)
     
  33. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    Can you repro this? I tried it on both macos and win10 on 5.2.1 and It Works For Me®
     
  34. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    Just tried again and i also can't repro this.
     
  35. Mikeysee

    Mikeysee

    Joined:
    Oct 14, 2013
    Posts:
    155
    Hi,

    Are there any plans to have Unit Tests able to run from within Visual Studio? It would be great if we didn't have to switch between the IDE and the Editor each time we want to run the tests..

    Mike
     
  36. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @Mikeysee yes, but the limitation are not on Unity side. One way of achieving this is by creating an NUnit extension. I tried to make it happen for the R# unit tests runner but it turned out that it's impossible because R# was using the plugin system differently than expected. I should try it again when the REST integration is ready.
    If any of you have any suggestion and ideas on how to approach it differently, I'm all ears.
     
  37. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    Tomek - the tests should still be executed within theeditor right? Only the results will be communicated back to VS.
     
  38. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @liortal yes, indeed. the runner would communicate with the editor, tell it to execute the tests and then get the results and render them in the runner of your choice. Still the execution would happen in the editor.
     
    liortal likes this.
  39. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    Can i use integration tests on "real" game scenes?

    For example - in case some of the game's logic cannot even be extracted into its own test scene since it's all a tangled mass of objects in the real scene (i didn't write that, of course !! haha).

    Can i define a dynamic integration test (from code), set its scene to the real scene from our game, and execute the tests in the context of that scene ? all objects added, etc at runtime shouldn't affect the scene since they are in play mode, right ?

    Has someone attempted this workflow before ?
     
  40. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
  41. Alex-Chouls

    Alex-Chouls

    Joined:
    Mar 24, 2009
    Posts:
    2,294
    I have some code based tests that ran fine in Unity 5.0 but are timing out in Unity 5.3.

    It seems to timeout when using this pattern:
    Code (CSharp):
    1.         void Start()
    2.         {
    3.             StartCoroutine(Test());
    4.         }
    5.  
    6.         IEnumerator Test()
    7.         {
    8.             yield return 0;
    9.             DoTest();
    10.         }
    DoTest uses IntegrationTest.Assert

    In Unity 5.3 (or maybe latest Test Tools since I had to import that for Unity 5.3 compatibility) this test times out, like the Assert is never registered...

    Is this a known issue?
     
  42. Alex-Chouls

    Alex-Chouls

    Joined:
    Mar 24, 2009
    Posts:
    2,294
    Actually it seems to be a more general problem with IntegrationTest.Assert.

    Maybe I'm using it wrong...?

    I was doing tests like this:

    Code (CSharp):
    1. public class Test_EaseActions : MonoBehaviour
    2. {
    3.     void FixedUpdate()
    4.     {
    5.         TestStartValues();
    6.         enabled = false;
    7.     }
    8.  
    9.     void TestStartValues()
    10.     {
    11.         var fsm = gameObject.GetComponent<PlayMakerFSM>().Fsm;
    12.  
    13.         var myFloat = fsm.GetFsmFloat("myFloat").Value;
    14.         IntegrationTest.Assert(gameObject, Math.Abs(myFloat - 1.23f) < Mathf.Epsilon, "myFloat = " + myFloat);
    15.  
    16.         var myRect = fsm.GetFsmRect("myRect").Value;
    17.         IntegrationTest.Assert(gameObject, myRect == new Rect(1f,2f,3f,4f), "myRect = " + myRect);
    18.  
    19.         var myColor = fsm.GetFsmColor("myColor").Value;
    20.         IntegrationTest.Assert(gameObject, myColor == Color.red, "myColor = " + myColor);
    21.  
    22.         var myVector3 = fsm.GetFsmVector3("myVector3").Value;
    23.         IntegrationTest.Assert(gameObject, myVector3 == new Vector3(1f,2f,3f), "myVector3 = " + myVector3);
    24.     }
    25. }
    This used to work. But now in Unity 5.3 it only seems to work if I setup Assertion Components with custom comparers.

    Setting up this test with AssertionComponents takes a lot longer than writing the above script.

    Is there a recommended way of doing tests with scripts instead of AssertionComponents?
     
    Last edited: Dec 29, 2015
  43. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @Alex Chouls if I understand correctly you are trying to use the option to succeed an integration tests when all assertions are checked and it doesn't work with IntegrationTest.Assert? This option would only work with the assertion component. if you want to assert from code you need IntegrationTest.Pass at the end of the test.
     
  44. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    mollermanden and Tomek-Paszek like this.
  45. ekt

    ekt

    Joined:
    Jul 9, 2012
    Posts:
    26
    @Tomek: just started playing with this and I think it is awesome. great job really!
    i hope the whole thing gets integrated in the ide at some point, not only the unit test one
     
  46. mollermanden

    mollermanden

    Joined:
    Apr 2, 2013
    Posts:
    23
  47. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    Just checked now, it's working fine ...
     
  48. mollermanden

    mollermanden

    Joined:
    Apr 2, 2013
    Posts:
    23
    Perfect thanks!
     
  49. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,201
    Hey all,

    Starting with Unity 5.3, there's a new feature called "multi scene editing" that allows having multiple scenes opened at the same time (http://docs.unity3d.com/Manual/MultiSceneEditing.html).

    While I haven't tested this feature myself (yet), i thought of new scenarios this may open in terms of automated testing, and was wondering whether these were considered (to be included in a future UTT release).

    Integration tests are currently defined in their own separate scene where test objects are placed and asserted against. This can be challenging for games that were not built with testing in mind (hard to separate prefabs and other game objects into a "clean scene").

    The MSE feature can assist this scenario by allowing to load a normal game scene together with a "test scene" that contains test objects or script code that will assert against the real objects from the game scene. This can make it much easier to create tests that are executed in the real context of the game, without having to define a new integration test scene.

    Is this something that is going to be implemented? anyone else thinks this can be a good idea? the test scene that will be loaded can also be dropped from the build, so it is essentially invisible to the actual game running on the device.

    What are your opinions on this? I thought of actually trying to implement it and submit as a pull request to the UTT repo, if there's demand of course.
     
  50. goldbug

    goldbug

    Joined:
    Oct 12, 2011
    Posts:
    679
    I just got the latest unity (5.3.3) and unity testing tools (1.5.7), and I have tons of warnings about obsolete methods, for example:

    Aside from triggering a bad case of OCD, these are really obnoxious since they make it hard to find the real warnings in my code.