Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Unity Unity Test Framework documentation

Discussion in 'Testing & Automation' started by Warnecke, Jul 16, 2019.

  1. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    40
    With Unity 2019.2, the Unity Test Framework (aka Unity Test Runner) will be a package. As a part of it, we’ve given the documentation a very much needed overhaul.

    The latest version of the documentation is available here: https://docs.unity3d.com/Packages/com.unity.test-framework@latest?

    Any feedback and questions to the documentation are welcome. We will be keeping the documentation up to date as we release new versions of the package.
     
    Last edited: Oct 3, 2019
    liortal and Stormy102 like this.
  2. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    11
    Beautiful; a hell of an improvement over the current documentation.

    One request; can you describe the order in which external actions within the same category run with respect to each other? E.g. in which order will implementations of
    OuterUnityTestAction
    run with respect to one another? Or implementations of
    IWrapSetUpTearDown
    ?

    EDIT: And maybe while you're at it, why would I want to use certain stages over others? Like, why would I want to use
    OuterUnityTestAction
    over
    IWrapSetUpTearDown
    /
    IWrapTestMethod
    , or vice versa?
     
  3. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    40
    Thanks.
    In regards to the order of each set of actions (e.g. OuterUnityTestAction), then they are run in the same order as other nunit attribute, which again is the order that System.Reflection provide the attributes. It is however not documented.
    In my experience, then the order seems to be first it takes the attributes in the order they are written, staring with the base most class, working upwards.

    Regarding OuterUnityTestAction vs IWrapSetUpTearDown/IWrapTestMethod, then they are a lot alike, but the need for the OuterUnityTestAction arose for performance measurements, which would preferably be at the outer most scope. Hence this can be used as the outer most action.
     
  4. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,257
    Thanks for updating the docs. here is some feedback from me (BEWARE!!! nitpicking ahead! also this is going to be very long).

    https://docs.unity3d.com/Packages/c...@1.1/manual/edit-mode-vs-play-mode-tests.html

    I am not using .asmdef but still, my editor code (which contains my editor tests) reference my "runtime" scripts. This phrasing (to me at least) made it sound as if you can only access editor code in these tests.

    We have a lot of code that is not MonoBehaviour based but still used at runtime in our game. This makes it perfectly suitable to be run as an edit mode test, so it's not really only for editor extensions.

    Not sure why this would be needed, but sounds like a bit of a hack isn't it? creating an edit mode test only to let it go into play mode.

    The page later mentions how to set up playmode vs. edit mode tests.
    For edit mode:
    And for play mode:
    Should probably describe both cases in a similar way to avoid any confusion.

    This section is not clear to me (sorry!)

    https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/workflow-create-test.html

    This is not a category... NUnit allows you to tag tests using the Category attribute. What's shown on the documentation page is the actual class name containing different tests. using this attribute has no effect for me.
    The categories are shown on the top right of the window:
    upload_2019-8-10_21-12-54.png

    https://docs.unity3d.com/Packages/c...al/workflow-run-playmode-test-standalone.html
    "Running as standalone" is a bit confusing, since it sounds like it's only possible to run on a standalone (e.g: Desktop) build.
    The UI itself calls it "Run in player" so i think the docs should be aligned with that.

    TestPlayerBuildModifier
    (not docs related, sorry..) First of all, why do we need this attribute? In order to modify the BuildPlayerOptions we already had this option available for a while.

    Also - this method of setting an attribute to point to a given type is a bit cumbersome. why is it designed that way? if my implementation needs to implement a certain interface, i suppose you can find that out via reflection and instantiate it.

    "Modify the Player build options for Play Mode tests" -- why is this needed? why does the location of the build player matters when executing PlayMode tests?

    Split build and run for Play Mode tests
    The code example is not very clear:
    1. Why do i need to modify the build location?
    2. A bit of a hacky path manipulation going on there
    3. Why are we exiting the app if run from the command line? isn't this provided by the -quit argument ?


    These are my comments so far. again, thank you for creating such a detailed guide with examples and API references. great step indeed :)
     
  5. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    40
    Hey. Thanks for your thorough feedback, it is much appreciated. :)
    I have applied changes as part of it and it should be part of the next version of the package.

    Regarding the split build and run:
    - We cannot use RegisterGetBuildPlayerOptionsHandler, as the build settings the test runner uses does not draw from that.
    - Regarding -quit, then it is not enough to specify that, as it would leave the test framework sitting waiting for the test results and first quit after receiving those. That is why we explicitly need to shut it down.
     
  6. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    15
    Hi, I try to start the Test runs with a script like described here:
    https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/extension-run-tests.html
    Code (CSharp):
    1. using UnityEditor.TestTools.TestRunner.Api;
    2. using UnityEngine;
    3.  
    4. namespace Editor
    5. {
    6.     public static class IntegrationTests
    7.     {
    8.         public static void StartTests()
    9.         {
    10.             Debug.Log("Game4Automation Tests started");
    11.            
    12.             var testRunnerApi = ScriptableObject.CreateInstance<TestRunnerApi>();
    But I am getting told that the Internal Class TestRunnerApi cannot be accessed. How could I start the tests with code=

    Best regards
    Thomas
     
  7. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,194
    This means you're using an older version of the com.unity.test-framework package. You need to upgrade to a newer version through the Package Manager.
     
  8. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    15
    Hmmm....

    I am using Unity 2018.4.8f1 and the Test-Framkework in the Package manager seems also to be updated (package manager just gives me 0.0.4)

    Any advice?

    Best regards
    Thomas
     
  9. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,194
    Does your manifest.json file have your project using a custom registry, by any chance?
     
  10. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    15
    This is my manifest.json (before installing Test-Framework):
    Code (JavaScript):
    1. {
    2.   "dependencies": {
    3.     "com.unity.ads": "2.0.8",
    4.     "com.unity.analytics": "3.2.2",
    5.     "com.unity.collab-proxy": "1.2.15",
    6.     "com.unity.package-manager-ui": "2.0.7",
    7.     "com.unity.purchasing": "2.0.3",
    8.     "com.unity.textmeshpro": "1.4.1",
    9.     "com.unity.modules.ai": "1.0.0",
    10.     "com.unity.modules.animation": "1.0.0",
    11.     "com.unity.modules.assetbundle": "1.0.0",
    12.     "com.unity.modules.audio": "1.0.0",
    13.     "com.unity.modules.cloth": "1.0.0",
    14.     "com.unity.modules.director": "1.0.0",
    15.     "com.unity.modules.imageconversion": "1.0.0",
    16.     "com.unity.modules.imgui": "1.0.0",
    17.     "com.unity.modules.jsonserialize": "1.0.0",
    18.     "com.unity.modules.particlesystem": "1.0.0",
    19.     "com.unity.modules.physics": "1.0.0",
    20.     "com.unity.modules.physics2d": "1.0.0",
    21.     "com.unity.modules.screencapture": "1.0.0",
    22.     "com.unity.modules.terrain": "1.0.0",
    23.     "com.unity.modules.terrainphysics": "1.0.0",
    24.     "com.unity.modules.tilemap": "1.0.0",
    25.     "com.unity.modules.ui": "1.0.0",
    26.     "com.unity.modules.uielements": "1.0.0",
    27.     "com.unity.modules.umbra": "1.0.0",
    28.     "com.unity.modules.unityanalytics": "1.0.0",
    29.     "com.unity.modules.unitywebrequest": "1.0.0",
    30.     "com.unity.modules.unitywebrequestassetbundle": "1.0.0",
    31.     "com.unity.modules.unitywebrequestaudio": "1.0.0",
    32.     "com.unity.modules.unitywebrequesttexture": "1.0.0",
    33.     "com.unity.modules.unitywebrequestwww": "1.0.0",
    34.     "com.unity.modules.vehicles": "1.0.0",
    35.     "com.unity.modules.video": "1.0.0",
    36.     "com.unity.modules.vr": "1.0.0",
    37.     "com.unity.modules.wind": "1.0.0",
    38.     "com.unity.modules.xr": "1.0.0"
    39.   }
    40. }
     
  11. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    15
    Hi, I checked again yesterday. It seems that even in a new empty project with Unity 2018.. there is only 0.0.4 available.
    With 2019... I was able to download newer versions (1.0,1.1....).
     
  12. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    40
    Hey. The unity test framework package is only available with 2019.2 and forward. The 0.0.4 version you see is visible by mistake. It was an intimidate version we had when moved from the build in module, into being a package, so it will not work. In 2018 the test framework is build into unity and you do not need a package. When you later upgrade to 2019.2 or newer, then the test framework will be automatically added.
     
  13. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    15
    Ah I see. So the API for the Test framework seems to be only available from 2019.2 upwards. Sorry - documentation wasn't clear about that. Thanks