Search Unity

Unity Unity Test Framework documentation

Discussion in 'Testing & Automation' started by Warnecke, Jul 16, 2019.

  1. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    72
    With Unity 2019.2, the Unity Test Framework (aka Unity Test Runner) will be a package. As a part of it, we’ve given the documentation a very much needed overhaul.

    The latest version of the documentation is available here: https://docs.unity3d.com/Packages/com.unity.test-framework@latest?

    Any feedback and questions to the documentation are welcome. We will be keeping the documentation up to date as we release new versions of the package.
     
    Last edited: Oct 3, 2019
    liortal and Stormy102 like this.
  2. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    193
    Beautiful; a hell of an improvement over the current documentation.

    One request; can you describe the order in which external actions within the same category run with respect to each other? E.g. in which order will implementations of
    OuterUnityTestAction
    run with respect to one another? Or implementations of
    IWrapSetUpTearDown
    ?

    EDIT: And maybe while you're at it, why would I want to use certain stages over others? Like, why would I want to use
    OuterUnityTestAction
    over
    IWrapSetUpTearDown
    /
    IWrapTestMethod
    , or vice versa?
     
  3. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    72
    Thanks.
    In regards to the order of each set of actions (e.g. OuterUnityTestAction), then they are run in the same order as other nunit attribute, which again is the order that System.Reflection provide the attributes. It is however not documented.
    In my experience, then the order seems to be first it takes the attributes in the order they are written, staring with the base most class, working upwards.

    Regarding OuterUnityTestAction vs IWrapSetUpTearDown/IWrapTestMethod, then they are a lot alike, but the need for the OuterUnityTestAction arose for performance measurements, which would preferably be at the outer most scope. Hence this can be used as the outer most action.
     
  4. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,506
    Thanks for updating the docs. here is some feedback from me (BEWARE!!! nitpicking ahead! also this is going to be very long).

    https://docs.unity3d.com/Packages/c...@1.1/manual/edit-mode-vs-play-mode-tests.html

    I am not using .asmdef but still, my editor code (which contains my editor tests) reference my "runtime" scripts. This phrasing (to me at least) made it sound as if you can only access editor code in these tests.

    We have a lot of code that is not MonoBehaviour based but still used at runtime in our game. This makes it perfectly suitable to be run as an edit mode test, so it's not really only for editor extensions.

    Not sure why this would be needed, but sounds like a bit of a hack isn't it? creating an edit mode test only to let it go into play mode.

    The page later mentions how to set up playmode vs. edit mode tests.
    For edit mode:
    And for play mode:
    Should probably describe both cases in a similar way to avoid any confusion.

    This section is not clear to me (sorry!)

    https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/workflow-create-test.html

    This is not a category... NUnit allows you to tag tests using the Category attribute. What's shown on the documentation page is the actual class name containing different tests. using this attribute has no effect for me.
    The categories are shown on the top right of the window:
    upload_2019-8-10_21-12-54.png

    https://docs.unity3d.com/Packages/c...al/workflow-run-playmode-test-standalone.html
    "Running as standalone" is a bit confusing, since it sounds like it's only possible to run on a standalone (e.g: Desktop) build.
    The UI itself calls it "Run in player" so i think the docs should be aligned with that.

    TestPlayerBuildModifier
    (not docs related, sorry..) First of all, why do we need this attribute? In order to modify the BuildPlayerOptions we already had this option available for a while.

    Also - this method of setting an attribute to point to a given type is a bit cumbersome. why is it designed that way? if my implementation needs to implement a certain interface, i suppose you can find that out via reflection and instantiate it.

    "Modify the Player build options for Play Mode tests" -- why is this needed? why does the location of the build player matters when executing PlayMode tests?

    Split build and run for Play Mode tests
    The code example is not very clear:
    1. Why do i need to modify the build location?
    2. A bit of a hacky path manipulation going on there
    3. Why are we exiting the app if run from the command line? isn't this provided by the -quit argument ?


    These are my comments so far. again, thank you for creating such a detailed guide with examples and API references. great step indeed :)
     
  5. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    72
    Hey. Thanks for your thorough feedback, it is much appreciated. :)
    I have applied changes as part of it and it should be part of the next version of the package.

    Regarding the split build and run:
    - We cannot use RegisterGetBuildPlayerOptionsHandler, as the build settings the test runner uses does not draw from that.
    - Regarding -quit, then it is not enough to specify that, as it would leave the test framework sitting waiting for the test results and first quit after receiving those. That is why we explicitly need to shut it down.
     
  6. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    18
    Hi, I try to start the Test runs with a script like described here:
    https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/extension-run-tests.html
    Code (CSharp):
    1. using UnityEditor.TestTools.TestRunner.Api;
    2. using UnityEngine;
    3.  
    4. namespace Editor
    5. {
    6.     public static class IntegrationTests
    7.     {
    8.         public static void StartTests()
    9.         {
    10.             Debug.Log("Game4Automation Tests started");
    11.            
    12.             var testRunnerApi = ScriptableObject.CreateInstance<TestRunnerApi>();
    But I am getting told that the Internal Class TestRunnerApi cannot be accessed. How could I start the tests with code=

    Best regards
    Thomas
     
  7. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    This means you're using an older version of the com.unity.test-framework package. You need to upgrade to a newer version through the Package Manager.
     
  8. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    18
    Hmmm....

    I am using Unity 2018.4.8f1 and the Test-Framkework in the Package manager seems also to be updated (package manager just gives me 0.0.4)

    Any advice?

    Best regards
    Thomas
     
  9. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,486
    Does your manifest.json file have your project using a custom registry, by any chance?
     
  10. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    18
    This is my manifest.json (before installing Test-Framework):
    Code (JavaScript):
    1. {
    2.   "dependencies": {
    3.     "com.unity.ads": "2.0.8",
    4.     "com.unity.analytics": "3.2.2",
    5.     "com.unity.collab-proxy": "1.2.15",
    6.     "com.unity.package-manager-ui": "2.0.7",
    7.     "com.unity.purchasing": "2.0.3",
    8.     "com.unity.textmeshpro": "1.4.1",
    9.     "com.unity.modules.ai": "1.0.0",
    10.     "com.unity.modules.animation": "1.0.0",
    11.     "com.unity.modules.assetbundle": "1.0.0",
    12.     "com.unity.modules.audio": "1.0.0",
    13.     "com.unity.modules.cloth": "1.0.0",
    14.     "com.unity.modules.director": "1.0.0",
    15.     "com.unity.modules.imageconversion": "1.0.0",
    16.     "com.unity.modules.imgui": "1.0.0",
    17.     "com.unity.modules.jsonserialize": "1.0.0",
    18.     "com.unity.modules.particlesystem": "1.0.0",
    19.     "com.unity.modules.physics": "1.0.0",
    20.     "com.unity.modules.physics2d": "1.0.0",
    21.     "com.unity.modules.screencapture": "1.0.0",
    22.     "com.unity.modules.terrain": "1.0.0",
    23.     "com.unity.modules.terrainphysics": "1.0.0",
    24.     "com.unity.modules.tilemap": "1.0.0",
    25.     "com.unity.modules.ui": "1.0.0",
    26.     "com.unity.modules.uielements": "1.0.0",
    27.     "com.unity.modules.umbra": "1.0.0",
    28.     "com.unity.modules.unityanalytics": "1.0.0",
    29.     "com.unity.modules.unitywebrequest": "1.0.0",
    30.     "com.unity.modules.unitywebrequestassetbundle": "1.0.0",
    31.     "com.unity.modules.unitywebrequestaudio": "1.0.0",
    32.     "com.unity.modules.unitywebrequesttexture": "1.0.0",
    33.     "com.unity.modules.unitywebrequestwww": "1.0.0",
    34.     "com.unity.modules.vehicles": "1.0.0",
    35.     "com.unity.modules.video": "1.0.0",
    36.     "com.unity.modules.vr": "1.0.0",
    37.     "com.unity.modules.wind": "1.0.0",
    38.     "com.unity.modules.xr": "1.0.0"
    39.   }
    40. }
     
  11. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    18
    Hi, I checked again yesterday. It seems that even in a new empty project with Unity 2018.. there is only 0.0.4 available.
    With 2019... I was able to download newer versions (1.0,1.1....).
     
  12. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    72
    Hey. The unity test framework package is only available with 2019.2 and forward. The 0.0.4 version you see is visible by mistake. It was an intimidate version we had when moved from the build in module, into being a package, so it will not work. In 2018 the test framework is build into unity and you do not need a package. When you later upgrade to 2019.2 or newer, then the test framework will be automatically added.
     
  13. in2sight

    in2sight

    Joined:
    Feb 24, 2018
    Posts:
    18
    Ah I see. So the API for the Test framework seems to be only available from 2019.2 upwards. Sorry - documentation wasn't clear about that. Thanks
     
  14. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,089
    The API and the documentation is coming along nicely!

    I've looked up the TestStatus Enum and didn't find any recommendations in the Test Framework scripting API about when to use Inconclusive or Skipped as a result. I did find information on the NUnit pages, but maybe we can add examples to the Unity docs for our specific context. For example, I found it helpful to know that it is possible to run a suite of tests to validate a Unity scene and then return Skipped/Ignored for tests which do not apply in very specific circumstances (e.g. no jump platforms present in the scene). Next, when loading a resource from the AssetDatabase that's no longer at that location or when anything goes wrong that breaks the test but does not indicate an issue with the scene or entity to be verified, we can return Inconclusive (like a warning to the user, but no error).

    Additionally, should we be more consistent with the naming between
    Assert.Ignore
    and
    TestStatus.Skipped
    ? Or are these different because of NUnit reasons?
     
  15. mbatutis

    mbatutis

    Joined:
    Jan 3, 2017
    Posts:
    4
  16. waldgeist

    waldgeist

    Joined:
    May 6, 2017
    Posts:
    221
    Hey, I just started using the Unity Testing Framework. I understood that all tested code has to be placed in its own Assembly for making the tests to work. You can't just test code that "lives" in the default assembly, correct?

    However, in a "real world" project that uses a lot of external assets and libraries, this turned into a real nightmare, because as soon as my own code is placed in an assembly, it can't access these 3rd party libraries anymore, so I have to define even more assemblies for them. Which sometimes doesn't seem to be possible at all. For instance, the Facebook SDK doesn't use an assembly by default, but as soon as I place it in one, it won't be able to access code in its own plugins, although the docs state that all plugin libraries are auto-referenced by assembly defs.

    Thus, every time I add a new assembly def, I get masses of (new) compile errors. And this continues like forever. Tbh, I am completely lost now. I am wondering why there isn't an easier way to define tests that can directly access all scripts that are already available in the project?

    I totally get that assemblies allow to exclude these tests from a build, but couldn't it just be done in a similar way that editor folder scripts are excluded from the build, without all that "assembly hell"? I am about to give up now, and I would consider myself as a pretty experienced developer...

    Also, the docs only describe basic Unit testing. But how to test a Monobehavior in the context of its scene where it belongs to, i.e. all the other Monobehaviors that are placed on the same game object? I mean actual UX tests, not just isolated units.
     
    Last edited: Jun 8, 2021
  17. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,089
    You can have tests mix with your regular code without issues, I just tested this although I wouldn't recommend it. The downside would be that you would also be shipping your test code with the final build.


    Did you keep in mind that you can reference DLLs directly without having to create ASMDEF? But other than that, yea 3rd party is a nightmare, but that's not much worse in Unity than it is in other environments. The issue with many Unity plugins is that they haven't upgraded to ASMDEF although they should. So, as a developer, you have to fix this by hand and it sucks, but I do it all the time. It's not that bad if you just do it and then look back and say "well ok that was an hour of work, but now the entire project works again, then next plugin we add will only take 10 minutes to setup".

    Don't! I was at that point with assembly hell, but then just went through it and know it doesn't bother me anymore. In every project I set up three main ASMDEF: Editor, Runtime and Tests. Editor references Runtime. Tests references Runtime. Then I add 3rd party plugins into a non-magic "Plugins" folder with a Plugins ASMDEF. Runtime references the Plugins ASMDEF. This covers 90% of all plugins/cases. Basically, they are all bundles up into one assembly, unless they define their own ASMDEF or have editor code. If they already have ASMDEF, well just reference the ones you need (if this gets out of hand, think about using less 3rd party code, it might bite you in many other ways). If the plugins haven't set up ASMDEF, but use the magig "Editor" folder, bite the bullet and create the ASMDEFs for the plugin. Usually, this just means creating two. And remember for everything else, if they use DLL you can simply reference this in your runtime ASMDEF via the dropdown. And finally, the separate test assembly is a proven idea. You simply reference runtime, because everything else should not be tested. Don't write tests that include the Facebook SDK, they ought to test their stuff on their own, etc.
     

    Attached Files:

    Last edited: Jun 9, 2021
    scarofsky likes this.
unityunity