Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Why are the tests of the Performance Testing package included in my project?

Discussion in 'Testing & Automation' started by DrummerB, Nov 16, 2020.

  1. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    135
    When I install the Performance Testing package, its own edit and play mode tests are included in my project. Is this on purpose? Should this not be internal tests that you do on your side that are not released as part of the package?
     
  2. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    53
    I believe this is fundamentally due to the fact that the package manager doesn't include assemblies with the UNITY_INCLUDE_TESTS define constraint, unless the package type is "tests". And if you specify this category, then all assemblies are included. So the only way to ship assemblies to be used only in tests, is to ship all test assemblies in the package.

    I ran into this with my testing utility, and had to use a non-standard layout, by moving the package tests out of the Packages directory to work around this issue. Would be nice if the package manager would support shipping testing utilities without their tests.

    Not sure if this is intentional or not (in general, or with the Performance Testing package).
     
  3. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    135
    Interesting insight. Would this not be something that could be solved as part of the CI setup of the package project? For example, when building a release of the package, the build script could remove the unneeded test assets.
     
  4. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,649
    Actually, it's intentional. One of the important use cases for packages is "I want to modify one of Unity's built-in packages," and so we ship the tests in order to help you make sure that the modifications you make haven't broken anything. They're also, sometimes, a useful reference for 'how do I use this API' - you can look at how the API is exercised in the tests for some insight.

    That said, if you aren't modifying a package, we don't want the tests to be in your way - they should not show up in the Test Runner window when the package has not been pulled into your project's Packages folder, for example. If you're seeing otherwise, that's a bug, please report it.
     
  5. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,294
    I'm getting tests from the Input system. I know why I get them, but it's kinda annoying.

    The input system ships with some testing features (ie. InputTestFixture) which makes it easier to write play mode tests that uses the input system. The problem is that in order to include those features, I have to add this to the manifest:

    Code (csharp):
    1.   "testables": [
    2.     "com.unity.inputsystem"
    3.   ]
    Which makes this happen:

    upload_2020-12-9_14-52-4.png

    So if I want the testing features, I also need to include the input system's integration tests. Is that the Input system distributing their test features in a silly way, or is it a corner case that's not been considered?

    I'm guessing that this is similar to what's happening with OP, can't be sure, though.
     
  6. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,649
    Discussing internally, looks like the latter.
     
    Baste likes this.
  7. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    53
    Putting a package in "testables" indeed has the same effect of specifying the package type as "tests".

    But in any case, I'd like to reiterate that it would be nice if the package manager would support shipping testing utilities without their tests. In addition to pure testing packages, I think this is an issue for all framework-like packages, which benefit from having testing utilities (I know at least Zenject has this issue).

    In practice, I believe it would require some kind of mechanism of differentiating between test assemblies and test utility assemblies. Currently, I believe UNITY_INCLUDE_TESTS is used to determine the assembly type, but that is not granular enough for this purpose. This data could live either in the assembly definition, or in the package manifest (thinking the latter would be better).
     
  8. sbergen

    sbergen

    Joined:
    Jan 12, 2015
    Posts:
    53
    I ran into this issue again, when trying to organise some of our internal packages better. Has the issue been considered more internally, or are we still stuck with either including package tests in consuming projects, or separating test utilities into separate packages?
     
  9. Stephen-Hodgson

    Stephen-Hodgson

    Joined:
    Jul 15, 2014
    Posts:
    35
    I'm also running into problems referencing external package test assemblies in my test assembly in packages I'm building.

    I wrote some test utilities I wanted to use in all my packages but am unable to do so.

    I added `testables` and asmdef references, but no dice.