Search Unity

Feature Request Ability to extend test runner window

Discussion in 'Testing & Automation' started by Xarbrough, Oct 9, 2020.

  1. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    I enjoy using the test runner window and test runner API, however, often I wished I could use these tools as a framework to add my own things.

    For example:

    Add entries to the context menu when clicking on a test in the runner window. Currently, it offers "Open error line", "Open source code" and "Run". I'd like to add things like "Select Asset" or "Open Asset" for specific tests that operate on prefabs or scenes. I'm using these tests for content validation and it would make sense to quickly navigate to a broken prefab for me.

    Or:

    Customize the way parameterized test names are generated. I use them extensively for content validation by providing a TestCaseSource that iterates over a set of assets, for example. Ideally, I would be able to return the asset GUID from the source as well as a test name or somehow customize the way the test names are generated. In my example, my TestCaseSource returns either an asset GUID or a path, both of which make unreadable and long test names. I'd probably simply want the name of the asset instead, however, the name alone may not be enough to reconstruct the actual path of the asset within the test. If this is all too specific, I'd still like to improve the test case name generation on a fundamental level, by e.g. stripping character from the middle of the name to shorten them. This would help with long file paths. Currently, if the test name is a file path and relatively long, it will cut off the end, which makes it impossible to differentiate between the two names in the window.

    And:

    Add custom action buttons or checkbox options to the header. This would make for a nice place for additional settings like whether the window should run a set of tests automatically before entering play mode.

    If it's too difficult to identify extension points that are valuable to a broad audience, maybe another approach would be to make more of the individual drawing parts public. I'd also be happy to write my own test runner window, if I can build on some of the blocks from the original window (e.g. draw the main part, but implement my own header).
     
    andrew-fray likes this.
  2. andrew-fray

    andrew-fray

    Joined:
    Jul 19, 2012
    Posts:
    157
    (for your customisation of parameterised test names, I ended up wrapping appropriate assets in a struct, and passing _that_ into TestFixtureSource. that way, I could override the ToString() of the struct and get a decent test name)
     
    Xarbrough likes this.
  3. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    I keep forgetting this works, so thanks for reminding me again! :D
     
  4. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    I have a new thing that bugs me about the otherwise awesome Test Runner window: It shows a lot of placeholder tests (somebody isn't doing their homework, Unity). Packages are internally required to have tests apparently, so Unity developers decide they'll put some placeholder tests into their packages to meet the requirement without adding any value. Kind of annoying that this now clutters my test runner view and probably costs time running all tests. So, I'd like a way to exclude certain tests.

    Currently, it's possible to filter tests by their category, but that only helps if I can ensure that developers on my team tag all test methods with our own say company name category. This way I could filter all tests that don't belong to our company. But that's not the optimal solution, since I do want to run reasonable tests from imported packages.

    Would it be possible to add a black-list to the test runner, to specifically exclude a list of tests following a glob or regex pattern? E.g. Exclude everything "Unity.2D.Psdimporter.*"