Search Unity

Resolved Running Only A Subset Of Tests Through Command Line

Discussion in 'Testing & Automation' started by Aceria_, Apr 6, 2019.

  1. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    64
    I was wondering if it's possible to run a specific subset of tests when running them through the command line. I'd like to run most of our tests all the time, but some of them aren't that critical and I'd rather have them run overnight instead of slowing down our day-to-day work.

    We run it all through Jenkins on a build server, so I can't just select them manually in the test runner in the editor.
     
  2. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,232
    The test runner supports a "test filter" option - this is pretty much undocumented (or underdocumented?)
    This allows you to run tests that match a certain name using a filter, so it may not be exactly what u need, but you can rename your tests in such a way that will allow you to run a subset.

    A more useful feature would've been to allow us to specify a category for executing tests via command line. As an old habit, i tag my test fixtures with categories (e.g: integration, slow, fast, etc).

    If i could, i would run a specific category (or a set of categories?) when running from the command line.
    Another option would've been to ignore certain categories (e.g: slow).

    But all of these are not available right now from what i know. the test filter is your best bet right now.
     
    Aceria_ likes this.
  3. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    34
    Hey. The testFilter is indeed available in order to filter test by names. It is however also possible to filter test by categories using the testCategory argument. Both are things that will be documented correctly in the documentation overhaul we are working on.

     
    Aceria_ and liortal like this.
  4. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    64
    The testCategory is exactly what I needed. I think in combination with testFilter that covers all of the use cases I can think of.

    Thanks for the answer liortal & Warnecke!
     
  5. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,232
    @Warnecke Is there any way to ignore tests or categories?
    For example - we have a lot of tests, but some are really slow. i would like our CI builds to run all tests (except those marked with category 'slow' or something similar).

    On our nightly build we will execute all tests, but for the ones that run many times a day, i would like to avoid running those.
     
  6. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    34
    Currently we do not support exclusive filters, but it should be possible for you to filter out some categories by explicitly specify the categories to include. You can give it "Uncategorized" to pick up tests without categories and then specify the categories to include. I believe that you can provide a regex like "^(?!SlowTest).*" (which would match any category that does not start with SlowTest). Note that it is important to pass both "Uncategorized" and the regex as tests without categories does not go trough the regex, but are just matched against the "Uncategorized" string.