Search Unity

Resolved Ability to specify custom ChildTests / create hierarchies of tests programmatically

Discussion in 'Testing & Automation' started by fherbst, Sep 12, 2019.

  1. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    While trying to get an adaptor for our Asset Testing pipeline (not using TestRunner currently) to work, I ran into a couple limitations:
    • there seems to be a hierarchy of ChildTests visible in TestRunner window, but is there a way to create such hierarchies from script and display them there?
    • while [TestCaseSource] with a getter property works, it's evaluated at compile time; I'm trying to test Assets in the Project which means that either
      • I need to recompile the tests every time someone changes anything in the project (not good)
      • I need a way to generate the ChildTests hierarchy before running the tests
      • I need a way to re-evaluate the [TestCaseSource] when running the Tests not at compile time.
    Ideally, I'd want to have a single script (test) that acesses the AssetDatabase when clicking "Run Tests" and builds me a hierarchy of little red and green things :)

    (we currently have a custom window that does that but I'd like to have an adaptor to TestRunner)

    Any ideas?

    To illustrate:

    upload_2019-9-12_12-13-22.png

    1. this list is only updated on script recompile
    2. I'd want "Cube" and "Rattail Fish" to be parent Tests of their respective Asset Tests
    Edit: This behaviour (only updating the TestCaseSource when recompiling) actually breaks Unit Testing; running tests after doing some modifications yields broken & wrong inbetween results:
    upload_2019-9-12_12-18-29.png

    Note that while some tests fail, some succeed, and some don't run at all, the "overall result" (top pixelated line) is "Passed"!
     
    Last edited: Sep 12, 2019
    fherbst, Sep 12, 2019
    #1

  2. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    92
    It does not seem like it is possible with the current version of nunit. The closest thing I could find was https://nunit.org/docs/2.5/suiteBuilders.html , which was never fully realized.

    Updating nunit to a newer version is on our radar, but I can not say anything about when it will be done currently.
     
    Warnecke, Sep 16, 2019
    #2
    fherbst likes this.

  3. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    1,053
    I'm looking forward to this!!
     
    bdovaz, Sep 16, 2019
    #3

  4. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    @Warnecke Thanks for verifying this. I think your answer only applies to "hierarchy" part of my question though.

    What about making sure the TestCaseSource is queried when actually running the tests compared to at compile time? This actually feels like a bug...
     
    fherbst, Sep 16, 2019
    #4

  5. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    92
    TestCaseSource is evaluated when we build the test tree, which is on two occasions: 1) When we are about to run a set of tests. 2) When we are retrieving the list of tests to display in the ui. If the result of the TestCaseSource differs between those two cases, then the ui will not update correctly. The same happens if you put e.g. values from random into test cases. So far I can see that is the same way as it works if you run the tests in e.g. Visual Studio, Resharper or Rider.
     
    Warnecke, Sep 17, 2019
    #5

  6. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    Hm, I can confirm that the above is not true -
    TestCaseSource is only evaluated at compile time of the tests currently. Should I file a bug report?

    (As for the usecase: I'm testing Assets. This means that somebody can change something in the AssetDatabase, and then I want to press on "Run Tests" and obviously want those changes to be tested against. This is not something that you usually have in Visual Studio or Resharper (a database of things you want to test against).)
     
    fherbst, Sep 17, 2019
    #6

  7. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    92
    I have tried to set up a test that verifies materials based on test case source:
    Code (CSharp):
    1. public class MaterialsTests
    2. {
    3.  
    4.     [Category("Materials")]
    5.     [Test, TestCaseSource(nameof(Cases))]
    6.     public void VerifyMaterial(Material material)
    7.     {
    8.         Debug.Log($"Verifying {material.name}");
    9.     }
    10.  
    11.     public static IEnumerable Cases()
    12.     {
    13.         return AssetDatabase.FindAssets("t:material", new[] {@"Assets\Temp\Tests\Materials"}).Select(guid =>
    14.             AssetDatabase.LoadAssetAtPath<Material>(AssetDatabase.GUIDToAssetPath(guid)));
    15.     }
    16. }
    It seems that the TestCaseSource gets evaluated on every test run, but it is the ui that does not update without a domain reload. I added updating of the ui whenever the test tree is rebuild (whenever any test is run).

    In this case it means that after I add a new material and run any test, the new test case will show up.



    The fix should become available in the next package version.
     
    Warnecke, Sep 18, 2019
    #7
    fherbst likes this.

  8. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    Sounds good. That was a quick fix, thanks for being so responsive!
     
    fherbst, Sep 18, 2019
    #8

  9. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    92
    You are welcome :)
     
    Warnecke, Sep 18, 2019
    #9

  10. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    Hey, I see the new package is out, does it contain the fix?

    Note: changelog has 2019-11-19 as date ;)
    upload_2019-9-20_14-22-38.png
     
    fherbst, Sep 20, 2019
    #10

  11. Pilus

    Pilus

    Joined:
    Aug 11, 2017
    Posts:
    1
    Yeah. Something went wrong with the date. But this version is a few days old. The fix will be in version 1.1.3
     
    Pilus, Sep 20, 2019
    #11
    fherbst likes this.