Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

How to use filter for tests with truncated name?

Discussion in 'Testing & Automation' started by Xarbrough, Jan 31, 2020.

  1. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    752
    I'm following the excellent talk about testing from the last Unite event but now I'm stuck at using the Filter for the ExecutionSettings of the test runner in this case:

    Code (CSharp):
    1. private void StartTestRun(Scene scene)
    2. {
    3.     hideFlags = HideFlags.HideAndDontSave;
    4.     RunnerAPI.Execute(new ExecutionSettings
    5.     {
    6.         filters = new[]
    7.         {
    8.             new Filter
    9.             {
    10.                 groupNames = new []{$"SaveKeyTests\\(\"{scene.path}\"\\)\\."},
    11.                 //testNames = new []{"SaveKeyTests(\"Assets/MyScenes/Canyon_Cobra.unity\")" },
    12.                 testMode = TestMode.EditMode
    13.             }
    14.         }
    15.     });
    16. }
    I'm trying to replicate the demo from the talk in which a context menu in for each scene calls all tests for that scene. However, I can't get the groupNames to correctly match my tests:

    upload_2020-2-1_0-18-44.png

    I believe the issue may be that the test names are truncated because of their length, but maybe I'm not using the right query. I've tested the commented line with a shorter path and it worked.
     
  2. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    752
    I experimented some more, but can't make my desired case work. It works, if I truncate the test name with three dots hardcoded like so:

    Code (CSharp):
    1. testNames = new [] {
    2.     "MyGame.SaveKeyTests(\"Assets/Runtime/S/A/Canyon_Hyena_AndLo...\")",
    3. },
    But once the path gets even longer, my scene Canyon_Hyena_AndLongerName.unity gets cut-off completely. I realise that long names are an issue to display in the test runner window, but first, I believe it would be better to truncate the start of the string in the runner view, since file paths are more specific at the end. Second, there should be a way to retrieve the non-truncated name of the generated test name in code, maybe I haven't found it yet, but if it doesn't exist, that's my feature request. :)
     
  3. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,337
    I've got a feeling that the name is being truncated by NUnit itself, rather than by the Test Runner Window. Take a look at this, and have a play with the TestName property of the TestFixtureAttribute.
     
  4. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    752
    Thanks for the tip! Here's what I found after some experimentation of searching NUnit documentation:

    • The TestName property works on the TestCase attribute and let's users format parameterized test methods
    • For unknown reasons it does not have any effect when used on a simple TestFixture class (at least in the Unity test runner)
    • It seems that using TestFixtureSource is another issue, because it autogenerates the names and it looks as if this part cannot be formatted. Instead, NUnit calls ToString on the returned values from the source collection.
    Maybe there is a more elegant solution, but for now I managed to at least get my tests working by returning the scene names instead of the full scene path. In the test class I then search for the scene in the AssetDatabase to get the full path again. This may be brittle if multiple scenes have similar names and may still break if the folder hierarchy is too deep.
     
  5. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,337
    Have you tried providing TestFixtureData objects from your source? I think they might let you override the name.
     
  6. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    752
    Yea, that sounds like a good place to modify this, but it's not working. Setting the TestName via the TestFixtureData object simply doesn't do anything. I tried a couple of workarounds with custom data objects, but none of them let me change the name of the test and also let my custom data be passed to the tests.

    I also found some sources about NUnit not really supporting the TestName property in TestFixtureData. I believe the NUnit community is discussing a couple of issue related to truncated test names when working with TestFixtures, but so far I'm not seeing a real solution.
     
unityunity