Search Unity

Resolved Generating NUnit-compatible XML output

Discussion in 'Testing & Automation' started by frank-ijsfontein, Oct 31, 2019.

  1. frank-ijsfontein

    frank-ijsfontein

    Joined:
    Sep 11, 2015
    Posts:
    16
    I'm trying to use the new test framework package (v1.1.5) to see how it integrates with our Jenkins setup. We used to run tests with the -runEditorTests parameter from the command line, and saving the resulting xml for the nunit Jenkins plugin to parse and report test status. Now, when I use the new test framework and run tests from code in our build script, the resulting xml seems incompatible.

    This is the code I use to save the XML:

    Code (CSharp):
    1.  
    2. public void RunFinished(ITestResultAdaptor result)
    3.             {
    4.                 Debug.Log($"{this}: run finished: passed {result.PassCount} tests");
    5.                 NUnit.Framework.Interfaces.TNode xml = result.ToXml();
    6.                 Debug.Log($"{ xml.OuterXml }");
    7.                 XmlWriter writer = XmlWriter.Create("../../TestResults.xml");
    8.                 xml.WriteTo(writer);
    9.                 writer.Flush();
    10.                 test.UnregisterCallbacks(this);
    11.                 if(Application.isBatchMode)
    12.                 {
    13.                     EditorApplication.Exit(0);
    14.                 }
    15.             }
    16.  
    The resulting XML looks a lot like it used to but one important line is missing: the <test-run ...> root element is not there, and looking at the xml specification for nunit that is required. Instead, the generated xml starts with a <test-suite> element right away.

    Am I doing something wrong here? Should I fix the XML output myself by adding the right root element? Or am I writing the wrong thing to xml?
     
    frank-ijsfontein, Oct 31, 2019
    #1

  2. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Not sure what you're doing wrong, but in the Unite Copenhagen talk they showed some code on how to serialize the results:


    (the link should take you to the exact time in the video where this is demonstrated).
     
    liortal, Oct 31, 2019
    #2
    frank-ijsfontein likes this.

  3. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    92
    Hey. You are doing it correctly, but the test result from the api does only include the top level. This is because you should be able to have more results in one test run.

    I believe the best approach is to wrap it in the test run node, with all the needed data. We are doing that as well when running tests from command line.

    Some snippets from our code:

    Code (CSharp):
    1. private const string k_nUnitVersion = "3.5.0.0";
    2.  
    3. private const string k_TestRunNode = "test-run";
    4. private const string k_Id = "id";
    5. private const string k_Testcasecount = "testcasecount";
    6. private const string k_Result = "result";
    7. private const string k_Total = "total";
    8. private const string k_Passed = "passed";
    9. private const string k_Failed = "failed";
    10. private const string k_Inconclusive = "inconclusive";
    11. private const string k_Skipped = "skipped";
    12. private const string k_Asserts = "asserts";
    13. private const string k_EngineVersion = "engine-version";
    14. private const string k_ClrVersion = "clr-version";
    15. private const string k_StartTime = "start-time";
    16. private const string k_EndTime = "end-time";
    17. private const string k_Duration = "duration";
    18.  
    19. private const string k_TimeFormat = "u";
    20.  
    21. void WriteResultsToXml(ITestResultAdaptor result, XmlWriter xmlWriter)
    22. {
    23.     // XML format as specified at https://github.com/nunit/docs/wiki/Test-Result-XML-Format
    24.  
    25.     var testRunNode = new TNode(k_TestRunNode);
    26.  
    27.     testRunNode.AddAttribute(k_Id, "2");
    28.     testRunNode.AddAttribute(k_Testcasecount, (result.PassCount + result.FailCount + result.SkipCount + result.InconclusiveCount).ToString());
    29.     testRunNode.AddAttribute(k_Result, result.ResultState.ToString());
    30.     testRunNode.AddAttribute(k_Total, (result.PassCount + result.FailCount + result.SkipCount + result.InconclusiveCount).ToString());
    31.     testRunNode.AddAttribute(k_Passed, result.PassCount.ToString());
    32.     testRunNode.AddAttribute(k_Failed, result.FailCount.ToString());
    33.     testRunNode.AddAttribute(k_Inconclusive, result.InconclusiveCount.ToString());
    34.     testRunNode.AddAttribute(k_Skipped, result.SkipCount.ToString());
    35.     testRunNode.AddAttribute(k_Asserts, result.AssertCount.ToString());
    36.     testRunNode.AddAttribute(k_EngineVersion, k_nUnitVersion);
    37.     testRunNode.AddAttribute(k_ClrVersion, Environment.Version.ToString());
    38.     testRunNode.AddAttribute(k_StartTime, result.StartTime.ToString(k_TimeFormat));
    39.     testRunNode.AddAttribute(k_EndTime, result.EndTime.ToString(k_TimeFormat));
    40.     testRunNode.AddAttribute(k_Duration, result.Duration.ToString());
    41.  
    42.     var resultNode = result.ToXml();
    43.     testRunNode.ChildNodes.Add(resultNode);
    44.  
    45.     testRunNode.WriteTo(xmlWriter);
    46. }
     
    Warnecke, Oct 31, 2019
    #3
    robal1991, recursive and frank-ijsfontein like this.

  4. frank-ijsfontein

    frank-ijsfontein

    Joined:
    Sep 11, 2015
    Posts:
    16
    Hey @Warnecke, that worked! Thanks.
    Any chance of incorporating this code into the framework, like a WriteTestRunReport() method, to avoid misunderstanding what is actually written?
    It took me a while to find the missing root element in the xml.
     
    frank-ijsfontein, Oct 31, 2019
    #4

  5. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    I am working on an extensions library for Unity test framework. Maybe i can include something like this in there.
     
    liortal, Oct 31, 2019
    #5

  6. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    92
    Rather than providing a helper function, in the long run, we would rather change to have this node included on the RunFinished event. We will likely do that, when we can make it happen in a way that does not break backwards compatibility.

    Yes. Feel free to do so. :)
     
    Warnecke, Nov 1, 2019
    #6

  7. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    liortal, Nov 6, 2019
    #7

  8. Thunderik

    Thunderik

    Joined:
    Apr 28, 2020
    Posts:
    32
    Hi, sorry for reviving an old thread but is this functionality still not part of the RunFinished event? Also it seems the XML report is all in the UnityEditor namespace but I am now trying make my built tests to generate XML, is there a way to do that without like copying the code inside of the ITestResultAdaptor, ITestAdaptor etc?
     
    Thunderik, Apr 28, 2023
    #8

  9. Warnecke

    Warnecke

    Unity Technologies

    Joined:
    Nov 28, 2017
    Posts:
    92
    Hey. You should be able to implement a callback listener using https://docs.unity3d.com/Packages/c...nual/reference-attribute-testruncallback.html on the runtime side. This will give you the raw NUnit ITestResult, which should contain the xml.
     
    Warnecke, Apr 28, 2023
    #9