Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Performance Testing package doesn't produce a test result when running on a device

Discussion in 'Testing & Automation' started by matros, May 15, 2022.

  1. matros

    matros

    Joined:
    Feb 1, 2014
    Posts:
    15
    Hello,

    I got 2 issues with test reports.

    1. There is no data in the Test Report window when running tests on a target device via Test Runner in the editor.
    This post tells that at least result.xml should be in the streaming assets folder, but it is not there either.
    So I tried to run it via the command line as this post suggests and got another issue.

    2. There is no test result saved at the given path when I am running tests using
    -testPlatform StandaloneWindows64 -buildTarget StandaloneWindows64
    or
    -testPlatform Android -buildTarget Android
    even though I can see that tests are finished successfully both on my android device and PC respectively.
    Here is the full command I use:

    ./Unity.exe -runTests -projectPath D:/Development/Unity/<projectDirectory>
    -testPlatform Android -buildTarget Android -mtRendering -scriptingbackend=il2cpp
    -testResults D:/Development/Unity/<projectDirectory>/TestReport/TestReport.xml
    -logfile D:/Development/Unity/<projectDirectory>/TestLogs/TestLog.txt

    However, if I replace -testPlatform with
    -testPlatform PlayMode
    then the test result is produced at the given path, but obviously, this argument makes the run be executed in the editor instead of a target platform, therefore results are kinda irrelevant.
    So without TestReport.xml I cannot even use Unity Performance Benchmark Reporter to produce a report and got no way to see performance results after testing on a target device.

    I tested it on:
    Performance Testing versions:
    2.7.0-preview,
    2.8.0-preview
    (2.5.0 from the docs is not working for me at all, as a package couldn't be found)

    Unity versions:
    2021.3.0f1,
    2022.1.0b8.

    The project where issues reproduce in 100% cases:
    GitHub

    What is the way to get a test report after running a performance test suite on a real device?
     
  2. matros

    matros

    Joined:
    Feb 1, 2014
    Posts:
    15
    Well, JFYI I tried https://github.com/Unity-Technologies/XRAutomatedTests/, particularly NoXR.
    Using 2019.3.3f1 and Performance Testing 1.1.31.

    A run on WindowsStandalone64 via the command line produced a test report file.

    So I tried to downgrade the Performance Testing package to the same version in my project, but the run still doesn't produce a test report. Looks like the next step should be downgrading the Unity version, going to create a sample project since the one I am using requires a new API that was introduced in 2020.
     
  3. matros

    matros

    Joined:
    Feb 1, 2014
    Posts:
    15
    I created an empty project, added the Performance Testing package, and checked running tests on a device using different unity versions.
    Here are the results:
    • Unity 2019.3:
      • Performance Testing 1.1.31: a report is produced when running tests on a device via the terminal. A report is also available inside the editor via the Test Report window when running tests using the Run All Tests (Android) button inside the Test Runner window.
      • Performance Testing 2.8.0: same as the version 1.1.31. Reports from a device run are produced both using the terminal and the editor.
    • Unity 2020.3/2021.3/2022.1
    • Performance Testing 1.1.31: a report is not created when running tests on a device via the terminal. After running tests on a device via the editor the Test Report window doesn't show the results (the data is updated only when running tests in the editor, not on a device).
    • Performance Testing 2.8.0: same as the version 1.1.31. Nothing is available after the test run on a device.
    Unity Technologies please fix.
     
  4. matros

    matros

    Joined:
    Feb 1, 2014
    Posts:
    15
    Somehow I was able to get the report in the editor after running performance tests on an Android device via the test runner window using the same project I linked in the first message and Unity 2021.3.

    Out of around 30 runs, I got 2 in a row, which results were available in the Test Report window.
    And I noticed in both runs that instead of showing a Performance Testing message "All test(s) succeeded" in the end, the app was automatically closed and after that I could see data in the test report window.
    But when I am closing the app via the close button there are no results available.

    So don't ask me how, but I found the following sequence of actions that allows me to get a report every time:
    1. Run tests on a target platform via the Test Runner window
    2. When a test run is finished manually connect Unity Profiler to your device
    3. In a few seconds the app is closed and the result is available in the Test Report window.​

    Sounds crazy, but works for me.