Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Performance testing results on builds / in player

Discussion in 'Scripting' started by Aithoneku, Nov 22, 2018.

  1. Aithoneku

    Aithoneku

    Joined:
    Dec 16, 2013
    Posts:
    67
    Short question:
    How can I get performance tests results when run on builds / in player?

    Long question:
    I'm trying to run some performance tests. Results in editor gives me weird results (i.e. it seems like the JIT C# compiler isn't able to inline even the simplest methods) and I read all over forums that profiling editor gives unreliable results which shouldn't be taken into account much and that I should measure release builds.

    So I created some performance tests using Unity performance module. Problem is, I cannot figure out how to get tests results when run in build. The xml file is not generated (or I couldn't find it) and Test Report window display nothing.

    So what's the correct way to execute build/player performance tests in order to get the reports?

    What I'm I doing:
    • Follow guide on https://docs.unity3d.com/Packages/com.unity.test-framework.performance@0.1/manual/index.html:
      • Add performance module to packages' manifest.json file.
      • Create assembly for performance tests.
      • Implement test using PerformanceTest attribute. Measuring is done with
        Measure.Method(...).WarmupCount(5).MeasurementCount(15).Run();
    • How I'm executing tests:
      • Variant a) (test runner in editor)
        • Window / General / Test Runner
        • Select play mode
        • Select my tests
        • Click "Run selected"
        • This works and I have results, but it's executed in editor, so according to what I read around the internet, it's unreliable. Results are available:
          • In the Test Runner
          • In Window / Analysis / Performance Test Report
      • Variant b) (test runner in build/player)
        • Window / General / Test Runner
        • Select play mode
        • Click "Run all in player (StandaloneWindows64)"
        • The player is built and executed. All tests pass.
        • Results are missing:
          • In Test Runner
          • In Performance Test Report window
          • In form of xml file
      • Variant c) (test runner from commandline)
    What kind of problem I'm trying to solve:
    I have lots of data (tens of megabytes) which I need to process. I already used profiler to find out what the bottleneck is and I found out that most of my assumptions (based on claims like compiler being smarter then programmer) are wrong - for example I needed to copy some continuous data (simple structure, with only blittable members). When I implemented variant which uses pointer magic I gained ~90% performance boost (certain calculation went from ~2000ms to ~200ms) when compared with simple loop and ~30% when compared with System.Array.Copy.

    My current problem is that I found out that another of my assumptions - about inlining even the simplest functions - is wrong, too, slowing down something about 25%.

    So now I want to measure the most critical parts and I want to find out whether C# JIT in Unity is so ineffective or my code is slow or just editor is ineffective.

    If you're interested, example project with my current tests are included. (In this case, I noticed that BitArray is quite slower when compared with using int[] directly.)
     

    Attached Files:

  2. monstercouch

    monstercouch

    Joined:
    Aug 23, 2016
    Posts:
    44
    Hi, have you found any solution for this? My Test Report window seems to show nothing regardless of the flow (in Unity 2018.3.10)
     
    JasonGriffithFF likes this.
  3. Aithoneku

    Aithoneku

    Joined:
    Dec 16, 2013
    Posts:
    67
    No, I spent too much time without progress so I gave up. Instead, I'm just doing basic profiling in editor, some parts I have to optimize blindly and measure how it affects fps. It's not that effective, but at least I made needed progress.
     
  4. JasonGriffithFF

    JasonGriffithFF

    Joined:
    Nov 24, 2019
    Posts:
    2
    So, I figured this out.

    It's because if you are on a version before 2019.1, PerformanceTestRunSaver.cs is saving the test data to the Application.streamingAssetsPath instead of the Application.persistentDataPath where the Test Report window is looking for it.

    I changed the code in TestReportWindow.cs to look for it at the streamingAssetsPath instead.

    Code (CSharp):
    1. private void LoadData()
    2. {
    3.      string filePath = Path.Combine(Application.streamingAssetsPath, "PerformanceTestResults.json");
    4.      ...
    5. }
     
    Last edited: Dec 13, 2019
    kerjemanov likes this.
  5. kerjemanov

    kerjemanov

    Joined:
    Aug 16, 2019
    Posts:
    7
    Thank you!