Search Unity

Automated rendering performance testing

Discussion in 'Testing & Automation' started by Aceria_, Jan 22, 2020.

  1. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    81
    I've been trying to get some tests running to ensure that we don't have a regression in performance, but I've been hitting my head against the wall and am not making much progress.

    The example found here: https://github.com/Unity-Technologies/XRAutomatedTests/releases/tag/2018.2_0.1.3 runs fine, but the results from it give me pretty useless graphs.

    The first frame in the FPS counter is always insanely high (anywhere between 30-80k), after which it goes to the regular state and evens out around the median. The same goes for the Camera.Render, a single lone spike makes the entire graph unreadable, see here for an example:



    While not completely unusable, it doesn't exactly help with my trust in the data.

    I also have been unable to get 2 scenes to run back-to-back when that scene instantiates GameObjects, as they are instantiate inside the Test Scene instead of the loaded scene. After the test is done they are then not cleaned up when the next scene test runs. I have not been able to find a solution to this so far.

    Does anyone have solutions/tips for the above solutions, or perhaps have an example where something similar does work as intended? I feel that the performance tests are extremely over-complicated compared to the regular UnitTests.
     
    Aceria_, Jan 22, 2020
    #1

  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,657
    Unfortunately, this kind of thing is - generally speaking - inevitable when you are measuring time information on a desktop platform. (We see much more stable numbers on platforms that do not have a general-purpose OS running in the background, such as consoles).

    The first frame will be higher because the rendering workload just changed, so this means that there is a lot of work to be done which other frames do not need - compiling shaders, uploading shader/texture/mesh data to graphics memory, etc. To some extent you get this even with pure CPU workloads, it can take a few frames to reach a state where caches have been populated in a consistent way. This is why we have 'warmup' functionality built into the framework.

    Other spikes can happen because you're sharing the machine with other processes. If e.g. Windows Defender suddenly decides to do a periodic process scan check while you're in the middle of a frame, then it can screw up your numbers - and obviously you would hope it doesn't do that, and you can try to configure the machine as much as possible to disable that sort of thing, but it's very hard to get everything. It's better to accept that a few spikes will happen and disregard them as outliers - this is one reason it's better to use the median and percentiles rather than average and min/max.

    The other option is to change what you're measuring. Looking at non-time things like the number of polygons rendered, or the number of SetPass calls, should be totally stable. The big downside is that they're only a proxy for what you actually care about.

    Have you tried SceneManager.SetActiveScene after loading the scene? GameObjects will be created into whichever scene is active.
     
    superpig, Jan 23, 2020
    #2

  3. Aceria_

    Aceria_

    Joined:
    Oct 20, 2014
    Posts:
    81
    I've managed to get rid of the initial spike by just ignoring the first result it tries to measure, which is good enough for now. The reason why I brought it up is because I use the template that was in the github I linked earlier. It starts with the cooldown (which I set to 2 seconds, though I don't think this is actually necessary for me at all), then there's a 10 second SettleTime. Even with this settle time the spike occurs.

    Code (CSharp):
    1.  
    2.         yield return CoolDown();
    3.         PlayerPrefs.SetString("testSave", fileName);
    4.         yield return SceneManager.LoadSceneAsync(defaultGameScene, LoadSceneMode.Additive);
    5.  
    6.         SetActiveScene(defaultGameScene);
    7.         // Instantiate performance test object in scene
    8.         var renderPerformanceTest = SetupPerfTest<DynamicRenderPerformanceMonoBehaviourTest>();
    9.  
    10.         // allow time to settle before taking measurements
    11.         yield return new WaitForSecondsRealtime(SettleTime);
    12.  
    13.         using (Measure.Scope()) {
    14.             yield return null;
    15.             using (Measure.Frames().Scope()) {
    16.                 // use ProfilerMarkers API from Performance Test Extension
    17.                 using (Measure.ProfilerMarkers(SamplerNames)) {
    18.                     // Set CaptureMetrics flag to TRUE; let's start capturing metrics
    19.                     renderPerformanceTest.component.CaptureMetrics = true;
    20.  
    21.                     // Run the MonoBehaviour Test
    22.                     yield return renderPerformanceTest;
    23.                 }
    24.             }
    25.         }
    26.  
    27.         yield return SceneManager.UnloadSceneAsync(defaultGameScene);
    The SetActiveScene seems to either not be triggered at all, or triggered after the Awake() has been called in the newly loaded scene, resulting in the generated GameObjects to be in the Test Scene, not in the loaded scene.
     
    Aceria_, Jan 27, 2020
    #3