Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Starting to use the Performance Test Runner: early feedback and questions

Discussion in 'Testing & Automation' started by sebas77, Nov 7, 2019.

  1. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,640
    Hi,

    at glance this seems a fantastic work and I am looking forward to use the Test Runner for integration tests/functional tests more than unit tests.

    I have some early comments:

    1) it appears that the tests do not load the RenderSettings.skybox settings from the lighting panel (I use a custom material). This can be fixed loading the material and set it in the setup
    2) I am not sure if addressable are supported, if so I probably need to build them somehow to make play mode tests work in editor (workaround: use Resource folder)
    3) this seems an important feature lack: IPrebuildSetup should support an async (IEnumerator) setup too. I.E.: in the case of loading a scene,I want to load a scene once, but use it with multiple tests. Is it possible to do so? Please read edit 1.
    4) not sure if I am supposed (read it's allowed) to perform multiple performance measures per test, for example using the Scope feature (this seems to work in this sense, but I am not sure why the option exists, although it solve the point 3 problem)
    5) I am not sure what's the best way to disable code outside a test when a test runs.
    6) I guess that most of the stuff listed here: https://blogs.unity3d.com/2018/09/25/performance-benchmarking-in-unity-how-to-get-started/ is now outdated, but the blog is still linked as source of reference, not sure what I should get there that is not in the official documentation
    7) not sure how to perform tests without the editor, it seems that run tests programmatically is a way to do so

    Edit 1: I may have misunderstood the reason why IPrebuildSetup exists. On second read it seems that it's there to setup stuff before the test are built and not executed. I am still not 100% sure what built means, i.e.: does it mean before the executable is built if I built a standalone client for testing? (this would open new questions in case).
    In any case I would still need an asynchronous way to initialize data for testfixture and not for test (or the other way around in case setup is already per testfixture and not per test, still working with one test only hence couldn't test it). My current suspect is that
    OuterUnityTestAction is what I am looking for.
     
    Last edited: Nov 8, 2019
  2. gintautass

    gintautass

    QA Minion Unity Technologies

    Joined:
    Oct 27, 2015
    Posts:
    46
    Hey,

    The performance testing package is still a very experimental feature and likely to change significantly in the future releases.

    > 1) it appears that the tests do not load the RenderSettings.skybox settings from the lighting panel (I use a custom material). This can be fixed loading the material and set it in the setup

    Could you reproduce this and file a bug report on a new unity project?

    > 2) I am not sure if addressable are supported, if so I probably need to build them somehow to make play mode tests work in editor (workaround: use Resource folder)

    I believe it should be support by test framework, not entirely sure though.

    > 4) not sure if I am supposed (read it's allowed) to perform multiple performance measures per test, for example using the Scope feature (this seems to work in this sense, but I am not sure why the option exists, although it solve the point 3 problem)

    Getting stable performance measurements is difficult due to so many factors impacting it. So we encourage running multiple iterations of your test to sample many times. We also allow and encourage people to collect extra measurements such as memory or setup times. So a single test can end up having main sample group named "Time" and multiple other such as allocated memory, gc allocations and setup time with some of them having multiple samples.

    > 5) I am not sure what's the best way to disable code outside a test when a test runs.

    depends on the codebase, whatever works for you will probably be the best way ;)

    > 6) I guess that most of the stuff listed here: https://blogs.unity3d.com/2018/09/25/performance-benchmarking-in-unity-how-to-get-started/ is now outdated, but the blog is still linked as source of reference, not sure what I should get there that is not in the official documentation

    Could you share where this is linked?

    > 7) not sure how to perform tests without the editor, it seems that run tests programmatically is a way to do so
    Performance testing is just an extension to unity test runner, refer to test framework package for how to run tests
    https://docs.unity3d.com/Packages/c...anual/reference-command-line.html?q=-runTests

    Current limitations are the tests can only run in development player and standalone runs with editor in the background which will affect performance. We have both tasks on the roadmap, no estimation on when it will be ready.


    IPrebuildSetup is a callback called before building the player, you can generate custom assets and include them in the project.


    It sounds like what you really want for setup of scene is UnitySetup attribute which lets you have a method returning IEnumerator https://docs.unity3d.com/Packages/c...nityEngine.TestTools.UnitySetUpAttribute.html
     
  3. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,640
    1) I will do it at a given point
    2) I will do some tests
    4) thanks
    5) I was hoping to have defines that are enabled only test-related builds, without being forced to add it manually. I tried to do something like this, but didn't seem to work:
    Code (CSharp):
    1.           _defines = PlayerSettings.GetScriptingDefineSymbolsForGroup(BuildTargetGroup.Standalone);
    2.             PlayerSettings.SetScriptingDefineSymbolsForGroup(EditorUserBuildSettings.selectedBuildTargetGroup, _defines.FastConcat(";UNITY_INCLUDE_RENDERING_TESTS"));
    3.             yield return new RecompileScripts();
    4.  
    to be honest I am not sure when I should do it, as I want to recompile the code once for all the tests (that's why I needed a UnitySetup per fixture and not per test)
    6) https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/resources.html
    7) development player is fine as long as I don't need the editor. Not sure if you are saying that I need the editor in the background in this case too. I did some tests and it seems that the
    TestRunnerApi is also meant to be used inside the Editor and not in play mode. If this is the case I have to stop using the Test Runner and do my tests manually.
    8) about the UnitySetup I didn't understand if it's per test and/or testfixture, need to do some tests.
    9) is there a way to measure the GPU time only of a frame? (CPU and GPU separated is even better)
     
    Last edited: Nov 18, 2019
  4. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,640
    @gintautass I noticed the performance package is not listed in the unity packages and I have to install it manually every time.