Search Unity

TestRunner -> Run all in player Profiling issues?

Discussion in 'Testing & Automation' started by sebas77, May 16, 2020.

  1. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,387
    Hi,

    I noticed that I can't make the editor build a release client to profile the tests. Regardless the build settings, the testrunner always build a dev client.
    You may say that there is no difference between the two, but that's not what I see at the moment. Running the test in the editor 2020 in release mode is more than twice as fast as the same test running in the client built by the test runner.

    I need to understand why there is this difference and my first guess atm is the fact that the client is built in dev mode.
     
  2. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,387
    These are the results of the same code profiled in 2020 in Release mode in the editor vs clicking Run all in player (StandaloneWindows64) WITH THE DEBUGGING SCRIPT CHECK DISABLED

    Editor:



    Run in player:



    If I do my own profiling with stopwatch, I get the same results of the run in player, so what's going on with the Editor profiling?
    To test it you can just iterate an array of 1 million int, should do the trick, otherwise, I will share my code.

    Edit: In case it wasn't clear, I am talking about the huge difference of profiling between the array iterations in editor and not in editor, while the rest is basically the same. Is no one bewildered by it?
     
    Last edited: May 18, 2020
  3. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,506
    @sebas77 may i ask if u are using any service or software for generating and storing the test results data?
     
  4. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,387
    @liortal I don't think so, I am not sure what you are asking. This is the standard unity "performance test report"
     
    Last edited: May 18, 2020
  5. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,387
    I did a quick test with a simple array to see if I was right. It seems I cannot reproduce it with a simple array, so I will spend more time when I can to see where the problem actually is.
     
  6. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,387
    OK last update from me: I wasn't wrong. It's just that I either get around 2ms or around 0.7ms randomly. I am not sure why it happens yet. For example the same code I ran yesterday now runs at 0.7ms even if I run it in the player. Instead a built client with stopwatches gives me 2ms, but I have seen giving me 0.7 as well.
    My bet is on the current state of the mono CLR at this point, but it's suspicious how constant those numbers are. It's either around 0.7 or around 2.0ms.

    The code with the stopwatch is this if you want to check:

    https://sharplab.io/#v2:EYLgxg9gTgp...aPyD2fZ4/ZPG6nh+RJDAekvnEaHxTAShDitjoUkSGIgA=

    while the code with the performance API looks like:

    Code (CSharp):
    1.  
    2.  
    3. var entityCollection = new EntityCollection() { count = dictionarySize };
    4. var testArray = new Test[dictionarySize];
    5. Measure.Method(() =>
    6. {
    7.     using (Measure.Scope("Iterate Managed Array"))
    8.     {
    9.         for (int index = 0; index < entityCollection.count; index++)
    10.         {
    11.             testArray[index].a = 4;
    12.         }
    13.     }
    14.     using (Measure.Scope("Iterate Managed Array constant size"))
    15.     {
    16.         for (int index = 0; index < dictionarySize; index++)
    17.         {
    18.             testArray[index].a = 4;
    19.         }
    20.     }
    21. })
    22. WarmupCount(3)
    23. MeasurementCount(10)
    24. IterationsPerMeasurement(10)
    25. Run();
    26.  
    If I can do more tests, it will be this weekend, but at this point I don't have much else to add.
     
    Last edited: May 18, 2020
unityunity