TestRunner -> Run all in player Profiling issues?

Hi,

I noticed that I can't make the editor build a release client to profile the tests. Regardless the build settings, the testrunner always build a dev client.
You may say that there is no difference between the two, but that's not what I see at the moment. Running the test in the editor 2020 in release mode is more than twice as fast as the same test running in the client built by the test runner.

I need to understand why there is this difference and my first guess atm is the fact that the client is built in dev mode.

 

These are the results of the same code profiled in 2020 in Release mode in the editor vs clicking Run all in player (StandaloneWindows64) WITH THE DEBUGGING SCRIPT CHECK DISABLED

Editor:



Run in player:



If I do my own profiling with stopwatch, I get the same results of the run in player, so what's going on with the Editor profiling?
To test it you can just iterate an array of 1 million int, should do the trick, otherwise, I will share my code.

Edit: In case it wasn't clear, I am talking about the huge difference of profiling between the array iterations in editor and not in editor, while the rest is basically the same. Is no one bewildered by it?

 

@sebas77 may i ask if u are using any service or software for generating and storing the test results data?

 

@liortal I don't think so, I am not sure what you are asking. This is the standard unity "performance test report"

 

I did a quick test with a simple array to see if I was right. It seems I cannot reproduce it with a simple array, so I will spend more time when I can to see where the problem actually is.

 

OK last update from me: I wasn't wrong. It's just that I either get around 2ms or around 0.7ms randomly. I am not sure why it happens yet. For example the same code I ran yesterday now runs at 0.7ms even if I run it in the player. Instead a built client with stopwatches gives me 2ms, but I have seen giving me 0.7 as well.
My bet is on the current state of the mono CLR at this point, but it's suspicious how constant those numbers are. It's either around 0.7 or around 2.0ms.

The code with the stopwatch is this if you want to check:

https://sharplab.io/#v2:EYLgxg9gTgp...aPyD2fZ4/ZPG6nh+RJDAekvnEaHxTAShDitjoUkSGIgA=

while the code with the performance API looks like:

Code (CSharp):

1.  
2.  
3. var entityCollection = new EntityCollection() { count = dictionarySize };
4. var testArray = new Test[dictionarySize];
5. Measure.Method(() =>
6. {
7.     using (Measure.Scope("Iterate Managed Array"))
8.     {
9.         for (int index = 0; index < entityCollection.count; index++)
10.         {
11.             testArray[index].a = 4;
12.         }
13.     }
14.     using (Measure.Scope("Iterate Managed Array constant size"))
15.     {
16.         for (int index = 0; index < dictionarySize; index++)
17.         {
18.             testArray[index].a = 4;
19.         }
20.     }
21. })
22. WarmupCount(3)
23. MeasurementCount(10)
24. IterationsPerMeasurement(10)
25. Run();
26.  

If I can do more tests, it will be this weekend, but at this point I don't have much else to add.

 

Sorry, I know the topic is old, but did you find a solution to run the testrunner in 'release-mode'. I'm trying to use testrunner with unity-performance-framework but I can't get unity to run the tests in release-mode (I guess there needs to be an option. Where is the point of performance tests if not done in release mode,right!?).
I was happy to see that there is now an option in Unity3d >2020 (Preferences).
View attachment 1056173
But this seems only to work if you go in playmode. Testrunner is still in debug mode. Any help appreciated.