Feedback Unity 2019.2 Performance Overview

Today I ran the performance tests of my game with Unity 2019.2.0b1, 2019.1.0b9 and 2018.1.9f2. I present the results below, just like I did before, during older beta cycles.

Spoiler: Older Reports

• Unity 2019.1 Performance Overview
  
• Unity 2018.3 Performance Overview
  
• Unity 2018.2 Performance Overview
  
• Unity 2018.1 Performance Overview
• Unity 2017.3 Performance Overview
• Unity 2017.2 Performance Overview

According to my test, 2019.2.0b1 is slower than 2019.1 and significantly slower 2018.1.

You can run the test using the project attached to the following bug-report:
(Case 1108597) Project to find performance regressions and bugs

Spoiler: How does the test work?

The (first-person) game I'm working on features an automated performance test. It works like this:

• Camera is placed at a defined position in the scene
• Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
  
• Every second the test captures the average CPU frame time of the last second

The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

I run this test in a Standalone Windows 64bit (Mono) Player:

• 2018 = .NET3.5
• 2019 = .NET4.x

I've also enabled "Incremental GC" on both 2019 versions.

The following Physics settings are applied:

• Physics.autoSyncTransforms=false
• Physics.autoSimulate=true
  
• Physics2D.autoSyncTransforms=false
• Physics2D.autoSimulate=false

The resolution is set to 320x240 (D3D11) to make sure the bottleneck isn't GPU.

The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.








These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.

 

The stuff I'm waiting for all beta.

Great job as always!

 

@Peter77 we all appreciate your taking the time to do this on every release.

Your project, like any would, is biased toward particular use cases and it would be a real effort to better represent more varied use cases. It seems like this is something Unity Tech would be doing on every build in an automated way, so that they know in realtime what perf impact of every change is. But based on your results over time, perf does *not* seem to get better over time. It's extremely curious.

Have you heard from anyone at Unity at any point about this?

 

Thank you @rastlin and @Claytonious for the kind words.

No, I haven't heard from Unity Technologies yet.

 

They just replied to the 2019.1 thread:
https://forum.unity.com/threads/performance-overview.637783/#post-4542484

 

Peter77 what do you say about 2019.3.0a3? I just test it, and according to my feelings, it's much faster.

 

I haven't profiled 2019.3 alpha yet, I think I'll wait once it turned into a beta.

I'm actually not sure how useful this performance test is for Unity Technologies, but it seems the community appreciates it, so I'll keep doing it for every beta.

 

I think it's probably worth doing once near the end of each beta rather than throughout just as a precautionary measure maybe?

Still would love to know where those missing millisecs have gone. Are the older version tests actually done again at this point in time to eliminate local machine issues?

 

My thought was to do it early to give UT more time to fix any potential performance issue before the beta ends. It makes sense to do another test near the end too, to check if performance changed.

According to Unity it's related to UI:
https://forum.unity.com/threads/performance-overview.637783/#post-4542484

Yes. I profiled all the versions you find in the graph above during the same session.

 

Well, the start of each beta is fresh off the heels of alpha rather than the stability of the released version, which means it's going to be pretty rough early on each time.

This actually hit me a few times but apparently not the project you have, because different I suppose!

 

As asked for in this post, here is new data for Unity 2019.2.0b7. I profiled all three builds listed in the graphs below during the same session.

 

It seem to be event worse than b1 D:

 

@vickycl Here is new data for Unity 2019.2.0f1. I profiled all three builds listed in the graphs below during the same session.

 

@Peter77 Would you mind benchmarking 2019.1 in the future as well? Trying to get an idea of the performance hit for switching to 2019.2. Thanks for all your hard work!

 

You can find my latest 2019.1 profile here:
https://forum.unity.com/threads/performance-overview.637783/page-2#post-4652356

My approach is to compare the fastest unity version against the latest. There seem to be no point in profiling "older" versions as performance does not seem to improve over the course of patch releases. Now with 2019.2 out of the door, there is probably not going on much with 2019.1 anymore?

 

That's really cool. Thanks for letting us know and kudos to the Release QA department!

Spoiler