Feedback Unity 2020.1 Performance Overview

Hello lovely people who like to look at performance graphs

Today I ran the performance tests of my game with Unity 2020.1.0b1 and 2018.1.9f2. I present the results below, just like I did before, during older beta cycles.

Spoiler: Older Reports

• Unity 2019.3 Performance Overview
  
• Unity 2019.2 Performance Overview
  
• Unity 2019.1 Performance Overview
  
• Unity 2018.3 Performance Overview
  
• Unity 2018.2 Performance Overview
  
• Unity 2018.1 Performance Overview
• Unity 2017.3 Performance Overview
• Unity 2017.2 Performance Overview

According to my test, 2020.1.0b1 is slower than 2018.1, which is still the fastest Unity version with my project. I profiled all builds listed in the graphs below during the same session.

You can run the test using the project attached to the following bug-report:
(Case 1108597) Project to find performance regressions and bugs

Spoiler: How does the test work?

The (first-person) game I'm working on features an automated performance test. It works like this:

• Camera is placed at a defined position in the scene
• Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
  
• Every second the test captures the average CPU frame time of the last second

The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

I run this test in a Standalone Windows 64bit (Mono) Player:

• 2018 = .NET3.5
• 2020 = .NET4.x

I've also enabled "Incremental GC" on both 2020 versions.

The following Physics settings are applied:

• Physics.autoSyncTransforms=false
• Physics.autoSimulate=true
  
• Physics2D.autoSyncTransforms=false
• Physics2D.autoSimulate=false

The resolution is set to 320x240 (D3D11) to make sure the bottleneck isn't GPU.

The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.








These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.

Please also process bug-report: (Case 1222370) 2019.3: Android performance regression
https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5577082

 

At this point I have to ask: Has Unity approached you to work in their QA team? The work you do is awesome. Thank you very much

 

I am curious about your experience with 2020 beta and 2019.3. if you have any?

 

LOL... they promoted me to QA Jesus, so in that regard they work for me, right?! If I work there, they would lose their most annoying customer. I can't let this happen. My mission is to annoy the hell out of them (pun intended)!

Thanks for the kind words, much appreciated!

 

I've no experience with 2020.1 other than the performance test you can see above.

I've little experience with 2019.3. I briefly used it at home and at work, in both cases for rather simple projects. I can't think of any major bug, of the top of my head, that was so bad to not use it.

Android performance is really really bad with my project compared to 2018.4 though:
https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5581939
... which is the reason why I decided to stay with 2018.4 at work.

 

Spoiler

Though I would love to know WHY it's not as fast as 2018. Is it just scripting or? what part is slow?

1ms is actually a big deal for the same game, same hardware, with just Unity changed. I mean it's a pretty large loss when you're doing VR or consoles.

If I knew what caused it, I'd be able to work around it.

If it's CPU side then it probably makes URP look uglier than it is performance wise.

 

Damn, I should make this my new profile picture!

Great question! So... I was just profiling scene_6_8 at sample 2 with Unity 2018.1.9f2 and 2020.1.0b1 to make a more detailed comparison and pulled the data in Profile Analyzer. It didn't make it easier that in 2018.1.9f2 you can't load profiler snapshots.

Having the Unity Profiler connected, makes the game actually run faster with 2020.1

Looking at the data, "Profiler" related methods in 2020 have almost 0 cost, where as in 2018, it's significantly more. Which is the reason why it runs faster in 2020.1 if the profiler is connected.

Great job people at Unity to get rid of the profiler overhead! Now make the remaining parts to have a 0 cost as well please

Here is a comparison of the profiler cost first. Keep this in mind if you look at the absolute frame time of the game later, because the profiler cost wouldn't exist in the game unless, well, you're profiling, therefore you'd need to subtract the profiler cost from the total frame-time.

Spoiler: 2020.1 Profiler Cost

Spoiler: 2018.1 Profiler Cost

Spoiler: 2020.1 Profiler Capture

Spoiler: 2018.1 Profiler Capture

The Profiler Analyzer versions are different, that's why the window displays different stuff. The spikes in the middle come from me alt-tabbing from the game to Unity.

Looking at this data, I would say:

• Script execution seems to be overall slower in 2020.1
• Physics seems to be faster in 2020.1
• Rendering seems to be slightly slower in 2020.1
• Profiling overhead seems to be non-existent in 2020.1

I was profiling windows standalone builds btw.

I hope this makes sense!

 

Please add 2019.3 to the graph, so we have a comparison to all current unity versions.

 

I've profiled 2019.3.4f1 a few days ago, see:
https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5562490

 

@Unity.
Please add an official Benchmark to Unity who is available for everyone. The Blender Foundation is Benchmark here.
see
https://opendata.blender.org/

@Peter77
Could you share the script to your benchmark.
When Unity did not has the ressources i will set up such an webpage.)

 

Thanks for the info! That makes sense, because I had all profilers but the CPU profiler disabled.

 

Unfortunately there is no benchmark script. The "benchmark" is a first person shooter game.

 

Hey! I thought that was me!

But staying on topic... And to play Devil's advocate ... I'm wondering how much the benchmarks actually tell performance wise? For example, it could be that for a simple game, 2018.1 beats out 2020.1 but perhaps the more demanding the game becomes the worse 2018.1 fares compared to 2020.1 ? Looking at the graphs in your first post I'm noticing that when the ms increase on 2018.1, the increase on 2020.1 appears slightly less. Also I'm assuming there could be quality differences when it comes to things like shadows?

 

Could be, I don't know. I just see that projects at hand regressed performance and that's alarming for me.

Quality-settings wise they're identical. The resolution is set to 320x240 during these tests, to make sure the bottleneck isn't GPU. If Unity change the shadows implementation in the meantime, maybe it's related to that. But I'm pretty sure the issue isn't on the GPU rendering side, because the content is quite optimized.

 

Today I re-ran the performance test with 2020.1.0b2 and of course 2018.1.9f1, which has been the fastest Unity build for me, thus I use it for performance comparison. I profiled all builds listed in the graphs below during the same session.

You probably notice the graphs changed! The graphs are a bit more zoomed in, but everything is also about 0.4ms faster now.

Why is that you ask?

The reason why everything is about 0.4ms faster now is that earlier tests had the "Script Debugging" build option enabled. I've now turned it off, for this and upcoming tests, because the script debugging cost won't exist in the final build, thus I don't care about it.

All earlier tests are still valid and continue to show regressions, they're not wrong, they just include a cost that's irrelevant for the final release that won't be included in the next tests.

2020.1.0b2 continues to be slower than Unity 2018.1.9f1 with my project.

 

I'm not aware of Unity changing the builtin renderer from 2018 to present day. They froze that when they started on SRP, so if anything things should have got faster, not slower over time, especially with redundant features being moved out to packages and even less pointless overheads and at that resolution, I don't expect it to be a GPU issue.

It might be an issue if realtime GI is used though. That got fiddled with. Probably not that if it's removed in 2020 though. Hmmm....

 

Today I re-ran the performance test with 2020.1.0b4 and 2018.1.9f1. I profiled all builds listed in the graphs below during the same session.

Performance seems to be the same as in earlier 2020 builds. The graphs look a little different to the previous ones, because I adjusted the vertical-range.

 

You swore an oath that you shall not rest until these two lines are one

 

Let the red one be many msec below the blue one.

 

I wonder if the automated test lab (https://blogs.unity3d.com/2016/12/21/graphics-tests-the-last-line-of-automated-testing/) was abandoned along the way.