Feedback Unity 2020.2 Performance Overview

Hello everybody, here we go again with another performance overview update. I know... I know... I'm early this time

Today I ran the performance tests of my game with Unity 2020.2.0a19 and 2018.1.9f2. I present the results below, just like I did before, during older beta cycles.

Spoiler: Older Reports

• Unity 2020.1 Performance Overview
  
• Unity 2019.3 Performance Overview
  
• Unity 2019.2 Performance Overview
  
• Unity 2019.1 Performance Overview
  
• Unity 2018.3 Performance Overview
  
• Unity 2018.2 Performance Overview
  
• Unity 2018.1 Performance Overview
• Unity 2017.3 Performance Overview
• Unity 2017.2 Performance Overview

According to my test, 2020.2.0a19 is slower than 2018.1, which is still the fastest Unity version with my project. I profiled all builds listed in the graphs below during the same session.

You can run the test using the project attached to the following bug-report:
(Case 1108597) Project to find performance regressions and bugs

Spoiler: How does the test work?

The indoor first-person shooter I'm working on features an automated performance test. It works like this:

• Camera is placed at a defined position in the scene
• Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
  
• Every second the test captures the average CPU frame time of the last second

The game runs just like it would normally do, but the camera/player does not move and the enemy AI in unable to see the player.

The performance test measures the "base-line" of the game basically (rendering, shadows, realtime lighting, physics, audio, NavMesh, UI, game code). If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

I run this test in a Standalone Windows 64bit (Mono) Player:

• 2018 = .NET3.5
• 2020 = .NET4.x

I've also enabled "Incremental GC" on 2020.

The following Physics settings are applied:

• Physics.autoSyncTransforms=false
• Physics.autoSimulate=true
  
• Physics2D.autoSyncTransforms=false
• Physics2D.autoSimulate=false

The resolution is set to fullscreen 320x240 (D3D11) to make sure the bottleneck isn't GPU. The game uses the built-in deferred renderer.

The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.

I profile each build three times and the graphs below display the minimum sample value (best performance) of these three profiling runs. I profile multiple times, because it improves the overall robustness of the test. In the past I only profiled a build once, but it caused that things like OS activity sometimes affected the timings and the overall graph was a bit more spiky.









These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.

I've also recently profiled various LTS versions for comparison. If you've missing the post, you find it here: https://forum.unity.com/threads/unity-2020-1-performance-overview.845506/page-2#post-6103425

 

@Peter77 do you have results for 2018 on .NET 4 runtime too ?
Surely there are more things at play, still surprised that 3.5 is noticeably - all things considered - faster

 

Yes, this was asked already, thus I have the numbers
https://forum.unity.com/threads/unity-2020-1-performance-overview.845506/#post-5684476

 

ok seems like runtime doesn't matter - which I found a little bit surprising tbh, that's why it's good to test it seems ) - I would, however, conduct all testing with the same one regardless - just to eliminate the variable, and given that new runtime is default on 2018, with 3.5 deprecated there already
But in any case I'm sure there are/were valid reasons for this, don't let internets distract you from doing these ]

 

Maybe I'm missing something, but why are results different for 2020.2 performance overview, and the one for 2020.1? For example, 2018.1 is starting at 3ms on Scene4_3 (current overview) but at 3.5ms on your older overview?

 

I explained it in this post:
https://forum.unity.com/threads/unity-2020-1-performance-overview.845506/#post-5625013

 

Makes sense, thanks!
Nice to see that 2020.2 is already faster than 2020.1 while still being an alpha.

 

if you ever add editor operations we'll get a sense of editor regression too and see
that
[editor refresh (1s)]
this
[editor refresh (2s)]
version
[editor refresh (5s)]
is...
so
zzzzz...
slow

 

I could repeat all the tests I did during the 2019.3 beta cycle:

Spoiler

(Case 1171423) 2019.3: Asset import significantly slower than in Unity 4.6
https://forum.unity.com/threads/cas...ignificantly-slower-than-in-unity-4-6.714986/

(Case 1171344) 2019.3: Focusing editor significantly slower than in Unity 4.6
https://forum.unity.com/threads/cas...-for-2-secs-before-it-gets-repsonsive.714791/

(Case 1161371) 2019.3: Compiling empty project significantly slower than in Unity 4.6
https://forum.unity.com/threads/cas...ty-project-takes-significantly-longer.691996/

(Case 1161373) 2019.3: Enter playmode time significantly slower than in Unity 4.6
https://forum.unity.com/threads/cas...playmode-time-significantly-increased.692005/

(Case 1161272) 2019.3: Time to open editor significantly slower than in Unity 4.6
https://forum.unity.com/threads/cas...o-open-editor-significantly-increased.691951/

(Case 1158368) 2019.3: Hierarchy window performance significantly slower than in Unity 4.6
https://forum.unity.com/threads/case-1158368-hierarchy-window-performance-degradation.684280/

The only issue is that it takes so much time and I'm so not interested in it at the moment

 

nm. One day maybe Unity will have the courage to publicize their internal benchmark.

 

Ha. I remember a runtime benchmark thread being locked because apparently, Unity having their internal benchmarks is a good reason to silence user benchmarks.

EDIT: Here it is.

 

Nice Work.

 

I think they locked it because it had evolved into users trying to help this one unity dev narrow down a problem - which he had - so there was no reason to keep it open. Seems fair although arbitrary given the number of other threads that go off topic and remain open.

But to go back to my point, which only had us as beneficiaries, I just thought of another benefit, for Unity: no one at Unity has to be the bad guy who tells a team "hey guys you're down 10% get your acts together" - we get to be the bad guys.
And some of us really enjoy it

 

It's graph time again!

Today I re-ran the performance test with 2020.2.0b4 against 2018.1.9f1 and 2020.2.0a19. I profiled all builds listed in the graphs below during the same session.

So far no performance improvement visible, perhaps even a tiny performance drop compared to a19.








PS: Don't miss to check out my new tool on github
https://github.com/pschraut/UnityMiniEditorIterationProfiler

 

Thanks for graphs @Peter77

I think I have missed something Can you elaborate you compare 2018.1.9 BuiltIn Renderer to 2020.2 BuiltIn Renderer
or 2018.1.9 X Renderer to 2020.2 URP 10 ?

In case you compare to 2020.2 BuiltIn can you please add line for last URP?

 

The project uses Unity's built-in deferred renderer in all tests.

That's unfortunately not as simple.

The project uses deferred rendering, which isn't available in URP at the time of writing. The last time I checked URP forward, it caused all sorts of light-flicker issues in this project. I'd also need to recreate all the custom shaders in URP.

When URP deferred is available, I'm looking into how to add this to the test. Until then, only built-in deferred renderer.

 

Just a few days after b4, we have b5 already. Lets see if this version runs as fast as they roll out new versions!

Today I re-ran the performance test with 2020.2.0b5 against 2018.1.9f1 and 2020.2.0a19. I profiled all builds listed in the graphs below during the same session. If you want to know what the test is actually doing, what rendering tech it uses, please see the first post.

So far no performance improvement visible in b5, perhaps even a little performance drop compared to a19.

 

Looking for the Performance Overview for Unity 2021.1? It's here: https://forum.unity.com/threads/unity-2021-1-performance-overview.993186/

 

Hey ! 2020 LTS is now out, are you planning on doing a performance overview for this ? Thanks a lot for your work btw

 

LTS is just bug fixes, so I don't think that it's worth doing within the first few versions. But who knows, maybe they F***ed something up while incrementing the version number and now everything is slower.

 

Thank you Nope, I don't plan on doing another performance overview. I mentioned it in the other thread that I'm not really interested in it anymore: https://forum.unity.com/threads/unity-2021-1-performance-overview.993186/#post-6472931

 

@Peter77 Thank you for your continued advocacy for holding the line on performance.