Feedback Unity 2020.1 Performance Overview

Hello lovely people who like to look at performance graphs

Today I ran the performance tests of my game with Unity 2020.1.0b1 and 2018.1.9f2. I present the results below, just like I did before, during older beta cycles.

Spoiler: Older Reports

• Unity 2019.3 Performance Overview
  
• Unity 2019.2 Performance Overview
  
• Unity 2019.1 Performance Overview
  
• Unity 2018.3 Performance Overview
  
• Unity 2018.2 Performance Overview
  
• Unity 2018.1 Performance Overview
• Unity 2017.3 Performance Overview
• Unity 2017.2 Performance Overview

According to my test, 2020.1.0b1 is slower than 2018.1, which is still the fastest Unity version with my project. I profiled all builds listed in the graphs below during the same session.

You can run the test using the project attached to the following bug-report:
(Case 1108597) Project to find performance regressions and bugs

Spoiler: How does the test work?

The indoor first-person shooter I'm working on features an automated performance test. It works like this:

• Camera is placed at a defined position in the scene
• Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
  
• Every second the test captures the average CPU frame time of the last second

The game runs just like it would normally do, but the camera/player does not move and the enemy AI in unable to see the player.

The performance test measures the "base-line" of the game basically (rendering, shadows, realtime lighting, physics, audio, NavMesh, UI, game code). If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

I run this test in a Standalone Windows 64bit (Mono) Player:

• 2018 = .NET3.5
• 2020 = .NET4.x

I've also enabled "Incremental GC" on both 2020 versions.

The following Physics settings are applied:

• Physics.autoSyncTransforms=false
• Physics.autoSimulate=true
  
• Physics2D.autoSyncTransforms=false
• Physics2D.autoSimulate=false

The resolution is set to fullscreen 320x240 (D3D11) to make sure the bottleneck isn't GPU. The game uses the built-in deferred renderer.

The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.








These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.

Please also process bug-report: (Case 1222370) 2019.3: Android performance regression
https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5577082

 

At this point I have to ask: Has Unity approached you to work in their QA team? The work you do is awesome. Thank you very much

 

I am curious about your experience with 2020 beta and 2019.3. if you have any?

 

LOL... they promoted me to QA Jesus, so in that regard they work for me, right?! If I work there, they would lose their most annoying customer. I can't let this happen. My mission is to annoy the hell out of them (pun intended)!

Thanks for the kind words, much appreciated!

 

I've no experience with 2020.1 other than the performance test you can see above.

I've little experience with 2019.3. I briefly used it at home and at work, in both cases for rather simple projects. I can't think of any major bug, of the top of my head, that was so bad to not use it.

Android performance is really really bad with my project compared to 2018.4 though:
https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5581939
... which is the reason why I decided to stay with 2018.4 at work.

 

Spoiler

Though I would love to know WHY it's not as fast as 2018. Is it just scripting or? what part is slow?

1ms is actually a big deal for the same game, same hardware, with just Unity changed. I mean it's a pretty large loss when you're doing VR or consoles.

If I knew what caused it, I'd be able to work around it.

If it's CPU side then it probably makes URP look uglier than it is performance wise.

 

Damn, I should make this my new profile picture!

Great question! So... I was just profiling scene_6_8 at sample 2 with Unity 2018.1.9f2 and 2020.1.0b1 to make a more detailed comparison and pulled the data in Profile Analyzer. It didn't make it easier that in 2018.1.9f2 you can't load profiler snapshots.

Having the Unity Profiler connected, makes the game actually run faster with 2020.1

Looking at the data, "Profiler" related methods in 2020 have almost 0 cost, where as in 2018, it's significantly more. Which is the reason why it runs faster in 2020.1 if the profiler is connected.

Great job people at Unity to get rid of the profiler overhead! Now make the remaining parts to have a 0 cost as well please

Here is a comparison of the profiler cost first. Keep this in mind if you look at the absolute frame time of the game later, because the profiler cost wouldn't exist in the game unless, well, you're profiling, therefore you'd need to subtract the profiler cost from the total frame-time.

Spoiler: 2020.1 Profiler Cost

Spoiler: 2018.1 Profiler Cost

Spoiler: 2020.1 Profiler Capture

Spoiler: 2018.1 Profiler Capture

The Profiler Analyzer versions are different, that's why the window displays different stuff. The spikes in the middle come from me alt-tabbing from the game to Unity.

Looking at this data, I would say:

• Script execution seems to be overall slower in 2020.1
• Physics seems to be faster in 2020.1
• Rendering seems to be slightly slower in 2020.1
• Profiling overhead seems to be non-existent in 2020.1

I was profiling windows standalone builds btw.

I hope this makes sense!

 

Please add 2019.3 to the graph, so we have a comparison to all current unity versions.

 

I've profiled 2019.3.4f1 a few days ago, see:
https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5562490

 

@Unity.
Please add an official Benchmark to Unity who is available for everyone. The Blender Foundation is Benchmark here.
see
https://opendata.blender.org/

@Peter77
Could you share the script to your benchmark.
When Unity did not has the ressources i will set up such an webpage.)

 

Thanks for the info! That makes sense, because I had all profilers but the CPU profiler disabled.

 

Unfortunately there is no benchmark script. The "benchmark" is a first person shooter game.

 

Hey! I thought that was me!

But staying on topic... And to play Devil's advocate ... I'm wondering how much the benchmarks actually tell performance wise? For example, it could be that for a simple game, 2018.1 beats out 2020.1 but perhaps the more demanding the game becomes the worse 2018.1 fares compared to 2020.1 ? Looking at the graphs in your first post I'm noticing that when the ms increase on 2018.1, the increase on 2020.1 appears slightly less. Also I'm assuming there could be quality differences when it comes to things like shadows?

 

Could be, I don't know. I just see that projects at hand regressed performance and that's alarming for me.

Quality-settings wise they're identical. The resolution is set to 320x240 during these tests, to make sure the bottleneck isn't GPU. If Unity change the shadows implementation in the meantime, maybe it's related to that. But I'm pretty sure the issue isn't on the GPU rendering side, because the content is quite optimized.

 

Today I re-ran the performance test with 2020.1.0b2 and of course 2018.1.9f1, which has been the fastest Unity build for me, thus I use it for performance comparison. I profiled all builds listed in the graphs below during the same session.

You probably notice the graphs changed! The graphs are a bit more zoomed in, but everything is also about 0.4ms faster now.

Why is that you ask?

The reason why everything is about 0.4ms faster now is that earlier tests had the "Script Debugging" build option enabled. I've now turned it off, for this and upcoming tests, because the script debugging cost won't exist in the final build, thus I don't care about it.

All earlier tests are still valid and continue to show regressions, they're not wrong, they just include a cost that's irrelevant for the final release that won't be included in the next tests.

2020.1.0b2 continues to be slower than Unity 2018.1.9f1 with my project.

 

I'm not aware of Unity changing the builtin renderer from 2018 to present day. They froze that when they started on SRP, so if anything things should have got faster, not slower over time, especially with redundant features being moved out to packages and even less pointless overheads and at that resolution, I don't expect it to be a GPU issue.

It might be an issue if realtime GI is used though. That got fiddled with. Probably not that if it's removed in 2020 though. Hmmm....

 

Today I re-ran the performance test with 2020.1.0b4 and 2018.1.9f1. I profiled all builds listed in the graphs below during the same session.

Performance seems to be the same as in earlier 2020 builds. The graphs look a little different to the previous ones, because I adjusted the vertical-range.

 

You swore an oath that you shall not rest until these two lines are one

 

Let the red one be many msec below the blue one.

 

I wonder if the automated test lab (https://blogs.unity3d.com/2016/12/21/graphics-tests-the-last-line-of-automated-testing/) was abandoned along the way.

 

I'm bringing an old project up to modern age Unity and 2018.3's [RPC] deprecation might be the proverbial straw so I'll go up to 2018.2 but is it even worth the effort?

I'm currently in 2017 LTS, it's working fine, how does 2018.1.9 compare in term of perfs and stability? Why are you using it as an etalon and not 2018 LTS ?

 

2018.1.9 is the fastest Unity release with my project since the Unity 2017. I always use the fastest release for comparison.

New releases should not be slower than earlier versions. Where would we end up if "the fastest" isn't the benchmark.

I've made a performance comparison on LTS and recent TECH releases here:
https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5613511
Please keep in mind the numbers might not be representative for your project.

From a stability point I view, I wouldn't use 2018.1, I would use 2018.4 instead. I would prefer 2018.4 over 2017.4, because 2017.4 LTS phase ends very soon and I also don't want to miss nested prefabs anymore.

The performance overview threads I created measure performance only and don't provide any measure of stability.

 

The older the project, more unfair the test will be for newer versions of Unity, since the project uses outdated API and there are new functions that increase performance (Jobs, Burst, GPU instancing). With optimations for a new Unity version an old project can be much faster then on the old Version.

 

I would expect the "outdated" APIs to be slower than the newer APIs, but not slower than those same APIs in older versions.

 

The complexity of Unity increases steadily and with it the code that is executed, so the execution speed also changes.
Of course, it would be good if Unity counteracts this with better optimizations, which is also not that easy.

 

Thanks, it looks like 2017 LTS is middle of the road perf wise. How is build stability?

And since I'm doing this for the current customers and wouldn't like it any slower, how do speed compare with Unity 4.7?

(I notice already that compile time are through the roof. 2020.1 is also catastrophic in this metric... what's going one!)

@Unity, what are the crash statistics per build?

 

@Peter77 one thing I am curious about is that since 2018.2 you can remove Unity's core packages through the Package Manager, have you tried to remove unused core stuff and see if there are any performance benefits? Never tried that myself

 

No I haven't tried that.

I'm pretty sure I didn't see any surprising things in the profiler in regards of features the project isn't using the last time I checked.

 

Do you use SRP?
It would be interesting to see the difference "2018.1 Build-in" and "2020.1 URP"

 

2018.x tests all use .NET 3.5
2019.3 and 2020.1 tests use .NET 4.x

 

I did this with 2019.1 a while ago too. In 2019 the newer .NET version seems to perform slower according to my earlier tests.
https://forum.unity.com/threads/performance-overview.637783/#post-4377760

Here is the 2018.1 .NET 3.5 vs .NET 4.x comparison you asked for:







I triple-checked the Scripting Runtime settings, because the curves are so close and I found this a little suspicious, but it's set to 3.5 and 4.x.

 

The project is not using SRP.

The performance overviews are all made with a copy of the project I made in 2017, to make sure to compare apples to apples.

However, I experimented with URP in the actual project (not the test copies). But the game is using deferred rendering and URP wasn't supporting it at the time and I believe it doesn't support it yet either. The game contains many light sources and it didn't work well with URP at that time. I didn't look at performance with URP, but lights flickered all over the place.

My point of these performance overviews is to show that you take an existing project, open it in a newer Unity version and it runs slower. This pattern does not only occur with the performance overview project, but basically with pretty much everything I tested recently (link).

My goal with these posts is not to find ways to make it run faster by replacing most major systems with new technology (Burst, Jobs, ECS, URP etc).

At work we have projects that are almost 10 years old, some of them still get updates from time to time. We try to keep them relatively up-to-date, upgrading from LTS to LTS since the concept of LTS exists. I don't have the luxury to rewrite all those projects with new tech, every time Unity releases a new version, just to make sure it performs not slower than before.

I find the idea to believe that it's ok that software worsens if you don't adopt their newest features highly questionable. It's thinking like this that let companies get away with providing unfinished and buggy software.

 

I think this is an inevitable side effect of software and hardware development. Computers are getting faster and faster, and programs that are growing use this more performance.
The developers have to make sure that it doesn't get out of hand and the resource requirements of the software grow faster than the hardware.

 

I watched Jonathan Blow talk "Preventing the Collapse of Civilization".
We could make software better and robust which takes time to fix bugs. But if bug list is growing then how to expect that software will get robust. This was one of his arguments.
It is like Achilles and the tortoise race story.
Tortoise represents the goal of robust and performant engine. Bugs are the distance we need to cross to reach the goal.
Can Achilles reach the goal? Of course, physically this is not paradox.

 

It's definitely an interesting topic, but please discuss it in another thread. Let's keep this thread clean and on-the-point for Unity. Thanks!

 

Hello,

I have been making a proceduraly generated game since some years now. I began on the builtin engine (4 years ago !). Last I checked on the builtin engine, I was between 60 and 70FPS. I ported the game on the HDRP, saw a framerate drop that got bigger as I added new features, until 25 to 40FPS. I recently ported the game on the URP, and got 90 to 110 FPS, while still adding more graphics FX and using more complex geometry. My point : for me, the new unity (with the URP) is really much faster than the original builtin engine.
I don't know if there's an overhead between old pre-SRP and new versions of unity but it feels like a huge step forward. As for stability : I've been using custom renders, custom post processes (on both HDRP and URP), and always the last alphas when available. It's been at least 6 months since I've had any stability problems with unity (except the imperfect connexion between Visual Studio and Unity). I know my post is not very helpful, but since satisfied people write less about their experience...

 

Today I re-ran the performance test with 2020.1.0b11 and 2018.1.9f1. I profiled all builds listed in the graphs below during the same session.

 

I compared your new results with the ones posted in the first post. In all new tests, the game is running about 0.5-0.7 ms faster, but in scene 6_3 It seems like the performance has improved a bit more than that in 2020.1.11 compared to 2020.1.1.

Something you should do with these tests is to check how big the difference is between 2018 and 2020 version. Simply subtract in the excel amount of ms from 2020 to 2018, do this with both the newest test and the one on the top of this thread, then compare them in one graph.

It should always give you a proper comparison between two versions.

 

Thanks for your feedback and ideas!

This observation is spot on. It's because older tests used "Script Debugging". I turned it off in newer tests, which caused everything go down, 2018 and 2020. I explained the change here.

Like in the following image?

2018 is the baseline, thus at 0ms always. Since 2020 is slower, it goes above 0ms. If 2020 would be faster, it would go below 0ms.

Is this what you had in mind?

I see how this can be interesting for Unity Technologies. However, the latest beta is really the only one that is important to me. I'm interested to see how latest beta compares to the fastest Unity version with my project. If it does not reach the performance of that fastest version, they need to fix it. It's as simple as it gets, I think.


Thanks for the feedback!

 

Well, I'm not the part of the company, however I'm actually interested in seeing the results. The graph I proposed has really no potential while shown without comparison and a classic way is probably easier to read for the most people.

And also, even If that would showcase something interesting, subtracting these numbers isn't really the right way to do It. Normally you would have to divide them and do a bunch of other stuff I'm not going to ask because there is really no point.

 

Thanks again for the reply.

The following graph shows the difference of b4 and b11 to 2018.1.
b4 would normally be the first beta.

Is this what you had in mind? Perhaps use a difference graph type? Let me know.

 

Yes. You can use different graphs, but that's a really minor thing, you used a really good one.

What does matter is the analysis. One thing is that beta version of the unity engine is slightly improving (newer version is faster around half of the millisecond on average, but also noticeably more stable in Its "slowness") and there is a case in your test in which the difference between "best" version and current one is significantly smaller than an average (sample 9, 10 and 11). This does mean that you can now detect which parts of the game are truly causing the game to slow down and potentially what to keep an eye on when upgrading your project to a newer version.

Unity team should have many, even more advanced tests like that and do even more advanced statistics that this but that's not our job to do. I gained really good information about newer version of unity, I hope you too, and I also hope you will do this test once the 2020.1 will release.

 

It's a little disheartening that sometime after 2018.1 it seems there was a drop in performance that never was recovered... there should be no objective reason to have a slower engine two years later.

 

#MakeUnityFastAgain

 

Today I re-ran the performance test with 2020.1.0b13 and 2018.1.9f1. I profiled all builds listed in the graphs below during the same session.

The graphs use the layout as suggested here. It compares the difference of 2020.1.0b13 and 2020.1.0b4 to 2018.1.9f1, that now allows to see how the beta improved. Unity 2020.1.0b4 is the oldest I have installed, it would be b1 otherwise.

Unity 2020.1.0b13 continues to be slower than 2018.1.9f1 with my project.
Unity 2020.1.0b13 is faster than 2020.1.0b4, which is a good trend.

Ultimately we want the bars go negative, which would indicate an improvement over 2018.1.9f1.