Feedback Unity 2019.3 Performance Overview

Today I ran the performance tests of my game with Unity 2019.3.0b1, 2019.2.0f1 and 2018.1.9f2. I present the results below, just like I did before, during older beta cycles.

Spoiler: Older Reports

• Unity 2019.2 Performance Overview
  
• Unity 2019.1 Performance Overview
  
• Unity 2018.3 Performance Overview
  
• Unity 2018.2 Performance Overview
  
• Unity 2018.1 Performance Overview
• Unity 2017.3 Performance Overview
• Unity 2017.2 Performance Overview

According to my test, 2019.3.0b1 is slower than 2019.2 and significantly slower 2018.1. I profiled all three builds listed in the graphs below during the same session.

You can run the test using the project attached to the following bug-report:
(Case 1108597) Project to find performance regressions and bugs

Spoiler: How does the test work?

The (first-person) game I'm working on features an automated performance test. It works like this:

• Camera is placed at a defined position in the scene
• Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
  
• Every second the test captures the average CPU frame time of the last second

The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

I run this test in a Standalone Windows 64bit (Mono) Player:

• 2018 = .NET3.5
• 2019 = .NET4.x

I've also enabled "Incremental GC" on both 2019 versions.

The following Physics settings are applied:

• Physics.autoSyncTransforms=false
• Physics.autoSimulate=true
  
• Physics2D.autoSyncTransforms=false
• Physics2D.autoSimulate=false

The resolution is set to 320x240 (D3D11) to make sure the bottleneck isn't GPU.

The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.









These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.

EDIT: The 2020.1 performance overview can be found at:
https://forum.unity.com/threads/unity-2020-1-performance-overview.845506/

 

Thanks

 

Personnaly I have much more problems with GPU performance. Granted, I use a GTX770M, but even with a simple shader on HDLit master node (or even PBR), I get near 2000 fragment shader code lines. I can spend a lot of time optimizing my shaders (and I do it through custom nodes), but it's a drop in the water next to what HDRP is generating at the end.
It's much easier to optimize the CPU side (no game objects except when absolutly necessary, instancing, threading, etc...). I have lots of behaviour trees AI running at the same time, some meteo calculations, minimaps, etc but I think that even with twice what I'm doing now, I'd still be GPU bound.

 

Yeah I'm also noticing a big performance drop on my mobile test. My android device used to run at 30fps no problem and it's usually around 24fps now. That's a 20% drop on performance from 2019.1 and I can't test it now but I think 2018 was even faster.

 

As far as I know, Unity collects hardware specs when submitting a bug-report. If that's true, you should be able to find the specs in Case 1108597.

Otherwise please let me know and I will add them to the report.

Edit: I added them to a forum thread a while ago too:
https://forum.unity.com/threads/performance-overview.637783/#post-4298446

 

Vsync in game mode is not present in 2019.3.
Could this be the cause of CPU/GPU overhead?

My CPU/GPU fans begins to spin rapidly when entering game mode in HDRP.

 

I've been following your performance test results in the Betas for quite a few versions now, always interesting stuff. There's something I don't really understand. I can understand the Unity Editor itself getting slower with each new major version. (And I think that's somewhat reasonable.) But I really don't understand why an optimized build of a game would have worse performance in each version of Unity. If anything, I would expect build performance to improve, not worsen.

Has there ever been an explanation of why build performance decays? It would be one thing if you were using new features of a newer version, and having to accept a performance hit to enjoy the new functionality. But it seems that's not the case with your performance tests. So, do we know why builds are slower in newer versions?

 

I think the best explanation would be this one:
https://forum.unity.com/threads/performance-overview.637783/page-2#post-4642300

I imagine depending on Unity versions, it's just different fixes/improvements that cause the build to run slower. Regression here, regression there.

That's correct. The project does not use any of the new features, except for things Unity Technologies created to improve performance. For example, starting with 2019.2?, the test uses the incremental GC. But otherwise it's really just a 2 years old copy of my actual project that I didn't change since then.

 

@Peter77 do you happen to know if Unity performance on Linux is better than on Windows?

I saw this video for Blender. Might be the same for Unity. Opening Unity project takes longer and longer.

 

Nope, I don't know. I've never used Unity on Linux.

 

Wait... They had a fix that actually improved the performances in 2019.2 and now it's not improved anymore .. I don't get it. What did I miss ? How come 2019.3 is worth that 2019.2 after all they found to be faulty ?

I ran some tests on my game project comparing the same project on 2018.3 (the version we're currently working on) and 2019.3 (the long awaited version that includes all the bugs we reported, being fixed...). And I found out that both in editor and in build, we have a rough 30% performance drop.

My game is a sprite based pixel art tactical, so nothing hardcore. And seeing the performances drop version after version is a bit worrying. Especially when we hear about performance improvement quit a lot from Unity.

 

It doesn't have to be the same issue.

The performance regression I reported during 2019.2 might be fixed and the regression we see in 2019.3 could be something differently.

Perhaps it's also a different regression in your project than the one I found in my project.

 

I never understand why almost every single release Unity posts: performance improvements. But all I see every single release on all my projects is: performance drops.
I have no idea what performance they improve, but it's definitely not the one for regular projects. At the beginning the drop was just a little but now it's very very noticeable. A game that ran just fine on an iPhone 5 compiled with unity 4? I don't even know how many years ago... same game without changes runs about the same speed on an iPhone 7 using unity 2019.3. The performance drop is gigantic.

 

It's probably becuase multiple version are sprinting by different teams and regressions are fixed afterwards in official version postlaunch.

 

It's quite hard to pin point what goes wrong exactly.
I tried to see why my project runs slower in 2019.3 but couldn't find anything specifically faulty.

I reported a bug (Case 1186691)
I made a simple 2D project.
2 sprites, 1 tile and 1 script (a simple fps counter).
In the main scene there are around 50 animated GameObjects and a pretty big tilemap (all prefabs).
It represents an average scene in my game project.

Here's the fps in 2018.3. (105.3 fps)


Here's the fps in 2019.3. (71.2 fps)


How can I detect what's wrong ?

 

It's very hard to report this thing without including full project. But to give you an example, this is my game:

That video was uploaded on 2015. Latest unity version back then. Running on android and iOS at 30 fps (or more) on an iPhone 5 and one of those crappy android phones (using android 4.4, I would need to look for the exact phone model).
It included:
Unity terrain+over 10k trees+water+all multiplayer going on+one directional light+clouds+ real time weather (you can see on the video raining, thunderstorms and such).

Today, the same game compiled on Unity 2019.2 or 2019.3, cannot be run at more than 10 fps on those phones. So currently:
- I had to remove that water quality and put a crappy one. Until a much improved version of the community ocean water was released but the original one was way slower on latest version of unity.
- Mesh terrain is used instead of unity terrain.
- I've removed real time weather, it's always sunny, no rain, no heavy clouds.

Everything else I was able to keep. But instead of adding stuff thanks to performance improvements on Unity, I keep removing stuff until the previous phones get so old that people upgrade I can finally put back what I had years before.

 

The fps counter I use seems fine according to Fraps.
As far as I know, the rendering statistic window only shows an estimated fps of what it might be in build:
https://forum.unity.com/threads/sta...66.1944009711.1569165747-108023183.1568208825

The project I sent with my bug report (Case 1186691) is as simple as it can be.
Make builds and you'll see that, even if the numbers are slightly higher in builds, there is still a significant difference in performance between 2018.3 and 2019.3.

 

Off topic, since this doesn't actually appear to be the problem here, but yeah, the thing to remember is that the editor *attempts* to remove its own update time from statistics about everything. In my experience it is rather unsuccessful at that. This is especially important to note if you are making anything like a rhythm game, where music is playing entirely unconstrained by the editor update and the latter doesn't actually run in real time as such. FRAPS will show you that the editor window is rendering a x number of FPS but the game view is only a subset of that.

 

Today I re-ran the performance test with 2019.3.0b9. I profiled all three builds listed in the graphs below during the same session.

2019.3.0b9 continues to be the slowest Unity release since 2018 with my project. Performance didn't improve between 2019.3.0b1 and b9.

Spoiler: 2019.2 spikes

What's fascinating (though not related to 2019.3) is that the spikes found in the 2019.2 graph happened again. Last time I thought it could be an one time thing due to some OS activity or so, but it seems it's actually related to Unity or it's really just a major coincidence.

 

Unity 2019.2.1f1 48fps in my project on my xiaomi redmi note 7 64go
Unity 2019.3.0b9 32fps in my project on my xiaomi redmi note 7 64go

Exactly same scene and configurations

What happened?

 

One of the many performance improvements. I keep saying that my game ran just fine on iphone 4 with unity 3 or 4. And now it needs a 64 bits processor, iphone 7 and runs slower. Same game. Not sure why they keep adding stuff and slowing it so much for existing games. I keep having to remove more and more stuff from my game so it can keep up the fps with every Unity performance improvement. (just been sarcastic because that's usually what the release notes say).

 

What about Unity 2020? nothing change?

 

yeah, many more performance improvements

 

I can just confirm, the performance dropped drasticaly between 2018 and 2019 and not only between these versions also earlier ones like Unity 5 to 2018 etc. Games once ran smooth and had a small build size on crap devices but not anymore. Also it's said that using always the most recent API's will increase performance but i noticed using vulkan decreases performance even more compared to open gl 2. For me it rather sounds like regression by default than perfomance by default.
Also i tried setting a lower framerate manually just to see if it stays stable like Application.targetFrameRate = 10; but even this function does not work anymore on Mobile as in ealier versions and just sets the framerate to maximum -.- (~60).

 

nvm, message deleted. It was just a big rant and I don't think Unity cares anymore so why should I.

Good luck with your performance enhancements.

 

i cant upload, my upload speed 80ko/s i will upload my 2go 2 years sorry.
Just create new scene 2019.2.1f1 and 2019.3.0b9 and use LWPR renderer with 512*512 shadow 80 distance with 1 terrain 4 texture + normal without antialiasing and run on Android device

 

Am I missing something or do we not have performance metrics for the 2018.4 LTS versions?

 

Wasn't it because 2018.3 was basically renamed and updated to be 2018.4? There were no betas I believe, which is where he usually does these reports.

 

Today I re-ran the performance test with 2019.3.0b11. I profiled all three builds listed in the graphs below during the same session.

Performance improved between 2019.3.0b9 and b11. However, 2019.3.0b11 continues to be the slowest Unity release since 2018 with my project.

Two things that changed:

1. I upgraded from b9 to b11
2. I deleted the library directory before opening the project (because of the magic that is sometimes going on)

I haven't profiled 2018.4. I can do this later this weekend.

 

I would have liked to update the 2018.3 thread, but I can't post there anymore. I also didn't want to start a new thread, so I post the LTS performance metrics here instead.

It's located under a spoiler to avoid confusion with actual 2019.3 data...

Spoiler: LTS Profiling Overview

2017.4 and 2018.4 because these are the current LTS releases. I've also added 2018.1 to graph, because it's the fastest Unity release for me so far.

PS: While I had 2017.4 open, I noticed how much faster/responsive the 2017 editor is compared to 2019.3! It's such huge difference like day and night. Yet the 4.6 editor is even more responsive than 2017!

 

Yeah, every new release is worse and worse. Not sure why this is not a critical thing to look at. Every single game is affected by performance degradation.

 

I changed all of my major projects over to 2018.4 and this graph make me happy to have done so.

But I have always wondered for many years, what made Unity releases less and less performant. This has been a consistent issue for every single release since the first 2017 release I believe. I know there are regressions and changes and more features, but it seems to almost always get worse and worse as time goes on. Not just talking game performance, but editor performance. I hope they really sit down and look at overall performance and iteration speed in the near future. (I doubt it though, with how fast everything is moving these days.)

 

Thank you very much for these tests! I have also noticed the blazingly fast editor usability in 2017.4 vs 2019.xx. Something as simple as right clicking and creating an object is even faster. I have no idea why the editor itself has become much slower and less responsive?

 

That's indeed an interesting question. I don't know that either, but perhaps Unity staff can shed some light into it.

 

Personally I started to notice slow down in editor when all these extra "services", such as ads, analytics, etc. started to appear. Looking from the styling of those screens they seem to be implemented in different way to the rest of the editor, but perhaps these days with the new theming and UI Elements stuff this is no longer the case (we're still in 2018.4, and will be for a good while). And of course there are likely other things contributing to the slowness but I suspect these may be one of the causes.

 

To me if it wasn't because of the Terrain improvements (they delayed 7 years to implement), I would still be using Unity 4/5 or maximum 2017. Anything after that is just a horrible performance drop not worth to update unless absolutely necessary. It's absolutely horrible and there is no mention whatsoever of anyone looking into it on the betas/alphas.

I don't care what caused it, I care that they fix it. It doesn't make sense you now need an iPhone 11 to run the same game you could easily run on iPhone 5 a couple years ago with same features and at the same FPS.

 

That's true. But it's also important to understand what caused this issue to learn from it. Otherwise we end up with the same problem over and over again, no matter how often it "got fixed".

 

At this point, this is my number one issue. I have an older version of our project in Unity 4.x that I sometimes open for reference and I am always amazed how fast and snappy everything is.

The Unity developer experience is in free fall for many years now, Unity keeps announcing they are fixing it, but it seems to keep getting worse.

 

So who knows who works on the Editor team(s) at Unity? Can we tag them in this thread and perhaps gain some insight?

 

This could be used for better frame stutter analysis.

 

The thing that's most concerning about such significant performance decline over each version is my previous understanding that moving Unity to a package-based approach was supposed to make Unity lighter and leaner. I was expecting that if enough functionality was moved into packages, and I only include a subset of functionality, then I should see faster performance overall. But it seems like things are actually going in the other direction. It makes me wonder if moving things into packages hasn't had the performance impact that was intended. Maybe to make packages work modularly enough, they needed to add a bunch of extra code that previously was just optimized into the engine. Who knows. I know I've certainly had to accept performance loss as a trade-off for flexibility in applications I've written.

 

My understanding wasn't this at all. I thought the reason Unity went with the package-based approach was to decouple development cycles of features from the core engine/toolchain development cycle.

A side benefit could be improved build times, and maybe smaller build sizes, but not a faster engine. I mean, if you didn't use a feature, it probably wasn't being executed as a part of the update loop / rendering path. Pulling out stuff like AR/VR or 2D Animation isn't going to speed the engine up if you never used them to begin with.