Feedback Performance Overview

Today I ran the performance test of my game with Unity 2019.1.0b4 and present the results below, just like I did before, during older beta cycles:

• Unity 2018.3 Performance Overview
  
• Unity 2018.2 Performance Overview
  
• Unity 2018.1 Performance Overview
• Unity 2017.3 Performance Overview
• Unity 2017.2 Performance Overview

According to my test, 2019.1.0b4 is slower than 2018 on average.

You can run the test using the project attached to the following bug-report:
(Case 1108597) Project to find performance regressions and bugs

Spoiler: How does the test work?

The (first-person) game I'm working on features an automated performance test. It works like this:

• Camera is placed at a defined position in the scene
• Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
  
• Every second the test captures the average CPU frame time of the last second

The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

I run this test in a Standalone Windows 64bit (Mono) Player. The following Physics settings are applied:

• Physics.autoSyncTransforms=false
• Physics.autoSimulate=true
  
• Physics2D.autoSyncTransforms=false
• Physics2D.autoSimulate=false

The resolution is set to 320x240 to make sure the bottleneck isn't GPU.

The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.







Playing the game inside the editor, is also significantly slower than in earlier Unity versions, like sometimes dropping to 20fps which I didn't observe in Unity 2017.

These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.

 

I'm thinking, maybe I should repeat the test with every Unity version.

This would make sure the performance regression is not related to some Windows and/or graphics card driver update that was installed in the meantime. I mean there were always several months between I ran those tests.

 

About 4 hours later, I'm done. I've updated the first post with the new numbers. The results look pretty similar to the ones I measured earlier though.

2019.1.0b4 is slower than Unity 2018. Unity 2018.1 continues to be the fastest Unity version with the provided test project.

 

Thank you for your hard work!

 

Thanks for noticing

 

It's great to know that the highest delta is around 0.5ms. By this knowledge, if you touched 16.16ms per frame you are at risk of not getting 60FPS from just changing Unity version. (probably will be magnified by a certain things in the game that's not in the test project though)

 

If you target VR, it's about 5% of your total frame time.

 

Is there a specific reason why Unity is getting consistently slower over time when the features you are using remain the same?

I guess I ask because someone will at some point.

 

Probably all the added features, aka everything running backend

 

thank you for making these charts, can you do tests with vulkan

 

Unfortunately I can't, I only have a Windows PC.

 

The audio in newer version is definitely glitching and sounds like buffer underrun, like Unity cost more spec but doesn't update the buffer size, so Unity can't keep up with the old buffer size anymore even in editor. (If you go set something not Best Latency it will improve, at the cost of latency)

Here's my case where UT said just change to something not "Best Latency" : https://fogbugz.unity3d.com/default.asp?1118164_ip87hse66aelc9od but the point is, it was working under Best Latency before the recent version. It is a regression.

Unfortunately I don't know how to benchmark the "buffer underrun" other than using my ears. If that is possible programmatically (you may have to ask audio driver that it found nothing to process and send to speaker.. but how?) we can dynamically adjust buffer size and restart audio system until it doesn't crack.

 

Please make your own topic, don't drag thread off topic.

 

Vulkan runs on Windows.

 

please

it runs on windows, please do a test?

 

Holy moly, will do a Vulkan test at the weekend then!

Do you know if I can start the build/player with a command-line to select a specific graphics api (Direct3D11, Vulkan, etc)?

 

there are a bunch of

-force-XXX

switches (https://docs.unity3d.com/Manual/CommandLineArguments.html), maybe try

-force-vulkan

or something

 

or just remove the automatic graphics setting for windows on player settings and force it to use vulkan there

 

I will give this a shot. I thought they're for the editor only. We'll see

Yeah I'm aware of this option. I was trying to find a way to build the player once only and then test all graphics api's, rather than having to build a new player for each test.

 

Here are the tests for the different graphics api's in Unity 2019.1.0b4. I tested D3D11, D3D12 (experimental) and Vulkan.

I've to admit that the game is not really graphics demanding. It's quite simple what's going on visually and it has only a few draw-calls. There are most likely other projects better suited for a graphics performance test than mine.

Anyway, here are the results.

Code (CSharp):

1. ------------------
2. System Information
3. ------------------
4.          Operating System: Windows 10 Home 64-bit (10.0, Build 17134) (17134.rs4_release.180410-1804)
5.       System Manufacturer: System manufacturer
6.              System Model: P5K/EPU
7.                      BIOS: BIOS Date: 06/19/08 17:02:36 Ver: 08.00.12 (type: BIOS)
8.                 Processor: Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz (4 CPUs), ~2.7GHz
9.                    Memory: 8192MB RAM
10.           DirectX Version: DirectX 12
11.  
12. ---------------
13. Display Devices
14. ---------------
15.            Card name: NVIDIA GeForce GTX 650 Ti
16. Driver File Version: 25.21.0014.1917 (English)
17.       Driver Version: 25.21.14.1917
18.          DDI Version: 12

 

Make own topic for discussions among yourselves, this is feedback for Unity.

Edit: was responding to some posts I deleted where users wanted to drag this topic off track for their own ends, hence the reminder that people keep noise low for Unity staff to appreciate the data in feedback posts.

 

This is a great thread, thank you. Unity should seriously check if they do reproduce theese and seriously take performance prior to fancy ui reworks etc.

 

Imagine if we have a special CI server service which we could upload any Unity project on it. Each user made their own project containing things they are concerned about regression. That project then include one special package which works with that CI server + contains reporting API. After the upload, CI server will run against all versions and output the graph. Maybe that server could automatically distribute builds to Firebase Test Lab devices. I would pay for that.

 

I wish Unity staff would leave a note if this thread has been seen and if it's being investigated or if I have to submit issues for that.

 

always submit issue. that "forces" them to look at it. then you can link to this thread.

 

Sad that that is the state of things.

 

Loaded up our game which is rather large and uses almost every major feature in Unity. it's a consistent drop of 8-10fps (vsync off) from the latest 2018.3. I couldn't notice anything that really stood out in the profiler it just seems to be across the board.

 

Good god, do you or anyone else know why D3D12 is so slow compared? I thought direct x 12 was touted to be super performant compared to previous versions, or at least thats what I was led to believe whey they were hyping it up before release?

edit: @Peter77 have you or anyone else here submitted an issue for this? This really needs to be submit as an issue for the team to look at ASAP and you seem best placed due to all the data you have acrued

 

I have not. I tried to reproduce the D3D12 performance issue in a test-project, but wasn't able to make D3D12 run slower than D3D11 and then didn't look further into it.

Unity Technologies have my project (Case 1108597) and instructions to run these tests themselves. If they're interested in doing that, they can. It should be reproducible with this project when running on slow(er) hardware.

 

may be since the tested beta was b4 can we get an updated benchmark on current¿?

 

Unity 2019.1.0b9 is slightly faster than 2019.1.0b4, but still slower than Unity 2018.1.

 

And here is a Graphics API comparison with 2019.1.0b9.

 

Here is a .NET3.5 vs .NET4.x (mono) comparison with 2019.1.0b9.

.NET 4.x is slightly slower than the now deprecated .NET3.5 Scripting Backend. Do you plan to get .NET 4.x as fast as 3.5 before actually removing it?

 

thanks a lot mate!!! at least there is hope seeing this going towards less worse. But still what a dissapointment huh, hope they don´t release until everything is fully fully polished. 2018.3 never again please

 

Are you using il2cpp or mono in your test builds?

 

This test was using mono.

 

As far as I understand, il2cpp can have substantially better performance (in some situations) than mono, and it works on more platforms and seems to be the universal intermediate target that Unity is favoring over mono. I'd probably test with il2cpp instead of mono, especially for things like the .NET upgrade.

 

il2cpp is still error prone, and not nessisarily faster, besides you can't build to il2cpp for linux soo.. I would hold the boat on il2cpp till it's more stable. testing mono makes a lot of sence. and i'm quite shocked mono 4x is slower than 3.5

 

Do you have any evidence to back up the claims of il2cpp being "error prone and not faster"?

Nothing in any of our projects has shown this, we have always found the complete opposite.

 

I tested this a while ago, see here:
https://forum.unity.com/threads/performance-overview.520116/#post-3411457

I'll see if I make an il2cpp comparison. This whole thing is just time consuming, since I've to rebuild "everything" and re-run the tests. I'll probably do this at the upcoming weekend.

 

Really interesting thanks for sharing the data.

 

@GameDevCouple_I Every single Unity patch release comes with more il2cpp fixes because it breaks on random things, especially with funnily enough manged to native interop. and other such cases, we simply cannot switch to il2cpp for this reason.
When testing though it is quite faster on float math, because mono is using doubles for everything. something that got fixed in mono, but never ported to unity, (come on guys its only a single flag!!!)
https://tirania.org/blog/archive/2018/Apr-11.html

Anyway.. Il2cpp is a code gen, and a horrible one at that, have you even seen what it generates? the most simple C# code becomes a giant mess of 1000's of cpp lines.. even the best compiler cannot make it much faster, that's why most code under il2cpp is only marginally faster.

The only reason of any speed up at all is that il2cpp overcomes the big problem of unity, that there is a LOT of managed to native interop overhead for every call to the engine from your c# code.

 

Performance becomes worse because while Unity slowly fixing/replacing/involving some stuff to the upgraded Mono, they don't bring any modern Jit intrinsics into it with the most noticeable performance-related changes.

 

Personally I don't find IL2CPP very relevant because it's missing features.

It doesn't support Linq.Expressions or runtime code generation and so it's missing an absolutely essential feature.
There is an interpreted mode now for expressions, but it doesn't help at all because it's slower.
Linq.Expressions/runtime code gen is used specifically to make code faster, and an interpreted version kinda (totally) defeats the point

Networking, serialization, saving/loading games, ... code gen is embedded into tons of systems, and is just way too useful to be missing (or crippled through low performance).

 

you can codegen at build time instead. that will be even faster than runtime
also some platforms (iOS, consoles, ...) do not support it at all

 

Personally, while I one of those who was waiting for IL2CPP on desktop platforms, I'm not using it today primarily due to long compilation time which means it impossible to make changes on the fly and quickly validate a particular behavior, suboptimal code that it produces which results in crashes in worst case scenarios, and unexpected failures in managed-native interop code (I encountered at least two, both was fixed by @JoshPeterson ).

 

maybe for stuff that can be done in advance...

I have an ingame script editor that absolutely needs to load/compile IL code...

 

Thank you very much for looking at it and keeping us in the loop. I've also created an Unity 2019.2 Performance Overview thread, in case you're interested (see here).