Performance Overview

Today I ran the automated performance test of my game with Unity 2018.1.0b6 and present the results below, just like I did before, during older beta cycles:

1. Unity 2017.2 Performance Overview
2. Unity 2017.3 Performance Overview

[EDIT: 2018.2 Performance Overview can be found here.]

2018.1.0b6 is the fastest Unity 201x.x version with my project yet. Unity 2018.1.0b6 performance is slightly faster (0.5ms in the best case) than 2017.3.0b6 for me.

Spoiler: How does the performance test work?

The (first-person) game I'm working on features an automated performance test. It works like this:

• Camera is placed at a defined position in the scene
• Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
  
• Every second the test captures the average CPU frame time of the last second

The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

I run this test in a Standalone Windows 64bit Player. The following Physics settings are applied:

• Physics.autoSyncTransforms=false
• Physics.autoSimulate=true
  
• Physics2D.autoSyncTransforms=false
• Physics2D.autoSimulate=false

The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test.

 

Could you include IL2CPP on your tests now that's an option on 2018.1?

 

Measuring an Il2Cpp build is actually a really good idea. Here is a comparison of 2018.1.0b9 between Mono and Il2Cpp (release mode).

In the best case scenario, Il2Cpp is almost 1ms faster with my project, awesome!







I'm going to post a zoomed-in version in the next post.

 

Here is the Mono vs. Il2Cpp test, but the charts zoomed in...

 

Thanks for the additional benchmark, it actually surprised me a little as I didn't expect overall better perf from IL2CPP. The figures I've seen from other users benches has often given worse results for IL2CPP (which is understandable as it can create it's own overhead).

 

If IL2CPP is the same performance, that will happen sometimes as you can't really be optimising basic logic that well, however if it's slower that should not ever happen IMHO and it's worth filing a bug.

 

Perhaps they didn't select the "Release Build Configuration"? If you create a debug build, Il2Cpp is significant slower, see images below.

 

I admit that I haven't benchmarked myself but I still think there are cases where IL2CPP will just be slower due to the extra work it can create. Of course when it works straight forwards, it'll be faster.

In this post: https://forum.unity.com/threads/job-system-not-as-fast-as-mine-why-not.518374/#post-3402526 @TBbadmofo benchmarked IL2CPP to run his task in 250ms where mono ran it in 120ms. I not sure what "Non Development" actually means in this context, you'd expect him to mean release build. Also worth noting that they might have not optimized c# jobs for IL2CPP much since they plan them to be compiled with Burst eventually.

Here's also article about possible IL2CPP overhead: https://jacksondunstan.com/articles/3916. He didn't actually benchmark the difference there so it's not fair to say IL2CPP would have been slower but for sure generated code will occasionally make lots of steps that one could bypass on manual design.

I've seen other occasions where IL2CPP hasn't really sped things up on mobile platforms but again, I don't have any concrete evidence myself, just what I've read from other peoples experiences. I'm mainly interested on IL2CPP on Windows Desktop and since it's brand new feat in Unity now, there's not much perf info around yet

 

@rizu some people do not understand that you can *only* benchmark IL2CPP, ECS, Jobs with standalone. This is because there are lots of extra checks and tests during editor, so people are naturally seeing slower results in editor.

 

This guy is doing a great job dissecting the Il2Cpp generated code. I wonder if his findings somehow find their way back into Unity. It's just too precious information to get lost, since there are probably a few quirks that shouldn't be too hard for UT to fix.

 

Yeah Unity replied to it, also it's a year old now almost. Good spot though.

 

That's music to my ears, thanks!

 

Release build (stand alone) is what I meant. I called it non-development build because I also unchecked the development build setting. Despite what @hippocoder said, I also complained about my code running faster in the editor than a mono release build; though, in IL2CPP it was better. I figured the editor was using mono and that it would have more overhead, so why does it run slower in a release?

 

If things run slower in release please report the bug and drop the case number. If IL2CPP is slower than mono then clearly it's purpose is broken there.

 

Today I ran the automated performance test of my game with Unity 2018.2.0b1. You can find the results here: https://forum.unity.com/threads/performance-overview.527846/

 

This is very interesting stuff, thank you Peter!

 

So is 2018 giving us an all time high in terms of performance? That's a nice surprise. Have you tried .net 4.6 vs .net 3.0?