Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Performance Overview

Discussion in '2018.1 Beta' started by Peter77, Mar 3, 2018.

  1. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,895
    Today I ran the automated performance test of my game with Unity 2018.1.0b6 and present the results below, just like I did before, during older beta cycles:
    1. Unity 2017.2 Performance Overview
    2. Unity 2017.3 Performance Overview
    [EDIT: 2018.2 Performance Overview can be found here.]

    2018.1.0b6 is the fastest Unity 201x.x version with my project yet. Unity 2018.1.0b6 performance is slightly faster (0.5ms in the best case) than 2017.3.0b6 for me.

    The (first-person) game I'm working on features an automated performance test. It works like this:
    • Camera is placed at a defined position in the scene
    • Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
    • Every second the test captures the average CPU frame time of the last second
    The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

    It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

    I run this test in a Standalone Windows 64bit Player. The following Physics settings are applied:
    • Physics.autoSyncTransforms=false
    • Physics.autoSimulate=true
    • Physics2D.autoSyncTransforms=false
    • Physics2D.autoSimulate=false

    The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
    Last edited: Apr 21, 2018
  2. rizu

    rizu

    Joined:
    Oct 8, 2013
    Posts:
    1,074
    Could you include IL2CPP on your tests now that's an option on 2018.1?
     
    Peter77 likes this.
  3. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,895
    Measuring an Il2Cpp build is actually a really good idea. :) Here is a comparison of 2018.1.0b9 between Mono and Il2Cpp (release mode).

    In the best case scenario, Il2Cpp is almost 1ms faster with my project, awesome!

    scene_4_3_il2cpp.png

    scene_5_4_il2cpp.png

    scene_6_8_il2cpp.png

    I'm going to post a zoomed-in version in the next post.
     
    Last edited: Mar 3, 2018
    rizu likes this.
  4. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,895
    Here is the Mono vs. Il2Cpp test, but the charts zoomed in...

    scene_4_3_il2cpp_zoom.png

    scene_5_4_il2cpp_zoom.png

    scene_6_8_il2cpp_zoom.png
     
    JamesArndt and Korindian like this.
  5. rizu

    rizu

    Joined:
    Oct 8, 2013
    Posts:
    1,074
    Thanks for the additional benchmark, it actually surprised me a little as I didn't expect overall better perf from IL2CPP. The figures I've seen from other users benches has often given worse results for IL2CPP (which is understandable as it can create it's own overhead).
     
  6. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,338
    If IL2CPP is the same performance, that will happen sometimes as you can't really be optimising basic logic that well, however if it's slower that should not ever happen IMHO and it's worth filing a bug.
     
  7. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,895
    Perhaps they didn't select the "Release Build Configuration"? If you create a debug build, Il2Cpp is significant slower, see images below.

    scene_4_3_il2cpp.png

    scene_5_4_il2cpp.png

    scene_6_8_il2cpp.png
     
    Ostwind likes this.
  8. rizu

    rizu

    Joined:
    Oct 8, 2013
    Posts:
    1,074
    I admit that I haven't benchmarked myself but I still think there are cases where IL2CPP will just be slower due to the extra work it can create. Of course when it works straight forwards, it'll be faster.

    In this post: https://forum.unity.com/threads/job-system-not-as-fast-as-mine-why-not.518374/#post-3402526 @TBbadmofo benchmarked IL2CPP to run his task in 250ms where mono ran it in 120ms. I not sure what "Non Development" actually means in this context, you'd expect him to mean release build. Also worth noting that they might have not optimized c# jobs for IL2CPP much since they plan them to be compiled with Burst eventually.

    Here's also article about possible IL2CPP overhead: https://jacksondunstan.com/articles/3916. He didn't actually benchmark the difference there so it's not fair to say IL2CPP would have been slower but for sure generated code will occasionally make lots of steps that one could bypass on manual design.

    I've seen other occasions where IL2CPP hasn't really sped things up on mobile platforms but again, I don't have any concrete evidence myself, just what I've read from other peoples experiences. I'm mainly interested on IL2CPP on Windows Desktop and since it's brand new feat in Unity now, there's not much perf info around yet :)
     
    Last edited: Mar 5, 2018
  9. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,338
    @rizu some people do not understand that you can *only* benchmark IL2CPP, ECS, Jobs with standalone. This is because there are lots of extra checks and tests during editor, so people are naturally seeing slower results in editor.
     
  10. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,895
    This guy is doing a great job dissecting the Il2Cpp generated code. I wonder if his findings somehow find their way back into Unity. It's just too precious information to get lost, since there are probably a few quirks that shouldn't be too hard for UT to fix.
     
  11. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,338
    Yeah Unity replied to it, also it's a year old now almost. Good spot though.
     
  12. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    3,874
    We are aware of the issues raised. We've been focused on other things recently, but I expect we will free up soon for some performance tweaks to generated IL2CPP code.
     
  13. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,895
    That's music to my ears, thanks! :)
     
  14. TBbadmofo

    TBbadmofo

    Joined:
    Apr 1, 2014
    Posts:
    14

    Release build (stand alone) is what I meant. I called it non-development build because I also unchecked the development build setting. Despite what @hippocoder said, I also complained about my code running faster in the editor than a mono release build; though, in IL2CPP it was better. I figured the editor was using mono and that it would have more overhead, so why does it run slower in a release?
     
    Last edited: Mar 31, 2018
  15. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,338
    If things run slower in release please report the bug and drop the case number. If IL2CPP is slower than mono then clearly it's purpose is broken there.
     
    LeonhardP and JamesArndt like this.
  16. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,895
  17. rastlin

    rastlin

    Joined:
    Jun 5, 2017
    Posts:
    100
    This is very interesting stuff, thank you Peter!
     
    Peter77 likes this.
  18. QFSW

    QFSW

    Joined:
    Mar 24, 2015
    Posts:
    2,352
    So is 2018 giving us an all time high in terms of performance? That's a nice surprise. Have you tried .net 4.6 vs .net 3.0?
     
    JamesArndt likes this.