Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Feedback Unity 2019.2 Performance Overview

Discussion in '2019.2 Beta' started by Peter77, May 10, 2019.

  1. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    Today I ran the performance tests of my game with Unity 2019.2.0b1, 2019.1.0b9 and 2018.1.9f2. I present the results below, just like I did before, during older beta cycles.

    According to my test, 2019.2.0b1 is slower than 2019.1 and significantly slower 2018.1.

    You can run the test using the project attached to the following bug-report:
    (Case 1108597) Project to find performance regressions and bugs

    The (first-person) game I'm working on features an automated performance test. It works like this:
    • Camera is placed at a defined position in the scene
    • Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
    • Every second the test captures the average CPU frame time of the last second
    The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

    It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

    I run this test in a Standalone Windows 64bit (Mono) Player:
    • 2018 = .NET3.5
    • 2019 = .NET4.x
    I've also enabled "Incremental GC" on both 2019 versions.

    The following Physics settings are applied:
    • Physics.autoSyncTransforms=false
    • Physics.autoSimulate=true
    • Physics2D.autoSyncTransforms=false
    • Physics2D.autoSimulate=false
    The resolution is set to 320x240 (D3D11) to make sure the bottleneck isn't GPU.

    The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png


    These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.
     
    Last edited: May 11, 2019
  2. rastlin

    rastlin

    Joined:
    Jun 5, 2017
    Posts:
    127
    The stuff I'm waiting for all beta.

    Great job as always!
     
  3. Claytonious

    Claytonious

    Joined:
    Feb 16, 2009
    Posts:
    893
    @Peter77 we all appreciate your taking the time to do this on every release.

    Your project, like any would, is biased toward particular use cases and it would be a real effort to better represent more varied use cases. It seems like this is something Unity Tech would be doing on every build in an automated way, so that they know in realtime what perf impact of every change is. But based on your results over time, perf does *not* seem to get better over time. It's extremely curious.

    Have you heard from anyone at Unity at any point about this?
     
    Peter77 likes this.
  4. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    Thank you @rastlin and @Claytonious for the kind words.

    No, I haven't heard from Unity Technologies yet.
     
    JonSWEDEN likes this.
  5. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    Prodigga likes this.
  6. id0

    id0

    Joined:
    Nov 23, 2012
    Posts:
    455
    Peter77 what do you say about 2019.3.0a3? I just test it, and according to my feelings, it's much faster.
     
  7. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    I haven't profiled 2019.3 alpha yet, I think I'll wait once it turned into a beta.

    I'm actually not sure how useful this performance test is for Unity Technologies, but it seems the community appreciates it, so I'll keep doing it for every beta.
     
  8. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    I think it's probably worth doing once near the end of each beta rather than throughout just as a precautionary measure maybe?

    Still would love to know where those missing millisecs have gone. Are the older version tests actually done again at this point in time to eliminate local machine issues?
     
  9. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    My thought was to do it early to give UT more time to fix any potential performance issue before the beta ends. It makes sense to do another test near the end too, to check if performance changed.

    According to Unity it's related to UI:
    https://forum.unity.com/threads/performance-overview.637783/#post-4542484

    Yes. I profiled all the versions you find in the graph above during the same session.
     
    hippocoder likes this.
  10. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Well, the start of each beta is fresh off the heels of alpha rather than the stability of the released version, which means it's going to be pretty rough early on each time.

    This actually hit me a few times but apparently not the project you have, because different I suppose!
     
  11. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    As asked for in this post, here is new data for Unity 2019.2.0b7. I profiled all three builds listed in the graphs below during the same session.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
  12. rastlin

    rastlin

    Joined:
    Jun 5, 2017
    Posts:
    127
    It seem to be event worse than b1 D:
     
  13. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    @vickycl Here is new data for Unity 2019.2.0f1. I profiled all three builds listed in the graphs below during the same session.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
    vickycl and hungrybelome like this.
  14. hungrybelome

    hungrybelome

    Joined:
    Dec 31, 2014
    Posts:
    336
    @Peter77 Would you mind benchmarking 2019.1 in the future as well? Trying to get an idea of the performance hit for switching to 2019.2. Thanks for all your hard work!
     
    Peter77 likes this.
  15. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    You can find my latest 2019.1 profile here:
    https://forum.unity.com/threads/performance-overview.637783/page-2#post-4652356

    My approach is to compare the fastest unity version against the latest. There seem to be no point in profiling "older" versions as performance does not seem to improve over the course of patch releases. Now with 2019.2 out of the door, there is probably not going on much with 2019.1 anymore?
     
  16. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    Just wanted to add the Release QA department in Unity have included this project into their automated testing system for performance profiling.
     
  17. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    One thing to add. The performance results can be dependent on the hardware being used. It is worth running the tests on similar hardware you intend to run your project on to get an improved measurement about performance.
     
  18. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,566
    That's really cool. Thanks for letting us know and kudos to the Release QA department!