Search Unity

  1. We would like to hear your feedback about Unity and our products. Click here for more information.
    Dismiss Notice

Feedback Performance Overview

Discussion in '2019.1 Beta' started by Peter77, Mar 2, 2019.

  1. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    Today I ran the performance test of my game with Unity 2019.1.0b4 and present the results below, just like I did before, during older beta cycles:
    According to my test, 2019.1.0b4 is slower than 2018 on average.

    You can run the test using the project attached to the following bug-report:
    (Case 1108597) Project to find performance regressions and bugs

    The (first-person) game I'm working on features an automated performance test. It works like this:
    • Camera is placed at a defined position in the scene
    • Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
    • Every second the test captures the average CPU frame time of the last second
    The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

    It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

    I run this test in a Standalone Windows 64bit (Mono) Player. The following Physics settings are applied:
    • Physics.autoSyncTransforms=false
    • Physics.autoSimulate=true
    • Physics2D.autoSyncTransforms=false
    • Physics2D.autoSimulate=false
    The resolution is set to 320x240 to make sure the bottleneck isn't GPU.

    The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.

    4_3.png

    5_4.png

    6_8.png

    Playing the game inside the editor, is also significantly slower than in earlier Unity versions, like sometimes dropping to 20fps which I didn't observe in Unity 2017.

    These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.
     
    Last edited: Mar 3, 2019
  2. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    I'm thinking, maybe I should repeat the test with every Unity version.

    This would make sure the performance regression is not related to some Windows and/or graphics card driver update that was installed in the meantime. I mean there were always several months between I ran those tests.
     
    Last edited: Mar 2, 2019
    luispedrofonseca and SugoiDev like this.
  3. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    About 4 hours later, I'm done. I've updated the first post with the new numbers. The results look pretty similar to the ones I measured earlier though.

    2019.1.0b4 is slower than Unity 2018. Unity 2018.1 continues to be the fastest Unity version with the provided test project.
     
  4. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    719
    Thank you for your hard work!
     
  5. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    Thanks for noticing :)
     
    TokyoWarfareProject likes this.
  6. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,195
    It's great to know that the highest delta is around 0.5ms. By this knowledge, if you touched 16.16ms per frame you are at risk of not getting 60FPS from just changing Unity version. (probably will be magnified by a certain things in the game that's not in the test project though)
     
    Peter77 likes this.
  7. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    If you target VR, it's about 5% of your total frame time. :eek:
     
    jrumps likes this.
  8. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,221
    Is there a specific reason why Unity is getting consistently slower over time when the features you are using remain the same?

    I guess I ask because someone will at some point.
     
  9. Chaz32621

    Chaz32621

    Joined:
    Apr 17, 2015
    Posts:
    198
    Probably all the added features, aka everything running backend
     
  10. AlteredPlanet

    AlteredPlanet

    Joined:
    Aug 12, 2013
    Posts:
    411
    thank you for making these charts, can you do tests with vulkan
     
    Peter77 likes this.
  11. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    Unfortunately I can't, I only have a Windows PC.
     
  12. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,195
    The audio in newer version is definitely glitching and sounds like buffer underrun, like Unity cost more spec but doesn't update the buffer size, so Unity can't keep up with the old buffer size anymore even in editor. (If you go set something not Best Latency it will improve, at the cost of latency)

    Here's my case where UT said just change to something not "Best Latency" : https://fogbugz.unity3d.com/default.asp?1118164_ip87hse66aelc9od but the point is, it was working under Best Latency before the recent version. It is a regression.

    Unfortunately I don't know how to benchmark the "buffer underrun" other than using my ears. If that is possible programmatically (you may have to ask audio driver that it found nothing to process and send to speaker.. but how?) we can dynamically adjust buffer size and restart audio system until it doesn't crack.
     
    jrumps likes this.
  13. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,221
    Please make your own topic, don't drag thread off topic.
     
    Peter77 likes this.
  14. rizu

    rizu

    Joined:
    Oct 8, 2013
    Posts:
    1,017
    Vulkan runs on Windows.
     
    Peter77 and AlteredPlanet like this.
  15. AlteredPlanet

    AlteredPlanet

    Joined:
    Aug 12, 2013
    Posts:
    411
    please
    it runs on windows, please do a test?
     
  16. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    Holy moly, will do a Vulkan test at the weekend then!

    Do you know if I can start the build/player with a command-line to select a specific graphics api (Direct3D11, Vulkan, etc)?
     
    AlteredPlanet likes this.
  17. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    423
    AlteredPlanet and Peter77 like this.
  18. rizu

    rizu

    Joined:
    Oct 8, 2013
    Posts:
    1,017
    or just remove the automatic graphics setting for windows on player settings and force it to use vulkan there :)
     
    Peter77 likes this.
  19. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    I will give this a shot. I thought they're for the editor only. We'll see :)

    Yeah I'm aware of this option. I was trying to find a way to build the player once only and then test all graphics api's, rather than having to build a new player for each test.
     
    AlteredPlanet likes this.
  20. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    Here are the tests for the different graphics api's in Unity 2019.1.0b4. I tested D3D11, D3D12 (experimental) and Vulkan.

    I've to admit that the game is not really graphics demanding. It's quite simple what's going on visually and it has only a few draw-calls. There are most likely other projects better suited for a graphics performance test than mine.

    Anyway, here are the results.

    4_3.png

    5_4.png

    6_8.png

    Code (CSharp):
    1. ------------------
    2. System Information
    3. ------------------
    4.          Operating System: Windows 10 Home 64-bit (10.0, Build 17134) (17134.rs4_release.180410-1804)
    5.       System Manufacturer: System manufacturer
    6.              System Model: P5K/EPU
    7.                      BIOS: BIOS Date: 06/19/08 17:02:36 Ver: 08.00.12 (type: BIOS)
    8.                 Processor: Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz (4 CPUs), ~2.7GHz
    9.                    Memory: 8192MB RAM
    10.           DirectX Version: DirectX 12
    11.  
    12. ---------------
    13. Display Devices
    14. ---------------
    15.            Card name: NVIDIA GeForce GTX 650 Ti
    16. Driver File Version: 25.21.0014.1917 (English)
    17.       Driver Version: 25.21.14.1917
    18.          DDI Version: 12
     
    5argon, AlteredPlanet and SugoiDev like this.
  21. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,221
    Make own topic for discussions among yourselves, this is feedback for Unity.

    Edit: was responding to some posts I deleted where users wanted to drag this topic off track for their own ends, hence the reminder that people keep noise low for Unity staff to appreciate the data in feedback posts.
     
    Last edited: Mar 11, 2019
    Peter77 likes this.
  22. TokyoWarfareProject

    TokyoWarfareProject

    Joined:
    Jun 20, 2018
    Posts:
    284
    This is a great thread, thank you. Unity should seriously check if they do reproduce theese and seriously take performance prior to fancy ui reworks etc.
     
  23. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,195
    Imagine if we have a special CI server service which we could upload any Unity project on it. Each user made their own project containing things they are concerned about regression. That project then include one special package which works with that CI server + contains reporting API. After the upload, CI server will run against all versions and output the graph. Maybe that server could automatically distribute builds to Firebase Test Lab devices. I would pay for that.
     
  24. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    I wish Unity staff would leave a note if this thread has been seen and if it's being investigated or if I have to submit issues for that.
     
  25. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    423
    always submit issue. that "forces" them to look at it. then you can link to this thread.
     
    TokyoWarfareProject likes this.
  26. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    719
    Sad that that is the state of things.
     
  27. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,174
    Loaded up our game which is rather large and uses almost every major feature in Unity. it's a consistent drop of 8-10fps (vsync off) from the latest 2018.3. I couldn't notice anything that really stood out in the profiler it just seems to be across the board.
     
  28. GameDevCouple_I

    GameDevCouple_I

    Joined:
    Oct 5, 2013
    Posts:
    1,539

    Good god, do you or anyone else know why D3D12 is so slow compared? I thought direct x 12 was touted to be super performant compared to previous versions, or at least thats what I was led to believe whey they were hyping it up before release?

    edit: @Peter77 have you or anyone else here submitted an issue for this? This really needs to be submit as an issue for the team to look at ASAP and you seem best placed due to all the data you have acrued
     
  29. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    I have not. I tried to reproduce the D3D12 performance issue in a test-project, but wasn't able to make D3D12 run slower than D3D11 and then didn't look further into it.

    Unity Technologies have my project (Case 1108597) and instructions to run these tests themselves. If they're interested in doing that, they can. It should be reproducible with this project when running on slow(er) hardware.
     
  30. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    1,411
    It has been seen and I'm optimistic that the project will get investigated. I'm expecting to hear back about it in the next couple of weeks. Thanks for keeping the ball rolling.
     
  31. TokyoWarfareProject

    TokyoWarfareProject

    Joined:
    Jun 20, 2018
    Posts:
    284
    may be since the tested beta was b4 can we get an updated benchmark on current¿?
     
  32. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    Unity 2019.1.0b9 is slightly faster than 2019.1.0b4, but still slower than Unity 2018.1.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
    TokyoWarfareProject likes this.
  33. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    And here is a Graphics API comparison with 2019.1.0b9.

    scene_4_3_gfx.png

    scene_5_4_gfx.png

    scene_6_8_gfx.png
     
  34. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    Here is a .NET3.5 vs .NET4.x (mono) comparison with 2019.1.0b9.

    .NET 4.x is slightly slower than the now deprecated .NET3.5 Scripting Backend. Do you plan to get .NET 4.x as fast as 3.5 before actually removing it?

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
    Last edited: Mar 31, 2019
  35. TokyoWarfareProject

    TokyoWarfareProject

    Joined:
    Jun 20, 2018
    Posts:
    284
    thanks a lot mate!!! at least there is hope seeing this going towards less worse. But still what a dissapointment huh, hope they don´t release until everything is fully fully polished. 2018.3 never again please
     
    Peter77 likes this.
  36. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    28
    Are you using il2cpp or mono in your test builds?
     
  37. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    This test was using mono.
     
  38. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    28
    As far as I understand, il2cpp can have substantially better performance (in some situations) than mono, and it works on more platforms and seems to be the universal intermediate target that Unity is favoring over mono. I'd probably test with il2cpp instead of mono, especially for things like the .NET upgrade.
     
  39. Vincenzo

    Vincenzo

    Joined:
    Feb 29, 2012
    Posts:
    45
    il2cpp is still error prone, and not nessisarily faster, besides you can't build to il2cpp for linux soo.. I would hold the boat on il2cpp till it's more stable. testing mono makes a lot of sence. and i'm quite shocked mono 4x is slower than 3.5
     
    dadude123 and nxrighthere like this.
  40. GameDevCouple_I

    GameDevCouple_I

    Joined:
    Oct 5, 2013
    Posts:
    1,539
    Do you have any evidence to back up the claims of il2cpp being "error prone and not faster"?

    Nothing in any of our projects has shown this, we have always found the complete opposite.
     
  41. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    I tested this a while ago, see here:
    https://forum.unity.com/threads/performance-overview.520116/#post-3411457

    I'll see if I make an il2cpp comparison. This whole thing is just time consuming, since I've to rebuild "everything" and re-run the tests. I'll probably do this at the upcoming weekend.
     
    TokyoWarfareProject and LaneFox like this.
  42. LaneFox

    LaneFox

    Joined:
    Jun 29, 2011
    Posts:
    6,433
    Really interesting thanks for sharing the data.
     
    Peter77 likes this.
  43. Vincenzo

    Vincenzo

    Joined:
    Feb 29, 2012
    Posts:
    45
    @GameDevCouple_I Every single Unity patch release comes with more il2cpp fixes because it breaks on random things, especially with funnily enough manged to native interop. and other such cases, we simply cannot switch to il2cpp for this reason.
    When testing though it is quite faster on float math, because mono is using doubles for everything. something that got fixed in mono, but never ported to unity, (come on guys its only a single flag!!!)
    https://tirania.org/blog/archive/2018/Apr-11.html

    Anyway.. Il2cpp is a code gen, and a horrible one at that, have you even seen what it generates? the most simple C# code becomes a giant mess of 1000's of cpp lines.. even the best compiler cannot make it much faster, that's why most code under il2cpp is only marginally faster.

    The only reason of any speed up at all is that il2cpp overcomes the big problem of unity, that there is a LOT of managed to native interop overhead for every call to the engine from your c# code.
     
  44. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    462
    Performance becomes worse because while Unity slowly fixing/replacing/involving some stuff to the upgraded Mono, they don't bring any modern Jit intrinsics into it with the most noticeable performance-related changes.
     
    dadude123 and Vincenzo like this.
  45. dadude123

    dadude123

    Joined:
    Feb 26, 2014
    Posts:
    784
    Personally I don't find IL2CPP very relevant because it's missing features.

    It doesn't support Linq.Expressions or runtime code generation and so it's missing an absolutely essential feature.
    There is an interpreted mode now for expressions, but it doesn't help at all because it's slower.
    Linq.Expressions/runtime code gen is used specifically to make code faster, and an interpreted version kinda (totally) defeats the point :p

    Networking, serialization, saving/loading games, ... code gen is embedded into tons of systems, and is just way too useful to be missing (or crippled through low performance).
     
    nxrighthere likes this.
  46. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    423
    you can codegen at build time instead. that will be even faster than runtime
    also some platforms (iOS, consoles, ...) do not support it at all
     
  47. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    462
    Personally, while I one of those who was waiting for IL2CPP on desktop platforms, I'm not using it today primarily due to long compilation time which means it impossible to make changes on the fly and quickly validate a particular behavior, suboptimal code that it produces which results in crashes in worst case scenarios, and unexpected failures in managed-native interop code (I encountered at least two, both was fixed by @JoshPeterson ;)).
     
    Last edited: Apr 3, 2019
  48. dadude123

    dadude123

    Joined:
    Feb 26, 2014
    Posts:
    784
    maybe for stuff that can be done in advance...

    I have an ingame script editor that absolutely needs to load/compile IL code...
     
  49. vickycl

    vickycl

    Unity Technologies

    Joined:
    Aug 23, 2018
    Posts:
    2
    Thanks for sending over this project, @Peter77 . We are looking into the regressions between versions that you reported, and have reproduced it locally. Unfortunately the regression doesn't occur on all platforms/hardware, which has slowed down identification of the root cause. From initial profiling, a large part of the regression seems to originate in the UI of the minimap, and we are continuing to investigate.
     
  50. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,601
    Thank you very much for looking at it and keeping us in the loop. I've also created an Unity 2019.2 Performance Overview thread, in case you're interested (see here).
     
    Chaz32621 and vickycl like this.