Search Unity

  1. Curious about what's going to be in 2020.1? Have a look at the 2020.1 beta blog post.
    Dismiss Notice
  2. Want to see 2020.1b in action? Sign up for our Beta 2020.1 Overview Webinar on April 20th for a live presentation from our evangelists and a Q&A session with guests from R&D.
    Dismiss Notice
  3. Interested in giving us feedback? Join our online research interviews on a broad range of topics and share your insights with us.
    Dismiss Notice
  4. New Unity Live Help updates. Check them out here!

    Dismiss Notice

Feedback Unity 2020.1 Performance Overview

Discussion in '2020.1 Beta' started by Peter77, Mar 12, 2020.

  1. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Hello lovely people who like to look at performance graphs :)

    Today I ran the performance tests of my game with Unity 2020.1.0b1 and 2018.1.9f2. I present the results below, just like I did before, during older beta cycles.

    According to my test, 2020.1.0b1 is slower than 2018.1, which is still the fastest Unity version with my project. I profiled all builds listed in the graphs below during the same session.

    You can run the test using the project attached to the following bug-report:
    (Case 1108597) Project to find performance regressions and bugs

    The (first-person) game I'm working on features an automated performance test. It works like this:
    • Camera is placed at a defined position in the scene
    • Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
    • Every second the test captures the average CPU frame time of the last second
    The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

    It measures the "base-line" of the game so to speak. If an actual gamer plays the game, more things are going to happen, which is missing in the test, such as visual and sound effects when firing weapons, additional AI to hunt the player and more navigation pathing.

    I run this test in a Standalone Windows 64bit (Mono) Player:
    • 2018 = .NET3.5
    • 2020 = .NET4.x
    I've also enabled "Incremental GC" on both 2020 versions.

    The following Physics settings are applied:
    • Physics.autoSyncTransforms=false
    • Physics.autoSimulate=true
    • Physics2D.autoSyncTransforms=false
    • Physics2D.autoSimulate=false
    The resolution is set to 320x240 (D3D11) to make sure the bottleneck isn't GPU.

    The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represents the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test. Fewer milliseconds (vertical axis) indicate better performance.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png


    These issues are probably not reproducible if you're running the game on high-end hardware. Please use a rig similar to the hardware I used to submit the bug-report.

    Please also process bug-report: (Case 1222370) 2019.3: Android performance regression
    https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5577082
     
  2. JoNax97

    JoNax97

    Joined:
    Feb 4, 2016
    Posts:
    128
    At this point I have to ask: Has Unity approached you to work in their QA team? The work you do is awesome. Thank you very much
     
  3. valentingurkov

    valentingurkov

    Joined:
    Jun 17, 2019
    Posts:
    10
    I am curious about your experience with 2020 beta and 2019.3. if you have any?
     
  4. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    LOL... they promoted me to QA Jesus, so in that regard they work for me, right?! If I work there, they would lose their most annoying customer. I can't let this happen. My mission is to annoy the hell out of them (pun intended)!

    Thanks for the kind words, much appreciated!
     
  5. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    I've no experience with 2020.1 other than the performance test you can see above.

    I've little experience with 2019.3. I briefly used it at home and at work, in both cases for rather simple projects. I can't think of any major bug, of the top of my head, that was so bad to not use it.

    Android performance is really really bad with my project compared to 2018.4 though:
    https://forum.unity.com/threads/unity-2019-3-performance-overview.735185/page-2#post-5581939
    ... which is the reason why I decided to stay with 2018.4 at work.
     
    jasonatkaruna likes this.
  6. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,457
    upload_2020-3-13_15-35-32.jpeg

    Though I would love to know WHY it's not as fast as 2018. Is it just scripting or? what part is slow?

    1ms is actually a big deal for the same game, same hardware, with just Unity changed. I mean it's a pretty large loss when you're doing VR or consoles.

    If I knew what caused it, I'd be able to work around it.

    If it's CPU side then it probably makes URP look uglier than it is performance wise.
     
  7. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Damn, I should make this my new profile picture!

    Great question! So... I was just profiling scene_6_8 at sample 2 with Unity 2018.1.9f2 and 2020.1.0b1 to make a more detailed comparison and pulled the data in Profile Analyzer. It didn't make it easier that in 2018.1.9f2 you can't load profiler snapshots.

    Having the Unity Profiler connected, makes the game actually run faster with 2020.1 :eek:

    Looking at the data, "Profiler" related methods in 2020 have almost 0 cost, where as in 2018, it's significantly more. Which is the reason why it runs faster in 2020.1 if the profiler is connected.

    Great job people at Unity to get rid of the profiler overhead! Now make the remaining parts to have a 0 cost as well please ;)

    Here is a comparison of the profiler cost first. Keep this in mind if you look at the absolute frame time of the game later, because the profiler cost wouldn't exist in the game unless, well, you're profiling, therefore you'd need to subtract the profiler cost from the total frame-time.

    upload_2020-3-13_18-14-18.png

    upload_2020-3-13_18-14-43.png

    upload_2020-3-13_18-16-24.png

    upload_2020-3-13_18-16-42.png

    The Profiler Analyzer versions are different, that's why the window displays different stuff. The spikes in the middle come from me alt-tabbing from the game to Unity.

    Looking at this data, I would say:
    • Script execution seems to be overall slower in 2020.1
    • Physics seems to be faster in 2020.1
    • Rendering seems to be slightly slower in 2020.1
    • Profiling overhead seems to be non-existent in 2020.1
    I was profiling windows standalone builds btw.

    I hope this makes sense!
     
    Last edited: Mar 13, 2020
  8. tessiof

    tessiof

    Joined:
    Dec 6, 2017
    Posts:
    14
    Please add 2019.3 to the graph, so we have a comparison to all current unity versions.
     
  9. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Last edited: Mar 14, 2020
  10. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    409
    Thanks ;)

    In 2018.3 we added a check for each profiler module to only collect stats if the chart of the module is active. If you want to recreate a similar overhead you'd need to add all modules in the Profiler or call Profiler.SetAreaEnabled() on each of them.
     
    konsic and Peter77 like this.
  11. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    548
    @Unity.
    Please add an official Benchmark to Unity who is available for everyone. The Blender Foundation is Benchmark here.
    see
    https://opendata.blender.org/

    @Peter77
    Could you share the script to your benchmark.
    When Unity did not has the ressources i will set up such an webpage.)
     
    Last edited: Mar 15, 2020
    konsic likes this.
  12. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Thanks for the info! That makes sense, because I had all profilers but the CPU profiler disabled.
     
    MartinTilo likes this.
  13. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Unfortunately there is no benchmark script. The "benchmark" is a first person shooter game.
     
    keeponshading likes this.
  14. transat

    transat

    Joined:
    May 5, 2018
    Posts:
    509
    Hey! I thought that was me! :)

    But staying on topic... And to play Devil's advocate ;) ... I'm wondering how much the benchmarks actually tell performance wise? For example, it could be that for a simple game, 2018.1 beats out 2020.1 but perhaps the more demanding the game becomes the worse 2018.1 fares compared to 2020.1 ? Looking at the graphs in your first post I'm noticing that when the ms increase on 2018.1, the increase on 2020.1 appears slightly less. Also I'm assuming there could be quality differences when it comes to things like shadows?
     
    Peter77 likes this.
  15. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Could be, I don't know. I just see that projects at hand regressed performance and that's alarming for me.

    Quality-settings wise they're identical. The resolution is set to 320x240 during these tests, to make sure the bottleneck isn't GPU. If Unity change the shadows implementation in the meantime, maybe it's related to that. But I'm pretty sure the issue isn't on the GPU rendering side, because the content is quite optimized.
     
    Last edited: Mar 16, 2020
    konsic likes this.
  16. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Today I re-ran the performance test with 2020.1.0b2 and of course 2018.1.9f1, which has been the fastest Unity build for me, thus I use it for performance comparison. I profiled all builds listed in the graphs below during the same session.

    You probably notice the graphs changed! The graphs are a bit more zoomed in, but everything is also about 0.4ms faster now.

    Why is that you ask?

    The reason why everything is about 0.4ms faster now is that earlier tests had the "Script Debugging" build option enabled. I've now turned it off, for this and upcoming tests, because the script debugging cost won't exist in the final build, thus I don't care about it.

    All earlier tests are still valid and continue to show regressions, they're not wrong, they just include a cost that's irrelevant for the final release that won't be included in the next tests.

    2020.1.0b2 continues to be slower than Unity 2018.1.9f1 with my project.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
  17. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,457
    I'm not aware of Unity changing the builtin renderer from 2018 to present day. They froze that when they started on SRP, so if anything things should have got faster, not slower over time, especially with redundant features being moved out to packages and even less pointless overheads and at that resolution, I don't expect it to be a GPU issue.

    It might be an issue if realtime GI is used though. That got fiddled with. Probably not that if it's removed in 2020 though. Hmmm....
     
  18. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Today I re-ran the performance test with 2020.1.0b4 and 2018.1.9f1. I profiled all builds listed in the graphs below during the same session.

    Performance seems to be the same as in earlier 2020 builds. The graphs look a little different to the previous ones, because I adjusted the vertical-range.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
    hippocoder and valarus like this.
  19. DoctorShinobi

    DoctorShinobi

    Joined:
    Oct 5, 2012
    Posts:
    126
    You swore an oath that you shall not rest until these two lines are one
     
    hippocoder and Peter77 like this.
  20. valarus

    valarus

    Joined:
    Apr 7, 2019
    Posts:
    39
    Let the red one be many msec below the blue one.
     
    Peter77 likes this.
  21. KokkuHub

    KokkuHub

    Joined:
    Feb 15, 2018
    Posts:
    150
  22. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,382
    Graphics tests are very much alive and kicking, but have nothing to do with performance testing - they are functional tests.

    We are working hard to bring our automated performance up to the same level as our functional testing, so we can prevent this constant “rot” of performance between versions. (Instead of always trying to play catch-up by fixing the performance regressions after they have gone into releases)
     
    Peter77, KokkuHub and valarus like this.
  23. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,382
    They are hugely valuable to us - they cover a large amount of use cases.

    But - you make a good point - there are also many more use cases they don’t cover. That’s always going to be the case in something as complex as Unity - one project can only ever hope to cover a small % of the total possible use cases.
     
unityunity