Search Unity

Feedback Performance Overview

Discussion in '2019.3 Beta' started by Peter77, Aug 27, 2019.

  1. Ethan_VisualVocal

    Ethan_VisualVocal

    Joined:
    Mar 23, 2016
    Posts:
    161
    What I don't understand is:
    Why do @Peter77 's quarterly performance overview posts seem to come as "new information" to Unity?

    Shouldn't whatever he's using have long since been integrated into automated nightly tests that help bisect the code and identify the sources of perf. regressions?

    I assume Unity has tests LIKE this, and maybe what Peter77 is running is catching something they are overlooking, whether it be due to the test environment, or some other factor? Or are the regressions known, and an accepted cost of other improvements or features?

    I DON'T mean for this post to be a dump on Unity, nor to trivialize what is obviously a very complex problem, I'm just truly confused here and looking to improve my understanding of the situation.
     
    Lorash, Cynicat, phobos2077 and 7 others like this.
  2. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,426
    Today I re-ran the performance test with 2019.3.0f1, which is Release Candidate 1. I profiled all three builds listed in the graphs below during the same session.

    2019.3.0f1 continues to be the slowest Unity release since 2018 with my project.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
  3. Roni92pl

    Roni92pl

    Joined:
    Jun 2, 2015
    Posts:
    310
    Nice. Did you checked if there is something specific that is getting slower in profiler or it is more general slowdown?
     
  4. interpol_kun

    interpol_kun

    Joined:
    Jul 28, 2016
    Posts:
    124
    I didn't catch up with Unity 4.X, but I started with 5.X and was always sticking to the recent version. The first time I noticed the performance problems was the 2018.1, then I started using SRP's and found out that performance (both editor and mobile runtime) became far worse than it was before.

    Every other version some new stuff started to pop-up. Slower entering play mode (it was always fine for me and my medium mobile projects), then script reloading times, asset importing, editor launch times, creating new resources, compilation times, etc.

    At first, I thought it was because of my dying hard drive, but I checked it and it's quite healthy. And I don't see any regression in games and other software.

    So, from my subjective experience, I noticed a few (at least this is what I noticed) significant changes which made Unity (both Editor and Runtime) slower:
    • Introduction of SRP
    • Package Manager and everything related to that
    They showed an Editor performance improvements that will come in 2020.X cycle, so maybe they'll compensate a bit.

    But anyway, I can't understand the situation here. Can someone clarify?
    Unity had performance regressions through 2017-2019 which were reported not only by @Peter77 (who did the best job) but by other users too. Okay, forums are like hit or miss, maybe you didn't notice (but it's obvious you did, you neglect it). So what about your QA, automatic tests, performance reports? Sorry to ask, but do you even have them? I know that you guys always saying that all performance regressions should be treated as bugs, but you do nothing.

    Ask anyone, you can even create a survey, about overall editor and runtime performance experience. Lots of people switched their projects from 5 or 4.X to 2018 LTS and they constantly crying about their FPS.

    I just don't understand. You have so many people talking about engine became slower through the past few years and all you can do is telling us to bug report that cause you don't see any regressions at all. How could you not notice that?! And I am not even talking about your "production-ready features" like LWRP. It was PrR since 2019.1, but even with 2019.2, you can catch a few serious bugs in the first day of your new project.

    I am really tired of that releases full of bugs, regressions and half-baked features. I sound like an old granddad who is tired of everything, but I really can't cope with that stuff. I don't enjoy Unity anymore. It used to be like that: I got an idea, created a project and launched the editor (like in a minute) and started my stutter-free development with fast iteration cycle. Now it's like open hub, create a new project and wait 4-6 minutes till all the stuff is imported and the editor is ready and you can start your semi-slow development.

    I recall someone from Unity said you have a goal to reduce script reload times to 0.5s (was that Joachim?) and you will have a new compiler do the job. Compiler did nothing, because times increased. 0.5s was an important number for a fast iteration workflow (and it's not my words, it's yours), but you failed with that goal and in the end, you gave us some hacky workaround which will literally disable the stuff you failed to optimise.

    I do really enjoy all the DOTS stuff (even considering I won't use any of ECS stuff except for educational purpose), but it can't come for the price of everything else. You know that a lot of people are not switching solely because of C#, not because they are happy with the recent Unity releases (me included, but I am already one foot UE4).

    Unity has to evolve in every part. You say it in every hype blogpost, but it's not true.

    This is one of my longest posts on the forum yet. And I wouldn't bother at all if I didn't care about Unity. But I do care, and sometimes I think that it's the dead end.
     
  5. creat327

    creat327

    Joined:
    Mar 19, 2009
    Posts:
    1,183
    I use 2020.1alphas. The performance is exactly the same if not worse than 2019.3 and Unity hasn't mentioned they are looking at it or anything. Basically they hope that you will rewrite all your existing apps and code to use their new DOTS system on the premise that that's the only way to get speed improvements. At least that's the feeling I'm getting. If you are able to keep using Unity 5... do so. Anything after that is a drastic slowdown on mobile and editor.
     
    joshcamas and laurentlavigne like this.
  6. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,426
    Hi Will,

    nobody reached out to me yet, which is fine if you don't need more info. If however you can only fix these issues with more details from my side, please let me know, so I can help.
     
    AskCarol, willgoldstone and Prodigga like this.
  7. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    2,335
    Oh man, same! I thought I was going nuts! 2019.3 is much faster with tons of objects onscreen though, faster than 4.7 at least as I haven't touched 5 but the editor feels sluggish with tons of hiccups, some times an orbit becomes a select for no reason and I think that's related.

    Given the global trend towards electron slugware, decay isn't surprising, what's surprising is the complete silence of @Joachim_Ante because he cares about performance.
     
  8. JohnC_Unity

    JohnC_Unity

    Unity Technologies

    Joined:
    Jun 7, 2017
    Posts:
    62
    Hey Peter,

    My team has been using your project for testing. I'm not sure how much I can share on the forums, but I will drop you a report sometime next week. I'll outline how the project is used, what new procedures it has triggered and the issues it's helped to find.

    I'll also let you know how we plan to use it for future testing.

    I want to say thanks again for sending it in, it is being used. If anyone else has performance regressions. Please log your bugs and send the files if possible, they will be taken seriously.
     
  9. willgoldstone

    willgoldstone

    Unity Technologies

    Joined:
    Oct 2, 2006
    Posts:
    738
    Hey Peter, sorry I didn't reply - i'm in same office as @JohnC_Unity and will catch up with him next week. Thanks so much for your work on this!
     
  10. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,426
    Hi John,

    thanks for your reply! I'm looking forward to the report and any improvements you came up with in the meantime.
     
    phobos2077 and JohnC_Unity like this.
  11. benperry

    benperry

    Joined:
    Nov 9, 2015
    Posts:
    2
    I've got another performance regression to add.
    IJobParallelFor job scheduling is completely broken for me in 2019.3.0f1.
    When updating from 2019.2.14f1 to Unity 2019.3.0f1, my PlayerLoop (in editor) went from ~4ms to ~10ms!!!
    IJobParallelFor jobs seem to be stuck running ONLY on the main thread most of the time... blocking main thread when it could be doing work. This is true in both editor and player.

    2019.2.14f1 profiler results:
    2019.2.14f1 perf.PNG
    2019.3.0f1 profiler results:
    2019.3.0f1 perf.PNG

    As a result, 2019.3 is completely unsusable if I want to ship a performant product.

    I've submitted a bug (1202317) with a small repro project.

    My guess it is a regression caused by this change in the 2019.3.0f1 changelog:
    • Kernel: Improve main thread utilisation for IJobParallelFor jobs
     
    joshcamas, Cynicat, Lynxed and 6 others like this.
  12. Freznosis

    Freznosis

    Joined:
    Jul 16, 2014
    Posts:
    217
    I remember how bad the performance was for Unity 2017. I tried it out and it was unbearable. You can still go back and see the crazy amount of threads about how much performance decreased. I think I was stuck using Unity 5 until 2017.4 had a couple of patch releases because it took them 4 cycles to finally fix the engine. And then when 2018.1 came out they did it again! I only recently switched to 2018.4, but it seems they are doing the same pattern over and over. So I think it's safe to say I won't be using 2019 until 2019.4 patch 11 is released. And by then, they will be pushing 2020.3 very hard. It's a never ending cycle, but since they won't bring a lot of improvements and features to older releases, you are stuck in between better performance and benefiting from new features while getting slow downs.

    I really hope Unity take a very hard look at where the engine is and where it should be by now. They weren't as big of a company before, so it was understandable to have some problems here and there. But in the whole time I've used Unity from 2 to Unity 5, this was never a problem. I wonder what changed starting with Unity 2017? It seems they are trying to do way too much, too fast. They have introduced several new systems, features, and workflows while the rest of the engine is still in a slump.

    I also agree with what was said above. It gets pretty annoying that most of the responses to horrible performance is "submit a bug report." Even if it's a completely new, blank, empty project. I understand it, because as developers we have to go through the same thing with people reporting stuff to us without any information to help us track down the problem. But if they can't track down this slow down after literally years and multiple life cycles of the engine since 2017. Then I don't know how much faith I can continue having in Unity going forward.
     
  13. Charles_Beauchemin

    Charles_Beauchemin

    Unity Technologies

    Joined:
    Jan 18, 2017
    Posts:
    370
    Hi Ben, would you mind trying the last beta version to see if it was indeed a regression introduced in f1?
     
    Lynxed, phobos2077 and Thriving like this.
  14. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,525
    This is a feature package (terrain tools), but something that absolutely shouldn't be happening off in individual features, this is a huge hit on load time iterating through all assemblies and types. Don't you have a single place in the loading cycle to do this that packages can hook into?

    NoiseLib in the terrain tools package, GatherNoiseTypes.
     
  15. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    2,335
    How is compile time when you change a script?
     
  16. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,426
    My very non-scientific approach to measure it was like:
    1. Hit "Reimport" on a script and start stopwatch on my cellphone at the same time
    2. Place mouse pointer over a window splitter in Unity, so it changes from "arrow" to "east-west" icon
    3. Wait until Unity freezes (domain reload) and move mouse pointer somewhere else
    4. Wait until mouse pointer changes from east-west to arrow again (domain reload done, Unity responsive again)
    5. Stop stopwatch

    Code (CSharp):
    1. 2017.4.34f1  11s
    2. 2018.1.9f2   11s
    3. 2018.4.12f1  13s
    4. 2019.3.0f4   9s
    2019.3 compiles quite fast, most of the time seems to be spent on the domain reload. From my observation I would say perhaps 4s compiling (rotating spinning icon), 5s domain reload (Unity frozen).
     
    Ethan_VisualVocal likes this.
  17. JohnC_Unity

    JohnC_Unity

    Unity Technologies

    Joined:
    Jun 7, 2017
    Posts:
    62
    Peter77 and Lars-Steenhoff like this.
  18. FcsVorfeed

    FcsVorfeed

    Joined:
    Aug 15, 2014
    Posts:
    30
    We don't want any "Enter Play Mode", because i have to change all of my 100000000+ scripts ,if not? when i "Enter Play", all of system will print errorBUG

    We just want click "enter play" and every thing is like Unity 5 fast

    If you really tell me have to change 100000000000000000+ script , i have a question:
    "Have you ever really developed a game? "
     
  19. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,426
    Thanks for the tip! I believe this new feature isn't going to help when compiling scripts, as in order to use the new script code, unity is doing the assembly/domain reload per definition. But I'm no expert on this, but @alexeyzakharov can most likely answer it.
     
  20. JohnC_Unity

    JohnC_Unity

    Unity Technologies

    Joined:
    Jun 7, 2017
    Posts:
    62
    Following up on the status of this issue.
    I looked into the bug it seems they have found where it regressed. Not sure on the status of a fix, but I'll find out and keep you posted.

    Many thanks.
     
  21. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    2,335
    Also the caveats of domain reload are substantial and require a lot of verbose code.
    I see that you folks already considered whiping the static variables, I skimmed over the reasoning, maybe it'll work in 50% of all cases and enough for us?
    Pretty nice, it seems slower than 4.7 which is the only version that etched itself on my mind.
     
    Last edited: Dec 20, 2019
  22. Ethan_VisualVocal

    Ethan_VisualVocal

    Joined:
    Mar 23, 2016
    Posts:
    161
    Is "domain reload" an all or nothing thing for the Unity "Enter Play" step? Or can it be limited to specific assemblies?

    For example, if I have my code broken up evenly between 10 assemblies (via asmdef files), and I adopt these manual reset-statics patterns, then change a script in 1 assembly:
    1. Do all of my 10 assemblies need to be reloaded? (along with various Unity assemblies)
      OR
    2. Can just the 1 assembly that got recompiled also be the only assembly needing a "domain reload"?
    (Thanks in advance!)
     
    hurleybird and Peter77 like this.
  23. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,426
    Ethan_VisualVocal likes this.
  24. rastlin

    rastlin

    Joined:
    Jun 5, 2017
    Posts:
    123
    In general, you cannot EVER unload assembly from app domain. Only way to replace the in-memory assembly is to dispose and start new AppDomain.
     
  25. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,426
    Today I re-ran the performance test with 2019.3.0f5, which is Release Candidate 5. I profiled all three builds listed in the graphs below during the same session.

    2019.3.0f5 continues to be the slowest Unity release since 2018 with my project.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
    mGMM, Griz, arkogelul and 13 others like this.
  26. spaceemotion

    spaceemotion

    Joined:
    Sep 29, 2015
    Posts:
    77
    Would love to see if the jump between f3, f4 and f5 did anything. You'e only listed f5 in your charts.
     
  27. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,426
    I've uninstalled f3 and f4 already, but I did profile f4 and have the results! :) No results for f3 though.
    scene_4_3.png

    scene_5_4.png

    scene_6_8.png

    I didn't post the f4 results on purpose, because it was during the holidays season. I thought it would be unfair to post it, if Unity staff would be on holidays and can't respond/comment. I don't know, felt wrong.
     
unityunity