Search Unity

  1. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice

Overall Performance degredation in 2017.2

Discussion in '2017.2 Beta' started by Peter77, Sep 24, 2017.

  1. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,589
    The (first-person) game I'm working on features an automated performance test. It works like this:
    • Camera is placed at a defined position in the scene
    • Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)
    • Every second the test captures the average CPU frame time of the last second
    The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

    I ran this test with three different builds (Standalone Windows 64bit):
    • 2017.1.1p3
    • 2017.2.0f1 without any modifications
    • 2017.2.0f1 with Physics.autoSyncTransforms=false, Physics2D.autoSyncTransforms=false, Physics2D.autoSimulate=false

    2017.2 performs worse in every scene compared to 2017.1. Some scenes are a bit slower only, while others are significantly slower. I picked a few random results and exported the data as images, which you can find below.

    The y-axis represents the average CPU time in milliseconds, how long "one frame took". The x-axis represent the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png

    As shown in the last image "SCENE 6_8", 2017.2 is up to 10ms slower than 2017.1 in my game. Using the new Physics Optimizations makes it run faster, but it's still 2ms slower than 2017.1.

    Is anybody seeing similar performance issues? Is Unity Technologies aware of these performance issues?
     
    Last edited: Sep 24, 2017
  2. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    976
    I can't find the thread right now but the unity devs were responding to some questions regarding the autosync stuff. From what I was able to gather, by default unity had autosync on so it worked as it always had. (No need to manually sync, raycasts always hitting the most "up to date" versions of objects, etc). It sounded like this should just perform as well as it did in 2017.1. But, now with the property exposed, the user could further optimise their game by disabling autosync and performing it when they need to. This should make their game run better as the sync isn't performed as often as when it is set to true (the default value).

    What's confusing was that having autosync set to true was for some reason slower than 2017.1, and from your tests it seems even having it turned off isn't as good as 2017.1.

    Seems something is wrong somewhere? One would expect autosync true to run as well as it did in 2017.1. the average user would just upgrade their project from 2017.1 to .2, expect everything to run as it did before (and it will, because autosync is true by default) without any performance drops (which it doesn't, it is a lot slower). They shouldn't be expected to make use of a feature targeted at more advance users just to achieve a frame rate that they were hitting in earlier version of the engine (where they weren't doing anything special - just using the physics as is out of the box)
     
  3. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    4,873
    Did you also run gfx only test ?

    My test was in-game with full physics ai and post stack running HDR linear with a controller so it is not automated at all.
    5.6 was around 140-70 fps
    .1 was 120-50
    .2 was 70-20fps
    Big use of kinematic dynamic physics 2d with 3d rotation which forces a collider rebuild, didn't auto off
     
  4. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    Can you pinpoint which profiler tasks seem to take more time than before? Is there a lof of UI or physics in your test scene? Animated skinned meshes? Things that might have used GPU instancing or static/dynamic batching? Occlusion Culling? Are you using .Net 4.5? DX11 or 12? Graphics Jobs? GPU Skinning? These are all things that seem to be at-risk in 2017.2

    Comparing all the physics times between 2017.2 autosync off and 2017.1 might already give us interesting info. Maybe they are the same, and the perf difference is due to something else

    Another possibility is that things are more and more getting converted to be multithreaded by the job system, but since the job system isn't fully there yet, it just ends up being slower than before

    I think there is some confusion as to what autoSync on is. Basically, Unity needed to make "breaking" changes to the physics engine in order to make it ready for multithreading with the jobs system (and I am really proud of them for doing that). This change means that all transform syncs must now be deferred, so it pretty much changes everything, and there is no real "equivalent" to how things were before (neither with autoSync on or off). But AutoSync off is the closest equivalent to how things were before in terms of performance. AutoSync on is just a "compatibility mode" that should give you the same end results as before in 99% of cases, but is extremely inefficient. AutoSync off is the "real" way of using the physics engine now.

    But the problem now is that AutoSync off should at least be as efficient as before

    I know for sure that having autoSync on shouldn't even be considered an option. It should be removed from the engine completely, because the performance hit is just preposterous. I've tested some scenes that ran at 200 fps in 2017.1, but ran at 8 fps in 2017.2 with autoSync on. However, with autoSync off, I didn't really notice a performance difference (but it was a really simple scene, too. Just X amount of rigidbody cubes moving around)

    But I'm all in favor of the manual transform sync feature. I'm glad Unity chose to do this. What I'm not at all happy about is that they are going ahead with 2017.2's release when it clearly isn't ready yet
     
    Last edited: Sep 24, 2017
    WildStyle69 and Prodigga like this.
  5. iamarugin

    iamarugin

    Joined:
    Dec 17, 2014
    Posts:
    636
    I see similar problems too, even if autosyncTransform is false.
     
  6. WildStyle69

    WildStyle69

    Joined:
    Jul 20, 2016
    Posts:
    291
    Thanks Peter for this report and all for follow-ups -- data you are presenting basically reinforces the behavior I've been seeing. I was on 5.6 patch releases for some time and recently tried moving firstly to 2017.1 and now 2017.2. Main reason was the touted improvements for features and performance, but honestly things are going backwards right now.

    As per the data and discussions here things have gradually degraded in performance. 5.6 >> 2017.1 >> 2017.2. Things seem way less stable and am losing performance in many ways, as per recent bug reports. Not complaining, however it's a tad confusing and disheartening at times, particularly when we are looking at imminent release (again).

    In hindsight I should have stayed on 5.6 and perhaps will go back to that once more, until I see more evidence that the newer versions of Unity are stable and ready for real usage.
     
    laurentlavigne and MrEsquire like this.
  7. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,589
    I could do that, but I decided to take a break from looking at new Unity releases for now. It's too frustrating to see all the (optimization) effort I put into the project falling apart with pretty much every new Unity release. I was trying to keep the project up-to-date with Unity, because I was looking forward the Job- and Input-System.

    It's really astonishing how Unity gets slower with (almost) every new release, I can't get this into my head. It takes so much time and effort to raise these issues. I feel like I spend more time running into Unity issues than actually working on the project, that's not how it should be when using middleware. I also don't get anything in return.

    I'll check 2017.2 after a few patch releases if performance improved. However, from past experience, I know these type of performance issues often lay around a really long time before being fixed.
     
  8. Muuuuo

    Muuuuo

    Joined:
    Apr 16, 2015
    Posts:
    57
    Hey but at least we can login to Unity editor with facebook now
     
    Ruslank100, Freznosis, mh114 and 6 others like this.
  9. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    I've decided to give this a try, and I made my own performance test.

    Video: https://i.gyazo.com/0a290f007cb591d3604491169a7d30b5.mp4


    I call this test...... "The Adams Family"

    This first test is mostly just to benchmark rendering performance. The scene has:
    • The entire Adam Exterior Environment
    • Static meshes
    • Skinned meshes (all the Adams are doing a walking animation)
    • Particles
    • Foliage
    • Shadows
    • Many lights
    • Transparency
    • PostFX v2
    • Etc....
    The project settings are:
    • DX11
    • .Net 3.5
    • Linear color space
    • Static batching on
    • Dynamic batching off
    • GPU Skinning on
    • Graphics jobs on
    • Deferred rendering
    • Very high res shadows (distance shadowmask)
    • VSync off
    I tested the scene both in 2017.1p5 and 2017.2b11, with the exact same settings (except I disabled autoSyncTransforms in .2, even though I don't think there were any moving rigidbodies in this scene.)

    And the results are.... 2017.2 performs a tiny little bit better. About 15ms per frame in 2017.2 versus 16ms per frame in 2017.1 (in-editor). In a release build, both hang out at around 44-48 fps. Hard to tell which one is better.

    HOWEVER!!!
    I noticed something pretty crucial. When I first built the executable after converting to 2017.2, it ran at 32 fps. Then my spider senses told me to try and delete the lighting data (just realtime GI in this one) and recalculate lighting, and see how it goes. After I did that, the performance was back on par with 2017.1 (a bit superior actually). I remembered having this problem in the past. Whenever you convert projects to new versions, be sure to recalculate lighting (maybe even delete the lighting asset completely first. You can never be too sure).

    Later this week I'll do a physics-centered test and report back here
     
    Last edited: Sep 26, 2017
  10. rastlin

    rastlin

    Joined:
    Jun 5, 2017
    Posts:
    127
    Hi guys, just wanted to let you know how great of a job you do with those comparisons, many thanks for those!
     
  11. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    It would all be so much simpler if Unity released its own internal performance benchmarks with each major engine version
     
    Ruslank100 and laurentlavigne like this.
  12. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,899
    Not really because a lot of them are user projects or asset store projects, ie real world scenarios... They can't do those. I guess they can do whatever other benches they've made but I think we can too.

    I'm not sure there's a lot of value here since Unity does run through those benchmarks as far as I understand it, and it still got released. I can only assume they're wilfully skipping doing benchmarks sometimes or it'd get caught.

    Would be nice to get more info.
     
  13. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    Just did the physics test

    The test scene is 3600 interpolated rigidbody spheres freefalling into a box, and 400 interpolated kinematic rigidbody cubes that are moving, rotating, and raycasting on every fixed update.


    I call this test... "Eh.... that'll do it"

    Here are the results in a release build:
    • 2017.1p5: Goes down to 56 fps when all the spheres are landing in the box, then gradually goes back up to 105 fps
    • 2017.2b11 - AutoSync OFF: Goes down to 65 fps on initial impact, goes back up to 115 fps
    • 2017.2b11 - AutoSync ON: Goes down to 16 fps on initial impact, goes back up to 45 fps
    So, have no fear: the physics performance is fine in 2017.2 as long as you remain in AutoSync OFF mode, which is no big deal at all. In several cases you may not even need to ever call Physics.SyncTransforms() at all. The only things that aren't fine are:
    • AutoSync ON by default, which will no doubt cause massive hysteria and make thousands of devs wonder why their game suddenly runs like a console game
    • this problem, which - thank god - has been resurrected from its "Won't Fix" status and is now back to "Active"

    This doesn't confirm beyond any doubt that 2017.2 has no performance regression whatsoever, but it's still somewhat reassuring I think. The only way to find out what did regress now would be to compare profilers for the same project in .1 and .2

    _________________

    Also, I tried the Adams rendering test with DX12 and Vulkan. The performances were abysmal with DX12 (it ran at 25 fps), and it just didn't work at all with Vulkan (1 fps and everything is purple). Just stay away from those for now, and stick with DX11
     
    Last edited: Sep 27, 2017
  14. ceebeee

    ceebeee

    Joined:
    Mar 7, 2017
    Posts:
    397
    AutoSync has always been on by default. they are just now giving a way to turn it off for the first time. If you turn it off, you're going to have to create a replacement system to track when items need synced. So it's questionable if the performance gain you might get is worth the headache.
     
  15. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    That's a common misunderstanding. It really sounded like this was the case in the draft documentation on AutoSync, but it isn't. See the explanation here : https://forum.unity.com/threads/cas...sforms-performance-issue.487556/#post-3192995

    Basically, the closest thing we have to how things were before is AutoSync OFF. But overall it's a totally different system now, and there is no real equivalent
     
    hopeful likes this.
  16. ceebeee

    ceebeee

    Joined:
    Mar 7, 2017
    Posts:
    397
    Ok they really need to put his stuff from comment #9 into the Google Doc, because the way the Google Doc reads, it's the complete opposite. Very Confusing.

    Now I just wonder what constitutes a "Physics Query".
     
  17. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    Raycasts, Overlap tests, etc....
     
    ceebeee likes this.
  18. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    976
    @PhilSA - The way it is worded here:


    Sounds like AutoSync ON is the old behaviour... This crap is so confusing. I wish the devs would chime in again and clear up this confusion.

    In the old system, any 'Physics Query' was always up to date.

    For example...:

    Prior to AutoSync..:
    Code (CSharp):
    1. transform.position = new Vector3(100,0,0);
    2. Physics.OverlapSphere(new Vector3(100,0,0),0.5f); //true = we find our object at 100,0,0
    With AutoSync On
    Code (CSharp):
    1. transform.position = new Vector3(100,0,0);
    2. Physics.OverlapSphere(new Vector3(100,0,0),0.5f); //true = we find our object at 100,0,0.
    3. //Position is Auto synced to 100,0,0 prior to Physics Query
    With AutoSync Off
    Code (CSharp):
    1. transform.position = new Vector3(100,0,0);
    2. Physics.OverlapSphere(new Vector3(100,0,0),0.5f); //false, transform has not yet been synced,
    3. //Physics thinks object is still at 0,0,0
    Alternatively...

    Code (CSharp):
    1. transform.position = new Vector3(100,0,0);
    2. Physics.SyncTransforms();
    3. Physics.OverlapSphere(new Vector3(100,0,0),0.5f); //true = we find our object at 100,0,0.
    4. //Position has been manually synced
     
  19. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    I'll admit my "the closest thing we have to how things were before is AutoSync OFF" statement was poorly worded and needed a pretty big clarification.

    What I meant is that AutoSync ON is just a big heavy brute-force 'hack' that was added just for the sake of backwards compatibility, even though it pretty much divides your framerate by 4, and it's not at all representative of how the physics operated before. The engine didn't call a "sync" before every physics query in 2017.1. It just operated in a way where it didn't have to do that

    But it's true that it gives more or less the same end results as before. I say "more or less" because there are cases where it doesn't.

    If, with AutoSync ON and within a single frame, you:
    1. move a transform to a position
    2. do some physics operation that needed to see what would happen if that transform was there
    3. then move the transform back where it originally was
    4. and THEN move a rigidbody on the same GameObject as the transform by using MovePosition (for interpolation)
    ..... it'll result in no movement at all. I know it sounds like a really far-fetched use case that no one would really need, but I encountered a situation where I needed that (it has something to do with character controllers and moving platforms). And the solution ended up being to call a Physics.SyncTransforms() between steps 3 and 4 even though AutoSync was on, so that it would un-dirty the transform, and let MovePosition do its thing. ( here's an unintelligible post that sort-of talks about that )
     
    Last edited: Oct 5, 2017
  20. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    976
    Yep, and this was my concern.. I sure hope the performance benefits with AutoSync Off pays off because a majority of developers will never take advantage of a feature like this. For a majority of devs upgrading to 2017.2, I can see 2 outcomes (neither of which are good)
    • TrueSync On by default: Experience a huge performance regression - up to 4x worse performance, just because they upgraded ("If I downgrade to 2017.1 my scene is 4x faster! 2017.2 is broken!!")
    • TrueSync Off by default: Experience strange new bugs and quirks in their logic that worked fine before that they have no way of explaining ("If I downgrade to 2017.1, it just works! 2017.2 is broken!!")
    And if TrueSync doesn't end up offering huge performance benifits, I am having trouble understanding what the point of all this is.

    Edit:
    Just to be clear, I am happy Unity is trying to innovate and experiment with new things!
     
    Ruslank100, WildStyle69 and PhilSA like this.
  21. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    My opinion on this is that Unity should come up and say "from now on you all just gotta sync your things manually, 'cause that's how it is now and once the Job System kicks in it'll give you multithreaded physics"

    ...but my opinions are often unpopular
     
  22. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,589
    There are probably (more) projects, just like mine, where the game runs slower even when switching autoSync off.
     
    WildStyle69 likes this.
  23. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    2,483
    ceebeee likes this.
  24. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,899
    I actually gave up trying to understand which sync is the normal sync as this thread has confused me so much about it that I decided I would just wait until everyone is sure :p (I just want the 'fastest' one, whatever that is).

    Well they're popular with Hippo. But then I do hate most of the world and wish the 80s stayed.


    Testing... also can you please add more clarity on this feature? It does seem like everyone is getting slightly muddled :D
     
    Prodigga likes this.
  25. HiddenMonk

    HiddenMonk

    Joined:
    Dec 19, 2014
    Posts:
    987
    What I dont understand is why would autosync on be so drastically bad in the first place. Are you guys just testing with thousands of objects in an extreme case?
    Shouldnt it all just be a list of all dirty transforms that when doing a physics query if autosync is on it checks if any in list and loops through and syncs them all? How majorly different can that be than from what it was before? Before they just synced right away, so they avoid adding to a list and doing a Count check every physics query or something, is that really whats causing you guys major frame loss, or is there just something weird going on in the backend that handles things in a real poor way currently and as they said, hopefully it will be improved?
    I need colliders to be synced when I move transforms, so if they make us do things manually and remove the whole autosync option all together, then that would be annoying and I would probably handle it the way I explained above anyways, which if thats how they are doing it now and causes such performance loss, am I just in a pickle?
    Guess time will tell =)
     
  26. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,589
    I repeated this test with 2017.2.0f2. Performance did improve, whether autoSync is set to true or false makes not so much of a performance difference anymore.

    However, overall 2017.2.0f2 continues to be slower than 2017.1.1p3 with my game.

    scene_4_3.png

    scene_5_4.png

    scene_6_8.png
     
    Last edited: Oct 5, 2017
  27. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    976
    I'm glad to see it is a huge step in the right direction.
     
  28. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,589
    I hope they just don't stop fixing the remaining performance issues now. I mean it isn't completely unusable anymore, but still slower than earlier versions.
     
    Prodigga likes this.
  29. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    4,873
    Good to see they fixed transform update as promised. They gutted out the entire transform update and traversal, replaced it with something jobs friendly, if they keep at it, we should see some impressive speedups in the next few releases.
     
  30. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    Yeah, it's entirely possible what without the jobs system, the current physics implementation is a little bit heavier than before because it all has to be deferred and multithreadable (even though it is not yet multithreaded to that point). Maybe it's a temporary price to pay while we wait for our lord and saviour; the Jobs System

    Also good job Unity for making autoSync seemingly have pretty much no performance overhead anymore. I was wrong!
     
    Last edited: Oct 5, 2017
  31. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    976
    @PhilSA is there a source on the job system improving overall physics performance in the future? The way I understand it, due to these changes we can now perform physics queries from jobs (ie the raycasts being performed across 10 cores in the job system demo at Unite), but I don't think it'll actually improve normal physics performance right? That's all Nvidias side.
     
  32. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,899
    Unity's physics developer said we will see more core usage of Physx in Unity starting with Physx 3.4. So it will slightly improve normal physics.

    Anyway I hope these things continue to be optimised because my fear would be that you'd get some gains going job-ified and this would mask underlying optimisation wins rather than be a win on a win.

    But I'll just have to trust Unity has top people on it.
     
    Peter77 and Prodigga like this.
  33. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    Lex4art, Peter77 and Prodigga like this.
  34. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,589
    Any idea what Unity version comes with PhysX 3.4?
     
  35. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,899
    Not a clue. @yant perhaps you know?
     
  36. scvnathan

    scvnathan

    Joined:
    Apr 29, 2016
    Posts:
    74
    I can't remember where I saw it, but I believe the dev said not before 2018.1.
     
  37. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,592
    I can confirm that my physics performance test (from earlier in the thread) now performs just as good whether autoSync is on or off in 2017.2f2
     
    laurentlavigne and Peter77 like this.
unityunity