Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug Physics.Processing spikes

Discussion in 'macOS' started by mons00n, Jan 10, 2022.

  1. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    303
    I am working on updating my project to 2021.2.x and attempting to get an Apple Silicon compatible desktop build going. One major discrepancy I am finding between the Intel build and the Universal Build on my M1Max is that Physics.Processing seems to spike at ~20ms every 8 frames or so. My physics is set to Multibox Pruning Broadphase and split into 16 subdivisions. I haven't tried other settings but this works fine on the Intel only builds; could this be an instance where the physics system is still going through rosetta?
     
  2. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    303
    I found this to be happening on 2021.2.12f1 on intel builds as well as apple silicon builds. It appears to be more severe if you call Physics.RebuildBroadphaseRegion. Submitted a bug report case #1406765.
     
  3. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    303
    As an additional experiment I built my project out with Unity 2020 and encountered the same exact problems. Could the issue be related to this issue? In the linked issue it is described as an editor spike, but it sounds eerily similar to what I am experiencing in built clients. It's been over a month and still no response on my bug submission. Has anyone else experienced this problem?
     
  4. adslitw

    adslitw

    Joined:
    Aug 23, 2012
    Posts:
    274
    I have definitely seen this before, but I just checked out my current project in the profiler and couldn't reproduce it. Is it only in builds that you see it?

    Also, your playerloop is taking 50ms??
     
    mons00n likes this.
  5. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    303
    Just tested my 2021 build (non-universal binary) on my intel mac and it too experiences giant Physics.Processing spikes, but less frequently than when I run it on my apple silicon machines. Editor performance is actually decent - haven't seen any appreciable spikes there. I wish Unity would just confirm that this is, or is not a bug. My project is stuck on 2019 until I can get this sorted :(
     
  6. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    Unfortunately it's very hard to tell just from that screenshot. Did you try looking at the physics profiler?

    You could always file a bug report through the editor, that way we would be able to drill into it.
     
    mons00n likes this.
  7. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    303
    Hi @Tautvydas-Zilys thanks for the response. This is happening in built clients; I am not seeing such spikes in the editor.
    As mentioned above I filed a bug report over a month ago and have yet to hear back. Case #1406765.
     
  8. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    The profiler can profile builds too. Sorry, totally missed the bug number.
     
    mons00n likes this.
  9. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    303
    Ah my apologies, I thought you were talking about the physics debugger. I see what you mean now. I checked the physics section of the profiler in my build and found no appreciable difference between a Physics.Processing 10ms spike and a regular frame:


    (10.59ms & 0.64ms are how long Physics.Processing took. I should also mention that these frames were back to back)

    In my bug report I found the easiest way to make this happen was to stick a camera in an empty scene, call Physics.RebuildBroadphaseRegion, then move the camera around and observe the Physics.Processing spikes.
     
  10. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    303
  11. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    Yeah I pinged them to make sure it didn't slip through the cracks :). We receive a lot of bogus reports and sometimes legitimate bugs get lost for a few weeks (they are usually found eventually, though). That's why posting the bug number on the forum can help it get processed faster.
     
    mons00n likes this.
  12. yant

    yant

    Unity Technologies

    Joined:
    Jul 24, 2013
    Posts:
    591
    Hey

    In case you're using multibox pruning on purpose, I would like to suggest switching over to Automatic Box Pruning. It's exactly the same stuff under the hood, with the nice advantage of not having to specify the world boundaries manually. It's now the default broadphase type in PhysX, so should be rather stable.

    As to the capture you shared, it doesn't directly appear as physics is what takes time here -- since there's nothing physics reported Physics.Processing that would take time. Keep in mind that the physics simulation code includes job stealing as part of the technical specifics. That means, on the main thread, it has a loop that checks if physics has done simulating yet, and if not -- will pick another job from the queue, that doesn't directly have to be physics at that point. Can be something from animation, graphics, sounds or anything else. That picked job can also be a very lengthy one. This can look pretty dramatic in tiny projects like the one attached to the fogbugz case, where time spent on "physics" can go from nothing to something if it manages to pick some other work.

    Hope it makes sense.

    Anthony
     
    radiantboy likes this.
  13. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    303
    Hi Anthony thanks for the detailed information. I switched my project to Automatic Box Pruning as you suggested and I'm still getting seemingly random spikes from 5-10ms in a built client registering as Physics.Processing in the profiler. If it is indeed another process stealing time what can I do to figure out what is doing the stealing? In this particular instance it seems that the profiler is not helpful at all.

    The repro project was just sent as an example of what I was seeing in my larger 80gb+ project. Right now these spikes are preventing me from updating from 2019 on macOS. These spikes are not present in my 2019 builds which use the same code, the same settings, the same everything other than the engine upgrade.
     
  14. JuozasK

    JuozasK

    Unity Technologies

    Joined:
    Mar 23, 2017
    Posts:
    84
    Hey! So there's not much you can do from the Unity Editor itself. I don't think the stolen jobs will show up in the profiler. What you could do is try to debug the Editor itself with an external program. If you're on MacOS the Instruments.app works very well and could possibly show what jobs are being executed.

    On the other hand, if you could save a profiler capture and send it to us, we may be able to at least examine this spike and either confirm our first suggestion or maybe find something else. To save a capture, record some frames in the Unity profiler and save those into a file, by clicking the appropriate button. Then just send us that file and we will be able to open it in the Editor on our side.
     
  15. Onat-H

    Onat-H

    Joined:
    Mar 11, 2015
    Posts:
    195
    Sorry to bring this up again, but this is definitely not stolen jobs... I'm on 2022.2.8 and this only happens on the mac and not on the pc, and it stops entirely when nothing physics related is in the scene. Adding just one box collider leads to constant 10ms physics processing markers, and this time is even multiplied when the frame time drops so much, that the physics loop is executed multiple times per frame.

    This is how my profiler looks in a scene that should run over 60fps on this system:
     

    Attached Files:

    adslitw likes this.
  16. Onat-H

    Onat-H

    Joined:
    Mar 11, 2015
    Posts:
    195
    This is a full scene, but the same thing also happens adding a box to an empty scene. The following two screenshots show an empty HDRP scene with a box. The one with the physics spikes is with the box collider on the box enabled, that other one is disabled. As you can see, there is in fact a drastic difference in performance. (And the only thing that is changed is enabling a box collider.) By the way, this is the apple silicon editor on an M1 Max if that helps.
     

    Attached Files:

    adslitw likes this.
  17. Onat-H

    Onat-H

    Joined:
    Mar 11, 2015
    Posts:
    195
    Bumping this up hoping to raise awarenes for this issue that's still present.
     
    NiklasP_, radiantboy and sickshredzz like this.
  18. NiklasP_

    NiklasP_

    Joined:
    Jul 6, 2020
    Posts:
    6
    This might be a general Unity 2022.2 issue. I'm having the exact same issue since upgrading to Unity 2022.2.11. Also happens only on my M1 MacBook and is no problem on a Windows PC with the exact same project.
     

    Attached Files:

    Onat-H likes this.
  19. Onat-H

    Onat-H

    Joined:
    Mar 11, 2015
    Posts:
    195
    I'm wondering why this isn't being worked on, given the fact that it's super easy to reproduce (just add a default unity cube, enable/disable the collider, look at the profiler)....
     
  20. adslitw

    adslitw

    Joined:
    Aug 23, 2012
    Posts:
    274
    My feeling is that Unity on mac is simply a low priority. It definitely feels unloved. Super frustrating.

    What's the vibe internally @Tautvydas-Zilys ? Are you guys aware of all these performance issues / crashes / instability etc - I'm guessing the move over to Apple Silicon hasn't exactly been an easy one?
     
    Onat-H and NiklasP_ like this.
  21. NiklasP_

    NiklasP_

    Joined:
    Jul 6, 2020
    Posts:
    6
    I'm having the exact same feeling, especially looking at the crash issue where it took nearly 4 months to fix this frequent crash occuring on all Apple silicon devices.


    I've now created a bug report with a simple URP scene with just one BoxCollider where you already see 10ms spikes. I'll let you know if I get any feedback from Unity.
     
    Onat-H and adslitw like this.
  22. GiantGrey

    GiantGrey

    Joined:
    Jul 2, 2012
    Posts:
    266
    I have the exact same issue on a PC. Unity 2022.2. Only happens in builds not in the editor.
     
  23. IsaacsUnity

    IsaacsUnity

    Unity Technologies

    Joined:
    Mar 1, 2022
    Posts:
    95
    Hey there, thank you for the feedback! We're actively looking into the issue, but we will need some time to investigate and identify a potential solution. We'll provide updates once they're available!
     
    NiklasP_ likes this.
  24. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    Thank you for the candid feedback. I will pass it along to relevant folks.
     
    NiklasP_ and adslitw like this.
  25. FeiYangShan

    FeiYangShan

    Joined:
    May 16, 2022
    Posts:
    1
    Hello, I believe I am experiencing the same problem on my M1 Macbook Air.

    A single cube with box collider causing a physics spike. In my project I have many box colliders and it causes significant performance issues.

    Has there been any update over the past couple of months on this issue? @IsaacsUnity @Tautvydas-Zilys

    Thank you! BoxColliderPhysicsSpike.png
     
  26. mominnnn

    mominnnn

    Joined:
    Jan 11, 2023
    Posts:
    7
    Same issue on 2022.3.2f1 M1 Macbook Pro. 2500ms per frame on Physics.Processing:

    EDIT: This was due to some large SphereColliders that were added to the scene
     

    Attached Files:

    Last edited: Jun 27, 2023
  27. adslitw

    adslitw

    Joined:
    Aug 23, 2012
    Posts:
    274
    Any updates on this @IsaacsUnity ? I'm not seeing huge spikes, but definitely still getting regular spikes. Is this expected behaviour?

    upload_2023-6-27_22-2-26.png
     
  28. adslitw

    adslitw

    Joined:
    Aug 23, 2012
    Posts:
    274
  29. NiklasP_

    NiklasP_

    Joined:
    Jul 6, 2020
    Posts:
    6
    I had filed a bug report about this, which was first confirmed and then marked as duplicated in June. They already have an internal issue about it, so at least they seem to be aware of it. But I would also be interested in the current status, as the problem has been going on for over 7 months without any solution.

    Edit: I just noticed that the issue was fixed in Unity 2023.2.0 Beta 1:
    • Physics: Fixed an issue on OSX where Physics.Simulate would experience hickus due to how the PhysX task system was being integrated with Unity's Job System. (UUM-27803)
    Hopefully this fix will also be implemented in the current Unity versions.
     
    adslitw likes this.
  30. alexrvn

    alexrvn

    Unity Technologies

    Joined:
    May 16, 2017
    Posts:
    53
    Heya! I'm currently in the process of back-porting the change mentioned above to older LTS versions (2022/2021), so we'll have the fix out to you as soon as possible. Will post back here with actual versions once the changes land.
     
    BACALL, NiklasP_, adslitw and 3 others like this.
  31. alexrvn

    alexrvn

    Unity Technologies

    Joined:
    May 16, 2017
    Posts:
    53
    Hey again!
    The fix for UUM-27803 has landed in the following versions of Unity:
    2023.1.9f1
    2022.3.8f1
    2021.3.30f1

    Cheers,
    Alex
     
  32. adslitw

    adslitw

    Joined:
    Aug 23, 2012
    Posts:
    274
    Amazing, thanks so much @alexrvn - will get this downloaded and give it a test asap.
     
    alexrvn likes this.
  33. adslitw

    adslitw

    Joined:
    Aug 23, 2012
    Posts:
    274
    @alexrvn any idea when 2022.3.8f1 will be out?
     
  34. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    721
    Was this only fixed for macOS? We have the same issue on Windows Server builds.
     
  35. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    721
    Examples :
     

    Attached Files:

  36. alexrvn

    alexrvn

    Unity Technologies

    Joined:
    May 16, 2017
    Posts:
    53
    2022.3.8.f1 is currently available.

    Hello nope, this was fixed globally. if you can submit a ticket I'd love to look at why that case is occurring for you on windows server ^^. The pictures uploaded do not show the job worker load, as the bulk of the work related to Physics.Processing happens off the main thread atm :3, either way it looks like a stall for some reason, which is worrying as we do not see this on our end at all.
     
    Last edited: Aug 25, 2023
  37. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    721
    This isn't on a new version btw, this is on 2022.2. So you think this would be fixed on the new version? My main question was if it would be fixed even if it isn't on macOS?

    Also, I don't see 2022.3.8 available. Only 2022.3.7, at least according to : https://unity.com/releases/editor/archive and according to Unity Hub

    EDIT : It must've just became available because I see it in Hub now.


    Also, attached a better screenshot. upload_2023-8-25_16-52-24.png
     
    Last edited: Aug 26, 2023
  38. alexrvn

    alexrvn

    Unity Technologies

    Joined:
    May 16, 2017
    Posts:
    53
    Ops I was a bit unclear there, when we fixed it, we did it for all platforms not just MacOS. Looking at the screenshot id defo say try the 2022.3.8 version.
     
  39. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    721

    Just tried 2022.3.8, it does not appear to be significantly better. It does appear to be slightly different though, in fact it will get caught up on other things, like Physics.UpdateBodies, or Physics.Raycast as well. It's very odd, as it seems quite obvious it is stalling on something.

    We would be well under our performance target if it weren't for the spikes. The other thing that is odd is we get spikes for other things, not even just physics, although it happens the most with physics. The spikes all seem to be related in that it appears to be a stall, no actual work seems to be getting done during the spike. For example in the third screenshot below, this method will normally take less than a 1 ms, but in the screenshotted frame, it takes 58 ms. And the method does the same thing every frame. There are a few other examples of random instances of this as well.

    (first screenshot is with vsync shown, second without vsync)

    upload_2023-8-26_9-9-34.png

    upload_2023-8-26_9-10-11.png

    upload_2023-8-26_9-15-38.png


    Also I can send this profiling data off to some email if that would help, I just can't post it publicly here. I really appreciate any and all help on this btw. Getting this fixed asap is very critical for us.
     
    Last edited: Aug 26, 2023
  40. alexrvn

    alexrvn

    Unity Technologies

    Joined:
    May 16, 2017
    Posts:
    53
    Hmm Physics.UpdateBodies does not have anything related to threading in it (what the original issue was caused by), it simply loops over all bodies in the scene, reads the new position and rotation from Physx and sets it to the transform of the GameObject(this happens once per frame for each Rigidbody and ArticulationBody inside your scene. Kinematic Rigidbodies are included in this cost.). How many bodies do you have in your scene?

    Submitting a bug even with just the profiling data would be really useful, and it is only visible to us and yourself.
     
  41. nina32176

    nina32176

    Joined:
    Jun 27, 2018
    Posts:
    17
    Is this fixed in 2020.3.48f1?

    Physics.Processing takes for me sometimes even 75ms, but child calls take less than 1 ms.
    At the time of spike, my scene stats:
    - active dynamic - 0
    - active kinematic - 12
    - static colliders - 5.79k
    - rigidbody 0
    - constraints 0
    - contacts 0
    What is taking so much time there?!
    upload_2023-11-15_21-18-11.png