Resolved Possible bug? App seems to lock up when using a lot of rigidbodies for an extended period of time

Unity Version: 2021.1.15f1
I currently need to use quite a lot of rigidbodies for an extended period of time (50-300 objects for possibly hours), but the app keeps freezing after around 10-30 minutes, and I can't find what triggers the freeze.
Looking at the C++ call stack, it seems like the app is waiting for

physx::shdfnd::SListImpl::pop

(this seems to be the topmost function on the stack),
(Links for reference? : https://github.com/NVIDIAGameWorks/...rce/foundation/src/windows/PsWindowsSList.cpp
https://docs.microsoft.com/en-us/wi...pi/nf-interlockedapi-interlockedpopentryslist)
Trying to debug in C# is basically impossible because Visual studio also freezes the moment the app locks up.

I've found that these help mitigate the problem, but none are really a good option:
1. Reduce the number of objects in contact with each other
2. Reduce the number of physics objects used in the scene
3. Shorten the physics timestep (We're already struggling with performance so..)

While disabling it didn't appear to help, I should probably mention that I'm using physics materials, and am using mesh colliders with said material. While using primitive colliders seems to fix the freeze, that might be because using the primitive colliders help reduce the number of objects in contact in the first place.

Don't know how useful this info will be, but as a test I've added something along the lines of

Code (CSharp):

1.  
2. // Awake
3. _logThread = new Thread(SubthreadLogger.Log);
4. shouldLog = true;
5. _logThread.Start();
6.  
7. /*
8. Somewhere else in code
9. void SubthreadLogger.Log()
10. {
11.    while (shouldLog) // false on destroy then followed by a [Join]
12.    {
13.        Debug.Log("Still Updating...");
14.        Thread.Sleep(1000);
15.    }
16. }
17. */

but this also stopped logging the moment the app freezes.
Oh and also, this problem only occurs in Build, it works flawlessly in Editor.
Any ideas what might be going on?

 

Tiny update, reducing the number of active Physics objects didn't actually fix anything, but it can now go for ~1 hour before freezing if I limit it to 100 objects.

Seeing that, I first thought that I may have a memory leak somewhere, but no, the memory usage seems constant throughout the hour.

I'm just wildly guessing at this point, but I assume having all the scene colliders in the same object would be better than having them in every single scene object so that's what I'm trying next. (Update: this did not fix it)

New clue: removing a _rb.AddForce line seems to fix the issue? still testing this build though. (Update: this also didn't fix it)
Will remove all code changing the _rb component properties and see how it goes now. (Update: don't want to get too exited, but 2 hours in, this seems to fix it; maybe using AddForce and/or changing the velocity property of the rigidbody is a bad idea?)

 

Another update, several hours into testing and yeah, seems like the problem was with me changing the force and velocity of the physics object. No idea why though.

Update: messing with the rigidbody's drag also causes freezing. Seems like I can't change any of the rigidbody's properties or it'll freeze sooner or later.

 

Soo, this is interesting, the build I thought wouldn't freeze, just froze.. in the same line.. (physx::shdfnd...)
Guess it'll be a silent night (of debugging) for me today. (pardon the joke)

 

Yeah.. I wouldn't say that I can reproduce it consistently right now.. (It'll freeze eventually, but I have no idea what triggers it.). I'm currently trying to at least make a minimal repro project that would mimic the freezing I'm experiencing, but that might take a while..

Update: Again, trying to not get my hopes up, but commenting out a kinematic rigidbody's MovePosition parameter seems to help fix things up, but I'm going to keep the build running for a while just to make sure.

Update 2: Aaand it froze!

 

Just a hunch: are you enabling/disabling any colliders in a Coroutine? Or anything relating to physics at all in Coroutines? I’ve had a case that reminds me a lot of yours with totally inexplicable crashes that I never really understood. In my case, the editor just froze and sometimes I could only recover my computer (macos by that time) doing a reboot.

anyhow, I’m following this thread in the best hopes that you’ll find the bug. I know how frustrating this can be!

 

Sadly no, I am not
However, I am using Coroutines to read the asset I'm instantiating from the asset bundle, which I don't think will mess things up, but I'll try a putting the reference object as a SerializedProperty in my script instead and see how that goes.
Sadly for my case, this only happens in build, so checking if the changes I made fixed things takes a bit of time.. (Need to wait at least an hour for asset bundle, 15 minutes for the actual build)

Thanks!

 

Last observation of the year: if I have a large number of objects colliding with a static rigidbody plane, it'll freeze within 15 minutes, but now I'm making it collide with a kinematic rigidbody plane, and after 30 minutes it still haven't froze. I want to test this for longer, but I need to finish up for today, Happy holidays Everyone!

 

I was able to get in touch with people from Unity Japan through a private forum, and they confirmed that it seems to be a bug with no foreseeable workaround. Currently being told to wait for a fix.

 

I currently don't have a case number, but I'll post it here if I get one.

 

So in the end I never received a case number, but it seems to be fixed on Unity 2022.1.0 and 2021.2.13?
Also heard that fix is coming for Unity 2020 and 2019
(The bugfix seems to not be referenced in the changelog though)

In the meantime, the staff I got in touch with suggested either
1. Turning Use Micro Sleep for Thread Yield Off
2. Turning Multithreaded Rendering Off

Both fixed the freezing issue, but it brings other problems for my project.
Might be useful for someone else though.

 

I've looked around my inbox for relevant information, but as far as I've checked, I never received any information regarding any case number, be it a new one or a duplicate.

 

I might be a bit paranoid here but I think the NDA disallows me to post things on private forum to a public forum.
Could I message you instead?