Bug Random memory related crashes in Unity versions above 2021.3.9

Hi.
For some time now we have been trying to update Unity beyond 2021.3.9.
Things have been going (mostly) smoothly during the upgrade process and in testing, but things have not been stable when we have rolled out live.
In our latest attempt with Unity 2022.3.7, we had around ~75% crash-free users over the course of a week (as reported by crashlytics) and more than 1.5% crashing sessions (User-perceived crash rate in google play console), compared to our normal metrics around ~96% crash-free users and ~0.5% crashing sessions.
This puts the updated releases well above the bad behavior threshold for google play, and our players has experienced and reported a much higher amount of crashes.

These are the Unity version we have tried, which all exhibit the same behavior: 2021.3.13, 2021.3.18, 2022.3.7
Whereas 2021.3.9 is stable.

We are having a very hard time reproducing these internally, which makes verifying any potential fix impossible without doing a risky and costly release to players.
The crashes we are getting, unique to releases with Unity > 2021.3.9, seems to all be engine internal and related to memory management(?).
Some of these happens at seemingly random times, others such as the RemapPPtrTransfer one always happens during synchronous load of a scene.

Any help or ideas as to what might be causing this is greatly appreciated.

Shortened stack trace examples:

Crashed: Thread: SIGSEGV  0x0000000000000090

#00 pc 0x115ee48 libunity.so (remove_free_block) (BuildId: 6c3ff32c7bec63b1)

#01 pc 0x115f294 libunity.so (tlsf_free) (BuildId: 6c3ff32c7bec63b1)

#02 pc 0x71d434 libunity.so (DynamicHeapAllocator::Deallocate(void*)) (BuildId: 6c3ff32c7bec63b1)

...

Crashed: Thread: SIGSEGV  0x0000000000000082

#00 pc 0x945080 libunity.so (MonoBehaviour::RemoveNodesFromLists()) (BuildId: 6c3ff32c7bec63b1)

#01 pc 0x947c9c libunity.so (MonoBehaviour::Deactivate(DeactivateOperation)) (BuildId: 6c3ff32c7bec63b1)

#02 pc 0x72f854 libunity.so (GameObject::ActivateAwakeRecursivelyInternal(DeactivateOperation, AwakeFromLoadQueue&)) (BuildId: 6c3ff32c7bec63b1)

...

Crashed: Thread: SIGSEGV  0x0000000000000000

#00 pc 0x0

#01 pc 0x993120 libunity.so (IDCollectorFunctor::GenerateInstanceID(int, TransferMetaFlags)) (BuildId: 6c3ff32c7bec63b1)

...

Crashed: Thread: SIGSEGV  0x003e003d003c007b

#00 pc 0x72fa8c libunity.so (GameObject::QueryComponentByType(Unity::Type const*) const) (BuildId: 6c3ff32c7bec63b1)

#01 pc 0xcf3b40 libunity.so (UI::AddRenderersToCanvas(JobFence&, Transform const*, UI::Canvas*, int&)) (BuildId: 6c3ff32c7bec63b1)

...

Crashed: Thread: SIGSEGV  0x0000007400000030

#00 pc 0xcf3b4c libunity.so (UI::AddRenderersToCanvas(JobFence&, Transform const*, UI::Canvas*, int&)) (BuildId: 6c3ff32c7bec63b1)

#01 pc 0xcf3b40 libunity.so (UI::AddRenderersToCanvas(JobFence&, Transform const*, UI::Canvas*, int&)) (BuildId: 6c3ff32c7bec63b1)

...

Crashed: Thread: SIGSEGV  0x000000003f800004

#00 pc 0x38acd4 libunity.so (void RemapPPtrTransfer::Transfer<ImmediatePtr<Unity::Component>>(ImmediatePtr<Unity::Component>&, char const*, TransferMetaFlags)) (BuildId: 30991d940037b998)

#01 pc 0x38ad45 libunity.so (void GameObject::ComponentPair::Transfer<RemapPPtrTransfer>(RemapPPtrTransfer&)) (BuildId: 30991d940037b998)

#02 pc 0x38ac99 libunity.so (void RemapPPtrTransfer::Transfer<dynamic_array<GameObject::ComponentPair, 0u>>(dynamic_array<GameObject::ComponentPair, 0u>&, char const*, TransferMetaFlags)) (BuildId: 30991d940037b998)

...

I'm also attaching a bunch of stacktraces from the more common crashes in full including all threads.
There are more but the upload limit is 5 files.

 

(bump)

 

Thanks for the response!

The stacktraces are from firebase crashlytics, which uses symbols we upload when builds are made.

Not 100 on the details of what happens on their end, but there is some mapping being done on BuildId for which symbols are used for symbolication.
We don't get symbolicated stacks unless we do the upload of symbols to firebase and we have crashes in our code that gets symbolicated correctly.

If you still think symbolication is an issue here, is there a specific crash I can force to test that we actually get the proper stacktrace?

 

Attached stacktraces using this method.

 

That is.. strange.
Would you say it's the same for stacks in OP?
And if so, since they're mostly libunity.so functions, do you have any insights as to what might be going on?

I'll try to figure out what could be happening with the incorrect symbolication, but we most definitely have symbolicated stacktraces for our own code (which would also be part of libil2cpp?) that makes sense and are correct.

 

I was referring to the stacktraces in the Original Post (OP), and if it was possible to deduce anything from them (assuming symbolication of libunity.so methods is correct).

To get the symbols we set

EditorUserBuildSettings.androidCreateSymbolsZip = true;

and build using

BuildPipeline.BuildPlayer(...)

Unity produces a .zip file (ex: android-build-1.11.73-v1595.symbols.zip) at the build destination containing libil2cpp.so, libmain.so, libunity.so for each architecture (arm64-v8a and armeabi-v7a).

The symbols are extracted and uploaded to firebase using their CLI tool and the command

firebase crashlytics:symbols:upload

as per https://firebase.google.com/docs/crashlytics/get-deobfuscated-reports?platform=unity

output:

i  Generating symbols for ./android-build//android-build-1.12.0-v11450.symbols.zip

i  Generated symbols for ./android-build//android-build-1.12.0-v11450.symbols.zip

i  Output Path: /var/folders/r5/nyxch5jn737f2vxw28509rhh0000gn/T/crashlytics-e507d298-a8df-4d2a-8fa2-a296898169a9/nativeSymbols/1-464341683993-android-a4549609e2dc980a177f88/breakpad

i  Uploading all generated symbols...

i  Successfully uploaded all symbols

I agree that parts of the stacktraces don't make sense and I'm looking into where things go wrong (thanks for highlighting this problem btw).

 

Less of a pointed suggestion, but here is another tool you can consider using in this investigation: GWP-ASan is a library you can add to your Unity Android builds to help diagnose and then solve memory related issues (including misalignment, use-after-free and bad memory (de)allocation).

This extra info available on randomly selected clients across your audience could help add more context to figuring this out.

 

Have been unable to focus on this for a while, but I gave the symbols a stab today.
Without success I have tested these things:

• Updated firebase CLI tools
  
• Use argument

   --generator=csym

  with the command

  firebase crashlytics:symbols:upload

• Set

  EditorUserBuildSettings.androidCreateSymbols = AndroidCreateSymbols.Public;

  along with the already set

  EditorUserBuildSettings.androidCreateSymbolsZip = true

Code stripping is already set to minimal, and Strip Engine code is not checked.

The one thing that did have an effect was setting

EditorUserBuildSettings.il2CppCodeGeneration =  Il2CppCodeGeneration.OptimizeSize;

instead of

Il2CppCodeGeneration.OptimizeSpeed

OptimizeSize results in a much shorter stack

I have also tested the same process in a different project, where the weirdly symbolicated methods are still weird but with a different symbol. I'll attach three symbolicated stack traces for comparison purposes.

Note: Unity 2021.3.9f1

 

We had wrong stacktraces on Android builds when Google Play Integrity protection is enabled.

Kind of related to GetELFImageBuildID bug as well, which is caused by Google Play Integrity protection as well. https://issuetracker.unity3d.com/is...yer-crashes-when-libil2cpp-dot-so-is-modified

Google Play integrity protection seems to re-arrange some stuff in the libilcpp.so and Unity doesn't expect that - or has some byte markers hard coded. For us the side-effect was that the first line of stacktraces was impossible to reach. But stacktrace lines after the first line were correct.

 

Note: Google Play Integrity protection is by default now enabled since recently for new releases on the Google Play store. Before that it was an optional / manual setting. Now you have to manually disable it.

 

These crash seems much more like with our project, have you solved these problems?