Huge jump on tgkill (signal 6) crashes after moving to IL2CPP.

We have seen about 500% increase on this crash after moving from 2017.4 to 2018.4 and switching using IL2CPP. Unfortunately stack trace doesn't tell much.

Pre-IL2CPP crash that occurred rarely:

Code (CSharp):

1. Caused by java.lang.Error: signal 6 (SIGABRT), code -6 (?), fault addr --------
2. Build fingerprint: 'samsung/j6ltedx/j6lte:8.0.0/R16NW/J600GDXU3ARK5:user/release-keys'
3. Revision: '2'
4. pid: 8907, tid: 9048, name: UnityMain  >>> net.test.text <<<
5.     r0 00000000  r1 00002358  r2 00000006  r3 00000008
6.     r4 000022cb  r5 00002358  r6 cb97ec80  r7 0000010c
7.     r8 00000056  r9 cd8d6e00  sl cb97ee90  fp cb97ecf4
8.     ip 00000000  sp cb97ec70  lr f59e44d7  pc f5a1580c  cpsr 00000056
9.  
10.        at libc.tgkill + 12(tgkill:12)
11.        at libc.abort + 54(abort:54)
12.        at libc.__libc_fatal + 24(__libc_fatal:24)
13.        at libc.__pthread_internal_find(long) + 88(__pthread_internal_find:88)
14.        at libc.pthread_gettid_np + 2(pthread_gettid_np:2)
15.        at libc.pthread_kill + 18(pthread_kill:18)
16.        at libmono.0023fef0()
17.        at libmono.mono_thread_suspend_all_other_threads + 1424(mono_thread_suspend_all_other_threads:1424)
18.        at libmono.mono_unity_jit_cleanup + 28(mono_unity_jit_cleanup:28)
19.        at libunity.0072fba0()
20.        at libunity.00705764()
21.        at libunity.002df31c()
22.        at libunity.002e2148()
23.        at base.00116113()

Post IL2CPP tgkill crashes that increased by a huge margin:

Code (CSharp):

1. Caused by java.lang.Error: signal 6 (SIGABRT), code -6 (?), fault addr --------
2. Build fingerprint: 'samsung/grandppltedx/grandpplte:6.0.1/MMB29T/G532GDXU1ASA5:user/release-keys'
3. Revision: '5'
4. pid: 3179, tid: 3466, name: Thread-6859  >>> net.test.text <<<
5.     r0 00000000  r1 00000d8a  r2 00000006  r3 833ae978
6.     r4 833ae980  r5 833ae930  r6 00000000  r7 0000010c
7.     r8 000004d6  r9 b433de44  sl 00000008  fp b4323820
8.     ip 00000006  sp 833ae7d0  lr b6cdfde5  pc b6ce21d4  cpsr 833ae4e0
9.  
10.        at libc.tgkill + 12(tgkill:12)
11.        at libc.pthread_kill + 32(pthread_kill:32)
12.        at libc.raise + 10(raise:10)
13.        at libc.__libc_android_abort + 34(__libc_android_abort:34)
14.        at libc.abort + 4(abort:4)
15.        at libc.abort + 4(abort:4)
16.        at libc.abort + 4(abort:4)
17.        at libc.abort + 4(abort:4)
18.        at libc.abort + 4(abort:4)
19.        at libc.abort + 4(abort:4)
20.        at libc.abort + 4(abort:4)
21.        at libc.abort + 4(abort:4)
22.        at libc.abort + 4(abort:4)
23.        at libc.abort + 4(abort:4)
24.        at libc.abort + 4(abort:4)
25.        at libc.abort + 4(abort:4)
26.        at libc.abort + 4(abort:4)
27.        at libc.abort + 4(abort:4)
28.        at libc.abort + 4(abort:4)
29.        at libc.abort + 4(abort:4)
30.        at libc.abort + 4(abort:4)
31.        at libc.abort + 4(abort:4)
32.        at libc.abort + 4(abort:4)
33.        at libc.abort + 4(abort:4)
34.        at libc.abort + 4(abort:4)
35.        at libc.abort + 4(abort:4)
36.        at libc.abort + 4(abort:4)
37.        at libc.abort + 4(abort:4)
38.        at libc.abort + 4(abort:4)
39.        at libc.abort + 4(abort:4)
40.        at libc.abort + 4(abort:4)
41.        at libc.abort + 4(abort:4)

Any ideas?

 

My bad, the first log was from pre-IL2CPP version, it was happening rarely back then, but the second one is the one that increased this crash report by 500% after IL2CPP swap, it's the one I'm concerned about right now.

 

I used to do some symbolication on Mono stacks but I don't know how to symbolicate IL2CPP stacks, would you guide me to it?

 

I have used that article to symbolicate mono crashes before, but it doesn't tell about IL2CPP symbolication sadly.

I have tried

Code (CSharp):

1. ndk-stack -sym /Applications/Unity/Hub/Editor/2018.4.3f1/PlaybackEngines/AndroidPlayer/Variations/ -dump myCrashDump.txt

but didn't work.

 

Hi,

Both logs are from the Fabric Crashlytics, there is nothing we do manually to enhance/symbolicate as far as I know.

Honestly, I have no idea what would have alter these crash dumps, are you sure they are not what they are supposed to be like?

 

Unfortunately I couldn't manage to reproduce this issue in any way so I don't have access to any logcat details.

 

Maybe I can find a way to attach the logcats to exceptions that are caught by Fabric, I will look into that.

Meanwhile this attached log is all the information it provides to us about an occurrence of this crash. Maybe that helps you somehow?

 

Terrible news, so we have actually nothing to help us to resolve this at all. What else would you suggest me to do to get some information about this crash?

 

Yeah, I have tried it on J6 and J7 but sadly couldn't produce the problem. This problem is not specific to those phones either, see the screenshots attached.

 

Thanks for your time anyway, I couldn't find anything related to this issue on 2018.4 and 2018.5 releases and to be honest kind of scared to upgrade, so far 2018 LTS upgrades introduced nothing but new crashes/bugs for us, but I guess I'll give it a try.

 

I'm finally able to reproduce the problem, and did it on development mode however log is exactly the same.

Code (CSharp):

1. 08-09 14:47:46.956 14765 15279 F art     : art/runtime/thread.cc:1238] Native thread exited without calling DetachCurrentThread: Thread[26,tid=15279,Native,Thread*=0x6539c200,peer=0x138d20a0,"Thread-12925"]
2. 08-09 14:47:46.956 14765 15279 F art     : art/runtime/runtime.cc:368] Runtime aborting...
3. 08-09 14:47:46.956 14765 15279 F art     : art/runtime/runtime.cc:368]
4. 08-09 14:47:46.956 14765 15279 E CRASH   : signal 6 (SIGABRT), code -6 (?), fault addr --------
5. ********** Crash dump: **********
6. Build fingerprint: 'samsung/kltejv/klte:6.0.1/MMB29M/G900FQJVS1CSB1:user/release-keys'
7. pid: 14765, tid: 15279, name: Thread-12925  >>> my.awesome.app <<<
8. Stack frame #00  pc 00041ff8  /system/lib/libc.so (tgkill+12)
9. Stack frame #01  pc 0003fc05  /system/lib/libc.so (pthread_kill+32)
10. Stack frame #02  pc 0001c38b  /system/lib/libc.so (raise+10)
11. Stack frame #03  pc 00019609  /system/lib/libc.so (__libc_android_abort+34)
12. Stack frame #04  pc 0001755c  /system/lib/libc.so (abort+4)
13. Stack frame #05  pc 0001755c  /system/lib/libc.so (abort+4)
14. Stack frame #06  pc 0001755c  /system/lib/libc.so (abort+4)
15. Stack frame #07  pc 0001755c  /system/lib/libc.so (abort+4)
16. Stack frame #08  pc 0001755c  /system/lib/libc.so (abort+4)
17. Stack frame #09  pc 0001755c  /system/lib/libc.so (abort+4)
18. Stack frame #10  pc 0001755c  /system/lib/libc.so (abort+4)
19. Stack frame #11  pc 0001755c  /system/lib/libc.so (abort+4)
20. Stack frame #12  pc 0001755c  /system/lib/libc.so (abort+4)
21. Stack frame #13  pc 0001755c  /system/lib/libc.so (abort+4)
22. Stack frame #14  pc 0001755c  /system/lib/libc.so (abort+4)
23. Stack frame #15  pc 0001755c  /system/lib/libc.so (abort+4)
24. Stack frame #16  pc 0001755c  /system/lib/libc.so (abort+4)
25. Stack frame #17  pc 0001755c  /system/lib/libc.so (abort+4)
26. Stack frame #18  pc 0001755c  /system/lib/libc.so (abort+4)
27. Stack frame #19  pc 0001755c  /system/lib/libc.so (abort+4)
28. Stack frame #20  pc 0001755c  /system/lib/libc.so (abort+4)
29. Stack frame #21  pc 0001755c  /system/lib/libc.so (abort+4)
30. Stack frame #22  pc 0001755c  /system/lib/libc.so (abort+4)
31. Stack frame #23  pc 0001755c  /system/lib/libc.so (abort+4)
32. Stack frame #24  pc 0001755c  /system/lib/libc.so (abort+4)
33. Stack frame #25  pc 0001755c  /system/lib/libc.so (abort+4)
34. Stack frame #26  pc 0001755c  /system/lib/libc.so (abort+4)
35. Stack frame #27  pc 0001755c  /system/lib/libc.so (abort+4)
36. Stack frame #28  pc 0001755c  /system/lib/libc.so (abort+4)
37. Stack frame #29  pc 0001755c  /system/lib/libc.so (abort+4)
38. Stack frame #30  pc 0001755c  /system/lib/libc.so (abort+4)
39. Stack frame #31  pc 0001755c  /system/lib/libc.so (abort+4)

Seems like a threading problem, but no idea how to progress based on this log, it actually says not much besides the new "Native thread exited without calling DetachCurrentThread:" part.

I also have bunch load of;

Code (CSharp):

1. 08-09 14:49:18.656 14765 14765 E CRASH   : other thread is trapped; signum = 6
2. 08-09 14:49:18.656 14765 14765 E CRASH   : main thread is trapped; signum = 6
3. 08-09 14:49:18.656 14765 14765 E CRASH   : other thread is trapped; signum = 6
4. 08-09 14:49:18.656 14765 14765 E CRASH   : main thread is trapped; signum = 6
5. 08-09 14:49:18.656 14765 14765 E CRASH   : other thread is trapped; signum = 6
6. 08-09 14:49:18.666 14765 14765 E CRASH   : main thread is trapped; signum = 6
7. 08-09 14:49:18.666 14765 14765 E CRASH   : other thread is trapped; signum = 6
8. 08-09 14:49:18.666 14765 14765 E CRASH   : main thread is trapped; signum = 6
9. 08-09 14:49:18.666 14765 14765 E CRASH   : other thread is trapped; signum = 6
10. 08-09 14:49:18.666 14765 14765 E CRASH   : main thread is trapped; signum = 6
11. 08-09 14:49:18.666 14765 14765 E CRASH   : other thread is trapped; signum = 6

How I can debug what's going wrong during runtime from this, any ideas?

 

I did more digging around and found this (it was in the code base for years not something we added after 2018 switch):

Code (CSharp):

1.         #region Local Notification
2.         public void ShowLocalNotification (string title, string message) {
3.             AndroidJNI.AttachCurrentThread();
4.             Facade.ShowLocalNotification(title, message);
5.         }
6.         #endregion
7.  

Then I have added

Code (CSharp):

1. AndroidJNI.DetachCurrentThread();

After calling native method here. Since this is the only instance where JNI calls happen in the code base.

Failed to produce any freezes after this change, maybe this fixed the problem, I will keep testing it.

That doesn't explain why it would increase dramatically after IL2CPP switch though, any ideas on that?

 

It does exit before closing the notification.

Did you mean `until you display the notification` by any chance?

 

It exits right after a notification is shown by the native code (native calls are blocking afaik?), so it exists while showing the notification and detaches currently.

 

After the fix I've mentioned, the exceptions have almost disappeared; from 0.1% of every sessions to less than 0.01%.

There are still some cases causes this exception, there seems to be some kind of race condition issue with the network thread (ShowLocalNotification method was also being called by the network thread) so I need to check through the network service to fix all tgkill exceptions.

Thanks for helping out @Tomas1856 and @JoshPeterson

 

Do you mean DetachCurrentThread needs to be in place always when using IL2CPP?