Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice

Bug Il2cpp crash at malloc

Discussion in 'Experimental Scripting Previews' started by Kamyker, Jul 20, 2021.

  1. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    We do have a few similar reports in our bug database, all for Unity version 2020.3 or earlier. Unfortunately we've not tracked down the cause of any of them.

    We have recently fixed a GC issue that is likely unrelated, but could be involved. If possible, please try the next Unity 2020.3 patch release. I'd be curious to know if that decreases the crash frequency you see with this call stack.
     
    RaventurnPatrick likes this.
  2. RaventurnPatrick

    RaventurnPatrick

    Joined:
    Aug 9, 2011
    Posts:
    250
    You are talking about 2020.3.34f1? We will definitely try it, once it is released.

    I also found a bug report for the Boehm GC (Mono) that has a very similar stack trace: https://bugzilla.xamarin.com/21/21939/bug.html (unfortunately it was never resolved) - but maybe this helps track the problem down, as the user in the bug report talks about a potential fix.
     
  3. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Looking at the bug status, I think you will need to wait until 2020.3.35 for the fix to be released. Unfortunately the bug report that looks similar is not public, so I don't have a public link you can track.

    It does sound very similar to that Mono bug report though, so this might be a promising avenue.
     
    RaventurnPatrick likes this.
  4. Badinha

    Badinha

    Joined:
    May 6, 2021
    Posts:
    5
    I don't see any clear indication of this bugfix on the .35 release notes, could you confirm if is there?
     
  5. sdg_unity

    sdg_unity

    Joined:
    Aug 28, 2019
    Posts:
    7
    We have released a new version with unity .35 and are still seeing this problem. So it does not seem to been fixed
     
  6. YVR_Unity

    YVR_Unity

    Joined:
    Jun 17, 2021
    Posts:
    3
    upload_2022-7-11_12-4-39.png
    We also faces the similiar issue in Unity 2020.3.35

    And, i am within android platform, not ios
     
    Last edited: Jul 11, 2022
  7. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    @sdg_unity @YVR_Unity

    The changes on our end that corrected the internally discovered issue are in Unity version 2020.3.35, so I suspect that there is some other cause of this problem.

    Can one or both of you submit bug reports for this issue? Often these problems require some deep debugging, so having a project that reproduces this in invaluable.
     
    WentaoWu likes this.
  8. WentaoWu

    WentaoWu

    Joined:
    Jul 12, 2022
    Posts:
    2
    Hi,We have a similar problems during the building process when ran into to GC_ push_all function. The "Unexpected mark stack overflow" exception is thrown and then show a message box, all threads is waiting for single object.

    This is the call stack from dump file in main thread. Unity version is 2019.4.35
    Code (Boo):
    1. # Child-SP          RetAddr               Call Site
    2. 00 000000b2`3c729b28 00007ffb`419bd6d1     win32u!NtUserWaitMessage+0x14
    3. 01 000000b2`3c729b30 00007ffb`419bd441     user32!DialogBox2+0x261
    4. 02 000000b2`3c729bd0 00007ffb`41a0bb4b     user32!InternalDialogBox+0x12d
    5. 03 000000b2`3c729c30 00007ffb`41a0a465     user32!SoftModalMessageBox+0x85b
    6. 04 000000b2`3c729d80 00007ffb`41a0b257     user32!MessageBoxWorker+0x341
    7. 05 000000b2`3c729f30 00007ffb`41a0b048     user32!MessageBoxTimeoutW+0x1a7
    8. 06 000000b2`3c72a030 00007ffb`41a0ac5e     user32!MessageBoxTimeoutA+0x108
    9. 07 000000b2`3c72a090 00007ffa`d6f87da8     user32!MessageBoxA+0x4e
    10. 08 (Inline Function) --------`--------     mono_2_0_bdwgc!GC_win32_MessageBoxA+0x18 [c:\build\output\unity-technologies\mono\external\bdwgc\misc.c @ 823]
    11. 09 000000b2`3c72a0d0 00007ffa`d6f9304a     mono_2_0_bdwgc!GC_default_on_abort+0x30 [c:\build\output\unity-technologies\mono\external\bdwgc\misc.c @ 1950]
    12. 0a 000000b2`3c72a100 00007ffa`d6c81bcc     mono_2_0_bdwgc!GC_push_all+0x46 [c:\build\output\unity-technologies\mono\external\bdwgc\mark.c @ 1351]
    13. 0b 000000b2`3c72a130 00007ffa`d6d98bbf     mono_2_0_bdwgc!monoeg_g_hash_table_foreach+0x9c [c:\build\output\unity-technologies\mono\mono\eglib\ghashtable.c @ 354]
    14. 0c 000000b2`3c72a180 00007ffa`d6f93c2d     mono_2_0_bdwgc!mono_push_other_roots+0x1f [c:\build\output\unity-technologies\mono\mono\metadata\boehm-gc.c @ 715]
    15. 0d 000000b2`3c72a1b0 00007ffa`d6f91531     mono_2_0_bdwgc!GC_push_roots+0xc9 [c:\build\output\unity-technologies\mono\external\bdwgc\mark_rts.c @ 938]
    16. 0e 000000b2`3c72a1e0 00007ffa`d6f9128d     mono_2_0_bdwgc!GC_mark_some_inner+0x24d [c:\build\output\unity-technologies\mono\external\bdwgc\mark.c @ 335]
    17. 0f 000000b2`3c72a210 00007ffa`d6f8a385     mono_2_0_bdwgc!GC_mark_some+0x9 [c:\build\output\unity-technologies\mono\external\bdwgc\mark.c @ 495]
    18. 10 000000b2`3c72a250 00007ffa`d6f8923a     mono_2_0_bdwgc!GC_collect_a_little_inner+0x49 [c:\build\output\unity-technologies\mono\external\bdwgc\alloc.c @ 649]
    19. 11 000000b2`3c72a280 00007ffa`d6f8dda8     mono_2_0_bdwgc!GC_allocobj+0x7a [c:\build\output\unity-technologies\mono\external\bdwgc\alloc.c @ 1545]
    20. 12 000000b2`3c72a2c0 00007ffa`d6f8da7f     mono_2_0_bdwgc!GC_generic_malloc_inner+0x124 [c:\build\output\unity-technologies\mono\external\bdwgc\malloc.c @ 191]
    21. 13 000000b2`3c72a300 00007ffa`d6f908bc     mono_2_0_bdwgc!GC_generic_malloc+0x16b [c:\build\output\unity-technologies\mono\external\bdwgc\malloc.c @ 252]
    22. 14 000000b2`3c72a340 00007ffa`d6d9745d     mono_2_0_bdwgc!GC_malloc_kind_global+0xd8 [c:\build\output\unity-technologies\mono\external\bdwgc\malloc.c @ 327]
    23. 15 000000b2`3c72a370 00007ffa`d6d3d3e1     mono_2_0_bdwgc!mono_gc_alloc_string+0x1d [c:\build\output\unity-technologies\mono\mono\metadata\boehm-gc.c @ 924]
    24. 16 000000b2`3c72a3a0 00007ffa`d6d500bf     mono_2_0_bdwgc!mono_string_new_size_checked+0xc1 [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 6242]
    25. 17 000000b2`3c72a440 000001da`6ab44947     mono_2_0_bdwgc!ves_icall_System_String_InternalAllocateStr+0x1f [c:\build\output\unity-technologies\mono\mono\metadata\string-icalls.c @ 41]
    26. 18 000000b2`3c72a4e0 000001da`6aba3ef3     0x000001da`6ab44947
    27. 19 000000b2`3c72a570 000001da`6aba3c23     0x000001da`6aba3ef3
    28. 1a 000000b2`3c72a620 000001dc`0792221b     0x000001da`6aba3c23
    29. 1b 000000b2`3c72a680 000001dc`0792f06b     0x000001dc`0792221b
    30. 1c 000000b2`3c72a730 000001dc`078c035a     0x000001dc`0792f06b
    31. 1d 000000b2`3c72a7a0 000001dc`078becbb     0x000001dc`078c035a
    32. 1e 000000b2`3c72a870 000001dc`078dadb3     0x000001dc`078becbb
    33. 1f 000000b2`3c72aa30 000001dc`078bfaf3     0x000001dc`078dadb3
    34. 20 000000b2`3c72aaf0 000001dc`078a3a43     0x000001dc`078bfaf3
    35. 21 000000b2`3c72acb0 000001dc`0784cafb     0x000001dc`078a3a43
    36. 22 000000b2`3c72add0 000001dc`0784cf89     0x000001dc`0784cafb
    37. 23 000000b2`3c72aed0 00007ffa`d6dae290     0x000001dc`0784cf89
    38. 24 000000b2`3c72af80 00007ffa`d6d32ac2     mono_2_0_bdwgc!mono_jit_runtime_invoke+0x530 [c:\build\output\unity-technologies\mono\mono\mini\mini-runtime.c @ 2835]
    39. 25 000000b2`3c72b160 00007ffa`d6d3bb1f     mono_2_0_bdwgc!do_runtime_invoke+0x82 [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 2921]
    40. 26 (Inline Function) --------`--------     mono_2_0_bdwgc!mono_runtime_try_invoke+0x47 [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 3026]
    41. 27 000000b2`3c72b1b0 00007ff7`0b99ceae     mono_2_0_bdwgc!mono_runtime_invoke+0x6f [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 2968]
    42. 28 000000b2`3c72b260 00007ff7`0b99684d     Unity!scripting_method_invoke+0xbe
    43. 29 000000b2`3c72b2a0 00007ff7`0b9901d3     Unity!ScriptingInvocation::Invoke+0x21d
    44. 2a 000000b2`3c72b3f0 00007ff7`0a3efb7f     Unity!ScriptingInvocation::Invoke<ScriptingArrayPtr>+0x13
    45. 2b 000000b2`3c72b420 00007ff7`0a3d1f37     Unity!BuildPipeline::GenerateAssemblyTypeInfos+0x27f
    46. 2c 000000b2`3c72b680 00007ff7`0a7c28b6     Unity!BuildPipeline::CompilePlayerScripts+0x9f7
    47. 2d 000000b2`3c72b920 000001db`908dbeeb     Unity!PlayerBuildInterface_CUSTOM_CompilePlayerScriptsNative_Injected+0x246
    48. 2e 000000b2`3c72ba90 000001db`908dbce3     0x000001db`908dbeeb
    49. 2f 000000b2`3c72bb20 000001db`908db683     0x000001db`908dbce3
    50. 30 000000b2`3c72bbc0 000001db`908db33b     0x000001db`908db683
    51. 31 000000b2`3c72bcd0 000001db`908da843     0x000001db`908db33b
    52. 32 000000b2`3c72bda0 000001da`e0cbb263     0x000001db`908da843
    53. 33 000000b2`3c72beb0 000001da`e0b976d3     0x000001da`e0cbb263
    54. 34 000000b2`3c72c0f0 000001da`e09e3163     0x000001da`e0b976d3
    55. 35 000000b2`3c72c260 000001da`e09e28e3     0x000001da`e09e3163
    56. 36 000000b2`3c72c2d0 000001da`e0b95d1b     0x000001da`e09e28e3
    57. 37 000000b2`3c72c3a0 000001da`e0b94fe3     0x000001da`e0b95d1b
    58. 38 000000b2`3c72c570 000001da`e0b904b3     0x000001da`e0b94fe3
    59. 39 000000b2`3c72cdc0 000001da`e0b7be33     0x000001da`e0b904b3
    60. 3a 000000b2`3c72d020 000001db`908d8a13     0x000001da`e0b7be33
    61. 3b 000000b2`3c72d180 000001da`da09b9b5     0x000001db`908d8a13
    62. 3c 000000b2`3c72d660 00007ffa`d6dae290     0x000001da`da09b9b5
    63. 3d 000000b2`3c72d710 00007ffa`d6d32ac2     mono_2_0_bdwgc!mono_jit_runtime_invoke+0x530 [c:\build\output\unity-technologies\mono\mono\mini\mini-runtime.c @ 2835]
    64. 3e 000000b2`3c72d8f0 00007ffa`d6d3b1dc     mono_2_0_bdwgc!do_runtime_invoke+0x82 [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 2921]
    65. 3f 000000b2`3c72d940 00007ffa`d6da97bd     mono_2_0_bdwgc!mono_runtime_class_init_full+0x2bc [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 477]
    66. 40 000000b2`3c72d9c0 00007ffa`d6dad742     mono_2_0_bdwgc!mono_jit_compile_method_inner+0x83d [c:\build\output\unity-technologies\mono\mono\mini\mini.c @ 4365]
    67. 41 000000b2`3c72da60 00007ffa`d6e89a76     mono_2_0_bdwgc!mono_jit_compile_method_with_opt+0x3b2 [c:\build\output\unity-technologies\mono\mono\mini\mini-runtime.c @ 2159]
    68. 42 000000b2`3c72dab0 00007ffa`d6e8bab4     mono_2_0_bdwgc!common_call_trampoline+0x606 [c:\build\output\unity-technologies\mono\mono\mini\mini-trampolines.c @ 716]
    69. 43 000000b2`3c72db70 000001da`66a803a3     mono_2_0_bdwgc!mono_magic_trampoline+0x74 [c:\build\output\unity-technologies\mono\mono\mini\mini-trampolines.c @ 848]
    70. 44 000000b2`3c72dc20 000001db`908d70ab     0x000001da`66a803a3
    71. 45 000000b2`3c72de40 000001db`908d3bb3     0x000001db`908d70ab
    72. 46 000000b2`3c72df10 000001da`e0b0238b     0x000001db`908d3bb3
    73. 47 000000b2`3c72e270 000001da`e0a19f7b     0x000001da`e0b0238b
    74. 48 000000b2`3c72e680 000001da`da09b9b5     0x000001da`e0a19f7b
    75. 49 000000b2`3c72ea60 00007ffa`d6dae290     0x000001da`da09b9b5
    76. 4a 000000b2`3c72eb10 00007ffa`d6d32ac2     mono_2_0_bdwgc!mono_jit_runtime_invoke+0x530 [c:\build\output\unity-technologies\mono\mono\mini\mini-runtime.c @ 2835]
    77. 4b 000000b2`3c72ecf0 00007ffa`d6d3bb1f     mono_2_0_bdwgc!do_runtime_invoke+0x82 [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 2921]
    78. 4c (Inline Function) --------`--------     mono_2_0_bdwgc!mono_runtime_try_invoke+0x47 [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 3026]
    79. 4d 000000b2`3c72ed40 00007ff7`0b99ceae     mono_2_0_bdwgc!mono_runtime_invoke+0x6f [c:\build\output\unity-technologies\mono\mono\metadata\object.c @ 2968]
    80. 4e 000000b2`3c72edf0 00007ff7`0b99684d     Unity!scripting_method_invoke+0xbe
    81. 4f 000000b2`3c72ee30 00007ff7`096f54a0     Unity!ScriptingInvocation::Invoke+0x21d
    82. 50 000000b2`3c72ef80 00007ff7`096eca7e     Unity!Application::ParseARGVCommands+0x830
    83. 51 000000b2`3c72f360 00007ff7`0a0b4553     Unity!Application::FinishLoadingProject+0x22e
    84. 52 000000b2`3c72f470 00007ff7`0d1b6f02     Unity!WinMain+0xc33
    85. 53 000000b2`3c73f740 00007ffb`40397034     Unity!__scrt_common_main_seh+0x106
    86. 54 000000b2`3c73f780 00007ffb`41ea2651     kernel32!BaseThreadInitThunk+0x14
    87. 55 000000b2`3c73f7b0 00000000`00000000     ntdll!RtlUserThreadStart+0x21
     
  9. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Does this happen in the Unity editor, or in a Unity player? And does this happen only with incremental GC is enabled?
     
  10. WentaoWu

    WentaoWu

    Joined:
    Jul 12, 2022
    Posts:
    2
    This happen in Unity Editor when BuildPipeline::CompilePlayerScripts is invoked. This problem occurs occasionally.

    Incremental GC is enabled in Build Player Setting. But I think this option only affect the final application.
     
  11. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Thanks for the details, can you submit a bug report for this? This is the kind of thing that we need to debug deeply.
     
  12. Morseliot

    Morseliot

    Joined:
    Jan 10, 2015
    Posts:
    74
    Hi,
    Have encountered same issue. Using unity 2020.3.35. Incremental GC is disabled.

    The game crashing very often. It is multiplayer, so it's crashing on multiple clients at the same time. Here is stack trace:
    IL2CPP
    Code (Boo):
    1. ======= OUTPUTTING STACK TRACE ==================
    2.  
    3. 0x00007FFAEF60EFE2 (GameAssembly) [c:\program files\unity\2020.3.35f1\editor\data\il2cpp\external\bdwgc\malloc.c:333] GC_malloc_kind
    4. 0x00007FFAEF5F743F (GameAssembly) [c:\program files\unity\2020.3.35f1\editor\data\il2cpp\libil2cpp\vm\array.cpp:128] il2cpp::vm::Array::NewSpecific
    5. 0x00007FFAF08F6388 (GameAssembly) [d:\escapismsoftworks\quad force\camps2020\library\il2cppbuildcache\windows\x64\il2cppoutput\photon3unity3d2.cpp:19349] StreamBuffer_ToArray_m84C066019D36E263104C521B4EB3CC7002C7D536
    6. 0x00007FFAF08F17D6 (GameAssembly) [d:\escapismsoftworks\quad force\camps2020\library\il2cppbuildcache\windows\x64\il2cppoutput\photon3unity3d2.cpp:11474] SocketTcp_ReceiveLoop_m90541B0DC94FC026611E1A2B6140F7D7D90BC29F
    7. 0x00007FFAF03DA632 (GameAssembly) [d:\escapismsoftworks\quad force\camps2020\library\il2cppbuildcache\windows\x64\il2cppoutput\mscorlib19.cpp:24069] ThreadStart_Invoke_m784444605A7ADF6A9948A4587DF33CDAC1B06536
    8. 0x00007FFAEFD1753D (GameAssembly) [d:\escapismsoftworks\quad force\camps2020\library\il2cppbuildcache\windows\x64\il2cppoutput\mscorlib2.cpp:43070] ContextCallback_Invoke_mF4F8496213E8F0925947DD8994A477AE2E54EFDF
    9. 0x00007FFAF04FF097 (GameAssembly) [d:\escapismsoftworks\quad force\camps2020\library\il2cppbuildcache\windows\x64\il2cppoutput\mscorlib8.cpp:18943] ExecutionContext_RunInternal_mC5D58D6EDE270B4CDA05181E9064E040D6692B2B
    10. 0x00007FFAF04FF220 (GameAssembly) [d:\escapismsoftworks\quad force\camps2020\library\il2cppbuildcache\windows\x64\il2cppoutput\mscorlib8.cpp:18788] ExecutionContext_Run_m818AB2408824BF47B0D639082CDC164DBF9259D4
    11. 0x00007FFAF03DA632 (GameAssembly) [d:\escapismsoftworks\quad force\camps2020\library\il2cppbuildcache\windows\x64\il2cppoutput\mscorlib19.cpp:24069] ThreadStart_Invoke_m784444605A7ADF6A9948A4587DF33CDAC1B06536
    12. 0x00007FFAEF42711C (GameAssembly) [d:\escapismsoftworks\quad force\camps2020\library\il2cppbuildcache\windows\x64\il2cppoutput\il2cppinvokertable.cpp:114168] RuntimeInvoker_TrueVoid_t700C6383A2A510C2CF4DD86DABD5CA9FF70ADAC5
    13. 0x00007FFAEF5F2C29 (GameAssembly) [c:\program files\unity\2020.3.35f1\editor\data\il2cpp\libil2cpp\vm\runtime.cpp:568] il2cpp::vm::Runtime::Invoke
    14. 0x00007FFAEF5CF61F (GameAssembly) [c:\program files\unity\2020.3.35f1\editor\data\il2cpp\libil2cpp\icalls\mscorlib\system.threading\thread.cpp:158] il2cpp::icalls::mscorlib::System::Threading::ThreadStart
    15. 0x00007FFAEF5B9892 (GameAssembly) [c:\program files\unity\2020.3.35f1\editor\data\il2cpp\libil2cpp\os\thread.cpp:186] il2cpp::os::Thread::RunWrapper
    16. 0x00007FFAEF5BAA53 (GameAssembly) [c:\program files\unity\2020.3.35f1\editor\data\il2cpp\libil2cpp\os\win32\threadimpl.cpp:28] il2cpp::os::ThreadStartWrapper
    17. 0x00007FFBA09F7034 (KERNEL32) BaseThreadInitThunk
    18. 0x00007FFBA21C2651 (ntdll) RtlUserThreadStart
    19.  
    20. ========== END OF STACKTRACE ===========
    Test with Mono. Crashed simultaneously on client and master.
    Player (Master)
    Code (Boo):
    1. ========== OUTPUTTING STACK TRACE ==================
    2.  
    3. 0x00007FFAEFA21A5A (mono-2.0-bdwgc) GC_malloc_kind
    4. 0x00007FFAEF827689 (mono-2.0-bdwgc) mono_object_hash
    5. 0x00007FFAEF7C9BC5 (mono-2.0-bdwgc) mono_object_new_alloc_specific
    6. 0x00007FFAEF7CA0FA (mono-2.0-bdwgc) mono_object_new_specific
    7. 0x00007FFAEF7CF503 (mono-2.0-bdwgc) mono_vtable_get_static_field_data
    8. 0x0000023A66BC1004 (Mono JIT Code) (wrapper managed-to-native) object:__icall_wrapper_ves_icall_object_new_specific (intptr)
    9. 0x0000023CC752603B (Mono JIT Code) System.NumberFormatter/CustomInfo:Parse (string,int,int,System.Globalization.NumberFormatInfo)
    10. 0x0000023CC752524B (Mono JIT Code) System.NumberFormatter:FormatCustom (string,System.Globalization.NumberFormatInfo)
    11. 0x0000023C14339463 (Mono JIT Code) System.NumberFormatter:IntegerToString (string,System.IFormatProvider)
    12. 0x0000023C143385BB (Mono JIT Code) System.NumberFormatter:NumberToString (string,int,System.IFormatProvider)
    13. 0x0000023C143384F3 (Mono JIT Code) System.Number:FormatInt32 (int,string,System.Globalization.NumberFormatInfo)
    14. 0x0000023CEC098FDB (Mono JIT Code) int:ToString (string,System.IFormatProvider)
    15. 0x0000023A66BCA201 (Mono JIT Code) System.Text.StringBuilder:AppendFormatHelper (System.IFormatProvider,string,System.ParamsArray)
    16. 0x0000023A66BC9253 (Mono JIT Code) string:FormatHelper (System.IFormatProvider,string,System.ParamsArray)
    17. 0x0000023CEC03CD3B (Mono JIT Code) string:Format (string,object,object)
    18. 0x0000023C1B94A0A3 (Mono JIT Code) MainBarUI:UpdateUITick ()
    19. 0x0000023C1B949E8C (Mono JIT Code) UIPersonal:OnLocalSyncTick ()
    20. 0x0000023CEC076DB2 (Mono JIT Code) (wrapper delegate-invoke) <Module>:invoke_void ()
    21. 0x0000023C1B946F90 (Mono JIT Code) GameManager:RpcSyncTick (int)
    22. 0x0000023CEE0AEDCC (Mono JIT Code) (wrapper runtime-invoke) <Module>:runtime_invoke_void__this___int (object,intptr,intptr,intptr)
    23. 0x00007FFAEF83F1E0 (mono-2.0-bdwgc) mono_get_runtime_build_info
    24. 0x00007FFAEF7C2AC2 (mono-2.0-bdwgc) mono_perfcounters_init
    25. 0x00007FFAEF7CBCF2 (mono-2.0-bdwgc) mono_runtime_invoke_array
    26. 0x00007FFAEF7CC489 (mono-2.0-bdwgc) mono_runtime_set_main_args
    27. 0x00007FFAEF7CBC86 (mono-2.0-bdwgc) mono_runtime_invoke_array
    28. 0x00007FFAEF7704A4 (mono-2.0-bdwgc) mono_lookup_internal_call
    29. 0x0000023CB9F64290 (Mono JIT Code) (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
    30. 0x0000023CB9F63DFB (Mono JIT Code) System.Reflection.MonoMethod:Invoke (object,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo)
    31. 0x0000023CB9F63D1E (Mono JIT Code) System.Reflection.MethodBase:Invoke (object,object[])
    32. 0x0000023CB9F6299B (Mono JIT Code) Photon.Pun.PhotonNetwork:ExecuteRpc (ExitGames.Client.Photon.Hashtable,Photon.Realtime.Player)
    33. 0x0000023CB9F609FB (Mono JIT Code) Photon.Pun.PhotonNetwork:RPC (Photon.Pun.PhotonView,string,Photon.Pun.RpcTarget,Photon.Realtime.Player,bool,object[])
    34. 0x0000023CB9F17DDB (Mono JIT Code) Photon.Pun.PhotonNetwork:RPC (Photon.Pun.PhotonView,string,Photon.Pun.RpcTarget,bool,object[])
    35. 0x0000023CB9F17C73 (Mono JIT Code) Photon.Pun.PhotonView:RPC (string,Photon.Pun.RpcTarget,object[])
    36. 0x0000023C1B9381E3 (Mono JIT Code) GameManager/<GameMainTickOp>d__134:MoveNext ()
    37. 0x0000023C1442837C (Mono JIT Code) UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)
    38. 0x0000023C144284A7 (Mono JIT Code) (wrapper runtime-invoke) <Module>:runtime_invoke_void_object_intptr (object,intptr,intptr,intptr)
    39. 0x00007FFAEF83F1E0 (mono-2.0-bdwgc) mono_get_runtime_build_info
    40. 0x00007FFAEF7C2AC2 (mono-2.0-bdwgc) mono_perfcounters_init
    41. 0x00007FFAEF7CBB1F (mono-2.0-bdwgc) mono_runtime_invoke
    42. 0x00007FFAF076DADD (UnityPlayer) UnityMain
    43. 0x00007FFAF076ABD2 (UnityPlayer) UnityMain
    44. 0x00007FFAF074E055 (UnityPlayer) UnityMain
    45. 0x00007FFAF04C6054 (UnityPlayer) UnityMain
    46. 0x00007FFAF05F839A (UnityPlayer) UnityMain
    47. 0x00007FFAF05F8440 (UnityPlayer) UnityMain
    48. 0x00007FFAF05FC6D8 (UnityPlayer) UnityMain
    49.   ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FFAF03BC95A)
    50. 0x00007FFAF03BC95A (UnityPlayer) (function-name not available)
    51.   ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FFAF03BAE2B)
    52. 0x00007FFAF03BAE2B (UnityPlayer) (function-name not available)
    53.   ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FFAF03BFF92)
    54. 0x00007FFAF03BFF92 (UnityPlayer) (function-name not available)
    55. 0x00007FFAF03C0F1B (UnityPlayer) UnityMain
    56.   ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF60F5411F2)
    57. 0x00007FF60F5411F2 (CAMPS) (function-name not available)
    58. 0x00007FFBA09F7034 (KERNEL32) BaseThreadInitThunk
    59. 0x00007FFBA21C2651 (ntdll) RtlUserThreadStart
    60.  
    61. ========== END OF STACKTRACE ===========
    Editor (Client)
    Code (Boo):
    1. ========== OUTPUTTING STACK TRACE ==================
    2.  
    3. 0x00007FFB24691A5A (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\external\bdwgc\malloc.c:308] GC_malloc_kind_global
    4. 0x00007FFB2449774D (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\boehm-gc.c:1083] mono_gc_alloc_string
    5. 0x00007FFB2443D3E1 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:6242] mono_string_new_size_checked
    6. 0x00007FFB2443D524 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:6147] mono_string_new_utf16_handle
    7. 0x00007FFB243D99F4 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\w32file.c:281] ves_icall_System_IO_MonoIO_FindNextFile
    8. 0x0000025D3DBAAAFA (Mono JIT Code) (wrapper managed-to-native) System.IO.MonoIO:FindNextFile (intptr,string&,int&,int&)
    9. 0x0000025D3DBAA78B (Mono JIT Code) System.IO.FileSystemEnumerableIterator`1<TSource_REF>:MoveNext ()
    10. 0x0000025CD14F936A (Mono JIT Code) System.Collections.Generic.List`1<T_REF>:.ctor (System.Collections.Generic.IEnumerable`1<T_REF>)
    11. 0x0000025D3DBA7C5B (Mono JIT Code) System.IO.Directory:InternalGetFileDirectoryNames (string,string,string,bool,bool,System.IO.SearchOption,bool)
    12. 0x0000025D3DBA7BA3 (Mono JIT Code) System.IO.Directory:InternalGetFileSystemEntries (string,string,System.IO.SearchOption)
    13. 0x0000025D3DBA7A9B (Mono JIT Code) System.IO.Directory:GetFileSystemEntries (string,string)
    14. 0x0000025D3DBA7873 (Mono JIT Code) System.IO.DefaultWatcher:DoFiles (System.IO.DefaultWatcherData,string,bool)
    15. 0x0000025D3DBA76EB (Mono JIT Code) System.IO.DefaultWatcher:UpdateDataAndDispatch (System.IO.DefaultWatcherData,bool)
    16. 0x0000025D3DBA687B (Mono JIT Code) System.IO.DefaultWatcher:Monitor ()
    17. 0x0000025D3C6EC6A6 (Mono JIT Code) System.Threading.ThreadHelper:ThreadStart_Context (object)
    18. 0x0000025D3C6EC36E (Mono JIT Code) System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
    19. 0x0000025D3C6EC15B (Mono JIT Code) System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
    20. 0x0000025D3C6EBF1B (Mono JIT Code) System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object)
    21. 0x0000025D3C6EBD73 (Mono JIT Code) System.Threading.ThreadHelper:ThreadStart ()
    22. 0x0000025D3C6EBCD4 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
    23. 0x00007FFB244AF1E0 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\mini\mini-runtime.c:2849] mono_jit_runtime_invoke
    24. 0x00007FFB24432AC2 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:2921] do_runtime_invoke
    25. 0x00007FFB2443BCF2 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\object.c:3073] mono_runtime_invoke_checked
    26. 0x00007FFB24456C2F (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\threads.c:1077] start_wrapper_internal
    27. 0x00007FFB24456976 (mono-2.0-bdwgc) [c:\build\output\unity-technologies\mono\mono\metadata\threads.c:1127] start_wrapper
    28. 0x00007FFBA09F7034 (KERNEL32) BaseThreadInitThunk
    29. 0x00007FFBA21C2651 (ntdll) RtlUserThreadStart
    30.  
    31. ========== END OF STACKTRACE ===========
     
    Last edited: Sep 5, 2022
  13. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    I don't think that we ever had a bug report for this issue. Can you submit one?
     
  14. Morseliot

    Morseliot

    Joined:
    Jan 10, 2015
    Posts:
    74
    Hi, I would like to report it, but the problem is that I still can't isolate the problem or get the precise pattern of it's appearing.
    Yesterday I've moved to 2020.3.38. Today we did some tests and the issue is still exists.

    The only thing I can say for sure, it mostly happens when something is "exploded" (allocated) during the game. But the pattern of this issue is unpredictable. It happen on both clients simultaneously, or might be only on one client.
    I did external tests with debug "explosion" button to spam explosions over the map. It may take over couple hundred explosions with no issue, but next time just few with exactly same crash for both clients.
     
  15. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Thanks for checking on this! I don't have a good sense yet about what the cause is from these call stacks alone. Please report any additional information you find here though. Every little bit of data will help us eventually track down the root cause.
     
    Morseliot likes this.
  16. Morseliot

    Morseliot

    Joined:
    Jan 10, 2015
    Posts:
    74
    Sure, I'll keep looking for a reason as it's high priority atm. Maybe even have to try it on previous versions of engine. The project is already third year in development, and there wasn't any issue before. Ill be right back after I have more details.
     
    JoshPeterson likes this.
  17. Morseliot

    Morseliot

    Joined:
    Jan 10, 2015
    Posts:
    74
    Ok, the issue was finally isolated. As I said it was on explosion during memory allocation. There is a method that modifying the terrain. You may try it yourself by putting this into infinity loop.
    Code (CSharp):
    1.  
    2.  
    3. private TerrainData GetTerrainData() => _targetTerrain.terrainData;
    4. private Vector3 GetTerrainSize() => GetTerrainData().size;
    5.  
    6. public Vector3 WorldToTerrainPosition(Vector3 worldPosition)
    7.     {
    8.         var terrainPosition = worldPosition - _targetTerrain.GetPosition();
    9.  
    10.         var terrainSize = GetTerrainSize();
    11.  
    12.         var heightmapResolution = GetHeightmapResolution();
    13.  
    14.         terrainPosition = new Vector3(terrainPosition.x / terrainSize.x, terrainPosition.y / terrainSize.y, terrainPosition.z / terrainSize.z);
    15.  
    16.         return new Vector3(terrainPosition.x * heightmapResolution, 0, terrainPosition.z * heightmapResolution);
    17.     }
    18.  
    19.     public Vector2Int GetBrushPosition(Vector3 worldPosition, int brushWidth, int brushHeight)
    20.     {
    21.         var terrainPosition = WorldToTerrainPosition(worldPosition);
    22.  
    23.         var heightmapResolution = GetHeightmapResolution();
    24.  
    25.         return new Vector2Int((int)Mathf.Clamp(terrainPosition.x - brushWidth / 2.0f, 0.0f, heightmapResolution), (int)Mathf.Clamp(terrainPosition.z - brushHeight / 2.0f, 0.0f, heightmapResolution));
    26.     }
    27.  
    28.     public Vector2Int GetSafeBrushSize(int brushX, int brushY, int brushWidth, int brushHeight)
    29.     {
    30.         var heightmapResolution = GetHeightmapResolution();
    31.  
    32.         while (heightmapResolution - (brushX + brushWidth) < 0) brushWidth--;
    33.  
    34.         while (heightmapResolution - (brushY + brushHeight) < 0) brushHeight--;
    35.  
    36.         return new Vector2Int(brushWidth, brushHeight);
    37.     }
    38.  
    39. public void LowerTerrain(Vector3 worldPosition, float strength, int brushWidth, int brushHeight)
    40.     {
    41.         var brushPosition = GetBrushPosition(worldPosition, brushWidth, brushHeight);
    42.  
    43.         var brushSize = GetSafeBrushSize(brushPosition.x, brushPosition.y, brushWidth, brushHeight);
    44.  
    45.         var tempHeights = GetTerrainData().GetHeights(brushPosition.x, brushPosition.y, brushSize.x, brushSize.y);
    46.         var tempDetails = GetTerrainData().GetDetailLayer(brushPosition.x, brushPosition.y, brushSize.x, brushSize.y, 0);
    47.  
    48.         for (var y = 0; y < brushSize.y; y++)
    49.         {
    50.             for (var x = 0; x < brushSize.x; x++)
    51.             {
    52.                 tempHeights[y, x] -= strength * Time.fixedDeltaTime;
    53.             }
    54.         }
    55.         GetTerrainData().SetHeights(brushPosition.x, brushPosition.y, tempHeights);// terrainData.SetHeightsDelayLOD(brushPosition.x, brushPosition.y, heights);
    56.         //terrainData.SyncHeightmap();
    57.     }
    The engine does silent crash at some point if "LowerTerrain" function are repeatedly invoking.
     
    Last edited: Sep 9, 2022
  18. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Morseliot likes this.
  19. sandeepsmartest

    sandeepsmartest

    Joined:
    Nov 7, 2012
    Posts:
    139
  20. YichenYu525

    YichenYu525

    Joined:
    Dec 8, 2020
    Posts:
    11
    So any solution for this GC problem? or any way to get around it ?
     
  21. jjxtra

    jjxtra

    Joined:
    Aug 30, 2013
    Posts:
    1,464
    Still see in Unity 2021 latest version
     
  22. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    I don't recall if we ever had a bug report on this issue. @jjxtra can you report one so that we can investigate?
     
  23. Armynator

    Armynator

    Joined:
    Feb 15, 2013
    Posts:
    67
    I constantly get "Fatal error in GC" "Unexpected mark stack overflow" in a IL2CPP server build with a high amount of GameObjects and colliders. Minimal reproduction project with 100% reproduction rate is attached to IN-64035.
    Incremental on/off doesn't make a difference. Mono seems to work fine, only IL2CPP is affected.

    After reading through this thread, this might be the issue. The usecase for us is a persistent game server with a fairly big static game world, always crashing with the GC error at startup when trying to load the world. With a smaller world or with Mono backend it's still working fine.

    Edit:
    There it is - UUM-59544
     
    Last edited: Dec 22, 2023
  24. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Thanks for the bug report! We will investigate it.
     
  25. Strom_CL

    Strom_CL

    Joined:
    Nov 17, 2012
    Posts:
    115
    Seeing a very similar "Fatal error in GC" crash with 30+ gameObjects with colliders all being interacted with via some Physics.OverlapShere calls, probably 4-5 calls per second on each of those game objects. Mono does not crash, but IL2CPP is blowing up maybe 1 in 30-40 play throughts with high densities of enemies being interacted with.
     
  26. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Is there anything reproducible here that we could look at in a bug report? This feels like something we will need to investigate for a while to pin down the cause.
     
  27. Strom_CL

    Strom_CL

    Joined:
    Nov 17, 2012
    Posts:
    115
    Not really, my tester has only seen the crash twice in about 90 sessions and since its only in the IL2CPP standalone builds, it makes it near impossible to see what's going on. The player.log also didn't have anything of use either. Is there any extra logging that can be enabled for il2cpp builds to help provide some more info?
     
  28. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    No, I don't think there is anything built in that will help with this, sorry!
     
  29. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    23
    We encountered a similar issue as our application also crashes with an "Unexpected mark stack overflow" exception so I thought I report it here since this thread was recently active. Our application exists for years now and this is the first time we came across this issue.
    We use a Windows 64 build using Mono and .NET. We tested using latest Unity LTS 2022.3.20f1 and 2023.2.12f1 with the same result. We didn't test with the latest beta yet. The crash happens in a time frame from a few hours to up to 24 hours. When we disable 'Incremental GC' the application runs for days. So we are pretty positive that this is what causes the crash. But of course we don't want to have to disable the incremental GC feature. We also found out that the problem seems to be related to part of our application that runs a thread to receive audio via a C++ plugin. If we don't run this thread the application also runs for days. But there is no obvious culprit there and as mentioned it also doesn't crash with incremental GC disabled. We already tried lots of different approaches on the audio thread and got rid of any unnecessary allocations. The crash logs show that the crash always happens at different parts of the application but the last lines of the stack trace are always the same:
    Code (CSharp):
    1. 0x00007FFD39ED14F6 (mono-2.0-bdwgc) mono_breakpoint_clean_code
    2. 0x00007FFD39C4F35B (mono-2.0-bdwgc) mono_gc_register_root
    3. 0x00007FFD39ECEB6A (mono-2.0-bdwgc) mono_breakpoint_clean_code
    4. 0x00007FFD39ECEA1B (mono-2.0-bdwgc) mono_breakpoint_clean_code
    5. 0x00007FFD39ED7D0D (mono-2.0-bdwgc) mono_breakpoint_clean_code
    6. 0x00007FFD39EC2DD6 (mono-2.0-bdwgc) mono_breakpoint_clean_code
    7. 0x00007FFD39EC1142 (mono-2.0-bdwgc) mono_breakpoint_clean_code
    8. 0x00007FFD39EC9D4F (mono-2.0-bdwgc) mono_breakpoint_clean_code
    9. 0x00007FFD39EC98BF (mono-2.0-bdwgc) mono_breakpoint_clean_code
    10. 0x00007FFD39ECDD33 (mono-2.0-bdwgc) mono_breakpoint_clean_code
    11. 0x00007FFD39C4F771 (mono-2.0-bdwgc) mono_gc_register_root
    12. 0x00007FFD39BB7332 (mono-2.0-bdwgc) mono_array_new_specific
    13. 0x00007FFD39BB739B (mono-2.0-bdwgc) mono_array_new_specific
    14. 0x0000016599852A2E (Mono JIT Code) (wrapper managed-to-native) object:__icall_wrapper_ves_icall_array_new_specific (intptr,int)
    Does this look familiar/are there comparable issues or should be create a new bug report? It will be hard though to create a project for recreating the issue. I attached a few more complete stack traces in a file and here is also the code from our audio receiving thread:

    Code (CSharp):
    1. private void AudioCaptureLoop()
    2.         {
    3.             while (!exitThread && NdiReceiver != IntPtr.Zero)
    4.             {
    5.                 var receivedFrameType = NDIlib.recv_capture_v2(NdiReceiver, null, ref receivedAudioFrame, null, 1000);
    6.  
    7.                 if (exitThread || receivedFrameType != NDIlib.frame_type_e.frame_type_audio ||
    8.                     volume == 0 || receivedAudioFrame.p_data == IntPtr.Zero || receivedAudioFrame.no_samples == 0)
    9.                 {
    10.                     NDIlib.recv_free_audio_v2(NdiReceiver, ref receivedAudioFrame);
    11.                     continue;
    12.                 }
    13.  
    14.                 if (!exitThread)
    15.                 {
    16.                     lock (audioLock)
    17.                     {
    18.                         ConfigureAudioOutput(receivedAudioFrame);
    19.                         if (!exitThread && AudioOutputInitialized)
    20.                             PlayReceivedAudio();
    21.                     }
    22.                 }
    23.  
    24.                 NDIlib.recv_free_audio_v2(NdiReceiver, ref receivedAudioFrame);
    25.             }
    26.          
    27.             Log.Trace($"Stopped audio capture for NDI stream '{NdiSource.SourceName}'");
    28.         }
    29.  
    30.         /// <summary>
    31.         /// Convert the received audio data to the required format and add to output buffer.
    32.         /// </summary>
    33.         private void PlayReceivedAudio()
    34.         {
    35.             // CSCore is expecting interleaved audio but NDI uses planar
    36.             interleavedFrame.sample_rate = receivedAudioFrame.sample_rate;
    37.             interleavedFrame.no_channels = receivedAudioFrame.no_channels;
    38.             interleavedFrame.no_samples = receivedAudioFrame.no_samples;
    39.             interleavedFrame.timecode = receivedAudioFrame.timecode;
    40.  
    41.             // We're working in bytes, so take the size of a 32 bit sample (float) into account
    42.             var bufferSize = receivedAudioFrame.no_samples * receivedAudioFrame.no_channels * sizeof(float);
    43.             var audioBuffer = GetOrCreateAudioBuffer(bufferSize);
    44.      
    45.             interleavedFrame.p_data = audioBuffer.Pointer;
    46.             NDIlib.util_audio_to_interleaved_32f_v2(ref receivedAudioFrame, ref interleavedFrame);
    47.             bufferingSource.Write(audioBuffer.Buffer, 0, audioBuffer.Buffer.Length);
    48.         }
     

    Attached Files:

  30. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Thanks for the information! I'm going to guess that the symbolication of the call stack in the Mono native code is not correct. These methods are only called when the script debugging option is enabled. To confirm - are you seeing this when script debugging is disabled?

    In this case, I would look closely at the memory behavior of the C++ code in the audio plugin. Without seeing that code, I would guess that it corrupting some managed memory. That kind of thing can be exposed by incremental GC, as the incremental mode is pretty strict about requiring write barriers when managed objects are changed.

    To be clear, if the native code is behaving correctly, then it should work find with incremental GC, but since there are crashes happening, I would look closely at that code first. We've seen similar issues like this before.
     
  31. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    23
    Thanks for your quick reply! We actually don't have script debugging or any other development option enabled in our build settings for these tests. Unfortunately the C++ code is Vizrt's SDK for receiving NDI streams and is closed source.
    In the meantime we ran the application with the latest Unity beta which also crashed after a few hours. We're also currently running a test where we commented out everything but the parts accessing the C++ SDK which seems to run fine for about 2 days now. Something we also tried was using NativeArrays for the audio data and passing its pointer to the C++ plugin. But that didn't change anything. If you have any other information/ideas for what we could try we would be very thankful.
     
  32. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Thanks for the additional details!

    So it does look like the Mono stack frames are not correct then, since you don't have script debugging enabled. The lack of correct stack frames might make tracking down the cause more difficult.

    I would still focus on that C++ plugin. Can you put it into a standalone .NET Framework executable? I'd be curious to know if it causes problems outside of Unity. That might help us indicate if the issue is with that library or with the Mono p/invoke implementation.
     
  33. daven_tk

    daven_tk

    Joined:
    Nov 15, 2022
    Posts:
    3
    I also encountered a similar problem, is there any solution?