Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Bug Il2cpp crash at malloc

Discussion in 'Experimental Scripting Previews' started by Kamyker, Jul 20, 2021.

  1. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    I'm trying to use ILRuntime that uses pointers heavily. Everything works fine in mono but il2cpp crashes overtime at something like:
    Code (CSharp):
    1. var arr = new float[bignumber];
    I'm not sure why or what is the issue exactly. Stack trace and logs in: https://github.com/Ourpalm/ILRuntime/issues/540

    Could there be issue that pointers aren't allocated correctly? If so then why il2cpp crashses and mono doesn't?

    Unity 2020.3.14
     
  2. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    Thanks for reporting this issue! We actually discovered it internally last week as well. We have a fix coming to all Unity versions. I'll reporting back here when the fix is available in 2020.3.
     
  3. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    Thanks, fingers crossed for it being included in 2020.3.16 as the bug is currently delaying my game update.


    Off-topic: When trying mono instead of il2cpp I've noticed that instantiating UI panels was much faster and smoother. I'll try to make repro project one day, it was something like 20+ panels in scroll rect.
     
    Last edited: Aug 1, 2021
  4. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    I'm not sure yet about the version, I'll let you know as soon as I find out.
     
    Kamyker likes this.
  5. devon_redgames

    devon_redgames

    Joined:
    Apr 3, 2020
    Posts:
    12
    @JoshPeterson we're seeing a similar issue with frequent calls to saving, which seems like an edge-case (perhaps race condition) where we get a GC_generic_malloc etc issue with EX_BAD_ACCESS on iOS.

    2020.3.11f1, 2020.3.13f1

    Here is an example stack of the issue we see:

    upload_2021-8-4_15-36-42.png
     
  6. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    I'm going to guess this is a different issue, as the original problem reported here specifically triggered an assert, where the call stack you provided does not seem to.

    Can you submit a bug report with a project that causes this crash?
     
  7. devon_redgames

    devon_redgames

    Joined:
    Apr 3, 2020
    Posts:
    12
    @JoshPeterson for various reasons, it's difficult for us to replicate this in a simple project due to the usage of native iOS plugins which would require various XCode configuration on your end in order to run.

    However, I do believe these are related to the original post. Consider that the original post concern was around a stack trace consisting of:

    Code (CSharp):
    1.      GameAssembly.dll!il2cpp_assert(const char * assertion, const char * file, unsigned int line) Line 12    C++
    2. >    GameAssembly.dll!GC_gcj_vector_proc(unsigned __int64 * addr, GC_ms_entry * mark_stack_ptr, GC_ms_entry * mark_stack_limit, unsigned __int64 env) Line 67    C++
    3.      GameAssembly.dll!GC_mark_from(GC_ms_entry * mark_stack_top, GC_ms_entry * mark_stack, GC_ms_entry * mark_stack_limit) Line 782    C
    4.      GameAssembly.dll!GC_mark_some_inner(char * cold_gc_frame) Line 361    C
    5.      GameAssembly.dll!GC_mark_some(char * cold_gc_frame) Line 506    C
    6.      GameAssembly.dll!GC_stopped_mark(int(*)() stop_func) Line 832    C
    7.      GameAssembly.dll!GC_try_to_collect_inner(int(*)() stop_func) Line 580    C
    8.      GameAssembly.dll!GC_collect_or_expand(unsigned __int64 needed_blocks, int ignore_off_page, int retry) Line 1484    C
    9.      GameAssembly.dll!GC_alloc_large(unsigned __int64 lb, int k, unsigned int flags) Line 64    C
    10.      GameAssembly.dll!GC_generic_malloc(unsigned __int64 lb, int k) Line 264    C
    11.      GameAssembly.dll!GC_malloc_kind_global(unsigned __int64 lb, int k) Line 327    C
    12.      GameAssembly.dll!GC_malloc_kind(unsigned __int64 lb, int k) Line 334    C
    13.      GameAssembly.dll!GC_malloc_atomic(unsigned __int64 lb) Line 342    C
    14.      GameAssembly.dll!il2cpp::vm::Object::AllocatePtrFree(unsigned __int64 size, Il2CppClass * typeInfo) Line 67    C++
    15.      GameAssembly.dll!il2cpp::vm::Array::NewSpecific(Il2CppClass * klass, unsigned __int64 n) Line 128    C++
    16.      GameAssembly.dll!SZArrayNew(Il2CppClass * arrayType, unsigned int length) Line 121    C++
    Which is an almost identical crash stack location of ours; consider:

    upload_2021-8-5_11-55-5.png

    It's the shared root of il2cpp::vm::Array::NewSpecific(Il2CppClass*, unsigned long) () seems like a regression in these versions of Unity.

    Meanwhile, we'll continue to try to create a small reproduction project.

    Best regards,
     
  8. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    @Kamyker The fix should be in 2020.3.16.
     
  9. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    You may indeed be correct. When 2020.3.16 is out, can you give that a try? It should correct this issue.
     
  10. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    Thank you! Now I only hope it will fix ilruntime crashes.

    That's possible as I was using debug cpp configuration and standalone build.
     
  11. devon_redgames

    devon_redgames

    Joined:
    Apr 3, 2020
    Posts:
    12
    @JoshPeterson unfortunately 2020.3.16 release has not resolve this issue for us. I've created a bug report and will private message you the link.

    Essentially, we're seeing a stack trace that is consistent with an IL2CPP memory allocation issue that seems to be related to BinaryFormatter and subsequent allocations that happen.

    Take the following stack trace as an example:

    upload_2021-8-12_14-32-0.png
     
  12. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    Same, ill try making repro project.

     
  13. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    #Edited

    Case 1357810 or Case 1358515 with much smaller repro project

    After updating some nuget dlls and removing unneeded ones it stopped crashing but only in Release build (or I wasn't lucky enough to hit bad memory cell yet). Debug crashes almost instantly.

    It's a bit scary to release an update with that possible crash, ill ask few players to test it.
     
    Last edited: Aug 16, 2021
  14. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    Thanks both for the information and the bug reports. There may be another issue here than the one we discovered internally - sorry for misleading you. We will investigate these reports.
     
  15. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    Thanks for the update, bug reporter worked finally -> case 1358515.

    I dig through il2cpp output trying to find the cause, here's the stack trace that later on crashes the project on GC.Collect:

    Code (CSharp):
    1. >    GameAssembly.dll!ArrayBuilder_1_ToImmutableAndFree_m8A43B45BFBEA0766547F96B07F3A2EAF5340E0AB_gshared(ArrayBuilder_1_tF965B3BB161639713A1ECA9E0647B21CECFA2798 * __this, const MethodInfo * method) Line 10077    C++
    2.      GameAssembly.dll!ArrayBuilder_1_ToImmutableAndFree_m8A43B45BFBEA0766547F96B07F3A2EAF5340E0AB(ArrayBuilder_1_tF965B3BB161639713A1ECA9E0647B21CECFA2798 * __this, const MethodInfo * method) Line 29200    C++
    3.      GameAssembly.dll!MembersAndInitializersBuilder_ToReadOnlyAndFree_m98071FD4483CE1FA07D02FDF074D798DC8A3E982(MembersAndInitializersBuilder_tEBEDEBF50A9D0A19E1B9DA47E328B4E5BA7BF165 * __this, const MethodInfo * method) Line 32315    C++
    4.      GameAssembly.dll!SourceMemberContainerTypeSymbol_BuildMembersAndInitializers_mA4D602F48CD34C0FBC1B7CA303CCA26A3EE853C5(SourceMemberContainerTypeSymbol_t313BD551412D035E45BF7B0B14FFF7664974CEE9 * __this, DiagnosticBag_t7D7AAB745EABCE30ADF71EA4C8421DDA328A1F45 * ___diagnostics0, const MethodInfo * method) Line 50015    C++
    5.      GameAssembly.dll!SourceMemberContainerTypeSymbol_GetMembersAndInitializers_m8DDC20A7F86D66E413DECF5E79D1B1AB06685531(SourceMemberContainerTypeSymbol_t313BD551412D035E45BF7B0B14FFF7664974CEE9 * __this, const MethodInfo * method) Line 45519    C++
    6.      GameAssembly.dll!SourceMemberContainerTypeSymbol_MakeAllMembers_m29489BFBA38403C2F43D60B6E0263FA16A5C88E2(SourceMemberContainerTypeSymbol_t313BD551412D035E45BF7B0B14FFF7664974CEE9 * __this, DiagnosticBag_t7D7AAB745EABCE30ADF71EA4C8421DDA328A1F45 * ___diagnostics0, const MethodInfo * method) Line 49392    C++
    7.      GameAssembly.dll!SourceMemberContainerTypeSymbol_GetMembersByNameSlow_m62E1F5926BF595BAC4162BEF7FA012F3BF30E9D1(SourceMemberContainerTypeSymbol_t313BD551412D035E45BF7B0B14FFF7664974CEE9 * __this, const MethodInfo * method) Line 45613    C++
    8.      GameAssembly.dll!SourceMemberContainerTypeSymbol_GetMembersByName_m4A43FFB890B7F6A989D885488BDBFCD2C3EEF9CE(SourceMemberContainerTypeSymbol_t313BD551412D035E45BF7B0B14FFF7664974CEE9 * __this, const MethodInfo * method) Line 45572    C++
    9.      GameAssembly.dll!SourceMemberContainerTypeSymbol_ForceComplete_m632E34218B14E9A71A1DFB1B4C7183B8D949A456(SourceMemberContainerTypeSymbol_t313BD551412D035E45BF7B0B14FFF7664974CEE9 * __this, SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * ___locationOpt0, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken1, const MethodInfo * method) Line 42994    C++
    10.      GameAssembly.dll!VirtActionInvoker2<SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 *,CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD>::Invoke(unsigned short slot, Il2CppObject * obj, SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * p1, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD p2) Line 56    C++
    11.      GameAssembly.dll!Symbol_ForceCompleteMemberByLocation_mEC1EDD3D8C57686403F4C49CD3502B9947B3F2CB(SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * ___locationOpt0, Symbol_t2A18793CC4D67272877DCA888FECBC91C98B4F18 * ___member1, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken2, const MethodInfo * method) Line 35137    C++
    12.      GameAssembly.dll!SourceNamespaceSymbol_ForceComplete_mDAA3FE3278B17992048242E31385617B856A1CF1(SourceNamespaceSymbol_tF9318C0E82D3B0292902A2598FB6FFC8B8280B56 * __this, SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * ___locationOpt0, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken1, const MethodInfo * method) Line 43211    C++
    13.      GameAssembly.dll!VirtActionInvoker2<SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 *,CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD>::Invoke(unsigned short slot, Il2CppObject * obj, SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * p1, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD p2) Line 56    C++
    14.      GameAssembly.dll!SourceModuleSymbol_ForceComplete_m44D6EC6B0FEE2E5DB95573A4BF7B3E358EE27966(SourceModuleSymbol_t715B30DA0D77D94F3D533E9679B70E5159039CEB * __this, SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * ___locationOpt0, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken1, const MethodInfo * method) Line 32499    C++
    15.      GameAssembly.dll!VirtActionInvoker2<SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 *,CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD>::Invoke(unsigned short slot, Il2CppObject * obj, SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * p1, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD p2) Line 56    C++
    16.      GameAssembly.dll!SourceAssemblySymbol_ForceComplete_m918F6BE2667526F4553F34AEE3BDCE67C22ECD55(SourceAssemblySymbol_t7FD1FE3518D845F42D877A553A22702BCA4D4AA5 * __this, SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * ___locationOpt0, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken1, const MethodInfo * method) Line 49425    C++
    17.      GameAssembly.dll!VirtActionInvoker2<SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 *,CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD>::Invoke(unsigned short slot, Il2CppObject * obj, SourceLocation_t957F3E4CA0504C73DA18AA5EAACF2573CC7C4232 * p1, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD p2) Line 56    C++
    18.      GameAssembly.dll!CSharpCompilation_GetSourceDeclarationDiagnostics_m88F893CEF32E9AAD00352767299B33C554E7C643(CSharpCompilation_tAF9E622B1BD76CA8394F28E53F36900BC2281706 * __this, SyntaxTree_tCCE97D1A1EDD940F5093E176B8C2AE3688FD384F * ___syntaxTree0, Nullable_1_t152ABB292321E9E09BBB2D5FB029AC6018F19CC5 ___filterSpanWithinTree1, Func_4_t892C0E76E5839586E1D8D8DD306969AA6E4E1451 * ___locationFilterOpt2, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken3, const MethodInfo * method) Line 34727    C++
    19.      GameAssembly.dll!CSharpCompilation_GetDiagnostics_m8B6D6F05728C269F1E71EB0179D3F34E91C93AF9(CSharpCompilation_tAF9E622B1BD76CA8394F28E53F36900BC2281706 * __this, int ___stage0, bool ___includeEarlierStages1, DiagnosticBag_t7D7AAB745EABCE30ADF71EA4C8421DDA328A1F45 * ___diagnostics2, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken3, const MethodInfo * method) Line 34445    C++
    20.      GameAssembly.dll!VirtActionInvoker4<int,bool,DiagnosticBag_t7D7AAB745EABCE30ADF71EA4C8421DDA328A1F45 *,CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD>::Invoke(unsigned short slot, Il2CppObject * obj, int p1, bool p2, DiagnosticBag_t7D7AAB745EABCE30ADF71EA4C8421DDA328A1F45 * p3, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD p4) Line 67    C++
    21.      GameAssembly.dll!CSharpCompilation_GetDiagnostics_m9D0159B87A1973698BC570C2E749EB371A26E0F5(CSharpCompilation_tAF9E622B1BD76CA8394F28E53F36900BC2281706 * __this, int ___stage0, bool ___includeEarlierStages1, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken2, const MethodInfo * method) Line 34025    C++
    22.      GameAssembly.dll!CSharpCompilation_CompileMethods_mBCD99E431AA4828973DE1F6D4BA2A314A8494B8E(CSharpCompilation_tAF9E622B1BD76CA8394F28E53F36900BC2281706 * __this, CommonPEModuleBuilder_t1943D6E05E10E288AE3043209AFB21F76DF8BAC4 * ___moduleBuilder0, bool ___emittingPdb1, bool ___emitMetadataOnly2, bool ___emitTestCoverageData3, DiagnosticBag_t7D7AAB745EABCE30ADF71EA4C8421DDA328A1F45 * ___diagnostics4, Predicate_1_tF8D01B705BFDDD9F897D4A55B28582F47E5AA6BE * ___filterOpt5, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken6, const MethodInfo * method) Line 35156    C++
    23.      GameAssembly.dll!VirtFuncInvoker7<bool,CommonPEModuleBuilder_t1943D6E05E10E288AE3043209AFB21F76DF8BAC4 *,bool,bool,bool,DiagnosticBag_t7D7AAB745EABCE30ADF71EA4C8421DDA328A1F45 *,Predicate_1_tF8D01B705BFDDD9F897D4A55B28582F47E5AA6BE *,CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD>::Invoke(unsigned short slot, Il2CppObject * obj, CommonPEModuleBuilder_t1943D6E05E10E288AE3043209AFB21F76DF8BAC4 * p1, bool p2, bool p3, bool p4, DiagnosticBag_t7D7AAB745EABCE30ADF71EA4C8421DDA328A1F45 * p5, Predicate_1_tF8D01B705BFDDD9F897D4A55B28582F47E5AA6BE * p6, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD p7) Line 89    C++
    24.      GameAssembly.dll!Compilation_Emit_mE15B88505AE8F54E6541376F205C6161BE9E58CC(Compilation_tD549E53F1A626524EC1A24C00023176A789E269B * __this, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___peStream0, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___metadataPEStream1, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___pdbStream2, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___xmlDocumentationStream3, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___win32Resources4, Il2CppObject * ___manifestResources5, EmitOptions_tBF744ECFE28B4FCB31272C70EC5ED5A5568BFBA1 * ___options6, Il2CppObject * ___debugEntryPoint7, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___sourceLinkStream8, Il2CppObject * ___embeddedTexts9, CompilationTestData_tC18A2CD93D11D0F130541710BD895D43D41FA06A * ___testData10, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken11, const MethodInfo * method) Line 26967    C++
    25.      GameAssembly.dll!Compilation_Emit_m795B6AA9529080AE428C3C9995D2D9AC18406B51(Compilation_tD549E53F1A626524EC1A24C00023176A789E269B * __this, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___peStream0, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___pdbStream1, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___xmlDocumentationStream2, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___win32Resources3, Il2CppObject * ___manifestResources4, EmitOptions_tBF744ECFE28B4FCB31272C70EC5ED5A5568BFBA1 * ___options5, Il2CppObject * ___debugEntryPoint6, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___sourceLinkStream7, Il2CppObject * ___embeddedTexts8, Stream_t5DC87DD578C2C5298D98E7802E92DEABB66E2ECB * ___metadataPEStream9, CancellationToken_tC9D68381C9164A4BA10397257E87ADC832AF5FFD ___cancellationToken10, const MethodInfo * method) Line 26830    C++
    26.      GameAssembly.dll!U3CCompileToStreamU3Ed__1_MoveNext_mDB88B245102676A0AAE04272A2DF8E7990D6D1F6(U3CCompileToStreamU3Ed__1_tC2395E9F78C31CFABB5A8842ACBF22108D64E8D4 * __this, const MethodInfo * method) Line 5024    C++
    27.      GameAssembly.dll!U3CCompileToStreamU3Ed__1_MoveNext_mDB88B245102676A0AAE04272A2DF8E7990D6D1F6_AdjustorThunk(Il2CppObject * __this, const MethodInfo * method) Line 5076    C++
    28.      GameAssembly.dll!InterfaceActionInvoker0::Invoke(unsigned short slot, Il2CppClass * declaringInterface, Il2CppObject * obj) Line 165    C++
    29.      GameAssembly.dll!MoveNextRunner_InvokeMoveNext_m6F87A8006A8DD916BAAA543205E987AC6E4E1906(Il2CppObject * ___stateMachine0, const MethodInfo * method) Line 31466    C++
    30.      GameAssembly.dll!ContextCallback_Invoke_mF4F8496213E8F0925947DD8994A477AE2E54EFDF(ContextCallback_t93707E0430F4FF3E15E1FB5A4844BE89C657AE8B * __this, Il2CppObject * ___state0, const MethodInfo * method) Line 41104    C++
    31.      GameAssembly.dll!ExecutionContext_RunInternal_mC5D58D6EDE270B4CDA05181E9064E040D6692B2B(ExecutionContext_t16AC73BB21FEEEAD34A017877AC18DD8BB836414 * ___executionContext0, ContextCallback_t93707E0430F4FF3E15E1FB5A4844BE89C657AE8B * ___callback1, Il2CppObject * ___state2, bool ___preserveSyncCtx3, const MethodInfo * method) Line 19393    C++
    32.      GameAssembly.dll!ExecutionContext_Run_mD1481A474AE16E77BD9AEAF5BD09C2819B60FB29(ExecutionContext_t16AC73BB21FEEEAD34A017877AC18DD8BB836414 * ___executionContext0, ContextCallback_t93707E0430F4FF3E15E1FB5A4844BE89C657AE8B * ___callback1, Il2CppObject * ___state2, bool ___preserveSyncCtx3, const MethodInfo * method) Line 19256    C++
    33.      GameAssembly.dll!MoveNextRunner_Run_mF9986F86D538F629861F62DD912B18CC58980D8B(MoveNextRunner_tFAEA0BEDD353E2E34E8E287C67B1F5572FD30C2D * __this, const MethodInfo * method) Line 31417    C++
    34.      GameAssembly.dll!Action_Invoke_m3FFA5BE3D64F0FF8E1E1CB6F953913FADB5EB89E(Action_tAF41423D285AE0862865348CF6CE51CD085ABBA6 * __this, const MethodInfo * method) Line 21669    C++
    35.      GameAssembly.dll!U3CU3Ec_U3C_cctorU3Eb__7_0_m0B62ED556B3855D4C64C5D7E638D6ED8AB0C4A88(U3CU3Ec_t97DE2C4F7EF16C425D7DB74D03F1E0947B3D9AF2 * __this, Il2CppObject * ___state0, const MethodInfo * method) Line 40837    C++
    36.      GameAssembly.dll!SendOrPostCallback_Invoke_m352534ED0E61440A793944CC44809F666BBC1461(SendOrPostCallback_t6B7334CE017AF595535507519400AC02D688DC3C * __this, Il2CppObject * ___state0, const MethodInfo * method) Line 19778    C++
    37.      GameAssembly.dll!WorkRequest_Invoke_m1C292B7297918C5F2DBE70971895FE8D5C33AA20(WorkRequest_tA19FD4D1269D8EE2EA886AAF036C4F7F09154393 * __this, const MethodInfo * method) Line 38347    C++
    38.      GameAssembly.dll!UnitySynchronizationContext_Exec_mC89E49BFB922E69AAE753887480031A142016F81(UnitySynchronizationContext_t9971A8B24E203428BF2E715ECC6019EE2D77EAD3 * __this, const MethodInfo * method) Line 28926    C++
    39.      GameAssembly.dll!UnitySynchronizationContext_ExecuteTasks_m323E27C0CD442B806D966D024725D9809563E0DD(const MethodInfo * method) Line 29039    C++
    40.      GameAssembly.dll!RuntimeInvoker_FalseVoid_t700C6383A2A510C2CF4DD86DABD5CA9FF70ADAC5(void(*)() methodPointer, const MethodInfo * methodMetadata, void * obj, void * * args) Line 221379    C++
    41.      GameAssembly.dll!il2cpp::vm::Runtime::Invoke(const MethodInfo * method, void * obj, void * * params, Il2CppException * * exc) Line 568    C++
    42.      GameAssembly.dll!il2cpp_runtime_invoke(const MethodInfo * method, void * obj, void * * params, Il2CppException * * exc) Line 1118    C++
    43.      UnityPlayer.dll!scripting_method_invoke(class ScriptingMethodPtr,class ScriptingObjectPtr,struct ScriptingArguments &,class ScriptingExceptionPtr *,bool)    Unknown
    44.      UnityPlayer.dll!ScriptingInvocation::Invoke(class ScriptingExceptionPtr *,bool)    Unknown
    45.      UnityPlayer.dll!ScriptingInvocation::Invoke<void>(class ScriptingExceptionPtr *,bool)    Unknown
    46.      UnityPlayer.dll!Scripting::UnityEngine::UnitySynchronizationContextProxy::ExecuteTasks(class ScriptingExceptionPtr *)    Unknown
    47.      UnityPlayer.dll!`InitPlayerLoopCallbacks'::`2'::UpdateScriptRunDelayedTasksRegistrator::Forward()    Unknown
    48.      UnityPlayer.dll!ExecutePlayerLoop(struct NativePlayerLoopSystem *)    Unknown
    49.      UnityPlayer.dll!ExecutePlayerLoop(struct NativePlayerLoopSystem *)    Unknown
    50.      UnityPlayer.dll!PlayerLoop(void)    Unknown
    51.      UnityPlayer.dll!PerformMainLoop()    Unknown
    52.      UnityPlayer.dll!MainMessageLoop()    Unknown
    53.      UnityPlayer.dll!UnityMainImpl(struct HINSTANCE__ *,struct HINSTANCE__ *,wchar_t *,int)    Unknown
    54.      UnityPlayer.dll!UnityMain()    Unknown
     
    JoshPeterson likes this.
  16. devon_redgames

    devon_redgames

    Joined:
    Apr 3, 2020
    Posts:
    12
    For the sake of history, we also see a similar crash stack when deserializing spine animations from an async asset bundle load. It's not consistent and seems to suggest a race condition that causes memory corruption in the Il2CPP vm runtime. The stack trace is as follows:
    Code (CSharp):
    1. GC_mark_from (in UnityFramework) (mark.c:851)
    2. GC_mark_from (in UnityFramework) (mark.c:0)
    3. ..
    4. GC_allocobj (in UnityFramework) (alloc.c:1568)
    5. ..
    6. il2cpp::vm::String::NewSize(int) (in UnityFramework) (String.cpp:113)
    7. String_CreateString_m16F181739FD8BA877868803DE2CE0EF0A4668D0E (in UnityFramework) (mscorlib15.cpp:0)
    8. JsonDecoder_ParseObject_mA5231DF6CB23559DF5F3E11330C36CDAB40E08F2 (in UnityFramework) (Assembly-CSharp17.cpp:0)
    9. ..
    10. JsonDecoder_ParseValue_m6EBE2BA3134A6DD3BEEB3446241DDE9BEB61F270 (in UnityFramework) (Assembly-CSharp17.cpp:0)
    11. ..
    12. Json_Deserialize_m73E914E7F7AEBD7F5A4FD0409CB2C5C7143609A9 (in UnityFramework) (Assembly-CSharp17.cpp:0)
    13. SkeletonJson_ReadSkeletonData_mF1FCFD42BEE7B107CC09D3F53BB85955F56B3984 (in UnityFramework) (Assembly-CSharp31.cpp:0)
    14. SkeletonDataAsset_ReadSkeletonData_mAC14964EE82711E95DFCDF7E44C3C3C77A01E0E4 (in UnityFramework) (Assembly-CSharp31.cpp:0)
    15. ..
    16. il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2
    Hopefully as we gather more information, these stacks can help point toward a solution.

    Best regards,
     
    JoshPeterson likes this.
  17. devon_redgames

    devon_redgames

    Joined:
    Apr 3, 2020
    Posts:
    12
    Another similar path for crashes, this time it appears when the stack trace is attempting to allocate memory for some exception that occurs:

    Code (CSharp):
    1. GC_allocobj (in UnityFramework) (alloc.c:1568)
    2. GC_generic_malloc_inner (in UnityFramework) (malloc.c:192)
    3. ..
    4. il2cpp::vm::Object::NewAllocSpecific(Il2CppClass*) (in UnityFramework) (Object.cpp:299)
    5. ..
    6. StackTraceUtility_ExtractFormattedStackTrace_m956907F6BE8EFF9BE9847275406FFBBB5FE7F093 (in UnityFramework) (UnityEngine.CoreModule1.cpp:0)
    7. ..
    8. il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**) (in UnityFramework) (Runtime.cpp:576)
    9. ..
    10. scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool) (in UnityFramework) (ScriptingApi_Il2Cpp.cpp:292)
    11. ..
    12. il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**) (in UnityFramework) (Runtime.cpp:576)
    13. ..
    14. MonoBehaviour::TryCreateAndRunCoroutine(ScriptingObjectPtr, ScriptingMethodPtr, Coroutine**) (in UnityFramework) (MonoBehaviour.cpp:691)
    15. ..
    16. Coroutine::Run(bool*) (in UnityFramework) (Coroutine.cpp:271)
    17. ..
    18. RuntimeInvoker_TrueVoid_t700C6383A2A510C2CF4DD86DABD5CA9FF70ADAC5(void (*)(), MethodInfo const*, void*, void**) (in UnityFramework) (Il2CppInvokerTable.cpp:91640)
    19. ..
    20. AwakeFromLoadQueue::PersistentManagerAwakeFromLoad() (in UnityFramework) (AwakeFromLoadQueue.cpp:191)
     
    JoshPeterson likes this.
  18. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    @devon_redgames are these stack traces isolated ie nothing was run before them? If not try to run GC.Collect before them to check if memory wasn't already corrupted.
     
  19. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
  20. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
  21. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
    Don't know whats wrong with that link, seems to work on my machine (as usual...).
    Copied from the other thread:

    We have been trying to find the source for GC crash. It seems to happen at least on OSX, Windows and Android.
    Our application creates quite a lot of threads for HttpWebrequests (which by itself seems to create lots of Tasks) and it will quite randomly within minutes or hours crash. Crash happens from random locations while GC is allocating some memory or from collect_incremental etc.

    Ironically on crash location, the GC_push_all source comments say that "Should only be used if there is no possibility of mark stack overflow." and well that precondition fails... Anyone got good ideas how to get rid of this problem? No bug report created yet as it's not sure if it is our fault.

    Editor version is 2020.3.8f1.

    On Android crash happens at: GC_mark_from, mark.c:851, stacktrace below.

    On OSX: GC_mark_from + 2058, stacktrace below.

    On Windows, it creates messagebox("Unexpected mark stack overflow") from :
    /*
    * Push all locations between b and t onto the mark stack.
    * b is the first location to be checked. t is one past the last
    * location to be checked.
    * Should only be used if there is no possibility of mark stack
    * overflow.
    */
    GC_API void GC_CALL GC_push_all(void *bottom, void *top)
    {
    word length;

    bottom = (void *)(((word)bottom + ALIGNMENT-1) & ~(ALIGNMENT-1));
    top = (void *)((word)top & ~(ALIGNMENT-1));
    if ((word)bottom >= (word)top) return;

    GC_mark_stack_top++;
    if ((word)GC_mark_stack_top >= (word)GC_mark_stack_limit) {
    ABORT("Unexpected mark stack overflow");
    }
    length = (word)top - (word)bottom;
    # if GC_DS_TAGS > ALIGNMENT - 1
    length += GC_DS_TAGS;
    length &= ~GC_DS_TAGS;
    # endif
    GC_mark_stack_top -> mse_start = (ptr_t)bottom;
    GC_mark_stack_top -> mse_descr.w = length;
    }

    Android Stacktrace:
    #00 pc 0000000004a90bb0 (GC_mark_from at C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\extra/../mark.c:851) /data/app/~~jllIATuZnpfl8Qeqp-Tdnw==/com.DeltaCygniLabs.Pointr--VckFJ-JdGNnCykrYEc8Zw==/lib/arm64/libil2cpp.so (BuildId: 513cf18ee78d45c8ae454b335f5fe0a84d655bd3)
    #01 pc 0000000004a8d550 (GC_mark_some at :?) /data/app/~~jllIATuZnpfl8Qeqp-Tdnw==/com.DeltaCygniLabs.Pointr--VckFJ-JdGNnCykrYEc8Zw==/lib/arm64/libil2cpp.so (BuildId: 513cf18ee78d45c8ae454b335f5fe0a84d655bd3)
    #02 pc 0000000004a8d35c (GC_collect_a_little_inner at C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\extra/../alloc.c:672) /data/app/~~jllIATuZnpfl8Qeqp-Tdnw==/com.DeltaCygniLabs.Pointr--VckFJ-JdGNnCykrYEc8Zw==/lib/arm64/libil2cpp.so (BuildId: 513cf18ee78d45c8ae454b335f5fe0a84d655bd3)
    #03 pc 0000000004a8edcc (GC_allocobj at C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\extra/../alloc.c:1563) /data/app/~~jllIATuZnpfl8Qeqp-Tdnw==/com.DeltaCygniLabs.Pointr--VckFJ-JdGNnCykrYEc8Zw==/lib/arm64/libil2cpp.so (BuildId: 513cf18ee78d45c8ae454b335f5fe0a84d655bd3)
    #04 pc 0000000004a8a108 (GC_generic_malloc_inner at C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\extra/../malloc.c:191) /data/app/~~jllIATuZnpfl8Qeqp-Tdnw==/com.DeltaCygniLabs.Pointr--VckFJ-JdGNnCykrYEc8Zw==/lib/arm64/libil2cpp.so (BuildId: 513cf18ee78d45c8ae454b335f5fe0a84d655bd3)
    #05 pc 0000000004a89ed0 (GC_gcj_malloc at :?) /data/app/~~jllIATuZnpfl8Qeqp-Tdnw==/com.DeltaCygniLabs.Pointr--VckFJ-JdGNnCykrYEc8Zw==/lib/arm64/libil2cpp.so (BuildId: 513cf18ee78d45c8ae454b335f5fe0a84d655bd3)
    #06 pc 0000000004a639b0 (il2cpp::vm::Object::AllocateSpec(unsigned long, Il2CppClass*) at C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\libil2cpp\vm/Object.cpp:78) /data/app/~~jllIATuZnpfl8Qeqp-Tdnw==/com.DeltaCygniLabs.Pointr--VckFJ-JdGNnCykrYEc8Zw==/lib/arm64/libil2cpp.so (BuildId: 513cf18ee78d45c8ae454b335f5fe0a84d655bd3)

    OSX stacktrace:
    0 GameAssembly.dylib 0x00000001171676ba GC_mark_from + 2058
    1 GameAssembly.dylib 0x000000011715f5d2 GC_mark_some + 562
    2 GameAssembly.dylib 0x000000011715f213 GC_collect_a_little_inner + 115
    3 GameAssembly.dylib 0x0000000117161afe GC_allocobj + 174
    4 GameAssembly.dylib 0x0000000117158be6 GC_generic_malloc_inner + 486
    5 GameAssembly.dylib 0x00000001171586c0 GC_gcj_malloc + 224
    6 GameAssembly.dylib 0x00000001170a1abe il2cpp::vm::Object::AllocateSpec(unsigned long, Il2CppClass*) + 94
    7 GameAssembly.dylib 0x00000001170c7f45 il2cpp::vm::Object::NewAllocSpecific(Il2CppClass*) + 149
    8 GameAssembly.dylib 0x00000001170b3085 il2cpp::vm::Object::New(Il2CppClass*) + 21
    9 GameAssembly.dylib 0x00000001171c17f5 il2cpp_codegen_object_new(Il2CppClass*) + 21
     
  22. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    The link works for me now! Maybe it was some momentary problem with the forum software.

    Do you know if this crash happens when incremental GC is disabled?
     
  23. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
    This bug is hideously random. For example on Windows standalone 64 bit it takes about a day to crash. On some Android devices it may crash in minutes. And so on. Crash is always in different places while GC activates due to new or CollectIncremental runs.
    But as far as we have tested it does not crash if run without incremental GC.

    Have been monitoring for memory leaks etc, but there is never significant increase, app mem alloc always stays around 300MB.

    My current theory of this bug that it is dependent on the available heap size and when the certain use pattern is active it has a nonzero probability to trigger. On my test when I disable certain part of application that heavily uses HttpWebrequests, Threads and within HttpWebrequest it uses lots of async calls. The problem does not trigger. But when used, then it will trigger while events like requests are created/stopped and those are heavily using allocations/frees.
     
  24. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    For the record my bug happens with both incremental gc on and off.
     
    JoshPeterson likes this.
  25. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    This one really feels like an incremental GC bug. Likely something in the libil2cpp runtime code is missing a write barrier.

    Can you put together a bug report with the C# code that you believe triggers the issue? In the IL2CPP test framework we have internally, we can enable a debugging mode that will catch missing write barriers. The mode is a dynamic analysis, so we need to execute code that actually causes the problem. But it will flag the issue 100% of the time. Often these problems happen in production but then go unnoticed for a while, since nothing else uses the corrupt memory.
     
  26. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
    I will try to make a repro case. Will take some time as nothing simple has been triggering it and cannot ship our application code. I think I can take the part of the application that causes problems and make a test application from that.
     
    JoshPeterson likes this.
  27. aking_wbgames

    aking_wbgames

    Joined:
    Sep 6, 2019
    Posts:
    2
    Possibly https://issuetracker.unity3d.com/is...-from-when-using-monopinvokecallbackattribute ?

    We ran into this issue recently with an identical stacktrace (gc_mark_from during allocations) as a result of having many threads calling back from native code using MonoPInvokeCallbackAttribute -- similarly they were spun up to handle web request responses, but the bug presents itself without any networking component. We were able to exacerbate the issue and rig up a minimal repro project by calling GC.Collect repeatedly.
     
  28. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    Oh, that is pretty interesting. I had not seen this bug report yet. It looks like our mobile team has it now, but I'll pull it over to the VM team for investigation. Thanks!
     
    Cromfeli likes this.
  29. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
    Without calling GarbageCollector.CollectIncremental I get this form of crash stacktrace:

    Code (CSharp):
    1. 0x00007FF88F7EE0D6 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\mark.c:1363] GC_push_all
    2. 0x00007FF88F80B239 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\libil2cpp\gc\BoehmGC.cpp:522] push_other_roots
    3. 0x00007FF88F7ECC2A (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\mark.c:441] GC_mark_some_inner
    4. 0x00007FF88F7ECB09 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\mark.c:506] GC_mark_some
    5. 0x00007FF88F7E5CA7 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\alloc.c:672] GC_collect_a_little_inner
    6. 0x00007FF88F7F3264 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\win32_threads.c:838] GC_unregister_my_thread
    7. 0x00007FF88F80A589 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\libil2cpp\gc\BoehmGC.cpp:300] il2cpp::gc::GarbageCollector::UnregisterThread
    8. 0x00007FF88F7D87CD (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\libil2cpp\vm\Thread.cpp:203] il2cpp::vm::Thread::Uninitialize
    9. 0x00007FF88F79D22C (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\libil2cpp\icalls\mscorlib\System.Threading\Thread.cpp:171] il2cpp::icalls::mscorlib::System::Threading::ThreadStart
    10. 0x00007FF88F784D82 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\libil2cpp\os\Thread.cpp:186] il2cpp::os::Thread::RunWrapper
    11. 0x00007FF88F786043 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\libil2cpp\os\Win32\ThreadImpl.cpp:28] il2cpp::os::ThreadStartWrapper
     
  30. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
    Is it possible for you to share your repro code?
     
  31. aking_wbgames

    aking_wbgames

    Joined:
    Sep 6, 2019
    Posts:
    2
    Sure, it's attached. This crashes iOS within seconds, unsure about android.
     

    Attached Files:

  32. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    I've had a chance to investigate the bug report submitted by @aking_wbgames today. This is https://issuetracker.unity3d.com/is...-from-when-using-monopinvokecallbackattribute

    This one looks like a long-standing bug in the IL2CPP runtime code that is exposed by code that jumps in an out of managed code via reverse p/invoke callbacks really often on many threads. The order of operation the IL2CPP runtime performs when a thread is detached leaves open a race condition with the garbage collector.

    I'm still waiting for our QA team to process the other bugs reported in this thread - sorry about that. I'm unclear yet if they have the same cause as this one or not.
     
  33. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    I don't think Microsoft.CodeAnalysis calls any p/invoke so prob not related. My repro also has
    ConcurrentBuild set to false but it doesn't make a difference.
     
  34. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
    Got a kind of repro on Standalone Windows x64. Created a bug report,
    1361799.Only threads are created in this one.

    Configure project with IL2CPP .net 4.0 and disable vsync.

    Code (CSharp):
    1. using System.Threading;
    2. using UnityEngine;
    3.  
    4. public class gcmarktest : MonoBehaviour
    5. {
    6.     int k = 0;
    7.     int r = 0;
    8.  
    9.     void Update()
    10.     {
    11.         if (k < 30)
    12.         {
    13.             Debug.Log("Starting thread " + r);
    14.             Thread t = new Thread(new ThreadStart(ThreadProc));
    15.             t.Start();
    16.             r++;
    17.         }
    18.         if (k == 50)
    19.         {
    20.             k = 0;
    21.         }
    22.         k++;
    23.         UnityEngine.Scripting.GarbageCollector.CollectIncremental(1000000);
    24.     }
    25.  
    26.     void ThreadProc()
    27.     {
    28.         Debug.Log("Starting thread from thread.");
    29.         Thread t2 = new Thread(new ThreadStart(ThreadProc2));
    30.         t2.Start();
    31.  
    32.         Debug.Log("Closing thread");
    33.     }
    34.  
    35.     static byte[] pff = null;
    36.     void ThreadProc2()
    37.     {
    38.         pff = new byte[1024];
    39.         pff[0] = 1;
    40.  
    41.         pff = new byte[1024*1024];
    42.         pff[0] = 1;
    43.     }
    44. }
    Will output messagebox with "Unexpected mark stack overflow" and stacktrace:
    Code (CSharp):
    1. 0x00007FF8A3F6D4F6 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\mark.c:1363] GC_push_all
    2. 0x00007FF8A3F88719 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\libil2cpp\gc\BoehmGC.cpp:522] push_other_roots
    3. 0x00007FF8A3F6C04A (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\mark.c:441] GC_mark_some_inner
    4. 0x00007FF8A3F6BF29 (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\mark.c:506] GC_mark_some
    5. 0x00007FF8A3F7195D (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\alloc.c:832] GC_stopped_mark
    6. 0x00007FF8A3F652EC (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\alloc.c:705] GC_collect_a_little_inner
    7. 0x00007FF8A3F712AC (GameAssembly) [C:\Program Files\Unity\Hub\Editor\2020.3.8f1\Editor\Data\il2cpp\external\bdwgc\alloc.c:382] GC_start_incremental_collection
     
    Last edited: Sep 6, 2021
  35. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
  36. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    Thanks, I'll do that.
     
  37. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
    Case 1361799 is also processed by QA.
     
  38. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    Sounds good, we will investigate it.
     
  39. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    I've now had a chance to investigate https://issuetracker.unity3d.com/product/unity/issues/guid/1361799/. It turns out this this is a problem with the debugger code leaking GC handles for each thread.

    It won't happen is the "Script Debugging" option is not selected in the Build Settings dialog. I do have a fix in process now that will prevent the GC handle leak though. The public issue tracker link should indicate which versions have the fix when they land in various places.
     
    futurlab_peterh likes this.
  40. tkumpumaki

    tkumpumaki

    Joined:
    Sep 26, 2018
    Posts:
    18
    Yes I can confirm that ticking "Script Debugging" off do prevent crash on repro.

    Is this "mark stack overflow" a common symptom for a leaked GC handle? And is it also possible to cause these from normal C# scripting by some kind of programming error?
     
  41. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    No, this is not a common error. It is not intended for user consumption, so the fact that it happens is a symptom of a weakness in the GC integration into Unity. Realistically, this should never happen, but there are corner cases like this where it can.

    I suspect that it is possible to cause from C# code if that code creates thousands of GC handles. However, I've never actually seen that in real production code, so I suspect the impact of this weakness is very small.
     
    futurlab_peterh likes this.
  42. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    futurlab_peterh and JoshPeterson like this.
  43. KurtGokhan

    KurtGokhan

    Joined:
    Jan 16, 2013
    Posts:
    37
    I see this error happening frequently in 2021.2.7f1. It works good in versions before 2021.2. Is there a regression?
     
    PokerDawg and futurlab_peterh like this.
  44. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    818
    Always make bug report with repro project.
     
  45. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    Yes, please make a bug report. We're not aware of any regression in this code, but it could have happened.

    https://unity3d.com/unity/qa/bug-reporting
     
    futurlab_peterh likes this.
  46. KurtGokhan

    KurtGokhan

    Joined:
    Jan 16, 2013
    Posts:
    37
    I created a ticket with the issue I am having (Case 1422458).

    I am using a library called ClearScript which uses V8 natively. So there is some heavy usage of native code. The curious thing is, the error does not happen in version 2020.3 and below. So I believe there is something wrong with Unity in newer versions.

    I am sorry for not being able to isolate the error better. The project I attached is not minimal. The error does not happen in small projects. It may also require you to restart the game a few times. Anyway, you have my full cooperation if there is anything I can do to help.
     
  47. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    Thanks for reporting this issue, we will investigate it.
     
    KurtGokhan likes this.
  48. RaventurnPatrick

    RaventurnPatrick

    Joined:
    Aug 9, 2011
    Posts:
    84
    We are also seeing lots of crashes on iOS (not on Android with il2cpp though), do you think this is the same issue?
    (Unity 2020.3.33f1)

    Crashed: com.apple.main-thread
    EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000048
    0 UnityFramework 0x19025a0 GC_make_disappearing_links_disappear + 40564
    1 UnityFramework 0x18fefb4 GC_finalize + 26760
    2 UnityFramework 0x18fe1c0 GC_finish_collection + 23188
    3 UnityFramework 0x18fd908 GC_try_to_collect_inner + 20956
    4 UnityFramework 0x18f9b6c GC_init + 5184
    5 UnityFramework 0x18fa288 GC_generic_malloc_inner + 7004
    6 UnityFramework 0x18f9f88 GC_gcj_malloc + 6236
    7 UnityFramework 0x1944e64 il2cpp::vm::Object::NewAllocSpecific(Il2CppClass*) + 83660
    8 UnityFramework 0x19618e0 il2cpp::vm::Thread::Attach(Il2CppDomain*) + 41476
    9 UnityFramework 0x1961870 il2cpp::vm::ScopedThreadAttacher::ScopedThreadAttacher() + 41364
    10 UnityFramework 0x137064 ReversePInvokeWrapper_RewardedAdClient_RewardedAdUserDidEarnRewardCallback_m42346DE78A69FA639E492123A79DACD94B8E18C2 + 13147 (GoogleMobileAds.iOS.cpp:13147)
    11 UnityFramework 0x19ec04c __22-[GADURewardedAd show]_block_invoke + 5332
    12 UnityFramework 0x899f64 __62-[GADFullScreenAd attachRewardObserversWithEarnRewardHandler:]_block_invoke + 399 (GADFullScreenAd.m:399)
    13 UnityFramework 0x8fe564 __68-[GADNotificationCenter addObserverForName:object:queue:usingBlock:]_block_invoke + 282 (GADEvents.m:282)
    14 Foundation 0x344a4 -[__NSObserver _doit:] + 348
    15 CoreFoundation 0x2aa44 __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ + 28
    16 CoreFoundation 0xc71d4 ___CFXRegistrationPost_block_invoke + 52
    17 CoreFoundation 0x9a3d0 _CFXRegistrationPost + 456
    18 CoreFoundation 0x40abc _CFXNotificationPost + 728
    19 Foundation 0x1a644 -[NSNotificationCenter postNotificationName:object:userInfo:] + 96
    20 UnityFramework 0x8fdf5c -[GADNotificationCenter postNotificationName:object:userInfo:] + 230 (GADEvents.m:230)
    21 UnityFramework 0x8fd5b4 GADPostNotification + 61 (GADEvents.m:61)
    22 UnityFramework 0x845de0 -[GADRTBMediationRewardBasedVideoAdRenderer didRewardUserWithReward:] + 267 (GADRTBMediationRewardBasedVideoAdRenderer.m:267)
    23 UnityFramework 0x438120 -[GADMAppLovinRewardedDelegate ad:wasHiddenIn:] + 88 (GADMAppLovinRewardedDelegate.m:88)
    24 UnityFramework 0x53a838 __68+[ALDelegateCallbackInvoker invokeAdDisplayDelegate:ad:wasHiddenIn:]_block_invoke + 100 (ALDelegateCallbackInvoker.m:100)
    25 UnityFramework 0x53a758 +[ALDelegateCallbackInvoker invokeAdDisplayDelegate:ad:wasHiddenIn:] + 31 (ALMacros.h:31)
    26 UnityFramework 0x4633d0 -[ALIncentivizedDelegateProxy handleAdHiddenForDirectAd:inView:] + 388 (ALIncentivizedInterstitialAd.m:388)
    27 UnityFramework 0x462d2c -[ALIncentivizedDelegateProxy ad:wasHiddenIn:] + 305 (ALIncentivizedInterstitialAd.m:305)
    28 UnityFramework 0x53a838 __68+[ALDelegateCallbackInvoker invokeAdDisplayDelegate:ad:wasHiddenIn:]_block_invoke + 100 (ALDelegateCallbackInvoker.m:100)
    29 UnityFramework 0x53a758 +[ALDelegateCallbackInvoker invokeAdDisplayDelegate:ad:wasHiddenIn:] + 31 (ALMacros.h:31)
    30 UnityFramework 0x43c7b8 -[ALDelegateCallbackUnrepeatableInvoker handleAd:wasHiddenIn:] + 64 (ALDelegateCallbackUnrepeatableInvoker.m:64)
    31 UnityFramework 0x44e6d0 -[ALBaseVideoViewController viewWillDisappear:] + 311 (ALBaseVideoViewController.m:311)
    32 UIKitCore 0x1ab830 -[UIViewController _setViewAppearState:isAnimating:] + 988
    33 UIKitCore 0x24a668 -[UIViewController __viewWillDisappear:] + 96
    34 UIKitCore 0x6bb804 __56-[UIPresentationController runTransitionForCurrentState]_block_invoke.424 + 888
    35 UIKitCore 0x37c0d4 +[UIPresentationController _scheduleTransition:] + 104
    36 UIKitCore 0x68a708 -[UIPresentationController runTransitionForCurrentState] + 1808
    37 UIKitCore 0x2aa60c -[UIPresentationController _dismissWithAnimationController:interactionController:target:didEndSelector:] + 620
    38 UIKitCore 0x50fc90 __99-[UIViewController _dismissViewControllerWithAnimationController:interactionController:completion:]_block_invoke_2 + 76
    39 UIKitCore 0x172e0c +[UIView(Animation) performWithoutAnimation:] + 104
    40 UIKitCore 0x2d2de0 -[UIViewController _dismissViewControllerWithAnimationController:interactionController:completion:] + 656
    41 UIKitCore 0x172e0c +[UIView(Animation) performWithoutAnimation:] + 104
    42 UIKitCore 0x477460 -[UIViewController _dismissViewControllerWithTransition:from:completion:] + 1628
    43 UIKitCore 0x3b66d4 -[UIViewController dismissViewControllerWithTransition:completion:] + 2748
    44 UIKitCore 0x25b18c -[UIViewController _performCoordinatedPresentOrDismiss:animated:] + 536
    45 UIKitCore 0x2b6c0c -[UIViewController dismissViewControllerAnimated:completion:] + 152
    46 UIKitCore 0x25b72c __68-[UIWindow _removeAllViewControllersFromWindowHierarchyFromDealloc:]_block_invoke + 64
    47 UIKitCore 0x1c4bdc +[UIViewController _performWithoutDeferringTransitionsAllowingAnimation:actions:] + 164
    48 UIKitCore 0x38c9dc -[UIWindow _removeAllViewControllersFromWindowHierarchyFromDealloc:] + 304
    49 UIKitCore 0x2d0ee0 -[UIWindow setRootViewController:] + 288
    50 UnityFramework 0x16b74 UnityCleanupTrampoline + 178 (UnityAppController.mm:178)
    51 UnityFramework 0x17e60 -[UnityAppController applicationWillTerminate:] + 535 (UnityAppController.mm:535)
    52 UIKitCore 0xf5abe8 -[UIApplication _terminateWithStatus:] + 244
    53 UIKitCore 0x636af8 -[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:] + 128
    54 UIKitCore 0x82c5f0 -[_UISceneLifecycleMultiplexer forceExitWithTransitionContext:scene:] + 224
    55 UIKitCore 0xf56f64 -[UIApplication workspaceShouldExit:withTransitionContext:] + 224
    56 FrontBoardServices 0x4efec __63-[FBSWorkspaceScenesClient willTerminateWithTransitionContext:]_block_invoke_2 + 80
    57 FrontBoardServices 0xa308 -[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:] + 240
    58 FrontBoardServices 0x4ef84 __63-[FBSWorkspaceScenesClient willTerminateWithTransitionContext:]_block_invoke + 132
    59 libdispatch.dylib 0x3a30 _dispatch_client_callout + 20
    60 libdispatch.dylib 0x74e0 _dispatch_block_invoke_direct + 264
    61 FrontBoardServices 0xbc70 __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ + 48
    62 FrontBoardServices 0xb040 -[FBSSerialQueue _targetQueue_performNextIfPossible] + 220
    63 FrontBoardServices 0xf700 -[FBSSerialQueue _performNextFromRunLoopSource] + 28
    64 CoreFoundation 0xbb614 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 28
    65 CoreFoundation 0xcc3a0 __CFRunLoopDoSource0 + 208
    66 CoreFoundation 0x58a4 __CFRunLoopDoSources0 + 268
    67 CoreFoundation 0xb26c __CFRunLoopRun + 828
    68 CoreFoundation 0x1edd8 CFRunLoopRunSpecific + 600
    69 GraphicsServices 0x1374 GSEventRunModal + 164
    70 UIKitCore 0x514c60 -[UIApplication _run] + 1100
    71 UIKitCore 0x2963bc UIApplicationMain + 364
    72 UnityFramework 0x1c1bc -[UnityFramework runUIApplicationMainWithArgc:argv:] + 96 (main.mm:96)
     
  49. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,038
    My initial reaction from this call stack is that this is a different issue, although I don't yet understand either of them completely, so I'm not certain. Are the other crashes like this happening with the same call stack? Or are there a few different call stacks that occur?
     
  50. RaventurnPatrick

    RaventurnPatrick

    Joined:
    Aug 9, 2011
    Posts:
    84
    Thanks for the quick response. The call stacks are all exactly the same and are happening for iOS devices all accross the board.
    It always happens when Google Admob tries to callback into the app (that is in background) - also it is likely not memory pressure related as there are devices with more than 1GB free ram (according to Firebase) experiencing this crash. It leads to approx. 0.2% crash rate accross all iOS Users.
     
unityunity