Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Bug Profiler Panel causes a Stackoverflow (2021.3.34)

Discussion in 'Editor & General Support' started by Whatever560, May 10, 2024.

  1. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    [Workaround] : See last post, we had to close the profiler panel. We still don't know what causes the issue.

    in 2021.3.34 I have a situation where some of my scenes consistently crash. The problem is I don't get any word of it in the logs (no crash log, dumps, no exception, no anything)

    I got to a point where I can log up to the last operation before the crash but it is after the end of a task. Not in the middle of it.

    What I suspect is an infinite loop or a multithread concurent access but even when debuging with rider it does not get to stop on any exception.

    The thing that I get is that Rider debugger actually stops like it is actually hitting some breakpoints but then it does not seem it is in any managed c# code as I have no line highlighted in none of the threads.

    I'd be more than interested by any pointers here to debug this.
    Thanks.
     
    Last edited: May 11, 2024
  2. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    To give more details :


    1.b
    Just before the crash, Rider debbugger hits a breakpoint, but seems it is inside unity c++ code as we have no stacktrace (31312 is the id of unity thread in this session) :
    upload_2024-5-10_14-36-50.png

    1. The crash is also coincidental with this unity accelerator related line.
    Code (CSharp):
    1. TcpProtobufClient - Transport error errorcode: 11001, details: No such host is known.
    2.  
    3.  
    4. AcceleratorClientConnectionCallback - disconnected - unityacc.[...].eu:10080
    2. Most of threads are waiting and unity main thread already crashed at this time. It is not in the thread view anymore
    upload_2024-5-10_11-39-13.png

    3. Our own AI thread is happily doing it stuff, it will actually continue to perform and log after the unity/main thread crash, so this is how I know it is not actually the one crashing. It will stop at a random point when the whole program quit.
    upload_2024-5-10_11-40-56.png

    4. I get a window event crash

    Faulting application name: Unity.exe, version: 2021.3.34.9831, time stamp: 0x65820ee0
    Faulting module name: mono-2.0-bdwgc.dll, version: 0.0.0.0, time stamp: 0x65398aad
    Exception code: 0xc0000005
    Fault offset: 0x00000000003719bc
    Faulting process id: 0x76c4
    Faulting application start time: 0x01daa2bbadee6f0d
    Faulting application path: []\2021.3.34f1\Editor\Unity.exe
    Faulting module path: []\UnityEditors\2021.3.34f1\Editor\Data\MonoBleedingEdge\EmbedRuntime\mono-2.0-bdwgc.dll
    Report Id: d9729076-872a-4565-ae4b-ec0370ce30b9
    Faulting package full name:
    Faulting package-relative application ID:

    And then that's it.
     
    Last edited: May 10, 2024
  3. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,922
    Crash is in the editor I presume? In playmode or edit mode? When loading the scene, or sometime later when working with it? Does it crash on other systems as well? Just in case it happens on just one machine.

    Did you try deleting the Library folder? Since you use the Accelerator I would disconnect it beforehand to make sure it doesn't synch you a corrupt file. If cleaning Library does fix the issue, I would subsequently also "clean" the Accelerator cache to be on the safe side before reconnecting to it.

    If nothing seems to help I would start to disect that particular scene. Remove parts you can easily remove. Keep removing stuff until the crash goes away. This may help narrow down what the crash may be related to.
     
  4. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    Thanks for your answer @CodeSmile

    Yes crash in the editor while in Play mode, after some initialization of the game is made (but can't reach gameplay though)
    Indeed already started scene bisect. I found the culript but I have no idea why it breaks as on an other scene (basically an other level) this setup does work. To make it short and relatable, it's a spawn for a 3rd team in the game, but it's tide to many things, as a 3rd team adds a 3rd player, a new AI, a new thread, etc ... so it can be anything related. It has to be linked to the way the 3rd team behaves with the map, because same spawns and team config is used on another level and no issue arrise. So next step is probably me trying to disable parts of the map and see how it goes.

    I won't go into library refresh yet, maybe as a last resort, as it has regularly proven to me not fixing my issues and It would take several hours. What fix things though is sometimes creating a new scene, moving all elements from the old scene and replacing it. This is something that I must do sometimes as unity corrupts scene files in the long run by having residual data.

    Regarding the accelerator, actually it can't be reached as the adresse is somewhat obsolete, I was just pointing that it's the only marker that show that unity thread crashed :(

    Now that I know what cause, I already have a workaround in mind but I'm still very surprised by the lack of exploitable log that could prby help me pinpoint the real issue.
     
  5. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    39,355
    You can get most of the same results (code-wise) by closing Unity and deleting the Library/ScriptAssemblies folder in your project, then reopening Unity. This just triggers a full rebuild.
     
    Whatever560 likes this.
  6. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    Very good to know !
     
  7. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    Nothing does work, it's absolutelly madening.
    - Cleared Library/ScriptAssemblies
    - Disabled accelerator
    - Tried any on/off possible on the scene.

    Unity team anyone could help us find out how we could debug this and catch an exception or anything ?

    Having no stacktrace at all makes it nearly impossible to get.
     
    Last edited: May 10, 2024
  8. CodeSmile

    CodeSmile

    Joined:
    Apr 10, 2014
    Posts:
    6,922
    Unity does not corrupt scene files. Crashes might. Or an unstable system might (eg RAM or disk errors). Or scripts that do bad things.

    But apart from that, this is something you need to investigate! You have an inexplicable crash issue which could very well be related to scenes becoming corrupt. The scene may be working in edit mode but it may already be corrupt in a way that loading it at runtime does what you experience right now.

    A Library rebuild taking several hours would indicate a resource hog of a project, or a relatively slow system. Perhaps the project could use some cleanup eg unused assets? It wouldn‘t be the first project that is unnecessarily bloated and runs into issues because it contains 3rd party assets that have not been tested with projects at scale or may be of questionable quality. You know that kind of thing that sometimes causes asset corruption.

    The best course of action is to rollback in source control to the last commit that doesn‘t crash, then diff it against the next commit that does crash. You could then cherry pick some of the changes until you find the one that causes the crash and hopefully that isn‘t thousands of lines of changes. But even then you have little choice but to look at those changes in detail.
     
  9. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    Indeed the steps you're citing are the logical ones however they're not really applicable in our situation, believe me it's not the first bug we encounter, nor the first silent one, however I'm getting older and more tired so if I could find a better solution than just wreck my brain against a wall I'd look into it.

    So basically i'd like to find a way to get a stacktrace.

    If I can't, as I said I'm almost considering just bypassing the issue by disabling the feature and mutate it as it might bring me more peace and sanity.
     
    Last edited: May 10, 2024
  10. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,737
    Whatever560 likes this.
  11. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    Thanks, looks like I'll dig into it. I'm debugging with rider and it does not seem to support mixed debugging (https://youtrack.jetbrains.com/issue/RIDER-11810/Support-mixed-mode-debugging) That might be why the native exception that seems to be caught is renderer with no frame.

    I'll go the VS 22 route and see where that gets me, also this resources seems pretty helpul :

    https://docs.unity3d.com/2021.3/Documentation/Manual/WindowsDebugging.html
    https://github.com/Unity-Technologies/UnityMixedCallstack?tab=readme-ov-file
     
  12. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    @Tautvydas-Zilys You did it, you helped me.
    Got this in VS : Exception thrown at 0x00007FF78F105FC4 in Unity.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000ACBACF9FE8).

    So it's a stackoverflow, at least a stepforward !

    You wouldn't now how I could get the actual managed call that caused this ? It seems that on attaching VS22 it warns on mixed-debugging being not supported by legacy .net fwk.

    I'll try getting a crash dump maybe it'll get more info

    [Edit] :
    With a dump made with "procdump -ma" when VS has a breakpoint on the error and then opened with WinDbg let us see the stacktrace thanks to the command "~0 kn 0xffff" (otherwise it's truncated).

    Here are the relevant bits of the the stack trace anything in betwen [...] is just a repetition of the same message:

    Code (CSharp):
    1.  
    2. # Child-SP          RetAddr               Call Site
    3. 00 0000009b`1d280eb0 00007ffc`5960fa27     mono_2_0_bdwgc!mono_aot_get_class_from_name+0x2c [C:\build\output\Unity-Technologies\mono\mono\mini\aot-runtime.c @ 2862]
    4. 01 0000009b`1d2813f0 00007ffc`59610491     mono_2_0_bdwgc!mono_class_from_name_checked_aux+0x197 [C:\build\output\Unity-Technologies\mono\mono\metadata\class.c @ 3288]
    5. 02 (Inline Function) --------`--------     mono_2_0_bdwgc!mono_class_from_name_checked+0x3c [C:\build\output\Unity-Technologies\mono\mono\metadata\class.c @ 3422]
    6. 03 0000009b`1d281900 00007ffc`59642512     mono_2_0_bdwgc!mono_class_load_from_name+0x61 [C:\build\output\Unity-Technologies\mono\mono\metadata\class.c @ 3471]
    7. 04 0000009b`1d2819c0 00007ffc`5964327f     mono_2_0_bdwgc!mono_exception_new_by_name_domain+0x72 [C:\build\output\Unity-Technologies\mono\mono\metadata\exception.c @ 102]
    8. 05 (Inline Function) --------`--------     mono_2_0_bdwgc!mono_exception_new_by_name+0x3e [C:\build\output\Unity-Technologies\mono\mono\metadata\exception.c @ 59]
    9. 06 0000009b`1d281a20 00007ffc`5964343e     mono_2_0_bdwgc!mono_exception_new_by_name_msg+0x7f [C:\build\output\Unity-Technologies\mono\mono\metadata\exception.c @ 288]
    10. 07 0000009b`1d281a80 00007ff7`8db02cc1     mono_2_0_bdwgc!mono_exception_from_name_msg+0xee [C:\build\output\Unity-Technologies\mono\mono\metadata\exception.c @ 324]
    11. 08 0000009b`1d281b70 00007ff7`8dae1fb4     Unity!scripting_method_invoke+0x51
    12. 09 0000009b`1d281bb0 00007ff7`8dadcbda     Unity!ScriptingInvocation::Invoke+0xc4
    13. 0a 0000009b`1d281c20 00007ff7`8dc2bf68     Unity!ScriptingInvocation::Invoke<bool>+0x1a
    14. 0b 0000009b`1d281c50 00007ff7`8dae263c     Unity!Scripting::UnityEngine::DebugProxy::CallOverridenDebugHandler+0x78
    15. [...]
    16. 9b 0000009b`1d287410 00007ff7`8dae263c     Unity!Scripting::UnityEngine::DebugProxy::CallOverridenDebugHandler+0x78
    17. 9c 0000009b`1d287580 00007ff7`8dae2012     Unity!Scripting::LogException+0x6c
    18. 9d 0000009b`1d2875e0 00007ff7`8dadcbda     Unity!ScriptingInvocation::Invoke+0x122
    19. 9e 0000009b`1d287650 00007ff7`8dc2bf68     Unity!ScriptingInvocation::Invoke<bool>+0x1a
    20. 9f 0000009b`1d287680 00007ff7`8dae263c     Unity!Scripting::UnityEngine::DebugProxy::CallOverridenDebugHandler+0x78
    21. a0 0000009b`1d2877f0 00007ff7`8da0d97f     Unity!Scripting::LogException+0x6c
    22. a1 0000009b`1d287850 00007ff7`8f220234     Unity!Application_Bindings::LogCallbackImplementation+0x13f
    23. a2 0000009b`1d2879e0 00007ff7`8f21ff9d     Unity!DebugStringToFilePostprocessedStacktrace+0x204
    24. a3 0000009b`1d287af0 00007ff7`8db03c81     Unity!DebugStringToFile+0x50d
    25. a4 0000009b`1d287cb0 00007ff7`8dae282a     Unity!scripting_stack_trace_info_for+0x9c1
    26. a5 0000009b`1d2880a0 00007ff7`8ce1f655     Unity!Scripting::LogExceptionFromManaged+0x10a
    27. a6 0000009b`1d2882a0 00000263`8588224c     Unity!DebugLogHandler_CUSTOM_Internal_LogException+0x175
    28. a7 0000009b`1d288330 00000263`858820bb     0x00000263`8588224c
    29. a8 0000009b`1d2883d0 00000263`85881f01     0x00000263`858820bb
    30. a9 0000009b`1d288430 00000263`85881d28     0x00000263`85881f01
    31. aa 0000009b`1d2884a0 00000263`85881b13     0x00000263`85881d28
    32. ab 0000009b`1d2884f0 00000260`7601ac11     0x00000263`85881b13
    33. ac 0000009b`1d288560 00007ffc`597e65fe     0x00000260`7601ac11
    34. ad 0000009b`1d288620 00007ffc`5971cd84     mono_2_0_bdwgc!mono_jit_runtime_invoke+0x94e [C:\build\output\Unity-Technologies\mono\mono\mini\mini-runtime.c @ 3445]
    35. ae 0000009b`1d2888d0 00007ffc`597229b2     mono_2_0_bdwgc!do_runtime_invoke+0xb4 [C:\build\output\Unity-Technologies\mono\mono\metadata\object.c @ 3068]
    36. af (Inline Function) --------`--------     mono_2_0_bdwgc!mono_runtime_try_invoke+0x4b [C:\build\output\Unity-Technologies\mono\mono\metadata\object.c @ 3175]
    37. b0 (Inline Function) --------`--------     mono_2_0_bdwgc!mono_runtime_try_invoke_handle+0x5a [C:\build\output\Unity-Technologies\mono\mono\metadata\object.c @ 3183]
    38. b1 (Inline Function) --------`--------     mono_2_0_bdwgc!mono_runtime_delegate_try_invoke_handle+0x11e [C:\build\output\Unity-Technologies\mono\mono\metadata\object.c @ 4463]
    39. b2 0000009b`1d288920 00007ffc`59722ed9     mono_2_0_bdwgc!call_unhandled_exception_delegate+0xa42 [C:\build\output\Unity-Technologies\mono\mono\metadata\object.c @ 4961]
    40. b3 0000009b`1d288b30 00007ffc`59722ad8     mono_2_0_bdwgc!mono_unhandled_exception_checked+0x259 [C:\build\output\Unity-Technologies\mono\mono\metadata\object.c @ 5195]
    41. b4 0000009b`1d288b80 00007ffc`598d99ca     mono_2_0_bdwgc!mono_unhandled_exception_internal+0x58 [C:\build\output\Unity-Technologies\mono\mono\metadata\object.c @ 5006]
    42. b5 0000009b`1d288c30 00007ffc`59a27c19     mono_2_0_bdwgc!mono_handle_exception_internal+0x11fa [C:\build\output\Unity-Technologies\mono\mono\mini\mini-exceptions.c @ 2774]
    43. b6 (Inline Function) --------`--------     mono_2_0_bdwgc!mono_handle_exception+0x1e [C:\build\output\Unity-Technologies\mono\mono\mini\mini-exceptions.c @ 3126]
    44. b7 0000009b`1d289440 00007ffc`59a264dc     mono_2_0_bdwgc!mono_arch_handle_exception+0x49 [C:\build\output\Unity-Technologies\mono\mono\mini\exceptions-amd64.c @ 815]
    45. b8 0000009b`1d289610 00007ffd`40528b1c     mono_2_0_bdwgc!seh_vectored_exception_handler+0xac [C:\build\output\Unity-Technologies\mono\mono\mini\exceptions-amd64.c @ 155]
    46. b9 0000009b`1d289650 00007ffd`405012d6     ntdll!RtlpCallVectoredHandlers+0x108
    47. ba 0000009b`1d2896f0 00007ffd`40550ebe     ntdll!RtlDispatchException+0x66
    48. bb 0000009b`1d289900 00007ff7`8f105fc4     ntdll!KiUserExceptionDispatch+0x2e
    49. bc 0000009b`1d28a000 00007ff7`8f105fdd     Unity!RecursiveAdjustChartForGroupChange+0x194
    50. [...]
    51. 3a12 0000009b`1d36f580 00007ff7`8f105fdd     Unity!RecursiveAdjustChartForGroupChange+0x1ad
    52. 3a13 0000009b`1d36f5c0 00007ff7`8f101bbd     Unity!RecursiveAdjustChartForGroupChange+0x1ad
    53. 3a14 0000009b`1d36f600 00007ff7`8f101f80     Unity!ProfilerHistory::AddFrame+0xbd
    54. 3a15 0000009b`1d36f6e0 00007ff7`8f13af84     Unity!ProfilerHistory::AddFrameDataAndTransferOwnership+0x40
    55. 3a16 0000009b`1d36f720 00007ff7`8f139413     Unity!profiling::ProfilerSession::IntegrateFramesToHistory+0x1d4
    56. 3a17 0000009b`1d36f850 00007ff7`8eb74d57     Unity!profiling::ProfilerSession::EditorTick+0xc3
    57. 3a18 0000009b`1d36f8b0 00007ff7`8eb79a9b     Unity!MainMessageLoop+0x147
    58. 3a19 0000009b`1d36fb50 00007ff7`8ff5b1ae     Unity!WinMain+0xddb
    59. 3a1a 0000009b`1d37fe20 00007ffd`3eaa7344     Unity!__scrt_common_main_seh+0x106
    60. 3a1b 0000009b`1d37fe60 00007ffd`405026b1     kernel32!BaseThreadInitThunk+0x14
    61. 3a1c 0000009b`1d37fe90 00000000`00000000     ntdll!RtlUserThreadStart+0x21
    So there is a stackoverflow and the method it calls embeds a "UnityEngine.Profiling.Profiler.BeginSample" isn't it ?
     
    Last edited: May 11, 2024
  13. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    Okay, it's fixed by : closing the embedded Profiler panel.

    I initially thought it was due to entangled calls of
    UnityEngine.Profiling.Profiler.BeginSample / EndSample but no. (Well it might be actually as these are probably called only if at least a profiler window is open ?)

    @Tautvydas-Zilys I'll file a bug report, I can't make a repro but I'll attach the dump. Thanks for the hint.
     
    Last edited: May 11, 2024
    Kurt-Dekker likes this.
  14. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,489
    This happens in code that calculates the stacked category colors for the CPU or GPU Usage charts. If you have the profiler window closed but profile to file via Profiler.logFile for the first 300 frames of you entering playmode on that scene and then open that .raw file later, does that also crash? And if so, could you please report a bug with that
    .raw file attached and post the issue ID here? That file would very probably be all that's needed to investigate and fix what's happening there.
     
    Whatever560 likes this.
  15. Whatever560

    Whatever560

    Joined:
    Jan 5, 2016
    Posts:
    564
    What if it's not crashing in this case, is it still interesting to you? Also the profiler windows was not the active tab when crashing. So does it mean that the profiler panel still processes frames when not the active tab ? On this machine we have a very large setup of panels (11 open docked panels each with its own tabs), I often saw performance issues in editor due to that (like scene hierarchy tree open and inspectors). Could this be related?
     
    Last edited: May 13, 2024
  16. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,489
    That's still interesting but then we need a project that reproduces the crash...

    You mean it was in the background and not drawn? Then there was no code run for drawing it but the code from that callstack is run after receiving and deserializing new frames regardless of whether or not they are shown. It just establishes the times spent in each category.

    It can also still be recording even if in background and yes, that can have a performance impact.
     
    Whatever560 likes this.
  17. MrMelSE

    MrMelSE

    Joined:
    Dec 10, 2017
    Posts:
    28
  18. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,489
    I doubt it. That sounds like a UI code bug whereas this here doesn't even need UI code to run, just profiler backend code.