Search Unity

mono-2.0-bdwgc.dll causes crash for some users. Cannot use IL2CPP.

Discussion in 'Windows' started by Lewnatic, Feb 6, 2019.

  1. Lewnatic

    Lewnatic

    Joined:
    Sep 29, 2012
    Posts:
    209
    I encountered a bug when using the mono backend. Some users of my game cannot play the game. They did not even get a configuration screen. Also no log file is produced because of that. It kinda indicates, that the game does not find the folders inside the game build.

    All users running into that issue also seem to run the game on newer machines. Some of them is russians and they tested it with other russian users and it did also not work for them as well on different machines.

    he send me this.

    Сигнатура проблемы Имя проблемного события: BEX64
    Имя приложения: Game.exe
    Версия приложения: 2018.3.0.39463
    Отметка времени приложения: 5c0e9ee8
    Имя модуля с ошибкой: mono-2.0-bdwgc.dll
    Версия модуля с ошибкой: 0.0.0.0
    Отметка времени модуля с ошибкой: 5c058d49
    Смещение исключения: 0000000000337050
    Код исключения: c0000409
    Данные исключения: 0000000000000007
    Версия ОС: 10.0.17763.2.0.0.256.48
    Код языка: 1049
    Дополнительные сведения 1: 54b2
    Дополнительные сведения 2: 54b2b85e8185fc9754ac9704cf2b16f0
    Дополнительные сведения 3: e362
    Дополнительные сведения 4: e362613a00cd1805058ff4eb462c691c

    We have the suspicion that mono backend only supports latin and not cyril. But i cannot confirm that.

    So i thought switching to IL2CPP might do the trick. Sadly IL2CPP builds are crashing for me in some scenes. So this is also not working for me. He can however run the game IL2CPP but when some huge scenes are loaded, it crashes too, so this is sadly also no solution.

    This is the Error Log he gets from the IL2CPP build.

    Code (CSharp):
    1. Initialize engine version: 2018.2.5f1 (3071d1717b71)
    2. GfxDevice: creating device client; threaded=1
    3. XR: OpenVR Error! OpenVR failed initialization with error code VRInitError_Init_PathRegistryNotFound: "Installation path could not be located (110)"!
    4. (Filename:  Line: 1780)
    5.  
    6. Direct3D:
    7.     Version:  Direct3D 11.0 [level 11.1]
    8.     Renderer: NVIDIA GeForce GTX 960 (ID=0x1401)
    9.     Vendor:  
    10.     VRAM:     2016 MB
    11.     Driver:   23.21.13.8813
    12. Begin MonoManager ReloadAssembly
    13. - Completed reload, in  0.121 seconds
    14. OnLevelWasLoaded was found on MoBlur
    15. This message has been deprecated and will be removed in a later version of Unity.
    16. Add a delegate to SceneManager.sceneLoaded instead to get notifications after scene loading has completed
    17. (Filename:  Line: 343)
    18.  
    19. OnLevelWasLoaded was found on OVRScreenFade
    20. This message has been deprecated and will be removed in a later version of Unity.
    21. Add a delegate to SceneManager.sceneLoaded instead to get notifications after scene loading has completed
    22. (Filename:  Line: 343)
    23.  
    24. <RI> Initializing input.
    25.  
    26. <RI> Input initialized.
    27.  
    28.  
     
    Last edited: Feb 6, 2019
    Lewnatic, Feb 6, 2019
    #1

  2. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Tautvydas-Zilys, Feb 6, 2019
    #2

  3. Lewnatic

    Lewnatic

    Joined:
    Sep 29, 2012
    Posts:
    209
    Thank you I am contacting the user.

    I am currently trying a final thing, which is building the game as UWP.
     
    Lewnatic, Feb 6, 2019
    #3

  4. Lewnatic

    Lewnatic

    Joined:
    Sep 29, 2012
    Posts:
    209
    I ve send you the dump file. :)
     
    Lewnatic, Feb 7, 2019
    #4

  5. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Thanks. The crash doesn't happen in mono-2-0-bdwgc-dll. It just so happens that it tries to handle the exception and it fails (rightfully so, and that's a bug. We'll be taking care of it).

    However, the real issue is in AudioPluginOculusSpatializer.dll:

    upload_2019-2-7_11-36-28.png

    It uses AVX instructions and the person's CPU doesn't seem to support AVX, so it raises invalid instruction exception.

    A workaround would be to remove this DLL. I am not sure where it comes from - do you? I'd try deleting the DLL and see if the game still works.
     
    Tautvydas-Zilys, Feb 7, 2019
    #5

  6. Lewnatic

    Lewnatic

    Joined:
    Sep 29, 2012
    Posts:
    209
    >However, the real issue is in AudioPluginOculusSpatializer.dll:

    Dang i read somewhere that currently Oculus asset store package is not supported with Unity2018.3 is this true? Sadly i cannot downgrade because of some new version dependencies.
    It seems like this issue appeared when i upgraded to 2018.3 because with unity 2017 the same user had no issues.

    Thanks for you help. I am currently trying things out.
     
    Last edited: Feb 8, 2019
    Lewnatic, Feb 8, 2019
    #6

  7. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    I asked a dev in our VR team to comment on this.
     
    Tautvydas-Zilys, Feb 8, 2019
    #7

  8. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    We recently updated to Unity 2022 LTS and am seeing a similar issue. How did you figure out that the issue is in AudioPluginOculusSpatializer.dll?

    I believe one of our DLLs are out of date, but I'm not sure where to start for figuring out which one (its a large project).
     
    ron_unity614, Oct 11, 2023
    #8

  9. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Can you share a dump file?
     
    Tautvydas-Zilys, Oct 11, 2023
    #9

  10. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    ron_unity614, Oct 11, 2023
    #10

  11. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    The dump shows a crash on exit, is that what you see? There seems to be a thread that is calling Thread.ThreadState or Thread.IsAlive and that is accessing some data that has already been freed from what I can tell. Do you have any code that calls these things on non-main thread?

    This doesn't seem related at all to the previous issue.
     
    Tautvydas-Zilys, Oct 11, 2023
    #11

  12. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    Didn't mean to take over the thread, just saw mono-2.0-bdwgc.dll as part of the crash and was wondering how you came up with the solution. I also posted this a week or two ago and didn't get a response, so I appreciate your help.

    I am seeing a Thread.IsAlive call in a destructor, I'm guessing I should make sure this gets called before allowing the application to quit via Application.wantsToQuit.
     
    ron_unity614, Oct 11, 2023
    #12

  13. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    No problem at all! I believe what you are proposing should work around this issue.
     
    Tautvydas-Zilys, Oct 11, 2023
    #13

  14. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    I attempted the solution and still experienced a crash. But then I tried changing Application.Quit() to System.Diagnostics.Process.GetCurrentProcess().Kill() and am no longer seeing a crash...
     
    ron_unity614, Oct 12, 2023
    #14

  15. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Tautvydas-Zilys, Oct 12, 2023
    #15

  16. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    ron_unity614, Oct 12, 2023
    #16

  17. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    It is the finalizer for the class named "MedicalDataLoader". Here's the callstack:

    Code (csharp):
    1. ntdll.dll!RtlpWaitOnCriticalSection()
    2. ntdll.dll!RtlpEnterCriticalSectionContended()
    3. ntdll.dll!RtlEnterCriticalSection()
    4. [Inline Frame] mono-2.0-bdwgc.dll!mono_os_mutex_lock(mono_mutex_t *) Line 288
    5. mono-2.0-bdwgc.dll!mono_coop_mutex_lock(_MonoCoopMutex * mutex) Line 57
    6. [Inline Frame] mono-2.0-bdwgc.dll!lock_thread(_MonoInternalThread *) Line 527
    7. [Inline Frame] mono-2.0-bdwgc.dll!ves_icall_System_Threading_Thread_GetState(MonoInternalThreadHandle) Line 2841
    8. mono-2.0-bdwgc.dll!ves_icall_System_Threading_Thread_GetState_raw(volatile void * a0) Line 1064
    9. mscorlib.dll!System.Threading.Thread.GetState()
    10. mscorlib.dll!System.Threading.Thread.IsAlive
    11. Assembly-CSharp.dll!MedicalDataLoader.IsMedicalDataThreadAlive
    12. Assembly-CSharp.dll!MedicalDataLoader.AbortMedicalDataThread()
    13. Assembly-CSharp.dll!MedicalDataLoader.Finalize()
    14. mono-2.0-bdwgc.dll!mono_gc_run_finalize(void * obj, void * data) Line 383
    15. mono-2.0-bdwgc.dll!finalize_domain_objects() Line 944
    16. [Inline Frame] mono-2.0-bdwgc.dll!mono_runtime_do_background_work() Line 978
    17. mono-2.0-bdwgc.dll!finalizer_thread(void * unused) Line 1036
    18. mono-2.0-bdwgc.dll!start_wrapper_internal(StartInfo * start_info, unsigned __int64 * stack_ptr) Line 1268
    19. mono-2.0-bdwgc.dll!start_wrapper(void * data) Line 1344
    20. kernel32.dll!BaseThreadInitThunk()
    21. ntdll.dll!RtlUserThreadStart()
    It seems you are trying to shut down a C# thread on exit. You don't need to do that: Unity will do that for you on exit. You'd only need to do that if the thread was stuck in native C++ code.

    That said, us crashing this way is a bug. So while you can work around it by not calling Thread.IsAlive from a finalizer, we would probably try fixing it if you reported it as a bug.

    As for now I did it: I opened the dump file, configured symbol server paths to resolve symbols for mono-2.0-bdwgc.dll (as outlined in https://docs.unity3d.com/Manual/WindowsDebugging.html), then looked at local variables of native code to find the finalizer it was invoking and the pointer to the Mono domain (both were available inside the mono_gc_run_finalize frame). From there, I accessed the "domain->jit_info_table" field and did a manual binary search in it to find all the C# function names.
     
    Tautvydas-Zilys, Oct 12, 2023
    #17

  18. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    Thank you for all the information!! I'll submit it as a bug.

    I removed the call and the crash is no longer happening, but now I'm running into an issue where the application is not responding when it is exited. I'm not seeing any insights in the logs or in the editor relating to this.

    Any ideas as to how I can diagnose the application not responding?
     
    ron_unity614, Oct 13, 2023
    #18

  19. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    What is the thread that you were trying to abort actually doing? I had assumed it was a C# thread which generally get shut down by Unity when it exits. However, if that for some reason fails, Unity will hang waiting for it to exit. Maybe try moving that abort logic to the OnApplicationQuit() callback instead of the finalizer?
     
    Tautvydas-Zilys, Oct 13, 2023
    #19

  20. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    The thread is reading and serializing a fairly large XML file. I've done some testing and found that the application still hangs on exit even after this thread is closed / aborted.

    It's possible that one of our 3rd party libraries / plugins is not aborting a thread properly.

    The following is the Application Hang event details, not much information though...
    Code (CSharp):
    1. - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    2. - <System>
    3.   <Provider Name="Application Hang" />
    4.   <EventID Qualifiers="0">1002</EventID>
    5.   <Version>0</Version>
    6.   <Level>2</Level>
    7.   <Task>101</Task>
    8.   <Opcode>0</Opcode>
    9.   <Keywords>0x80000000000000</Keywords>
    10.   <TimeCreated SystemTime="2023-10-18T16:36:56.3116454Z" />
    11.   <EventRecordID>11582</EventRecordID>
    12.   <Correlation />
    13.   <Execution ProcessID="0" ThreadID="0" />
    14.   <Channel>Application</Channel>
    15.   <Computer>RON-AMD-RZR.sonosim.local</Computer>
    16.   <Security />
    17.   </System>
    18. - <EventData>
    19.   <Data>SonoSimulator.exe</Data>
    20.   <Data>2022.3.7.27451</Data>
    21.   <Data>69e4</Data>
    22.   <Data>01da01e069e74772</Data>
    23.   <Data>5</Data>
    24.   <Data>C:\Program Files\SonoSim\SonoSimulator\SonoSimulator.exe</Data>
    25.   <Data>c1da8969-fef0-4a17-bab4-af1e3a723deb</Data>
    26.   <Data />
    27.   <Data />
    28.   <Data>Unknown</Data>
    29.   <Binary>55006E006B006E006F0077006E0000000000</Binary>
    30.   </EventData>
    31.   </Event>
     
    ron_unity614, Oct 18, 2023
    #20

  21. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Can you also capture a dump file when the hang happens by finding the process in the task manager details tab, right clicking it and pressing "Create dump file". I'd be able to look at it and tell you what it hangs on.
     
    Tautvydas-Zilys, Oct 18, 2023
    #21

  22. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    ron_unity614, Oct 18, 2023
    #22

  23. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    How long was it hung when you captured the dump?
     
    Tautvydas-Zilys, Oct 18, 2023
    #23

  24. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    Maybe 30 seconds? I can leave it hanging for longer if needed
     
    ron_unity614, Oct 19, 2023
    #24

  25. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Can you leave it for 5 minutes and then capture the dump?
     
    Tautvydas-Zilys, Oct 19, 2023
    #25

  26. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    ron_unity614, Oct 19, 2023
    #26

  27. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Here's the callstack for the hang:

    Code (csharp):
    1. [Inline Frame] UnityPlayer.dll!handle_ring_t::try_dequeue(ujob_control_t *)
    2. UnityPlayer.dll!lane_try_steal(ujob_control_t * con, ujob_lane_t * self, int steal_hint, ujob_handle_t & handle, ujob_job_t * & job, unsigned int & work_left)
    3. [Inline Frame] UnityPlayer.dll!lane_guts(ujob_control_t *)
    4. UnityPlayer.dll!guest_work_until(ujob_control_t * con, int(*)(ujob_control_t *, void *) fn_ptr, void * user_data, ujob_lane_t * lane, int stealing_allowed)
    5. UnityPlayer.dll!ujob_schedule_parallel_for_internal(ujob_control_t * con, JobsCallbackFunctions & job_funcs, void * user_data, WorkStealingRange * steal_range, unsigned int count, unsigned int deps, const ujob_handle_t * dep_count, int flags, unsigned char)
    6. [Inline Frame] UnityPlayer.dll!ujob_schedule_single(ujob_control_t *)
    7. [Inline Frame] UnityPlayer.dll!JobQueue::ScheduleJob(void(*)(void *))
    8. UnityPlayer.dll!JobBatchDispatcher::ScheduleJobDependsInternal(JobFence & fence, void(*)(void *) jobFunc, void * userData, const JobFence & dependsOn)
    9. [Inline Frame] UnityPlayer.dll!JobBatchDispatcher::ScheduleJobDepends(JobFence &)
    10. UnityPlayer.dll!VideoMediaTextureOutput::ScheduleClearJob(JobFence & fence, JobBatchDispatcher & dispatcher, const JobFence & dependsFence)
    11. UnityPlayer.dll!VideoClipPlayback::Stop()
    12. UnityPlayer.dll!VideoPlayback::StopPlayback()
    13. UnityPlayer.dll!VideoPlaybackMgr::ReleaseVideoPlayback(VideoPlayback * playback)
    14. [Inline Frame] UnityPlayer.dll!VideoPlaybackMgr::{dtor}()
    15. UnityPlayer.dll!VideoPlaybackMgr::`scalar deleting destructor'(unsigned int)
    16. [Inline Frame] UnityPlayer.dll!delete_internal(VideoPlaybackMgr * ptr, MemLabelId)
    17. [Inline Frame] UnityPlayer.dll!MediaAPI::Playback::Shutdown()
    18. [Inline Frame] UnityPlayer.dll!VideoManager::CleanupClass()
    19. UnityPlayer.dll!VideoPlayer::CleanupClass()
    20. [Inline Frame] UnityPlayer.dll!TypeManager::CleanupAllTypes()
    21. UnityPlayer.dll!CleanupEngine()
    22. UnityPlayer.dll!PlayerCleanup(bool forceQuit)
    23. UnityPlayer.dll!DoQuit(bool postQuitMsg)
    24. UnityPlayer.dll!PlayerMainWndProc(HWND__ * hWnd, unsigned int message, unsigned __int64 wParam, __int64 lParam)
    25. user32.dll!UserCallWinProcCheckWow()
    26. user32.dll!DispatchClientMessage()
    27. user32.dll!__fnDWORD()
    28. ntdll.dll!00007ffd95af0f34()
    29. win32u.dll!00007ffd93231124()
    30. user32.dll!RealDefWindowProcWorker()
    31. user32.dll!RealDefWindowProcW()
    32. uxtheme.dll!00007ffd905ce782()
    33. uxtheme.dll!00007ffd905c7122()
    34. uxtheme.dll!00007ffd905d0869()
    35. uxtheme.dll!00007ffd905d02d1()
    36. user32.dll!DefWindowProcW()
    37. UnityPlayer.dll!PlayerMainWndProc(HWND__ * hWnd, unsigned int message, unsigned __int64 wParam, __int64 lParam)
    38. user32.dll!UserCallWinProcCheckWow()
    39. user32.dll!DispatchClientMessage()
    40. user32.dll!__fnDWORD()
    41. ntdll.dll!00007ffd95af0f34()
    42. win32u.dll!00007ffd93231124()
    43. user32.dll!RealDefWindowProcWorker()
    44. user32.dll!RealDefWindowProcW()
    45. uxtheme.dll!00007ffd905ce782()
    46. uxtheme.dll!00007ffd905c7077()
    47. uxtheme.dll!00007ffd905d0869()
    48. uxtheme.dll!00007ffd905d02d1()
    49. user32.dll!DefWindowProcW()
    50. UnityPlayer.dll!PlayerMainWndProc(HWND__ * hWnd, unsigned int message, unsigned __int64 wParam, __int64 lParam)
    51. user32.dll!UserCallWinProcCheckWow()
    52. user32.dll!DispatchMessageWorker()
    53. [Inline Frame] UnityPlayer.dll!TranslateAndDispatchEventMsg(const tagMSG &)
    54. UnityPlayer.dll!TranslateAndDispatchFilteredMessage(const tagMSG & msg)
    55. UnityPlayer.dll!MainMessageLoop()
    56. UnityPlayer.dll!UnityMainImpl(HINSTANCE__ * szCmdLine, HINSTANCE__ * nCmdShow, wchar_t *)
    57. UnityPlayer.dll!UnityMain(HINSTANCE__ * hInst, HINSTANCE__ * hPrev, wchar_t * szCmdLine, int nCmdShow)
    58. SonoSimulator.exe!00007ff70fe211f2()
    59. kernel32.dll!BaseThreadInitThunk()
    60. ntdll.dll!00007ffd95aa26f1()
    It hangs while trying to stop video playback on exit. It is not entirely clear to me what's going on here, I will talk to our job system and video team about it.
     
    Tautvydas-Zilys, Oct 20, 2023
    #27

  28. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    Okay, great, thank you! In the meanwhile I'll take a look at the codebase and try to find misusages of the video player component.
     
    ron_unity614, Oct 20, 2023
    #28

  29. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Is there anything getting written into the log file at all after the hang happens?
     
    Tautvydas-Zilys, Oct 20, 2023
    #29

  30. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    Nothing after the hang happens, but I'm seeing `Setting up 8 worker threads for Enlighten.` right when application is trying to exit.
     
    ron_unity614, Oct 20, 2023
    #30

  31. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    Hello! Any updates on this?

    I've tried to disable all Video Player components in the scene and was no longer experiencing the application not responding.

    Is there a way to fully unload the video player? There's VideoPlayer.Stop but its not fully unloading the player.
     
    ron_unity614, Oct 24, 2023
    #31

  32. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    The video folks said they will look into it.

    Do you have any code to seek the video clip?
     
    Tautvydas-Zilys, Oct 24, 2023
    #32

  33. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    We have some being called on Reset, but I removed it and am still seeing the issue.
     
    ron_unity614, Oct 24, 2023
    #33

  34. ron_unity614

    ron_unity614

    Joined:
    Oct 29, 2020
    Posts:
    24
    I might have figured it out!

    I went through each video player component and turned them off one by one, then ran a build to see if the app would hang. I found one that was problematic and checked the video file itself and saw that it was the only video not being transcoded. I enabled the transcoding in the import settings and set it to VP8 and now I'm not experiencing the hang anymore.

    I'm handing the build over to QA for some extra testing to be sure, and I'll keep you updated on whether this fixed the issue.
     
    ron_unity614, Oct 25, 2023
    #34

  35. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,674
    Nice!
     
    Tautvydas-Zilys, Oct 25, 2023
    #35