Bug Freeze when running on Android and iOS devices(Async Upload Pipeline)

Recently, we found our game may become freezing when loading scene. The following is the callstack of main thread:

Spoiler: Issue 1

#0 0x00000001b899f540 in semaphore_wait_trap ()
#1 0x00000001810b7c00 in _dispatch_sema4_wait ()
#2 0x00000001810b82b8 in _dispatch_semaphore_wait_slow ()
#3 0x0000000110d4d114 in DarwinApi::detail::Acquire(UnityClassic::Baselib_SystemSemaphore_Handle, unsigned long long) [inlined] at /Users/x/Unity/unity2020.3.30f1/External/baselib/builds/Source/Darwin/Baselib_SystemSemaphore_DarwinApi.inl.h:35
#4 0x0000000110d4d108 in DarwinApi::Baselib_SystemSemaphore_Acquire(UnityClassic::Baselib_SystemSemaphore_Handle) [inlined] at /Users/x/Unity/unity2020.3.30f1/External/baselib/builds/Source/Darwin/Baselib_SystemSemaphore_DarwinApi.inl.h:54
#5 0x0000000110d4d108 in UnityClassic::Baselib_SystemSemaphore_Acquire(UnityClassic::Baselib_SystemSemaphore_Handle) at /Users/x/Unity/unity2020.3.30f1/External/baselib/builds/Source/CProxy/Baselib_SystemSemaphore_CProxy.inl.h:14
#6 0x0000000110783228 in Baselib_CappedSemaphore_Acquire [inlined] at /Users/x/Unity/unity2020.3.30f1/External/baselib/builds/Include/C/Internal/Baselib_CappedSemaphore_SemaphoreBased.inl.h:33
#7 0x0000000110783220 in WaitForSignal [inlined] at /Users/x/Unity/unity2020.3.30f1/Runtime/Threads/CappedSemaphore.h:29
#8 0x0000000110783220 in ::SyncForMainThread() at /Users/x/Unity/unity2020.3.30f1/Runtime/Graphics/Mesh/MeshAsyncUpload.cpp:231
#9 0x0000000110c0b79c in ::CompleteUploadCommand() at /Users/x/Unity/unity2020.3.30f1/Runtime/Graphics/Mesh/Mesh.cpp:3157
#10 0x0000000110c0b73c in ::AwakeFromLoad() at /Users/x/Unity/unity2020.3.30f1/Runtime/Graphics/Mesh/Mesh.cpp:3104
#11 0x0000000110910070 in ::InvokePersistentManagerAwake() at /Users/x/Unity/unity2020.3.30f1/Runtime/Serialize/AwakeFromLoadQueue.cpp:504
#12 0x000000011090fe6c in :ersistentManagerAwakeFromLoad() at /Users/x/Unity/unity2020.3.30f1/Runtime/Serialize/AwakeFromLoadQueue.cpp:320
#13 0x000000011090fdc0 in :ersistentManagerAwakeFromLoad() at /Users/x/Unity/unity2020.3.30f1/Runtime/Serialize/AwakeFromLoadQueue.cpp:192
#14 0x00000001108ff76c in ::IntegrateAllThreadedObjects() at /Users/x/Unity/unity2020.3.30f1/Runtime/Serialize/PersistentManager.cpp:1256
#15 0x00000001108fff3c in ::LoadAndIntegrateAllPreallocatedObjects() at /Users/x/Unity/unity2020.3.30f1/Runtime/Serialize/PersistentManager.cpp:1402
#16 0x0000000110901298 in ::LoadObjects() at /Users/x/Unity/unity2020.3.30f1/Runtime/Serialize/PersistentManager.cpp:1884
#17 0x000000011055d364 in :rocessAssetBundleEntries() at /Users/x/Unity/unity2020.3.30f1/Modules/AssetBundle/Public/AssetBundleLoadAssetUtility.cpp:50
#18 0x000000011055d234 in ::LoadNamedObjectFromAssetBundle() at /Users/x/Unity/unity2020.3.30f1/Modules/AssetBundle/Public/AssetBundleLoadAssetUtility.cpp:70
#19 0x00000001105541d8 in ::LoadAsset_Internal() at /Users/x/Unity/unity2020.3.30f1/Modules/AssetBundle/Public/AssetBundle.cpp:272
#20 0x00000001105d0a20 in ::AssetBundle_CUSTOM_LoadAsset_Internal() at /Users/x/Unity/unity2020.3.30f1/artifacts/iOS/AssetBundle/iOS_arm64_nondev_i_r/AssetBundleBindings.gen.cpp:489

Spoiler: Issue 2

#0 0x00000001b899f540 in semaphore_wait_trap ()
#1 0x00000001810b7c00 in _dispatch_sema4_wait ()
#2 0x00000001810b82b8 in _dispatch_semaphore_wait_slow ()
#3 0x000000010dcc9d44 in DarwinApi::detail::Acquire(UnityClassic::Baselib_SystemSemaphore_Handle, unsigned long long) [inlined] at /Users/x/Unity/unity2020.3.30f1/External/baselib/builds/Source/Darwin/Baselib_SystemSemaphore_DarwinApi.inl.h:35
#4 0x000000010dcc9d38 in DarwinApi::Baselib_SystemSemaphore_Acquire(UnityClassic::Baselib_SystemSemaphore_Handle) [inlined] at /Users/x/Unity/unity2020.3.30f1/External/baselib/builds/Source/Darwin/Baselib_SystemSemaphore_DarwinApi.inl.h:54
#5 0x000000010dcc9d38 in UnityClassic::Baselib_SystemSemaphore_Acquire(UnityClassic::Baselib_SystemSemaphore_Handle) at /Users/x/Unity/unity2020.3.30f1/External/baselib/builds/Source/CProxy/Baselib_SystemSemaphore_CProxy.inl.h:14
#6 0x000000010d55727c in Baselib_Semaphore_Acquire [inlined] at /Users/x/Unity/unity2020.3.30f1/External/baselib/builds/Include/C/Internal/Baselib_Semaphore_SemaphoreBased.inl.h:41
#7 0x000000010d557274 in WaitForSignal [inlined] at /Users/x/Unity/unity2020.3.30f1/Runtime/Threads/Semaphore.h:23
#8 0x000000010d557274 in ::CreateGpuProgram() at /Users/x/Unity/unity2020.3.30f1/Runtime/GfxDevice/threaded/GfxDeviceClient.cpp:1076
#9 0x000000010d7a203c in ::Compile() at /Users/x/Unity/unity2020.3.30f1/Runtime/Shaders/ShaderImpl/ShaderProgram.cpp:433
#10 0x000000010d7a2ec8 in EnsureCompiled [inlined] at /Users/x/Unity/unity2020.3.30f1/Runtime/Shaders/ShaderImpl/ShaderProgram.cpp:283
#11 0x000000010d7a2eac in ::EnsureCompiledAndMarkUnsupportedIfFailed() at /Users/x/Unity/unity2020.3.30f1/Runtime/Shaders/ShaderImpl/ShaderProgram.cpp:908
#12 0x000000010d7a3138 in ::GetMatchingSubProgram() at /Users/x/Unity/unity2020.3.30f1/Runtime/Shaders/ShaderImpl/ShaderProgram.cpp:1097
#13 0x000000010d7a7b98 in ::FindSubProgramsToUse() at /Users/x/Unity/unity2020.3.30f1/Runtime/Shaders/ShaderImpl/ShaderState.cpp:408
#14 0x000000010d7a7e70 in ::ApplyShaderState() at /Users/x/Unity/unity2020.3.30f1/Runtime/Shaders/ShaderImpl/ShaderState.cpp:532
#15 0x000000010d7a0cd4 in ::ApplyPass() at /Users/x/Unity/unity2020.3.30f1/Runtime/Shaders/ShaderImpl/ShaderPass.cpp:72

They are reproducible in auto test for our game(Loading scene again and again), but I can't create a simple repro project.

After futher investigation, I found they can be fixed by adding memory barrier at the proper position. But I'm not 100% sure about it, and our enterprise support ticket is not responding. So I post here and hope unity developers can take a look at these two issues.

Here are the bug reports(Including the full callstack of all threads, my thoughts and solution) for them:

• https://unity3d.atlassian.net/servicedesk/customer/portal/2/IN-8601
  
• https://unity3d.atlassian.net/servicedesk/customer/portal/2/IN-8605

 

@aleksandrk

 

Hi,

I sent the support person name and ticket number in the conversation(Not sure about wthether it's DM).

 

I have the same issue, but absolutely have no clue, either. Some of my TestFlight testers are reporting crashes when in the loading scene. In the loading scene, we are initializing some plugins like Firebase, Facebook SDK, Unity Mobile Notification, Addressables, etc. But the stack trace of the crash is so vague.

 

Try to right click the callback and select 'Show in Finder'. There should be some detailed logs in the opened folder.

 

Here it is. I see that it has something to do with loading AssetBundle. We are indeed loading some AssetBundles via Addressables in the loading scene. Maybe we are not the same issue.

 

It seems to be different to the issues I had, but similar to this one:
https://forum.unity.com/threads/deadlock-in-assetbundlemanager.1196116/

 

What's you Unity version? Maybe you can try to upgrade to the latest versions, e.g. 2019.4.40, 2020.3.37

 

We are using Unity 2019.4.22f1.

 

I noticed Unity 2021.3.11 includes a fix:

Does it fix the issue about freezing in mesh upload in my first post? Will it be backported to Unity 2020.3.x?

If it's not the same issue, hope Unity QA can process my bug report. It has been over 3 months since I submitted and it's still in OPEN status.

 

The first issue about mesh has been fixed in Unity 2020.3.41. But the second issue about shader seems to be not fixed yet: https://issuetracker.unity3d.com/issues/ios-freeze-when-compiling-shaders.

I saw other people also encountered this issue: https://forum.unity.com/threads/occ...vices-seeking-help-with-stack-traces.1425012/

Any update on this?