Search Unity

Question signal 11 (SIGSEGV), code 1 (SEGV_MAPERR)

Discussion in 'Android' started by GrumpyRhino, May 23, 2020.

  1. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    Hey,

    So on our most recent game we're getting a high crash rate on Google Play. Our general crash rate is 0.77% but on Android 10 it's higher, about 1.46% (above the bad behaviour threshold). We have a bunch of similar crashes all signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), but the two below are the most common.

    We've spent a couple of weeks trying to work out what's going on and have tried a bunch of different things, but in all honesty I don't really know where to start, so was wondering if anyone could give us a pointer. We're on 2019.3.13f1. We've updated all our plugins to the most recent versions (Google Play Games, IronSource, PlayFab etc) and this hasn't made a difference to the crash rate.

    We haven't been able to reproduce these issues locally and we're not getting reports from players (we have quite a tight reddit community, so I would expect at least some reports if it was disrupting gameplay), so I don't think the crashes are player facing. I'm wondering if the game crashes silently when you resume playing or when you close it.

    Any help is massively appreciated.

    pid: 0, tid: 0 >>> com.grumpyrhinogames.idlemastermind <<< backtrace:

    #00 pc 00000000001424fc /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/lib/arm64/libunity.so

    #01 pc 00000000006244a0 /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/lib/arm64/libunity.so

    #02 pc 0000000000624454 /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/lib/arm64/libunity.so

    #03 pc 0000000000622340 /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/lib/arm64/libunity.so

    #04 pc 0000000000621814 /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/lib/arm64/libunity.so

    #05 pc 000000000062033c /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/lib/arm64/libunity.so

    #06 pc 000000000064e678 /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/lib/arm64/libunity.so

    #07 pc 000000000004883c /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/oat/arm64/base.odex (art_jni_trampoline+124)

    #08 pc 0000000000088fc4 /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/oat/arm64/base.odex (com.unity3d.player.UnityPlayer.resume+948)

    #09 pc 0000000000089c1c /data/app/com.grumpyrhinogames.idlemastermind-atYXsde-gpBYoX2KKqZdTA==/oat/arm64/base.odex (com.unity3d.player.UnityPlayerActivity.onResume+76)

    #10 pc 00000000003e3b60 /system/framework/arm64/boot-framework.oat (android.app.Instrumentation.callActivityOnResume+80)

    #11 pc 00000000006c1afc /system/framework/arm64/boot-framework.oat (android.app.Activity.performResume+444)

    #12 pc 00000000004dfdf0 /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.performResumeActivity+640)

    #13 pc 00000000004dd634 /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.handleResumeActivity+276)

    #14 pc 00000000006ddc90 /system/framework/arm64/boot-framework.oat (android.app.servertransaction.ResumeActivityItem.execute+128)

    #15 pc 000000000040df24 /system/framework/arm64/boot-framework.oat (android.app.servertransaction.TransactionExecutor.executeLifecycleState+340)

    #16 pc 000000000040e6e8 /system/framework/arm64/boot-framework.oat (android.app.servertransaction.TransactionExecutor.execute+776)

    #17 pc 00000000004c3350 /system/framework/arm64/boot-framework.oat (android.app.ActivityThread$H.handleMessage+496)

    #18 pc 000000000074dfc4 /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+180)

    #19 pc 0000000000751560 /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+1440)

    #20 pc 00000000004d4b14 /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.main+788)

    #21 pc 00000000001375b8 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568)

    #22 pc 000000000014600c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+276)

    #23 pc 00000000004b1884 /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)

    #24 pc 00000000004b3428 /apex/com.android.runtime/lib64/libart.so (art::InvokeMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1480)

    #25 pc 000000000043e1c8 /apex/com.android.runtime/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*)+52)

    #26 pc 00000000000c2c54 /system/framework/arm64/boot.oat (art_jni_trampoline+180)

    #27 pc 00000000009c39f8 /system/framework/arm64/boot-framework.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)

    #28 pc 00000000009cbfa4 /system/framework/arm64/boot-framework.oat (com.android.internal.os.ZygoteInit.main+3124)

    #29 pc 00000000001375b8 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568)

    #30 pc 000000000014600c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+276)

    #31 pc 00000000004b1884 /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)

    #32 pc 00000000004b1474 /apex/com.android.runtime/lib64/libart.so (art::InvokeWithVarArgs(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+408)

    #33 pc 00000000003bbb9c /apex/com.android.runtime/lib64/libart.so (art::JNI::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+624)

    #34 pc 00000000000ed9e4 /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+116)

    #35 pc 00000000000f0b04 /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+792)

    #36 pc 00000000000034f8 /system/bin/app_process64 (main+1192)

    #37 pc 000000000007e898 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+108)


    pid: 0, tid: 0 >>> com.grumpyrhinogames.idlemastermind <<< backtrace:

    #00 pc 00000000001424fc /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/lib/arm64/libunity.so

    #01 pc 00000000006244a0 /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/lib/arm64/libunity.so

    #02 pc 0000000000624454 /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/lib/arm64/libunity.so

    #03 pc 0000000000622340 /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/lib/arm64/libunity.so

    #04 pc 0000000000621808 /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/lib/arm64/libunity.so

    #05 pc 000000000062033c /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/lib/arm64/libunity.so

    #06 pc 000000000064e6d0 /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/lib/arm64/libunity.so

    #07 pc 0000000000045758 /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/oat/arm64/base.odex (art_jni_trampoline+152)

    #08 pc 00000000000724d4 /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/oat/arm64/base.odex (com.unity3d.player.UnityPlayer$e$1.handleMessage+1956)

    #09 pc 0000000000733578 /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+136)

    #10 pc 0000000000736b50 /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+1440)

    #11 pc 0000000000072ef8 /data/app/com.grumpyrhinogames.idlemastermind-kVtZIMYn8sDsSfNr-GwXag==/oat/arm64/base.odex (com.unity3d.player.UnityPlayer$e.run+216)

    #12 pc 0000000000136334 /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_stub+548)

    #13 pc 000000000014506c /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+244)

    #14 pc 00000000004a9668 /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)

    #15 pc 00000000004aa6fc /apex/com.android.runtime/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue const*)+416)

    #16 pc 00000000004ea494 /apex/com.android.runtime/lib64/libart.so (art::Thread::CreateCallback(void*)+1176)

    #17 pc 00000000000e10a0 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36)

    #18 pc 0000000000083ab0 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)
     
    JustAnotherDude likes this.
  2. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    Code (CSharp):
    1. void VFXSystemDescT<VFXTaskDesc>::Transfer<StreamedBinaryWrite>(StreamedBinaryWrite&)
    2. glsl::ExtractDefineBlock(core::basic_string<char, core::StringStorageDefault<char> > const&, core::basic_string<char, core::StringStorageDefault<char> > const&, core::basic_string<char, core::StringStorageDefault<char> >*)
    3. glsl::ExtractDefineBlock(core::basic_string<char, core::StringStorageDefault<char> > const&, core::basic_string<char, core::StringStorageDefault<char> > const&, core::basic_string<char, core::StringStorageDefault<char> >*)
    4. RenderMultipleMeshes(RenderNodeQueue const&, RenderMultipleData const&, VertexInputMasks)
    5. ProcessDynamicBatchGeometryJob(SharedGeometryJobData*, unsigned int)
    6. void std::__ndk1::__sort<vector_map<int, InstancingBatcher::PropertyType, std::__ndk1::less<int>, stl_allocator<std::__ndk1::pair<int, InstancingBatcher::PropertyType>, (MemLabelIdentifier)1, 16> >::value_compare&, std::__ndk1::pair<int, InstancingBatcher::PropertyType>*>(std::__ndk1::pair<int, InstancingBatcher::PropertyType>*, std::__ndk1::pair<int, InstancingBatcher::PropertyType>*, vector_map<int, InstancingBatcher::PropertyType, std::__ndk1::less<int>, stl_allocator<std::__ndk1::pair<int, InstancingBatcher::PropertyType>, (MemLabelIdentifier)1, 16> >::value_compare&)
    7. insphereexact(double*, double*, double*, double*, double*)
    8. ??
    9. AvatarBuilder::BuildAvatarConstantFromTransformHierarchy(Avatar*, GameObject const&, RuntimeBaseAllocator&)
    10. AvatarMask::RemoveTransformPath(Transform&, bool)
    11. TrailRenderer_CUSTOM_get_startColor_Injected(ScriptingBackendNativeObjectPtrOpaque*, ColorRGBAf&)
    12. void NoiseModule::CalculateNoiseJob<1, true, true>(NoiseJobData*)
    13. void LightProbes::Transfer<SafeBinaryRead>(SafeBinaryRead&)
    14. OcclusionArea::VirtualRedirectTransfer(GenerateTypeTreeTransfer&)
    15. void ApplyForceField<(ParticleSystemForceFieldShape)3>(ParticleSystemParticles&, unsigned int, unsigned int, math::affineX const&, math::affineX const&, ParticleSystemForceFieldParameters const&, float __vector(4) const&, MinMaxCurve const&)
    16. ExportCoreBindingsNew()
    17. ExportCoreBindingsNew()
    18. LensFlare::VirtualRedirectTransfer(SafeBinaryRead&)
    19. unsigned int std::__ndk1::__sort3<bool (*&)(UnityVRTrackedObjectInfo const&, UnityVRTrackedObjectInfo const&), UnityVRTrackedObjectInfo*>(UnityVRTrackedObjectInfo*, UnityVRTrackedObjectInfo*, UnityVRTrackedObjectInfo*, bool (*&)(UnityVRTrackedObjectInfo const&, UnityVRTrackedObjectInfo const&))
    20. Rigidbody_CUSTOM_get_position_Injected(ScriptingBackendNativeObjectPtrOpaque*, Vector3f&)
    21. void Light::Transfer<StreamedBinaryWrite>(StreamedBinaryWrite&)
    22. VFXExpressionNoiseFunctions::Cellular::Generate2D(float __vector(2) const&)
    23. VFXSpawnerSystem::Validate(VFXSystemDesc const&, dynamic_array<VFXCPUBuffer, 0u> const&)
    24. GlslGpuProgramGLES::LoadFromBinaryShaderCache(unsigned int&, core::basic_string<char, core::StringStorageDefault<char> > const&, ShaderChannelMask&, char const*, int*, int*)
    25. CompileFragmentShader(core::basic_string<char, core::StringStorageDefault<char> > const&, bool, bool)
    26. Camera_Set_Custom_PropAllowDynamicResolution(ScriptingBackendNativeObjectPtrOpaque*, unsigned char)
    27. InvalidatePropertySceneHandleBoundCurvesCallback(void*, void*, int)
    28. physx::pvdsdk::PvdMarshalling<short, float>::marshalBlock(unsigned char const*, unsigned char*, unsigned int)
    29. physx::NpShape::setFlagsInternal(physx::PxFlags<physx::PxShapeFlag::Enum, unsigned char>)
    30. VFXExpressionNoiseFunctions::Cellular::Generate2D(float __vector(2) const&)
    31. VFXSpawnerSystem::Validate(VFXSystemDesc const&, dynamic_array<VFXCPUBuffer, 0u> const&)
    32. GlslGpuProgramGLES::LoadFromBinaryShaderCache(unsigned int&, core::basic_string<char, core::StringStorageDefault<char> > const&, ShaderChannelMask&, char const*, int*, int*)
    33. GlslGpuProgramGLES::SwitchProgram(unsigned int)
    34. ControllerFilterCallback::filter(physx::PxController const&, physx::PxController const&)
    35. RegisterModule_HotReload
    36. void Transfer_ManagedObject<JSONRead, false>(SerializationCommandArguments const&, RuntimeSerializationCommandInfo&)
    37. ??
    38. Scripting::UnityEngine::RenderTextureProxy::InitProxy()
    So I've done addr2line (heh, didn't know that was a thing) and this is what's come out. So it looks like a shader issue if I'm reading that right?
     
  3. ARPAplus

    ARPAplus

    Joined:
    Jun 5, 2015
    Posts:
    32
    We have a lot of similar crashes, with the latest 2019.2 or 2019.3. Nobody really cares. Some of the games has 0.70% crash rate, other crossed bad behaviour with 1.25% or so. Downloads are constantly drops. Very sad, that Unity Team doing nothing for this ;(

    In our games we have AdMob (and Chartboost + Applovin mediation) and Unity Ads. Ads and ads plugins - all the latest versions. Unity 2019.3.14f + Api Level 29

    Updated just yesterday, still no success, same crashes everywhere.

    We tried to fix it with "backup prefs" solution (.xmls and androidmanifest), but this is does not worked.

    http://i.imgur.com/T4OibdL.png
     
  4. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    Are you sure you're using correct path for symbols, the resolved stacktrace doesn't look correct.
     
  5. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    Hi Tomas, thanks very much for the reply. We followed the instructions on https://support.unity3d.com/hc/en-us/articles/115000292166-Symbolicate-Android-crash, but needed to use a different path to what's on there and used the wrong one.

    We tried again using the following which I think is the correct symbol file.

    Code (CSharp):
    1. ./toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/bin/aarch64-linux-android-addr2line -f -C -e /Applications/Unity/Hub/Editor/2019.3.13f1/PlaybackEngines/AndroidPlayer/Variations/il2cpp/Release/Symbols/arm64-v8a/libunity.sym.so    00000000001424fc  00000000006244a0  0000000000624454  0000000000622340  0000000000621808  000000000062033c  000000000064e678  000000000004e83c  00000000000a39b4  00000000000a469c  000000000044a930  000000000074b3cc  0000000000558540  0000000000555cf4  000000000076b460  000000000047cf04  000000000047d6c8  000000000053a670  00000000007e1104  00000000007e4770  000000000054c8d4  00000000001375b8  000000000014600c  00000000004b1718  00000000004b32bc  000000000043e05c  00000000000c9d34  0000000000a97268  0000000000a9fa24  00000000001375b8  000000000014600c  00000000004b1718  00000000004b1308  00000000003bba30  00000000000ee4dc  00000000000f15fc  00000000000034f8  000000000007e898  
    This produces the following.

    Code (CSharp):
    1.  
    2. void std::__ndk1::__tree_balance_after_insert<std::__ndk1::__tree_node_base<void*>*>(std::__ndk1::__tree_node_base<void*>*, std::__ndk1::__tree_node_base<void*>*)
    3. ??:?
    4. std::__ndk1::__tree<std::__ndk1::__value_type<int, sigaction>, std::__ndk1::__map_value_compare<int, std::__ndk1::__value_type<int, sigaction>, std::__ndk1::less<int>, true>, std::__ndk1::allocator<std::__ndk1::__value_type<int, sigaction> > >::__insert_node_at(std::__ndk1::__tree_end_node<std::__ndk1::__tree_node_base<void*>*>*, std::__ndk1::__tree_node_base<void*>*&, std::__ndk1::__tree_node_base<void*>*)
    5. ??:?
    6. std::__ndk1::pair<std::__ndk1::__tree_iterator<std::__ndk1::__value_type<int, sigaction>, std::__ndk1::__tree_node<std::__ndk1::__value_type<int, sigaction>, void*>*, long>, bool> std::__ndk1::__tree<std::__ndk1::__value_type<int, sigaction>, std::__ndk1::__map_value_compare<int, std::__ndk1::__value_type<int, sigaction>, std::__ndk1::less<int>, true>, std::__ndk1::allocator<std::__ndk1::__value_type<int, sigaction> > >::__emplace_unique_key_args<int, std::__ndk1::piecewise_construct_t const&, std::__ndk1::tuple<int const&>, std::__ndk1::tuple<> >(int const&, std::__ndk1::piecewise_construct_t const&, std::__ndk1::tuple<int const&>&&, std::__ndk1::tuple<>&&)
    7. ??:?
    8. NativeRuntimeException::chain_handler(int, sigaction&)
    9. ??:?
    10. NativeRuntimeException::install_signal_handlers()
    11. ??:?
    12. NativeRuntimeException::GetExceptionState()
    13. ??:?
    14. nativeRestartActivityIndicator(_JNIEnv*, _jobject*)
    15. ??:?
    16. ??
    17. ??:0
    18. ??
    19. ??:0
    20. ??
    21. ??:0
    22. Cubemap::VirtualRedirectTransfer(SafeBinaryRead&)
    23. ??:?
    24. vk::CommandBuffer::FlushBarriers(bool)
    25. ??:?
    26. TransformChangeDispatch::RemoveTransformHierarchy(TransformHierarchy&)
    27. ??:?
    28. dynamic_array<dynamic_array<StreamingTextureMipLevel, 0ul>, 0ul>::emplace_back()
    29. ??:?
    30. prcore::StretchRGB9e5Bilinear(prcore::InnerInfo const*)
    31. ??:?
    32. AmbientProbeBaker::CalculateAmbientFaceProcessor::operator()(Image&, int, Vector3f const&, Vector3f const&, Vector3f const&, bool)
    33. ??:?
    34. AmbientProbeBaker::CalculateAmbientFaceProcessor::operator()(Image&, int, Vector3f const&, Vector3f const&, Vector3f const&, bool)
    35. ??:?
    36. std::__ndk1::pair<core::pair<ShaderTagID, ShaderTagID, false>*, long> std::__ndk1::get_temporary_buffer<core::pair<ShaderTagID, ShaderTagID, false> >(long)
    37. ??:?
    38. FrustumCullReflectionProbesJob(ReflectionProbeCullJobData*, unsigned int)
    39. ??:?
    40. DeferredRenderLoopJob(GfxDeviceAsyncCommand::ArgScratch*, GfxDeviceAsyncCommand::Arg const*)
    41. ??:?
    42. ShaderLab::IntShader::ResolveFallback(ShaderLab::SerializedShader const&, Shader*, std::__ndk1::vector<PPtr<Shader>, std::__ndk1::allocator<PPtr<Shader> > >&, bool)
    43. ??:?
    44. AnimationState::UpdateFading(float)
    45. ??:?
    46. std::__ndk1::vector<AnimationClip::FloatCurve, stl_allocator<AnimationClip::FloatCurve, (MemLabelIdentifier)30, 16> >::__vallocate(unsigned long)
    47. ??:?
    48. void GrowableBuffer::WriteArrayType<Matrix4x4f>(Matrix4x4f const*, int, unsigned long)
    49. ??:?
    50. RenderingCommandBuffer::ExecuteCommandBufferWithState(ShaderPassContext&, RenderNodeQueue&, RenderingCommandBufferState*, unsigned int, ComputeQueueType) const
    51. ??:?
    52. DoDeCruncherJob(TextureUploadInstruction*)
    53. ??:?
    54. ??
    55. ??:0
    56. ClipperLib::Clipper::Clipper(int)
    57. ??:?
    58. AnchoredJoint2D::ConfigureDistance(b2Vec2 const&, b2Vec2 const&)
    59. ??:?
    60. AnimationState::UpdateFading(float)
    61. ??:?
    62. std::__ndk1::vector<AnimationClip::FloatCurve, stl_allocator<AnimationClip::FloatCurve, (MemLabelIdentifier)30, 16> >::__vallocate(unsigned long)
    63. ??:?
    64. void GrowableBuffer::WriteArrayType<Matrix4x4f>(Matrix4x4f const*, int, unsigned long)
    65. ??:?
    66. void GrowableBuffer::WriteValueType<RenderCommandSetGlobalTexture>(RenderCommandSetGlobalTexture const&, unsigned long)
    67. ??:?
    68. void std::__ndk1::__insertion_sort_3<std::__ndk1::__less<Hash128, Hash128>&, Hash128*>(Hash128*, Hash128*, std::__ndk1::__less<Hash128, Hash128>&)
    69. ??:?
    70. _GLOBAL__sub_I_Runtime_Graphics_9.cpp
    71. ??:?
    72. _GLOBAL__sub_I_Modules_Physics_2.cpp
    73. ??:?
    74. ??
    75. ??:0
    76. ??
    77. ??:0
    78.  
    Does this look correct and does it mean anything to you if so?
     
  6. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    Lines [1-15] looks correct, everything else doesn't look correct. Can you copy paste unresolved stacktrace?
     
  7. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    This is the one.

    Code (CSharp):
    1. *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
    2. pid: 0, tid: 0 >>> com.grumpyrhinogames.idlemastermind <<<
    3.  
    4. backtrace:
    5.   #00  pc 00000000001424fc  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/lib/arm64/libunity.so
    6.   #01  pc 00000000006244a0  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/lib/arm64/libunity.so
    7.   #02  pc 0000000000624454  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/lib/arm64/libunity.so
    8.   #03  pc 0000000000622340  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/lib/arm64/libunity.so
    9.   #04  pc 0000000000621808  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/lib/arm64/libunity.so
    10.   #05  pc 000000000062033c  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/lib/arm64/libunity.so
    11.   #06  pc 000000000064e678  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/lib/arm64/libunity.so
    12.   #07  pc 000000000005883c  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/oat/arm64/base.odex (art_jni_trampoline+124)
    13.   #08  pc 00000000000d31d4  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/oat/arm64/base.odex (com.unity3d.player.UnityPlayer.resume+932)
    14.   #09  pc 00000000000d3d2c  /data/app/com.grumpyrhinogames.idlemastermind-qsgtog0u09-7C6OGF0ODFg==/oat/arm64/base.odex (com.unity3d.player.UnityPlayerActivity.onResume+76)
    15.   #10  pc 00000000003dad10  /system/framework/arm64/boot-framework.oat (android.app.Instrumentation.callActivityOnResume+80)
    16.   #11  pc 00000000006a5a18  /system/framework/arm64/boot-framework.oat (android.app.Activity.performResume+360)
    17.   #12  pc 00000000004d0318  /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.performResumeActivity+600)
    18.   #13  pc 00000000004cdf5c  /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.handleResumeActivity+268)
    19.   #14  pc 00000000006c3d20  /system/framework/arm64/boot-framework.oat (android.app.servertransaction.ResumeActivityItem.execute+128)
    20.   #15  pc 0000000000403534  /system/framework/arm64/boot-framework.oat (android.app.servertransaction.TransactionExecutor.executeLifecycleState+340)
    21.   #16  pc 0000000000403cf8  /system/framework/arm64/boot-framework.oat (android.app.servertransaction.TransactionExecutor.execute+776)
    22.   #17  pc 00000000004b5070  /system/framework/arm64/boot-framework.oat (android.app.ActivityThread$H.handleMessage+496)
    23.   #18  pc 00000000007335a4  /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+180)
    24.   #19  pc 0000000000736b50  /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+1440)
    25.   #20  pc 00000000004c5d24  /system/framework/arm64/boot-framework.oat (android.app.ActivityThread.main+756)
    26.   #21  pc 00000000001365b8  /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568)
    27.   #22  pc 000000000014508c  /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+276)
    28.   #23  pc 00000000004a9668  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
    29.   #24  pc 00000000004ab090  /apex/com.android.runtime/lib64/libart.so (art::InvokeMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1476)
    30.   #25  pc 0000000000437ab0  /apex/com.android.runtime/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*)+52)
    31.   #26  pc 00000000000c2c34  /system/framework/arm64/boot.oat (art_jni_trampoline+180)
    32.   #27  pc 00000000009a9988  /system/framework/arm64/boot-framework.oat (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+136)
    33.   #28  pc 00000000009b155c  /system/framework/arm64/boot-framework.oat (com.android.internal.os.ZygoteInit.main+2076)
    34.   #29  pc 00000000001365b8  /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_static_stub+568)
    35.   #30  pc 000000000014508c  /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+276)
    36.   #31  pc 00000000004a9668  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
    37.   #32  pc 00000000004a92d4  /apex/com.android.runtime/lib64/libart.so (art::InvokeWithVarArgs(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+408)
    38.   #33  pc 00000000003b67c4  /apex/com.android.runtime/lib64/libart.so (art::JNI::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+628)
    39.   #34  pc 00000000000be560  /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+116)
    40.   #35  pc 00000000000c13e8  /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector<android::String8> const&, bool)+780)
    41.   #36  pc 00000000000034e0  /system/bin/app_process64 (main+1168)
    42.   #37  pc 000000000007d780  /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+108)
     
  8. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    so as you see only frames from [#00 .. #06] are from libunity.so, others are not, so you can't use libunity symbols on them.

    Sadly, It's not clear what exactly is wrong here, it seems it crashed on application start.
     
  9. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    Re:libunity. Are you saying here that everything beyond frame #06 is from before Unity is invoked, and so not relevant, or that it could be relevant but we can't see what it is? If it's from before unity is invoked and unity is being killed within the first lines, what could be causing it to crash so early?

    Here's the second most common crash which I assume is connected.

    Code (CSharp):
    1. void std::__ndk1::__tree_balance_after_insert<std::__ndk1::__tree_node_base<void*>*>(std::__ndk1::__tree_node_base<void*>*, std::__ndk1::__tree_node_base<void*>*)
    2. ??:?
    3. std::__ndk1::__tree<std::__ndk1::__value_type<int, sigaction>, std::__ndk1::__map_value_compare<int, std::__ndk1::__value_type<int, sigaction>, std::__ndk1::less<int>, true>, std::__ndk1::allocator<std::__ndk1::__value_type<int, sigaction> > >::__insert_node_at(std::__ndk1::__tree_end_node<std::__ndk1::__tree_node_base<void*>*>*, std::__ndk1::__tree_node_base<void*>*&, std::__ndk1::__tree_node_base<void*>*)
    4. ??:?
    5. std::__ndk1::pair<std::__ndk1::__tree_iterator<std::__ndk1::__value_type<int, sigaction>, std::__ndk1::__tree_node<std::__ndk1::__value_type<int, sigaction>, void*>*, long>, bool> std::__ndk1::__tree<std::__ndk1::__value_type<int, sigaction>, std::__ndk1::__map_value_compare<int, std::__ndk1::__value_type<int, sigaction>, std::__ndk1::less<int>, true>, std::__ndk1::allocator<std::__ndk1::__value_type<int, sigaction> > >::__emplace_unique_key_args<int, std::__ndk1::piecewise_construct_t const&, std::__ndk1::tuple<int const&>, std::__ndk1::tuple<> >(int const&, std::__ndk1::piecewise_construct_t const&, std::__ndk1::tuple<int const&>&&, std::__ndk1::tuple<>&&)
    6. ??:?
    7. NativeRuntimeException::chain_handler(int, sigaction&)
    8. ??:?
    9. NativeRuntimeException::install_signal_handlers()
    10. ??:?
    11. NativeRuntimeException::GetExceptionState()
    12. ??:?
    13. nativeSetLaunchURL(_JNIEnv*, _jobject*, _jstring*)
    14.  
     
  10. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    Everything is relevant, the problem it's not enough information... It's sad that google doesn't provide logcat together with crash stacktrace.

    Do you have information about which devices produce stacktrace containing

    NativeRuntimeException::install_signal_handlers()
     
  11. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    They're all Android 10 devices, main culprits are Pixel 4 XL, Pixel 3, Pixel 2, Galaxy S9.
     
  12. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    I did some digging, and there is one place which might be suspicious and might cause a crash, but it would be a blind fix since there's no repro.
     
  13. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    We're happy to try anything at this point!
     
  14. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    Is this fix something we can try out? Or will it be coming out in a new version of Unity?
     
  15. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    The fix has to first land into our master branch, only then we can backport it to previous Unity versions, assuming it's not causing any regressions. Sadly it might take few weeks to happen, and another few weeks for the new Unity release
     
  16. Mythree__

    Mythree__

    Joined:
    Aug 1, 2016
    Posts:
    2
    Anything you can share regarding the cause of the issue or a way to find a workaround?
    Our crash rate also skyrocketted on our live version and mostly points at this stack.
     
  17. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    Without the repro case, it's hard to say. But please provide Unity version and stacktraces where this happens, this will help identify if this is a regression.

    The specific code above haven't been touched for several years, so I doubt the issue is there, there's probably an external factor. Like Android API level upgrade.

    If it's really Android API problem, I guess you could always lower the target API level
     
  18. ARPAplus

    ARPAplus

    Joined:
    Jun 5, 2015
    Posts:
    32
    Yes, we are also have unacceptable tons of crashes in several games built with Unity 2019.3. We are always stick with the latest API level 29, so I saw hope in your post. Yesterday our team have uploaded game built with the latest 2019.3 and API level 28 and I confirm that it does not help, same crashes. It does not make a sense.

    Unity Technologies team, please do something about this, because the latest releases of Unity is not good for Android Developer, 2018.4, 2019.2, 2019.3 - have different problems, but they share unacceptable level of crashes.

    FYI, few days ago Google just mailed a letter to all developers about incoming changes, so they won't accept API 28 soon, only 29 will be acceptable since August for a new apps, and since November for an updates.
     
    Last edited: Jun 4, 2020
  19. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    Can you share more information, like stacktraces?
     
  20. ARPAplus

    ARPAplus

    Joined:
    Jun 5, 2015
    Posts:
    32
    We should raise verbosity level and update the game we are testing on, will let you know later.
     
  21. scareface124

    scareface124

    Joined:
    Aug 19, 2017
    Posts:
    5
    Hi this error happened for me. I was using Co-routines to load images and assign to a image component in gameobject but when this request came, the gameobject was destroyed. So I suggest you to try to disable all coroutines and try if it crashes or add null checks.
     
  22. ARPAplus

    ARPAplus

    Joined:
    Jun 5, 2015
    Posts:
    32
    Last edited: Jun 5, 2020
  23. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    Yep, to confirm we also get crashes when targeting API level 28 and 29, we've tried both over the last few weeks.

    Our game is still in beta, so we may try disabling some of our plugins temporarily to see if that does anything, as I reckon the issue would be far more heavily reported if it was happening on every Android game built with 2019.3. So I figure it's some combination of plugin and 2019.3.
     
  24. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    But you didn't resolve stacktraces, how can you know if the crash stack trace is the same as in this thread?
     
  25. ARPAplus

    ARPAplus

    Joined:
    Jun 5, 2015
    Posts:
    32
    How can I get stacktraces for you? Anything is switched to "FULL" in Player Settings->Stacktraces.

    Tomas, I have a good idea for you. Upload your own blank project to Google Play built with latest Unity and latest versions of Unity Ads and AdMob (better with mediations). And you will see what is up. Also you will see no stacktraces, but a bunch of crashes.

    You always asking for test projects, stacktraces, additional info, etc, making everbody think that problem is on our side. But believe me, the first thing we do is checking if the problem is on our side. Our code is simple and clean, and I'm strictly doubt it can actually cause crashes like that. We have not changed code anyhow in most of the latest updates, but crashes raised significantly.
     
  26. ARPAplus

    ARPAplus

    Joined:
    Jun 5, 2015
    Posts:
    32
    The best choice for now is to use 2018.4, overall project built with this has much less crashes then 2019.2 and 2019.3, but it still have a lot of issues with Unity Ads (a lot of Proxy errors). But all our projects is migrated to 2019.2 long time ago and now it's impossible to go backwards to 2018.4.
     
  27. ARPAplus

    ARPAplus

    Joined:
    Jun 5, 2015
    Posts:
    32
    Last edited: Jun 6, 2020
  28. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    libil2cpp.so stacktraces cannot be resolved without libil2cpp symbol file, libil2cpp is a combination of generic C++ and C++ files generated from your scripts, you should have symbol package after you perform the build, I think it's called symbols.zip. I assume you didn't upload that file to Unity Diagnostics, thus it wasn't able to resolve stacktrace automatically.
     
  29. JustAnotherDude

    JustAnotherDude

    Joined:
    Oct 28, 2013
    Posts:
    279
    I'm also now getting massive amount of crashes after I updated to the most recent unity.

    This is terrible.

    It happens on many devices, android 8, 8.1 and 9.




    The only thing I changed on my game is that I removed other ad providers and left only Admob.

    Error is the same as OP.
     
  30. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    Interesting. One of the things we're going to try with our next update is actually removing AdMob from ours. There's a possibility that the crash rate rose once we added AdMob in (it's hard to tell as we added it shortly after launching beta, so hard to untangle whether it was AdMob or not).

    We've also updated to 3.15f and are making some other changes to a couple of plugins. Will let you know if we see any changes.
     
  31. JustAnotherDude

    JustAnotherDude

    Joined:
    Oct 28, 2013
    Posts:
    279
    I tried with the 3.12 and the errors remained.

    I'll try 3.5 now.
     
  32. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    We've upgraded to version 2019.3.15f1. We also took out AdMob from our mediation just to check it's not that, as from googling a few people with similar-ish looking crashes that were potentially caused by AdMob.

    No luck on either, still getting the crashes on the newest version. Is there anything else that we can try? It's pretty much killing our game at this point. I'm considering downgrading the game to a previous version of Unity (I know this isn't supported, but I don't really know what else we can do).
     
  33. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    Just in case, is it still pointing to NativeRuntimeException::install_signal_handlers() ? You could submit a bug with apk attached, maybe our QA will be able to reproduce it, but please specify which device/Android OS has this issue, the more info you'll provide the better
     
  34. Zeeppo-Games

    Zeeppo-Games

    Joined:
    May 13, 2015
    Posts:
    103
    theres nothing to do with admob. Its unity 2019. I have tried everything possible with no luck. About sending an apk, just upload any apk built with unity 2019 and you will see tons of crashes.


     
  35. Zeeppo-Games

    Zeeppo-Games

    Joined:
    May 13, 2015
    Posts:
    103
    Correction: its unity with app bundles. The crashes do not happen with apks
     
  36. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    In 2019.4.3f1 there will be a fix
    Fix native exception handler code to be thread safe, potential fix for crashes which contain stacktrace with install_signal_handlers entry.
     
    Munchy2007 likes this.
  37. Munchy2007

    Munchy2007

    Joined:
    Jun 16, 2013
    Posts:
    1,735
    @Tomas1856 I've noticed that as @Zeeppo-Games reported, the crashes are far more frequent when building with app bundles (by a factor at least 5 in my case), so for the time being I'm back to building an apk.

    Is it likely that the above bug would have been an underlying cause of this also?
     
  38. GrumpyRhino

    GrumpyRhino

    Joined:
    Jun 5, 2013
    Posts:
    22
    Oh great, good to know. I actually ended up downgrading to 2019.2.17f1 (the same as another project) and this has largely fixed the issue for us. We're still getting an occasional similar-looking crash but it's at a significantly lower volume (like 30 times lower).

    Eventually we do want to move to LTS, but will hold off upgrading for a while, now that the crashes are stable.
     
  39. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    Can't say, I don't think I've seen reports regarding those.
     
  40. Munchy2007

    Munchy2007

    Joined:
    Jun 16, 2013
    Posts:
    1,735
    Okay, I guess I'll wait for the fix to be released then compare results with apk vs app bundle builds again.
     
  41. Zeeppo-Games

    Zeeppo-Games

    Joined:
    May 13, 2015
    Posts:
    103
    Is it possible to send an APK on google play console? I thought it only accept App Bundles.
    Thanx
     
  42. Zeeppo-Games

    Zeeppo-Games

    Joined:
    May 13, 2015
    Posts:
    103
    It's a fact, Bundles are A LOT worse than APK. There's several references to it in this very forum, here's an example: https://forum.unity.com/threads/nat...ativerestartactivityindicator-v.546271/page-3.
    But now, people from Unity don't remeber it anymore or didn't read it. As I said before, they don't have a clue on what's is happening. Android builds from Unity are totally broken. Since 2018.
    Maybe you should report a bug. Another one. To be forgetten like the other 50 bug reports without a fix about this issue.
    Or send them a project. Because is "very hard" to reproduce it. Well, we have been very unlucky here then. Every app bundle we send from any game we have counts thousands of crashes as soon as they get released on the store. And, again, ONLY when downloaded from Play Store (App Bundles). You can't reproduce them locally.
     
  43. Zeeppo-Games

    Zeeppo-Games

    Joined:
    May 13, 2015
    Posts:
    103
    Sorry if I'm being rude. This is making our business not viable. Our downloads have dropped dramatically because of so many crashes. The games rate dropped. And I spend more time trying to solve this, in vain, than actually working on our games.
     
  44. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    3,920
    You can try submitting a bug report with the link to your game, also specifying on which devices it crashes, if our QA have such device and will be able to reproduce, we might be able to pinpoint the issue, if there's really an issue.

    In app bundle cases, there seems to be a lot of factors in play, for ex., it's possible that it wasn't even downloaded from Google Play, so when looking at the crash log in Google Play, check if it's installed from Google Play or from Anywhere.
     
  45. Munchy2007

    Munchy2007

    Joined:
    Jun 16, 2013
    Posts:
    1,735
    There really is an issue! After 5 days with around 11k installs I have had 5 crashes in total for the apk build, and the exact same version of the game, built as an app bundle was getting in the region of 10-15 crashes a day, after the same period of time and similar amount of installs. In both cases the crashes are all signal 11 (SIGSEGV), code 1 (SEGV_MAPERR) libunity.so.

    In my case, I'm only referring to crashes for downloads through Google Play, and the crashes happen on many different devices, but not on any of the devices I have available for testing. (Although some of the reported crashes seem to be on the same devices as those that I have.)

    I'll submit a bug report sometime over the weekend listing as many of the affected devices as I can and I'll post the bug report number here once I've got it.
     
    Last edited: Jun 20, 2020
  46. Munchy2007

    Munchy2007

    Joined:
    Jun 16, 2013
    Posts:
    1,735
    Yes it seems so, it allowed me to revert to uploading an apk instead of an app bundle, it just gave a warning about size optimisation.
     
  47. Munchy2007

    Munchy2007

    Joined:
    Jun 16, 2013
    Posts:
    1,735
    2019.4.3f1 seems to have fixed the SIGSEGV crash for me :)
     
    Last edited: Jul 16, 2020
    markopetkovic likes this.
  48. modernator24

    modernator24

    Joined:
    Apr 7, 2017
    Posts:
    205
    Me too, I updated 2019.4.3f1 and now no more crash.
     
    markopetkovic likes this.
  49. PopsawayGames

    PopsawayGames

    Joined:
    Jan 2, 2018
    Posts:
    25
    Unfortunately not the same for me! I upgraded project from Unity 2018.3.14f1 to 2019.4.3f1 and we're getting crashes ( crashes happened also in the previous version )...I'm building an APK, not App Bundle. We're near to pre-registration and beta version, I don't know what to do ( we're a small company and I'm the only client side developer ). I will provide a crash log and symbols.zip soon
     
  50. PopsawayGames

    PopsawayGames

    Joined:
    Jan 2, 2018
    Posts:
    25
    Symbols.zip is too large to be uploaded, this is tombstone file generated on my test device, it's a Huawei p8 lite 2017...it's also happening on other devices like Honor 9, Xiaomi mi a2 lite and others...
     

    Attached Files: