Mesh Crash - What is this code doing?

We are seeing a very high number of crashes coming from iOS devices. These appear to be happening in Unity code while doing some mobile mesh optimizations.

I have found a couple of other issues I have included below that appear similar to ours, in which Unity has responded with: "Won't fix: issue was fixed by rebuilding assets for a later version of Unity". The memory workaround is not an option for us. The later version (assuming they mean Unity version) also is not something we can currently do.

For starters, am I correct in assuming that this is a skinned mesh optimization based off the call stack below? Secondly, can anyone clarify exactly what those methods are doing? Is this something that every mesh goes through, only skinned, only while playing animations, etc? This is causing tens of thousands of crashes at this point, and any help would be greatlety appreciated.

Spoiler: Crash Thread

Crashed: com.apple.main-thread
0 xxxxxxxxxxxxxxxx 0x101872a8c s_SkinVertices4Bones_Tangents_NEON_loop (MeshSkinningNeon64_Loop.inc:355)
1 xxxxxxxxxxxxxxxx 0x10144d5a8 SkinMeshOptimizedMobile(SkinMeshInfo&) (MeshSkinningMobile.h:149)
2 xxxxxxxxxxxxxxxx 0x10144c9b4 DeformSkinnedMesh(SkinMeshInfo&) (MeshSkinning.cpp:73)
3 xxxxxxxxxxxxxxxx 0x10144ca24 DeformSkinnedMeshJob(SkinMeshInfo*) (MeshSkinning.cpp:161)
4 xxxxxxxxxxxxxxxx 0x1013ee1cc JobQueue::Exec(JobInfo*, long long, int) (JobQueue.cpp:356)
5 xxxxxxxxxxxxxxxx 0x1013ee13c JobQueue::Steal(JobGroup*, JobInfo*, long long, int, bool) (JobQueue.cpp:563)
6 xxxxxxxxxxxxxxxx 0x1013eecdc JobQueue::WaitForJobGroup(JobGroupID, bool) (JobQueue.cpp:1367)
7 xxxxxxxxxxxxxxxx 0x1013ed27c CompleteFenceInternal(JobFence&) (Jobs.cpp:27)
8 xxxxxxxxxxxxxxxx 0x101466304 GeometryJobTasks:: PutGeometryJobFence(GfxDevice&, unsigned int) (GeometryJob.cpp:37)
9 xxxxxxxxxxxxxxxx 0x10142f158 SkinnedMeshRenderer::EndCurrentSkinningInstance() (SkinnedMeshRenderer.cpp:1585)
10 xxxxxxxxxxxxxxxx 0x10142f8d0 SkinnedMeshRenderer::Render(int, ChannelAssigns const&) (SkinnedMeshRenderer.cpp:1049)
11 xxxxxxxxxxxxxxxx 0x1013ae498 ForwardShaderRenderLoop:: PerformRendering(JobFence&, ActiveLight const*, ShadowCullData const&, bool, bool, bool, int) (ForwardShaderRenderLoop.cpp:789)
12 xxxxxxxxxxxxxxxx 0x1013af2d8 DoForwardShaderRenderLoop(RenderLoopContext&, dynamic_array<RenderObjectData, 8ul>&, bool, bool, ActiveLights&, bool, int) (ForwardShaderRenderLoop.cpp:165)
13 xxxxxxxxxxxxxxxx 0x1013b5b50 DoRenderLoop(RenderLoop&, RenderingPath, CullResults&, ShadowMapCache&, bool) (RenderLoopPrivate.cpp:436)
14 xxxxxxxxxxxxxxxx 0x10135ad60 Camera:: DoRender(CullResults&, void (*)(Camera&, RenderLoop&, CullResults&), int) (Camera.cpp:1254)
15 xxxxxxxxxxxxxxxx 0x10135c488 Camera::Render(CullResults&, ShaderPassContext&, CameraRenderingParams const*, int) (Camera.cpp:1986)
16 xxxxxxxxxxxxxxxx 0x10138d818 RenderManager::RenderCameras(int) (RenderManager.cpp:129)
17 xxxxxxxxxxxxxxxx 0x1014cbbdc PlayerRender(bool) (Player.cpp:1344)
18 xxxxxxxxxxxxxxxx 0x1014cc594 PlayerLoop(bool, bool, IHookEvent*) (Player.cpp:1855)
19 xxxxxxxxxxxxxxxx 0x1017f9570 UnityPlayerLoopImpl(bool) (LibEntryPoint.mm:240)
20 xxxxxxxxxxxxxxxx 0x10004b84c UnityRepaint (UnityAppController+Rendering.mm:234)
21 xxxxxxxxxxxxxxxx 0x10004b68c -[UnityAppController(Rendering) repaintDisplayLink] (UnityAppController+Rendering.mm:49)
22 QuartzCore 0x18344f814 CA:: Display:: DisplayLinkItem::dispatch() + 40
23 QuartzCore 0x18344f6c8 CA:: Display:: DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 424
24 IOKit 0x180f7a1e8 IODispatchCalloutFromCFMessage + 372
25 CoreFoundation 0x180ca71f8 __CFMachPortPerform + 180
26 CoreFoundation 0x180cbd634 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 56
27 CoreFoundation 0x180cbcd6c __CFRunLoopDoSource1 + 436
28 CoreFoundation 0x180cbaac4 __CFRunLoopRun + 1800
29 CoreFoundation 0x180be9680 CFRunLoopRunSpecific + 384
30 GraphicsServices 0x1820f8088 GSEventRunModal + 180
31 UIKit 0x185a60d90 UIApplicationMain + 204
32 xxxxxxxxxxxxxxxx 0x10004556c main (main.mm:32)

https://issuetracker.unity3d.com/is...4bones-tangents-neon-loop-when-reloading-mesh
https://community.unity.com/t5/iOS-...Skinning-Optimizations-on-iOS-5-0/td-p/878510

 

Bump...any @Unity Developer help here?

 

We have rebuilt the bundles, which were originally built with the same version anyway, and still seeing the crash. We are not able to reproduce the issue on our end, but we know it is an asset issue because when we point users to a different version of those same assets we no longer get the crash.

As for my other questions in the post (only skinned meshes, animation issue based off the DeformSkinnedMesh(), the SkinVertices4Bones)? Is there any additional information to be had there? Am I right in assuming this is a pipeline for only skinned meshes?

At this point we are doing what we can to single out and find the bad asset, but with thousands of them it is quite the task. Any additional information would be appreciated.

 

We have more info on this, but still not a real fix. During processing of our assets, different versions (low, mid, high) we change the animation errors (rotation, position, scale) from 5 on low to 2 on mid. When we set mid to match the 5 from low, the crash no longer appears.

So it would seem that this is crashing based off some bundles have more precise animation data than others, but since we did not see this until 5.3 was there a change on the Unity side to anything related? The fix for now results in a lot of animation sliding, which is not something we obviously want to deal with long term.

It would be great to have more information regarding any of those points if possible.

 

I wish I could, but as I said earlier we are unable to reproduce the issue. I was hoping to get some more information as to which actual assets would be going through that pipeline, such as only skinned meshes, only those with animations, etc., so that I could further limit it down on my end and hopefully find the culprit asset. At that point I would hope to be able to determine if it is an issue with one of our assets, or something worth submitting a bug report and project over.

If that pipeline is something you are just not familiar with and cannot obtain from someone else, I will be at Unite LA next week and can try to get some more clarification there.

Thank you for your time.

 

Thank you Karl, that is exactly the type of information I was looking for. I will update this post and follow up through the bug report system if we determine it to be an issue outside of our own assets.