Debugging Native Crashes: Missing Symbols

Hello everyone,

I have been using Performance Reporting for some time now.
For finding basic exceptions it has been great but I keep seeing seemingly random native crashes which as usual, I can never reproduce locally. My app is built targeting IL2CPP for both Android and iOS.

The reports are always "missing symbols" so I don't have any insight into the issue. Such as:

Spoiler: Android 5.0-> 5.1

Spoiler: Android 7.1.1

Spoiler: iOS 11.2.2

All my builds are created through UCB so I assume any symbols should be automatically transferred.
The iOS one is weird because I do not use any lightmaps and thought I disable all settings related to lightning.

Is there something I need to do so I can get a clearer picture from the report? This is just a sample, there are many "Unknown File" crashes.
Any insight into what is causing these issues? Especially the Android Lollipop crash.

Normally, I can figure this stuff out and I am willing to debug anything I can reproduce but I have hit a dead end with these. I'll take any tips/suggestions/information anyone may have.

Thank you!

 

The 3rd item for Android 5.0 is actually Android 4.3 (dalvik-heap error). Can't edit the post for some reason.

 

Same thing here. Performance reporting is simply broken and needs to be fixed. Symbols are not being uploaded/processed by Android builds. At least with IOS you can download the dSYM file. But even then, even after uploading symbols to process, nothing seems to change.

I have so many native crash reports with missing symbols. This is just a screenshot from the last day. If I leave this for a week. I usually end up with about 6 pages of native crashes, all with missing symbols.

Unity, are you guys aware of this issue? It's always been like this.

 

Thanks for sharing! At least I am not alone.

 

Yeah, and as you said, I can never replicate any crashes on my test devices.

 

I've tried using mono and il2cpp, cloud build and local build, manually uploading symbols.. nothing works.

 

Thanks for contributing to the chat, now let's hope we hear from Unity. =)

 

I've done separate x86 and ARMv7 builds also, no luck.

 

I also have no symbols in any of my apps for native, crashes, just a pile of "missing symbols" crashes like below. The project in question is a Unity-only app (that is, it includes no external SDKs or other thirdparty native code, just C# Unity code.) The project is build by Unity Cloud Build with Mono for Android and IL2CPP for iOS, and uploaded to the app stores using those builds. I'm really not sure how to simplify this case further.


Native Crash - <symbols missing for uuid: a3d8672ba14ad199de7c22043ad3cb885ba0f06a> (Unknown File)

Thread 0 (crashed)
0 libunity.so <symbols missing for uuid: a3d8672ba14ad199de7c22043ad3cb885ba0f06a>
[and so on]

 

Same thing here

 

@thomasn_unity @andykoval_unity @johng_unity @bretth_unity or any other Unity team members have any info for us? Does this summon you?

 

I've tried with Android Unity 2017.3.0f3 + 2017.3.0p3

- Cloud build.
- Manual build and symbol upload.
- Mono builds.
- il2cpp builds.
- Separate x86 and ARMv7 builds.
- Internal build system vs gradle.

Nothing works.

 

We are running into the same problem with 2017.2.1p2, from ~/Library/Logs/Unity/symbol_upload.log we can see that USYM_UPLOAD_AUTH_TOKEN is never set. I tried manually setting this to my API key in the Unity console however it then logs 401.

time="2018-01-30T15:58:24+02:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
time="2018-01-31T10:25:35+02:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
time="2018-01-31T10:40:32+02:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
time="2018-01-31T12:05:49+02:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
time="2018-01-31T12:31:53+02:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
time="2018-01-31T13:15:16+02:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
time="2018-01-31T14:03:40+02:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
time="2018-01-31T15:47:07+02:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"

time="2018-01-31T16:50:49+02:00" level=error msg="Signed URL request failed with status: 401"
time="2018-01-31T16:50:49+02:00" level=error msg="Signed URL request failed with status: 401"

 

I believe that the USYM_UPLOAD_AUTH_TOKEN issue only happens when you've signed out or lost your connection to Unity Services. Just going to the services window and letting it refresh its connection tends to fix it. I haven't seen that error since Jan 24th, when I do have vague recollections of it happening.

We should probably report a bug rather than posting on the forums. Would somebody who's made more investigation than me (frankly, I only CARE about it working on cloud builds, as that's where our retail builds are currently made) mind submitting a bug report and pinging back to this thread.

In fairness to the various Unity developers, it does seem reasonable that they have a process around bugs (which it would appear this is) rather than just ad-hoc replying to forum posts.

Also, we signed up for Fabric yesterday and switched to crashlytics so I'm not in a position to investigate deeper right now. I would obviously rather use Unity's built-in system, as we all know that the golden rule of Unity and Android is "one external tool works. Two (say, Fabric+Firebase) and you might just be able to talk them into working in one build config. Three is right out."

 

I'm not so sure that we have been logged out of UnityServices, I suspect it's something else related to our setup but I will check with the team tomorrow. Our build runs via Jenkins which executes a script that triggers the build like so..

$UNITY \
-logfile \
-batchmode \
-nographics \
-projectPath $PWD/$UNITY_PROJ \
-executeMethod AutomatedBuild.Make_$UNITY_BUILD_PROFILE \
-buildNumber "$BUILD_NUMBER"

AutomatedBuild.cs then configures the environment and invokes BuildPipeline.BuildPlayer.

 

I haven't really done any investigation except for ruling out different build configs like @SevenPointRed did. My expectation as a customer of this service is that if I build my app using UCB and integrate Analytics and Performance Reporting into my app then it should all work.

I can't tell if this is a bug, lack of feature, or by design. I am hoping someone from Unity can answer that for me, hence the forum post.

There seems to be some instability in Unity in general that leads to random crashes and no way of tracking them down and this seems to be prevalent on Android OS 5.0 for some reason. I have more users on 4.1-4.4 and 6->7 but the majority of my native crashes are on 5.0.

I don't have access to the devices so I am relying on the performance reports. For all I know there is a huge user base on these devices with custom OSes or rooted devices that cause the problem.

I'm honestly just getting frustrated at this point. This may be unreasonable but Unity is marketed towards Indies and I work alone. I pay money for a license and product and have certain expectations I am tired of tracking issues in the engine and services. It is not my job. They have vast resources compared to me they need to step up.

 

Where do you even submit bug reports relating to cloud build?

I agree with Zebec, Unity should be routinely bug testing these services, not waiting on people to complain about it.
It's not hard to imagine or expect a Unity developer to visit this dark cobweb infested corner of the Unity forums at least once every few days. You know, just to check up on things. And maybe @ZebecGames is right, this could be by design, but who knows? Without any confirmation from Unity, we're all just sitting here twiddling our thumbs.

 

It might not be the best place to discuss this but I am a solo dev yet everyday I manage to:

1. Check reviews on Google Play & iTunes
2. Check Reddit
3. Check Twitter
4. Check Facebook
5. Check Emails (multiple)
6. Check Forums

If any of my customers have written to me I respond right away. Often times they bash me for the game crashing on them. I ask for them to help me figure it out with not response (since I cannot reproduce them on my test devices). My app receives a bad rating and review for which I have no recourse.

Unity is insulated from this. I lose money, ranks, and reputation while I pay them monthly for the service and they collect ad revenue. I am completely willing to work with Unity on resolving these issues in any way that I can.

 

Thank you for the response! I am taking a look at the attached document. Much appreciated.

 

So I read the document and that definitely provides some context to the issue.
The information for iOS is very interesting and I use BitCode so that definitely solves part of my problem.

Taking a closer look at my UCB log I see an issue similar to @Johnou

Spoiler: Cloud Error

[Unity] DisplayProgressbar: Zipping symbols
time="2018-01-30T20:53:11-06:00" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
[Unity] Android PostProcess task "Zipping symbols" took 3208.099 ms
[Unity] DisplayProgressbar: Performance Reporting Symbol Upload
[Unity] Android PostProcess task "Performance Reporting Symbol Upload" took 29.405 ms

 

@johng_unity we are seeing the same problem with IOS builds, if you require further information please don't hesitate to ask.

 

Thank you so much for the feedback so far.

 

I've had a look at the pdf and it seems mostly iOS related, so just to add our issues are with Android.

 

Yeah, our issues seem to be Android too - the (relatively few) ios crashes we have seem to symbolicate OK.

(We do explicitly disable bitcode - partly because Crashlytics had previously documented issues with bitcode, and also because I really don't see the point in providing Apple with bitcode except in cases like tvOS where it's required. I mean, Apple is concerned with old apps NOT hogging space in the store, and makes continual changes to mitigate this - giving them the means to auto-update your app won't help you with long-tail revenue if it isn't worth your effort to submit new builds.)

https://docs.fabric.io/apple/crashlytics/missing-dsyms.html

The document is a well written explanation of how to use the ios tools, and hopefully helps anyone having ios issues.

 

Also the only plugin im using is google play games unity.

 

Thanks @johng_unity my issues are with mostly with Android though. Is there a way to generate the .so files on cloud?

 

There are multiple bugs/issues they are investigating on the Android side. It sounded like there was a general issue with "certain Android builds missing symbols" and another related to UCB (USYM_UPLOAD_AUTH_TOKEN error) - not sure if this one shows up in your log too.

Unfortunately, that means we do not have a solution in hand but at least we have confirmation it is being addressed. Hopefully we'll see a fix soon and not "soon".

 

Ah right, yeah I'm seeing that error on every Android build log.
Sleep mode activated...

 

Thank you very much.

 

UPDATE!

I've manged to get SOME symbols ( Still a lot of unknown )

I had to also manually upload Unitys symbols in:

C:\Program Files\Unity2017.3.0f3\Editor\Data\PlaybackEngines\AndroidPlayer\Variations\il2cpp\Release\Symbols
C:\Program Files\Unity2017.3.0f3\Editor\Data\PlaybackEngines\AndroidPlayer\Variations\mono\Release\Symbols

I'm waiting on more logs to come through but I'm now getting reports I have never seen:

Spoiler: Native Crash - core::hash_map<int, int, core::hash<int>, std::equal_to<int> >::allocate_nodes(int) (Unknown File)

Code (CSharp):

1.  
2. Thread 0 (crashed)
3. 0   libunity.so                         core::hash_map<int, int, core::hash<int>, std::equal_to<int> >::allocate_nodes(int)
4. 1   libunity.so                         core::hash_map<int, int, core::hash<int>, std::equal_to<int> >::grow(int)
5. 2   libunity.so                         GarbageCollectSharedAssets(bool)
6. 3   libunity.so                         UnloadUnusedAssetsOperation::IntegrateMainThread()
7. 4   libunity.so                         PreloadManager::UpdatePreloadingSingleStep(PreloadManager::UpdatePreloadingFlags, int)
8. 5   libunity.so                         PreloadManager::UpdatePreloading()
9. 6   libunity.so                         PlayerLoop()
10. 7   libunity.so                         UnityPlayerLoop()
11. 8   libunity.so                         nativeRender(_JNIEnv*, _jobject*)
12. 9   base.odex                           <unknown>

Spoiler: Native Crash - BatchAllocator::Commit(MemLabelId) (Unknown File)

Code (CSharp):

1.  
2. Thread 0 (crashed)
3. 0   libunity.so                         BatchAllocator::Commit(MemLabelId)
4. 1   libunity.so                         RenderDynamicBatch(GfxDevice&, RenderNodeQueue const&, BatchInstanceData const*, unsigned int, unsigned int, unsigned int, ShaderChannelMask, ShaderChannelMask, VertexChannelsLayout const&, GfxPrimitiveType)
5. 2   libunity.so                         RenderMultipleMeshes(RenderNodeQueue const&, RenderMultipleData const&, ShaderChannelMask)
6. 3   libunity.so                         BatchRenderer::RenderBatch(BatchInstanceData const*, unsigned int, ShaderChannelMask)
7. 4   libunity.so                         ForwardRenderLoopJob(GfxDeviceAsyncCommand::ArgScratch*, GfxDeviceAsyncCommand::Arg const*)
8. 5   libunity.so                         GfxDevice::ExecuteAsync(int, void (*)(GfxDeviceAsyncCommand::ArgScratch*, GfxDeviceAsyncCommand::Arg const*), GfxDeviceAsyncCommand::ArgScratch**, GfxDeviceAsyncCommand::Arg const*, JobFence const&)
9. 6   libunity.so                         ForwardShaderRenderLoop::StartRenderJobs(JobFence&, bool, bool, ShaderPassContext&)
10. 7   libunity.so                         ForwardShaderRenderLoop::performRendering(JobFence&, ActiveLight const*, ShadowJobData const&, bool, bool, bool, Camera::RenderFlag)
11. 8   libunity.so                         DoForwardShaderRenderLoop(RenderLoopContext const&, dynamic_array<RenderObjectData, 4u>&, CullResults const&, bool, bool, bool, Camera::RenderFlag)
12. 9   libunity.so                         DoRenderLoop(RenderLoop&, RenderingPath, CullResults&, ShadowMapCache&, bool)
13. 10  libunity.so                         Camera::DefaultPerformRenderFunction::eek:perator()(Camera*, RenderingPath, CullResults*)
14. 11  libunity.so                         Camera::DoRender(CullResults&, SharedRendererScene const&, Camera::RenderFlag, Camera::performRenderFunction*)
15. 12  libunity.so                         Camera::CustomRender(CullResults&, ShaderPassContext&, CameraRenderingParams const*, Camera::RenderFlag, Camera::performRenderFunction*, void (*)())
16. 13  libunity.so                         RenderManager::RenderCameras(int, void (*)())
17. 14  libunity.so                         PlayerRender(bool)
18. 15  libunity.so                         PlayerLoop()
19. 16  libunity.so                         UnityPlayerLoop()
20. 17  libunity.so                         nativeRender(_JNIEnv*, _jobject*)
21. 18  base.odex                           <unknown>

The bad news is that looks like unity bugs and I can't find any reports about it so far.

 

Yeah I think you are right. It is possible when the hash_map grew it ran out of memory? Or perhaps some kind of access violation. I am not sure how those would manifest in the logs.

I build production builds using UCB so I don't have symbols to upload locally.

 

If this is an expected manual process (Unity can clarify, but I'm assuming that the bug is that NO symbols are being uploaded, and that when that is fixed these symbols will be uploaded automatically too) then theoretically, the symbols for the AndroidPlayer itself should be the same between UCB and Unity, provided that you're using the same versions of Unity, so in this case you should just be able to upload your local ones, assuming this works.

This is slightly complicated by the fact that there is currently a bug in UCB that prevents specifying explicit patch release of Unity 2017, so my cloud builds are built with 2017.3.0Final, but by local builds are 3.0p3, although there is nothing stopping me doing a sidebyside install of .0f if I wanted to upload the symbols for libUnity from that.

https://fogbugz.unity3d.com/default.asp?995639_bqeqsdanjufvb7qr

 

The symbols should be automatically copied as part of UCB. This seems to be related to the "USYM_UPLOAD_AUTH_TOKEN" bug mentioned earlier.

Wait, so if I select use Latest 2017.3 that does not use the latest patch? Only if I select the patch explicitly which is currently broken? I definitely misunderstood what that setting meant then.

I use 2017.3.0p3 locally too so as a temporary solution I will SxS 2017.3.0 and grab the symbols. This is definitely not ideal but I am glad to have some kind of path forward.

 

I can't be much help with cloud build, our builds never work with cloud build, everything is local. I would try uploading the symbol files for Unity anyway.I seem to be getting mostly decent reports now.

Seems utterly stupid that Unity's own performance reporting doesn't have Unity's own symbols by default.

 

@johng_unity could said issue cause problems with IOS builds too? We are finding that running BuildPipeline.BuildPlayer from CI (see spoiler) is resulting in the same error message "Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable".

Spoiler: CI Unity command

$UNITY \
-logfile \
-batchmode \
-nographics \
-projectPath $PWD/$UNITY_PROJ \
-executeMethod AutomatedBuild.Make_$UNITY_BUILD_PROFILE \
-buildNumber "$BUILD_NUMBER"

 

Thanks @johng_unity!

 

Thanks @johng_unity from me too!

We'll likely sit with Crashlytics for a few weeks until this lands, then re-test Unity Perf reporting.

 

@johng_unity I don't get the USYM_UPLOAD_AUTH_TOKEN error, it says upload successful for me but I still don't get symbols without my earlier hack:

https://forum.unity.com/threads/debugging-native-crashes-missing-symbols.514307/#post-3377696

 

Hi All,

It's been a couple of weeks, and I'd like to get this thread to some kind of conclusion so that I can try again with Unity Performance Reporting.

Goal: Properly symbolicated reporting of native crashes, via a project built on Unity Cloud Build on both iOS and Android. Assume that our cloud build scripts explicitly disable bitcode so we won't have the recompilation issues that affect all crash symbolication on iOS.

From this thread it seems:

- There were some problems with uploading Android symbols from cloud build. This is fixed in some Unity version. The version is unknown. There may be other issues.

- Can anyone at least confirm which version yet (I know release processes are, err, processes) so that we can determine when to try again?

- Have Unity, internally, built and tested an iOS and Android app that crashes natively (via a forced crash in native code) and confirmed that the crash stacks are symbolicated in both cases?

- Do Unity, internally, conduct the same test when building the test app with UCB?

Also, it would be great if Unity could give some kind of statement of the kind of volume of professional developers using Perf Reporting for active crash monitoring, as the state of perf reporting as opposed to, say, Crashlytics, and the relatively low volume of traffic on the message board leads me to believe the answer is "not many, if any." It would be GREAT to have Unity performance reporting be our only crash reporting tool, as it saves all manner of futzing around overriding the Android activity for thirdparty solutions (which in turn becomes a hassle with other things that also want to override the activity...)

Additionally, given the aggressive approach to data collection taken by Unity with, for example, Unity IAP, I'm surprised a similar approach isn't taken to accumulating in-the-wild crash reports.

 

I think Unity might be ignoring this, I gave a workaround to get some symbols and they have not responded to it.
My guess is they are going to redo the performance reporting completely as there are a lot of issues with it and a lot of missing/broken features. If they ignore this long enough they can push a new version out without having to bother with any fixes in the meantime.

 

@johng_unity can you post here when this has been addressed in unity cloud build? Thanks

 

Excellent, thanks for the confirmation. We've moved all of our existing products to Crashlytics now, but our next new ios/Android product (~2wks) will use the Unity crash reporting flow. We'll post in a new thread if it's not producing native symbols via cloud-created builds.

 

Any news on this? still doesnt work in 2018