IL2CPP missing line numbers in stack trace

We are using UCB but stacktraces from IL2CPP release builds on Windows/Mac/Linux without script debugging, always shows lines 00 in stacktraces from Cloud Diagnostics.

Any information to try and fix this would be greatly appreciated,

Thanks!

 

Ah forget that, we are on latest TECH and this is only available in Unity 2022 - my bad!

Cheers,

 

Is it confirmed it's working in Unity 2022? Working on 2020.3 here but we want those line numbers..

 

Is my understanding correct here that it's impossible to get line numbers in stack traces with Unity? We build for iOS and Android and there are errors that happen on test devices that would be nice to be able to track down. Having all line numbers be 00000 is somewhat annoying. We're using the 2021.3.0f1 LTS version. Seems this has been an issue for close to a decade now? I assume it's not exactly a priority when there are more interesting shiny render pipelines to work on instead of things that developers actually want?

 

You understood well, feature coming in unity 2022 (if lucky)

 

Unity likes to prioritize features which increase the number of people starting projects with it, not finishing and maintaining them.

Anyway, the problem with stack traces is that with IL2CPP the code that runs (and crashes) on device is not the code you wrote: your C# is compiled into IL bytecode, which is then converted into a very convoluted C++ code, which is then compiled into machine code by Xcode/Clang/MSVC.

The native symbols produced by the C++ compiler can contain line information, but those lines point to the C++ files generated by IL2CPP. You need some sort of map that associates the lines in the IL2CPP files to the IL bytecode they were generated from. From there, it's possible for find which C# line that instruction maps to using the PDB file produced by Unity's C# compiler.

There are two problems: first, that IL2CPP-to-DOTNETIL mapping seems to not be generated for release builds. Second, Android/iOS and all other platforms don't care about that: they can only see the native C++ symbols to produce call stacks.

To produce call stacks with C# line information, Unity would need to manipulate the symbol generation process to change the file and line number mapping of each native instruction. This is certainly possible, but they need to care enough to put people to work on this, even if it's too complicated and doesn't produce pretty pictures to woo investors.

 

Our build is (I guess) a development build, not a release build. We set

BuildOptions.Development

for the

BuildPipeline.BuildPlayer()

call. But this limitation also holds for development builds, I guess? We just migrated our entire project to the brand new 2021 LTS version, but I assume that this version will never ever have properly working backtraces?

 

Welcome to the Unity debugging experience (until 2022 LTS, at least). Here are some survival tips;

- Breaking large functions in smaller ones helps narrowing down the actual crash site.
- 99% of the crashes which are not memory-related are null reference exceptions and out-of-bounds array accesses, so check the offending function for where those could happen. If you have such exceptions in the editor, fix them! A game should not have any of such exceptions happening as part of its normal operation (and no, adding try/catch to them is not a fix).

 

Really, in my opinion even in development builds and in the editor Unity should stop everything immediately on null/missing reference and out-of-range exceptions and display some large error information or jump straight to the line in Visual Studio.

These will crash optimized IL2CPP builds, and even if they didn't, exceptions like those put the game in undefined state which manifest as weird seemingly unrelated issues later or, which range from a button no longer working to an infinite loading screen or some other random bizarre occurrence.

 

Pretty frustrating how you can't even get line numbers in 2021. Feels like such a standard feature, and it would be only for slower development builds. But I guess we're all out of luck for a year, but then 2022 will most likely break something else.

 

Haha, I implemented this for a company I worked for a few years ago. It definitely caused people to look at those problems.

 

Looking forward to it! Will this work for all platforms?

 

Just to clarify something with this 2022.x update. Is this ONLY for cloud diagnostics? Or would it also provide correct line numbers when using
Application.logMessageReceivedThreaded for example?

 

You can have line numbers on your C# Exceptions with IL2CPP builds now when using Sentry. Works on iOS, Android, Windows, macOS and Linux players.

Works on 2020.3 and newer.

https://docs.sentry.io/platforms/unity/configuration/il2cpp/

 

Thanks for sharing
We are using Crashlytics but this is interesting as we can have PC + mobile reporting from the same library.

 

We have been using Sentry for a few months and it's the best you can find out there. Totally recommended.

 

@JoshPeterson I understand Unity creates certain optimizations on the c++ generated code. If those are enabled is still possible to retrieve the c# line numbers on 2022.2 ?

 

I was finally able to upgrade to Unity 2020.3. Only to find out that I still do not see any line numbers in Xcode when an exception occurs. I enabled Script Debugging. What elso do I have to do to see line numbers in Xcode?

 

Ok. So you first told me that I can get line numbers in 2020, but sadly not in 2019. And now it has been postponed to 2022? Seriously?

 

Wait, with script debugging enabled should work on 2019,2020 and 2021 too, am I wrong @JoshPeterson ?

 

2023.1.0a24 ? Is this supposed to be good news? How much until 2023.1 becomes LTS and we can migrate "safely" our projects in production to this one ?

Still, thanks for the effort

 

Hey Josh, great news! Thanks a lot!!
Is it possible to backport the solution to LTS versions?
This is super important and will improve the quality of our games tremendously...

 

Would it be possible to use it on per file or class basis in the future? I guess problem with delivering builds with full stack traces would be easier decompilation by hackers/thieves/modders. Here's how I see it working:
1. Game uses previous stack traces without line numbers
2. Devs notice bug being reported in online analytics
3. After being unable to identify root cause they decide to mark file/class with a header to use full stacktraces
4. Upload new build
5. Get better stacktrace for only this exact bug

If that's not possible, is there any other method? Maybe instead of per class/file it could be per asmdef? Rest of pdbs wouldn't generate.

 

Hello.

I have installed and launched Unity 2023.1.0a24.
However, I could not find the menu "IL2CPP Stacktrace Information" in Player Settings.



Am I doing something wrong in the procedure?

 

I understand! I am looking forward to it.

 

I've upgraded from 2021.3 LTS to 2022.3 LTS expecting to see line numbers in managed stacktraces for release builds via the Unity Cloud Diagnostics platform, but that's not the case. Is there some setting somewhere I need to enable?

 

Thanks!

 

Any update here @JoshPeterson? Tempted to open a premium support ticket for this

 

I just found this thread and I'm a bit confused now. So starting in 2022.1, regular, release builds of the game (for Standalone, iOS and Android), should show line numbers in Cloud Diagnostics without any extra steps, correct? The build sent for users, without symbol files, will still work as intended?

Then, the change in 2023.1 alpha is that you will also get line numbers, if you so choose, in the local logs, like Player.log.

Am I understanding it correctly?

 

Correct! But as fair as I am aware from my own testing, and other comments in this thread, Unity 2022.x builds don't actually produce line numbers in Unity Cloud Diagnostics

 

Did anyone get a chance to try this with 2023.1?
Wanted to know if it works with Firebase for Android and iOS.

Thanks.

 

I dug into this a bit more and noticed the following warning when building the iOS XCode project:

The project calls process_symbols.sh, which uploads build symbols to Unity Cloud Diagnostics via usymtool:

Code (Boo):

1. #!/bin/sh
2.  
3. #  process_symbols
4. #
5. #  Copyright (c) 2015 Unity Technologies. All rights reserved.
6.  
7. if [ "$(arch)" == "arm64" ]; then
8.   usymtool="usymtoolarm64"
9. else
10.   usymtool="usymtool"
11. fi
12.  
13. if [ "${SYNCHRONOUS_SYMBOL_PROCESSING}" = "TRUE" ]; then
14.     "$PROJECT_DIR/$usymtool" -symbolPath "$DWARF_DSYM_FOLDER_PATH/$DWARF_DSYM_FILE_NAME" -il2cppOutputPath "$PROJECT_DIR/Il2CppOutputProject/Source/il2cppOutput/"
15. else
16.     nohup "$PROJECT_DIR/$usymtool" -symbolPath "$DWARF_DSYM_FOLDER_PATH/$DWARF_DSYM_FILE_NAME" -il2cppOutputPath "$PROJECT_DIR/Il2CppOutputProject/Source/il2cppOutput/" > /dev/null 2>&1 &
17.     disown
18. fi

Notice no il2cppFileRoot argument? I made a post-process script to modify this shell script to add that:

Code (Boo):

1. #!/bin/sh
2.  
3. #  process_symbols
4. #
5. #  Copyright (c) 2015 Unity Technologies. All rights reserved.
6.  
7. if [ "$(arch)" == "arm64" ]; then
8.   usymtool="usymtoolarm64"
9. else
10.   usymtool="usymtool"
11. fi
12.  
13. if [ "${SYNCHRONOUS_SYMBOL_PROCESSING}" = "TRUE" ]; then
14.     "$PROJECT_DIR/$usymtool" -symbolPath "$DWARF_DSYM_FOLDER_PATH/$DWARF_DSYM_FILE_NAME" -il2cppFileRoot "$PROJECT_DIR/Il2CppOutputProject/Source/il2cppOutput/" -il2cppOutputPath "$PROJECT_DIR/Il2CppOutputProject/Source/il2cppOutput/"
15. else
16.     nohup "$PROJECT_DIR/$usymtool" -symbolPath "$DWARF_DSYM_FOLDER_PATH/$DWARF_DSYM_FILE_NAME" -il2cppFileRoot "$PROJECT_DIR/Il2CppOutputProject/Source/il2cppOutput/" -il2cppOutputPath "$PROJECT_DIR/Il2CppOutputProject/Source/il2cppOutput/" > /dev/null 2>&1 &
17.     disown
18. fi
19.  

And the warning disappeared. Made a build and forced a crash and hey presto, line numbers!

 

With this setting and "Copy PDB files" enabled, the line numbers showing don't match the line numbers in our source code. Presumably it's showing the CPP line numbers, is there an extra step required for the line numbers to match the C#?