Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Symbols for native stack traces on Linux builds

Discussion in 'Linux' started by Alloc, Oct 3, 2016.

  1. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Hi,

    is there a way to get the native stack traces on crashes in the Linux builds readable? Ideally written readable to the log already like the Windows version does, alternatively by "postprocessing" the stack trace?

    Currently it typically looks like this:
    Code (CSharp):
    1.  
    2. *** Error in `/home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64': double free or corruption (!prev): 0x000000001715a590 ***
    3. Stacktrace:
    4.  
    5.  
    6. Native stacktrace:
    7.  
    8.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie_Data/Mono/x86_64/libmono.so(+0x91772) [0x7f2de0c97772]
    9.        /lib/x86_64-linux-gnu/libpthread.so.0(+0x10d10) [0x7f2de6390d10]
    10.        /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7f2de4bd71c7]
    11.        /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f2de4bd8e2a]
    12.        /lib/x86_64-linux-gnu/libc.so.6(+0x78ba3) [0x7f2de4c1aba3]
    13.        /lib/x86_64-linux-gnu/libc.so.6(+0x80bb9) [0x7f2de4c22bb9]
    14.        /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c) [0x7f2de4c267ec]
    15.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64() [0x64b835]
    16.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64() [0x4a0f2a]
    17.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64() [0x4a102e]
    18.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64() [0x77f4c0]
    19.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64() [0x575b87]
    20.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64() [0x78e7a4]
    21.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64() [0x464fbe]
    22.        /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f2de4bc2ac0]
    23.        /home/ci/.local/share/Steam/steamapps/common/7 Days To Die/7DaysToDie.x86_64() [0x46e709]
    24.  
    Obviously not really giving us any clue what it might be related to at all :(

    Regards,
    Chris
     
  2. Tak

    Tak

    Joined:
    Mar 8, 2010
    Posts:
    1,001
    We've improved this for 5.5, but the change is large enough that we haven't backported it to 5.3/5.4.
     
  3. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Thanks, that sounds good:)
    5.5 isn't going to drop tomorrow though (at least I suppose so ;) ) so is there anything we can do with the traces from 5.3 builds for now?
     
  4. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Can I consider this a "no" or just missed it? :)
     
  5. Tak

    Tak

    Joined:
    Mar 8, 2010
    Posts:
    1,001
    Sorry, I don't have any advice apart from the standard linux tools: use gdb, enable core dumps, etc.
     
  6. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Ok, thanks anyway :)
    Oh, but what about the actual symbols of the player? Gdb won't help if there's no symbols or am I wrong here?
     
  7. Tak

    Tak

    Joined:
    Mar 8, 2010
    Posts:
    1,001
    The players do have minimal symbols embedded - they're just not getting picked up correctly by the old stacktrace generation code.
     
  8. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Hi,

    while I couldn't find any documentation on either the forum or the docs about debugging on Linux, maybe adding such a section would help.

    Think I figured it out myself though: Enabled core dumps ("ulimit -c unlimited") and then loaded the dump into gdb ("gdb game.x86_64 core_dump"). Using "bt" I get the trace but at least in my tests (killed the server with SIGSEGV) the trace only contained libmono-entries. So I can't say if it would work on code addresses in the actual player executable but it definitely does not print any information for libmono, seems like that one is missing the symbols? Are they provided anywhere to download separately?

    Regards,
    Chris
     
  9. Tak

    Tak

    Joined:
    Mar 8, 2010
    Posts:
    1,001
  10. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Hm, maybe I was doing something wrong here as I'm not used to gdb at all, but tried this with the running gdb instance:
    Code (CSharp):
    1. (gdb) symbol-file ...._Data/Mono/x86_64/libmono.so
    2. Load new symbol table from "...._Data/Mono/x86_64/libmono.so"? (y or n) y
    3. Reading symbols from ...._Data/Mono/x86_64/libmono.so...(no debugging symbols found)...done.
    So I thought that means that libmono does not have any debug symbols?

    Also thought that while I can't get symbols resolved for managed code at least part of the trace should always lead to unmanaged code and as long as that's part of libmono or the player executable resolution should work on those, even on the dump file after the process has terminated? Printing the stack trace in gdb resulted in this output though:
    Code (csharp):
    1.  
    2. #0  0x00007fe05b2d31c7 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
    3.   resultvar = 0
    4.   pid = 29850
    5.   selftid = 29850
    6. #1  0x00007fe05b2d4e2a in __GI_abort () at abort.c:89
    7.   save_stage = 2
    8.   act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {140601582831968, 0, 140601579371343, 0, 23, 140601582827072, 23, 140601572195227,
    9.   140601579373295, 520918584, 140601582827072, 140601572195227, 23, 456050000, 140601594091711, 520918584}}, sa_flags = 1529926616, sa_restorer = 0x7fe05ac3fb38}
    10.   sigs = {__val = {32, 0 <repeats 15 times>}}
    11. #2  0x00007fe05ab1f976 in ?? () from ...._Data/Mono/x86_64/libmono.so
    12. No symbol table info available.
    13. #3  0x00007fe05aac28e4 in ?? () from ...._Data/Mono/x86_64/libmono.so
    14. No symbol table info available.
    15. #4  <signal handler called>
    16. No locals.
    17. #5  0x00007fe05aad5bbf in ?? () from ...._Data/Mono/x86_64/libmono.so
    18. No symbol table info available.
    19. #6  0x00007fe05aac4572 in ?? () from ...._Data/Mono/x86_64/libmono.so
    20. No symbol table info available.
    21. #7  0x00007fe05aac5c75 in ?? () from ...._Data/Mono/x86_64/libmono.so
    22. No symbol table info available.
    23. #8  0x00007fe05aac6329 in ?? () from ...._Data/Mono/x86_64/libmono.so
    24. No symbol table info available.
    25. #9  0x00007fe05ab209c9 in ?? () from ...._Data/Mono/x86_64/libmono.so
    26. No symbol table info available.
    27. #10 0x000000004133f171 in ?? ()
    28. No symbol table info available.
    29. #11 0x00007ffe8b2dd6e8 in ?? ()
    30. No symbol table info available.
    31. #12 0x0000000000000000 in ?? ()
    32. No symbol table info available.
    33.  
    As you can see the only resolved symbols are from system libs due to the signal I sent to the process. Is there really nothing I can do to at least get the symbols for the unmanaged code?

    Thing is a lot of these crashes happen on end user systems, especially on servers (long running, high stress, lots of action, more likely to crash). And I can't really tell them to run the server process within gdb to debug these issues, I'm already happy a lot of the users are as helpful as letting me have a look at the stuff after it crashed ;)