Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

5.2 sporadic SIGSEGV crash in linux headless builds

Discussion in 'Multiplayer' started by any_user, Sep 10, 2015.

  1. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    I'm experiencing a SIGSGV crash since updating to 5.2, which happens not all the time, but around 1 of 3 times when starting the build. I still have to build a smaller repro, but maybe somebody here already has an idea what it could be..?

    OS: Ubuntu 14.04
    Architecture: x64

    Code (CSharp):
    1. Stacktrace:
    2.  
    3.  
    4. Native stacktrace:
    5.  
    6.     /home/krautscape/krautscape/node/bin/game/Krautscape_Data/Mono/x86_64/libmono.so(+0x915e6) [0x7f1e16c3b5e6]
    7.     /home/krautscape/krautscape/node/bin/game/Krautscape_Data/Mono/x86_64/libmono.so(+0x3483b) [0x7f1e16bde83b]
    8.     /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f1e181b7340]
    9.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0xd1a8d0]
    10.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0xd1adef]
    11.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x843a20]
    12.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x843839]
    13.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x843975]
    14.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x843a06]
    15.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x843839]
    16.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x843975]
    17.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x843a06]
    18.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x843839]
    19.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x845207]
    20.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x842be9]
    21.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0xd01dd1]
    22.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x46f26a]
    23.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x7198f2]
    24.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x715bd3]
    25.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x716714]
    26.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x5ca827]
    27.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x7257f4]
    28.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x45e213]
    29.     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f1e173daec5]
    30.     /home/krautscape/krautscape/node/bin/game/Krautscape.x86_64() [0x46684d]
    31.  
    32. Debug info from gdb:
    33.  
    34.  
    35. =================================================================
    36. Got a SIGSEGV while executing native code. This usually indicates
    37. a fatal error in the mono runtime or one of the native libraries
    38. used by your application.
    39. =================================================================
    40.  
     
  2. CipSoft

    CipSoft

    Joined:
    Jan 30, 2014
    Posts:
    5
    Same problem here.
    3 out of 5 times crashed on CentOS7.
    Windows works fine.
     
  3. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    I finally reported the bug with repro, case id: 732667

    Could somebody of Unity have a look at it?
     
  4. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    Any ideas what it could be? Could somebody from Unity look at the bug report?
     
  5. holliebuckets

    holliebuckets

    Moderator

    Joined:
    Oct 23, 2014
    Posts:
    496
    Hi folks! I'm looking into your bug status right now, but I'm also going to move this to the L̶i̶n̶u̶x̶ Multiplayer Networking part of the forums :D

    - EDIT - Its being worked on actively by QA to reproduce and find out what is causing your issues :D That is good news! :D
     
    any_user and ma1onso like this.
  6. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    Great! Let me know if you need any more information about the issue.
     
    holliebuckets likes this.
  7. chrismarch

    chrismarch

    Joined:
    Jul 24, 2013
    Posts:
    470
    What is the most recent version of Unity Editor that creates builds with this issue? 5.2.1p2?
     
  8. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    I had it with p1 and f1 last time I checked.
     
  9. CipSoft

    CipSoft

    Joined:
    Jan 30, 2014
    Posts:
    5
    I tried 3 times today with a fresh Unity 5.2.1p2 build.
    It crashed 3/3 times under Linux.

    Works perfect under Windows.

    @any_user: ["I finally reported the bug with repro, case id: 732667"]

    Would you mind posting the full link to the fogbugz issue? So we can all look up the progress.
    Thanks in advance.
     
  10. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    there's no progress, i never received an update. will post here if i get one.

    @holliebuckets do you know if there is an update about this issue?

    for us this crash means it's impossible to update to 5.2 in two projects which use linux servers..
     
  11. chrismarch

    chrismarch

    Joined:
    Jul 24, 2013
    Posts:
    470
    Are people seeing this with 5.2.1p3?
     
  12. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    I investigated a bit. It still happens in 5.2.1p3.

    I made a new repro that consists of just two scenes. In Start of a script in scene 1, scene 2 is loaded. The first scene also includes a canvas with an image inside. Starting a build with just this inside is enough to trigger the crash. I reported it as a new case with more precise description as case 734921.

    It might be somehow connected to this:
    http://forum.unity3d.com/threads/un...d-at-start-with-ui-in-the-first-scene.353709/

    But I'm only able to reproduce it in linux headless builds, not in a Mac build or in the Editor, so I'm not completely sure if it's the same.
     
    chrismarch likes this.
  13. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    any update on this? It's still not fixed in 5.2.3 and nobody looked at the bug report..

    edit: the workaround of removing all canvases in the first scene works so far, but that's not really how it should be.
     
    Last edited: Nov 20, 2015
  14. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Same here. Lot of users reporting their dedicated servers don't work anymore with the latest release (Unity 5.2.3f1 vs 5.1.1 in our previous release). All get that stack trace as shown in the OP.
    It does work on some machines though, for example both my own desktop and dedi.

    Also could maybe be related to sound changes (maybe previous headless builds properly disabled audio output which does make sense but the newer ones do not?). Reason: quite a bunch of the users reported it working after installing pulseaudio.

    @Unity: Any updates on this one?

    Regards,
    Chris
     
  15. lazytiger

    lazytiger

    Joined:
    Sep 10, 2015
    Posts:
    3
    Hi, guys

    Same problem found on centos 7.1 and unity 5.3.1f1.
    Does anyone has any progress about this?

    Native stacktrace:

    /tmp/Demo-HeadLess/DemoLinux64_Data/Mono/x86_64/libmono.so(+0x915e6) [0x7f57acda45e6]
    /tmp/Demo-HeadLess/DemoLinux64_Data/Mono/x86_64/libmono.so(+0x347fc) [0x7f57acd477fc]
    /lib64/libpthread.so.0(+0xf100) [0x7f57ae31a100]
    ./DemoLinux64() [0x46a5ae]
    ./DemoLinux64() [0x46a916]
    ./DemoLinux64() [0x46a988]
    ./DemoLinux64() [0x46826f]
    ./DemoLinux64(_Znwm+0x45) [0x4687b5]
    /lib64/libstdc++.so.6(_ZNSs4_Rep9_S_createEmmRKSaIcE+0x59) [0x7f57adeb8c69]
    /lib64/libstdc++.so.6(_ZNSs12_S_constructIPKcEEPcT_S3_RKSaIcESt20forward_iterator_tag+0x21) [0x7f57adeba521]
    /lib64/libstdc++.so.6(_ZNSsC2EPKcRKSaIcE+0x38) [0x7f57adeba958]
    ./DemoLinux64() [0x87c6ca]
    ./DemoLinux64() [0x87cb14]
    ./DemoLinux64() [0x46a719]
    ./DemoLinux64() [0x46a916]
    ./DemoLinux64() [0x46a988]
    ./DemoLinux64() [0x46826f]
    ./DemoLinux64() [0x469979]
    ./DemoLinux64() [0x846489]
    /lib64/libpthread.so.0(+0x7dc5) [0x7f57ae312dc5]
    /lib64/libc.so.6(clone+0x6d) [0x7f57ad61821d]
     
  16. CipSoft

    CipSoft

    Joined:
    Jan 30, 2014
    Posts:
    5
    This issue disappeared after switching to Unity 5.3.1.
    Right now we are running 5.3.1p2.
     
  17. lazytiger

    lazytiger

    Joined:
    Sep 10, 2015
    Posts:
    3
    I have done more research and found that it depends. On my dell r410 and centos7, it's ok; but dell r420 and centos7, it crashed. I can't imagine why this happens. So my guess is that this issue is related with hardware.
     
  18. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    241
    Maybe it does, but as it was working fine before 5.2 on all those machines it's some kind of regression that's been ignored again :(
     
  19. DimensionU

    DimensionU

    Joined:
    Aug 1, 2013
    Posts:
    43
    I'm having the same issue with Unity 5.3.2 f1. Are you guys as well?
     
  20. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    I don't think it was fixed in 5.3.2, the only solution for now is the no-UI-in-first-scene workaround.
     
  21. DimensionU

    DimensionU

    Joined:
    Aug 1, 2013
    Posts:
    43
  22. holliebuckets

    holliebuckets

    Moderator

    Joined:
    Oct 23, 2014
    Posts:
    496
    @DimensionU thank you for pinging me. I have pinged the dev directly to see why/if it was lost between two teams or what the outcome has been. I will update you as soon as I hear back.
     
  23. DimensionU

    DimensionU

    Joined:
    Aug 1, 2013
    Posts:
    43
    @holliebuckets - thank you so much!

    I'm finding that the workaround that @any_user prescribed definitely helps, but doesn't completely eliminate the
    SIGSEGV for me. Thanks again.
     
    holliebuckets likes this.
  24. DimensionU

    DimensionU

    Joined:
    Aug 1, 2013
    Posts:
    43
    @holliebuckets - have you heard anything back from the developers on this? I searched for both case numbers listed above (732667 and 734921) in the release notes for both 5.3.3f1 and 5.3.3p1, but didn't see anything. This SIGSEGV issue is a pretty big problem for me as our headless, Linux dedicated servers are throwing it and once that happens, they don't shut down properly. This means that we end up spawning a bunch of unusable instances on EC2 that have to be manually dealt with. A fix (or a workaround that is 100% effective) would be AWESOME. Thanks.
     
  25. holliebuckets

    holliebuckets

    Moderator

    Joined:
    Oct 23, 2014
    Posts:
    496
    @DimensionU I have not heard back. I check the status and the Linux team has set it to a severity of 1 (most severe) so they are well aware it is a big problem. I'm on the US West Coast and they are in CPH so pinging back and forth can take a few days. I have set the bug to auto ping me when there is a change or comment :)
     
  26. DimensionU

    DimensionU

    Joined:
    Aug 1, 2013
    Posts:
    43
  27. holliebuckets

    holliebuckets

    Moderator

    Joined:
    Oct 23, 2014
    Posts:
    496
    Hi @DimensionU just to update you - the team has not been able to reproduce the issue with 734921 with the provided project BUT 732667 is still under evaluation :)
     
  28. DimensionU

    DimensionU

    Joined:
    Aug 1, 2013
    Posts:
    43
    Hi @holliebuckets

    I wanted to let you know that shortly after your last post, I tried something on my end. I increased the RAM in the Linux VM I am using in ec2. I've now been running that way for about 2 weeks without the SIGSEGV.
     
  29. Arsinx

    Arsinx

    Joined:
    Apr 14, 2014
    Posts:
    55
    I am getting exactly the same error, I start my Unity Headless builds and it crashes immediately with the error above. Same build in non headless mode runs fine in the same OS. please Help!
    I've Tried everything I can think of.

    If any of you have any work arounds for how to make it work would really appreciate it. The strangest thing is that it was working before, It just stopped working this week, The only changes in the project were the addition of more assets and models and some animations. No plugins or anything that would suggest any reason for a crash. Its really frustating.
     
    Last edited: Apr 19, 2016
  30. DimensionU

    DimensionU

    Joined:
    Aug 1, 2013
    Posts:
    43
    Hi @holliebuckets
    I spoke too soon and my attention was drawn elsewhere. I still do get the SIGSEGV. That 2 weeks without an issue may have been during a time of low demand for us. I've been restarting servers nightly for the past couple of months. Are they still unable to reproduce?

    This may be a tangent, but it could also help us all. I think we could get more information from logging if we could run a development linux build in headless mode. Currently, the two are mutually exclusive.