Crash during Scene Change / Unload

Alright, we're over 20 hours into trying to debug this and running out of ideas, and steam... Any guidance would be much appreciated.

Problem:
Loading into our 'loader' (or main menu) scene from our 'main gameplay' scene results in a crash. This only occurs on Windows x86_64 builds though. =(

Our game is almost exclusively driven via procedural means, but we use 3 scenes for state control / management:
Scene 1) gameplay / UI;
Scene 2) loader/loading screen;
Scene 3) main menu;

For legacy reasons, our primary (gameplay/UI) scene needs to be loaded non-async. Our main menu, and the loader scene, can be loaded async though. That said, I've tried every different way to load them, and I've even created a new 4th scene called 'Nothing' (contains absolutely & literally nothing, just a camera, no components) -- still end up with a crash during the unload.

When loading the GameLoader scene (contains 1 component/script -- GameLoader.cs) the Start does not get called before the crash.

The Setup:

• Unity 2018.1.9f1 (editor running on macOS)
• Runtime: .NET 4.x Equivalent / Mono / .NET 4.x
• DOES occur on Windows x86_64 standalone build
• Does NOT occur in the Editor, nor on Mac builds; unknown if impacts Windows x86 builds
• Reproduces on our Windows test rig (i7-7700K, 32GB RAM, Windows 10 64bit, GTX1080)

Details + .DMP + Log file
Screenshots (code + dmp): https://imgur.com/a/E2FSXbo
Log file: https://pastebin.com/5Ys3Qyty

More than happy to send the .dmp over if it'll be helpful; it's quite large though, 1-2GB uncompressed. Can also send the full log, etc...

Further Thoughts & Things We've Considered / Tried / Misc...
So this reproduces every single time when exiting from save certain save games / state that the 'gameplay scene' is in. I'm obviously not sure what specific state is the culprit, though I'm also a bit dumbfounded at what state could do this. Some thoughts we've had but had no dice in finding issues with include...

• Thread / background runners -- they are stopped via CancellationToken + .Join(3000) + Abort if they don't exit within the 3000ms. We're confident they're being stopped appropriately, safely, before any of the above runs...
  
  
• We long-suspected this to be a RAM issue, and it had long been not reliably reproducible (but was intermittently reported -- at one point users reported 'quickly forcing scene changes [save/load cycles] caused crash]), though now we are able to reproduce it with full consistency (yay). It really doesn't appear to be RAM related at this point, at least AFAICT. The scenes we're loading into are tiny, and the scene we're loading 'out of' run just fine (though are large) -- it's not until they're unloaded that we have issues. Furthermore, we clean up quite well -- calling Dispose() where appropriate (on Meshes [we handle rendering mostly w/out SRs], but on all else too).
  
  
• We use a Steam integration library -- we've disabled it, is not instantiated at all, etc... and crash persists. Nonetheless, this report has similarities (we don't use Steamworks.NET, using the Rust library instead, but seems awfully similar): https://github.com/rlabrecque/Steamworks.NET/issues/195
  
  
• We don't use physics, rigidbodies, etc.
  
  
• The 'WaitForEndOfFrame' is in an effort to be able to drop the reference to game state after everything (including any coroutines) have finished running for the frame so that they don't end up in a nullref storm. Found some indications online that this was one way to avoid issues, but hasn't helped us here.
  
  
• None of our threaded stuff is hitting Unity API during gameplay obviously, so don't believe there's anything unusual happening there during this sequence (threads should all be stopped by this point anyway).
  
  
• Is almost surely something fairly 'dumb' going on I think, and likely leaving the gameplay scene in a state that causes this; may be purely use code, but we're basically at the point where we aren't sure how to debug this further without expending tons of time (literally well over 70 "trial & error" builds the last 24hrs or so); any thoughts / ideas / guidance would be so, so much appreciated!!!

 

Alright, so about 6 hours further into it today.

Went ahead and got Unity 2018.1.9f2 setup directly on the Windows machine to see if it'd reproduce in the Windows editor and potentially cut the feedback-loop timing down to be able to iterate/test stuff faster on this.

It does reproduce in the 2018.1.9f2 Editor when running on Windows. So this appears to be specific to Windows, not just the standalone Windows build.

Will continue testing & crashing Unity repeatedly, though any ideas or guidance would be most welcomed!

 

More stack from WinDBG: https://imgur.com/a/BkNvFft

 

Day #3 on this. =(

I could really use some help from Unity on this -- I really can't fathom how this can be expected behavior when it works flawlessly on macOS (both in editor & standalone mac builds) but crashes on Windows (in editor & standalone builds).

I've emailed via the Unity "Inquire About Paid Support" form already as well, and we'd be more than happy to pay for guidance on this issue, but haven't received any response thus far. Heck, we'd frankly be thrilled to pay a year's worth++ of extra Unity dues if it meant being able to get this resolved & get back to gameplay code / adding actual value to our game itself, and away from burning time on this kind of boilerplate debugging. =(

Will continue to update with findings.

 

Stepping through it in the debugger (which takes quite a while), it appears as though all of the OnDestroy & OnDisables are called & everything runs as as expected and then it just dies. =(

The last thing it hits, which may or may not be relevant (and seems likely to be expected, I think...?), are a repeated bunch of OnCanvasHierarchyChanged within TMPro_UGUI_Private.cs -- it hits those and then seems to be "out" of C# land and freezes for a few seconds before crashing.

Am going to stop trying to actually debug it at this point and try some different scene-switching paradigms (ever-present "loader" scene with additive Load + Unloads) because I'm pretty convinced it's a Unity bug at this point. Unfortunately I have no idea what specifically I'm doing to trigger it.

FWIW, the issue persists in 2018.2 which I briefly tried to see if it maybe had been resolved via luck; unfortunately it wasn't, and there were some new issues in that version of Unity as well (sprites in SpriteAtlas that we then draw onto a Mesh manually -- the mesh wasn't showing up, unsure if the underlying Textures are no longer available or what, but anyways, back to the issue at hand).

Thanks to those of you following along -- if anyone has any ideas, I'm all ears. If someone from Unity pings me, I'd buy you multiple kegs of beer...!

 

I basically stopped 'actual debug' on this the last few hours, and decided to try blindly working around the issue. My plan was first to try changing the paradigm by rewriting the GameLoader to be a "persistent scene" and doing everything additively; if that didn't work then I was going to really hack-around it, and try writing a "recovery_state_scene" string into playerPrefs, fire up a new application process, and terminate the application process (gross, I know).

Luckily, the 'persistent scene' approach appears to have worked around the issue from my testing so far though still need to test further to be positive, but initial tests on the previously 100%-reproducible case look promising. Assuming it holds, I'm almost positive this is going to be a Unity-side bug -- probably triggered by some specific thing we're doing somewhere (I'd guess perhaps a late/early SetParent, Instantiate, Resource.Load, Destroy, or something along those lines but really no idea).

Was lucky enough to have been in touch with someone from Unity who is going to take a closer look at the .dmp file next week, so will hopefully be able to identify the root cause of the issue and get the process started to put it to rest, but will definitely report back with outcome on that for anyone else who eventually runs across this same issue -- because I'd wish this class of bug on no one!!

 

Did you ever solve this?

 

Sorry for the delay.

We just worked around it as described above. By all accounts what we had originally should have worked fine, and we were told that the stack trace from the crash indicated a bug within Unity (that was confirmed by a Unity dev who looked at the .dmp for us); but after spending multiple days on it we essentially just worked around it via a persistent scene which ended up being fine for us.

We don't do much 'classic' scene/GameObject stuff anyways, so it really wasn't a big deal to use a work around -- in hindsight we should have just managed the swap ourselves more-so, which is basically what we ended up doing.

That said, it was a large (and unexpected) time-sink figuring it out & debugging, since it really wasn't anything unusual / was being used exactly as documented. Oh well.

 

Have Unity ever looked into/resolved this issue?
I stumbled onto something similar where going from our main game scene into another level will crash. I have found through testing that it is the unloading the main scene that crashes, loading the main scene from a empty scene works without any problem.

This is happening on Quest (Android), Unity 2019.3.0f6, IL2CPP or mono, doesn't happen on Mac Editor.

Crash log isn't too helpful and because it only happens on the Quest, it is complicated to debug it.
--------- beginning of crash
01-30 00:47:19.855 712 22100 F libc : Fatal signal 6 (SIGABRT), code -6 in tid 22100 (Mapper0)
01-30 00:47:21.104 22116 22116 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
01-30 00:47:21.104 22116 22116 F DEBUG : Build fingerprint: 'oculus/vr_monterey/monterey:7.1.1/NGI77B/434260.5710.0:user/release-keys'
01-30 00:47:21.104 22116 22116 F DEBUG : Revision: '0'
01-30 00:47:21.104 22116 22116 F DEBUG : ABI: 'arm64'
01-30 00:47:21.105 22116 22116 F DEBUG : pid: 712, tid: 22100, name: Mapper0 >>> /system/bin/trackingservice <<<
01-30 00:47:21.105 22116 22116 F DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
01-30 00:47:21.109 22116 22116 F DEBUG : Abort message: 'Check '0 <= pIndex < map_.num_points()' failed
01-30 00:47:21.109 22116 22116 F DEBUG : 0: 0
01-30 00:47:21.109 22116 22116 F DEBUG : pIndex: -1
01-30 00:47:21.109 22116 22116 F DEBUG : map_.num_points(): 515
01-30 00:47:21.109 22116 22116 F DEBUG : '
01-30 00:47:21.109 22116 22116 F DEBUG : x0 0000000000000000 x1 0000000000005654 x2 0000000000000006 x3 0000000000000008
01-30 00:47:21.109 22116 22116 F DEBUG : x4 0000007f72994630 x5 000000000000003f x6 0000000000000001 x7 00000055593fded4
01-30 00:47:21.109 22116 22116 F DEBUG : x8 0000000000000083 x9 0000007f72995450 x10 b155ed72af558bdb x11 0000000000000000
01-30 00:47:21.109 22116 22116 F DEBUG : x12 0000000000000001 x13 0000000000000020 x14 ffffffffffffffdf x15 0032fb449cad3104
01-30 00:47:21.109 22116 22116 F DEBUG : x16 0000007f9888cec8 x17 0000007f9882ded0 x18 000000000000005f x19 0000007f729954f8
01-30 00:47:21.109 22116 22116 F DEBUG : x20 0000000000000006 x21 0000007f72995450 x22 0000000000000000 x23 0000007f65bef200
01-30 00:47:21.109 22116 22116 F DEBUG : x24 0000007f6d9ca588 x25 0000007f7c828e20 x26 0000000000000043 x27 0000007f65bef250
01-30 00:47:21.109 22116 22116 F DEBUG : x28 bfbb0ee54fad38fa x29 0000007f729944f0 x30 0000007f9882ac84
01-30 00:47:21.109 22116 22116 F DEBUG : sp 0000007f729944d0 pc 0000007f9882ded8 pstate 0000000060000000
01-30 00:47:21.127 22116 22116 F DEBUG :
01-30 00:47:21.127 22116 22116 F DEBUG : backtrace:
01-30 00:47:21.127 22116 22116 F DEBUG : #00 pc 0000000000079ed8 /system/lib64/libc.so (tgkill+8)
01-30 00:47:21.127 22116 22116 F DEBUG : #01 pc 0000000000076c80 /system/lib64/libc.so (pthread_kill+64)
01-30 00:47:21.127 22116 22116 F DEBUG : #02 pc 0000000000024ac0 /system/lib64/libc.so (raise+24)
01-30 00:47:21.127 22116 22116 F DEBUG : #03 pc 000000000001cdcc /system/lib64/libc.so (abort+52)
01-30 00:47:21.127 22116 22116 F DEBUG : #04 pc 0000000000003d58 /system/bin/trackingservice (__android_log_assert+224)
01-30 00:47:21.127 22116 22116 F DEBUG : #05 pc 0000000000b919a0 /system/lib64/libMontereyTracker.so (_ZN9visionlog12doLogFatalOpIRKiRiRmJEEEvOT_PKcOT0_S8_OT1_S8_S8_S8_DpOT2_+192)
01-30 00:47:21.127 22116 22116 F DEBUG : #06 pc 0000000001070760 /system/lib64/libMontereyTracker.so (_ZN5viper29VisualBundleAdjustmentProblem38createAllReprojectionErrorsForKeyframeERNS_8KeyframeEb+732)
01-30 00:47:21.127 22116 22116 F DEBUG : #07 pc 00000000010702ac /system/lib64/libMontereyTracker.so (_ZN5viper29VisualBundleAdjustmentProblem15initializeLocalERNSt6__ndk16vectorIiN9optimizer9allocator13PoolAllocatorIiEEEERNS1_3setIiNS1_4lessIiEES6_EE+3020)
01-30 00:47:21.127 22116 22116 F DEBUG : #08 pc 000000000112704c /system/lib64/libMontereyTracker.so (_ZN5viper21localBundleAdjustmentEiiiRNS_23BundleAdjustmentProblemEPNSt6__ndk16vectorIiNS2_9allocatorIiEEEES7_+1460)
01-30 00:47:21.127 22116 22116 F DEBUG : #09 pc 0000000000ed6c14 /system/lib64/libMontereyTracker.so (_ZN5viper17ViperMapOptimizer24wrappedLocalBundleAdjustERNS_3MapEiiPNSt6__ndk16vectorIiNS3_9allocatorIiEEEE+244)
01-30 00:47:21.127 22116 22116 F DEBUG : #10 pc 0000000000ed7784 /system/lib64/libMontereyTracker.so
01-30 00:47:21.127 22116 22116 F DEBUG : #11 pc 0000000001209ebc /system/lib64/libMontereyTracker.so (_ZN5viper27MapBundleAdjustmentStrategy25runBundleAdjustFuncIfTimeEN13enum_wrapper_22BundleStrategySizeImplERKNSt6__ndk18functionIFNS1_24BundleAdjusterStatusImplEvEEE+60)
01-30 00:47:21.127 22116 22116 F DEBUG : #12 pc 0000000000ed6204 /system/lib64/libMontereyTracker.so (_ZN5viper17ViperMapOptimizer3runERNS_3MapE+316)
01-30 00:47:21.127 22116 22116 F DEBUG : #13 pc 0000000000ec56e0 /system/lib64/libMontereyTracker.so (_ZN5viper11ViperMapper13process_queueEv+456)
01-30 00:47:21.127 22116 22116 F DEBUG : #14 pc 0000000000ec3ec4 /system/lib64/libMontereyTracker.so (_ZN5viper11ViperMapper18process_queue_loopEv+204)
01-30 00:47:21.127 22116 22116 F DEBUG : #15 pc 0000000000ec88c0 /system/lib64/libMontereyTracker.so (_ZNSt6__ndk114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMN5viper11ViperMapperEFvvEPS8_EEEEEPvSD_+64)
01-30 00:47:21.127 22116 22116 F DEBUG : #16 pc 0000000000076454 /system/lib64/libc.so (_ZL15__pthread_startPv+204)
01-30 00:47:21.127 22116 22116 F DEBUG : #17 pc 000000000001e204 /system/lib64/libc.so (__start_thread+16)
01-30 00:47:21.655 1061 1220 W NativeCrashListener: Couldn't find ProcessRecord for pid 712
01-30 00:47:21.665 1061 1118 I BootReceiver: Copying /data/tombstones/tombstone_02 to DropBox (SYSTEM_TOMBSTONE)
01-30 00:47:22.508 1061 1250 I ActivityManager: START u0 {act=com.oculus.vrshell.intent.action.LAUNCH flg=0x10000000 cmp=com.oculus.vrshell/.MainActivity (has extras)} from uid 10024 on display 0
01-30 00:47:27.295 1061 1563 I ActivityManager: START u0 {act=com.oculus.vrshell.intent.action.LAUNCH flg=0x10000000 cmp=com.oculus.vrshell/.MainActivity (has extras)} from uid 10024 on display 0
01-30 00:47:29.841 1061 1207 E TaskPersister: File error accessing recents directory (directory doesn't exist?).

 

I’m so glad I found this thread in 2020. I’ve been doing a beta test for my game on iOS, Mac and Windows simultaneously (solo dev, dying inside) and windows has by far the most issues. Ram management on iOS is one thing, MacOS being metal and tested on my laptop is another. But windows CONSTANTLY crashes, time and time again, with fairly consistent crash conditions (normal gameplay things people should do to enjoy the game, of course...) when changing to the next scene from the main game overworld. It’s maddening.

I have a lot of data persisting through scenes but not as crazy as many games out there. Inventory, stats, that kinda stuff. And it works flawlessly on mobile... and MacOS. But why windows? There’s no separate build, no platform specific code that should change scene loading. It’s so weird.
And I’m on 2019.3 release. Soooo.. not sure what’s going on haha

 

Sorry to hear you're having crash issues -- I know how frustrating it can be. I clearly remember the awful week I was having when we were desperately trying to debug this one. It was definitely not fun! Just to clarify though, in an effort to help, the issue we observed in this case did not appear to have anything to do with "normal gameplay" or the persistence of data during gameplay.

The crash that we experienced (and detailed in the OP) was occurring only at/during explicit scene load/unload sequences and (on Windows at least) it reproduced 100% of the time, every time. Meanwhile, the same code worked fine in Mac builds (and editor).

Over the last few years we've had to debug all kinds of crashes, though in every instance (except this one, as detailed above), we've eventually been able to trace it back to either a hardware issue (ie out of memory, old drivers, etc) or -- and most commonly -- "user" code. Meaning, code we wrote / mistakes we made.

Getting to a point where you can repro it 100% of the time is the main thing -- it's all downhill from there, just a question of how long (and/or steep) the hill might be!

 

Yeah this is quite the steep hill haha. It’s such an odd crash. I look at the crash log and it says it’s a StackOverflorException but everything beyond that line in the log is linked to internal Unity backend things and Debug code. I’m not sure what to do.

I removed all of the fonts I was using to get rid of “non dynamic font” warnings. Changed shaders to get rid of THOSE warnings, trying to go through and minimize any sort of throwable issue Unity detects. To no avail, however. It’s crazy. Crashes eeeeeverytime the player decides to enter a dungeon.

sometimes it’ll work just once after the initial build, then crash every time again. Same useless StackOverflowException in the log. I can’t figure it out!

 

You're seeing a "StackOverflowException" in the log though at least -- it's somewhere to start from, at least! If you can get it to reproduce with the "Development" build flag on, and set logging to 'Full' or similar, then perhaps try posting the stack trace here on the forum and someone may recognize something or at least be able to provide some guidance on how to continue debugging.

What's happening in your code when the player "enters a dungeon" -- are you invoking a scene change? If so, perhaps post the relevant lines of code that encompass the scene change specifically; a second+ set of eyes can't hurt.

I know the feeling for sure, all you can do is continue to debug & perhaps post some more details/info to help others be able to possibly "get lucky" and spot something that may prove useful as a clue, even if not spot the issue itself directly.