How to figure out a crash in release build

Our game is crashing rarely only when running as a Release build. We're unable to figure out what could be the cause. It completely freezes the process and we have to end the task via the task manager to exit. We've never seen it happen in a Debug build.

What are some ways that we could figure this out? Is there some way we can profile the Release build of our game using an external tool and know what caused the freeze? (infinite loop, out of memory, exception etc).

 

This will probably be fairly difficult to track down and fix. There are a few general things you can do, but usually when I have a release-only bug, it will require a lot of careful thought and imagination to deduce what could be going on.

You mentioned it's freezing up, not fully crashing? Normally if it actually crashes, there will be a crush dump somewhere on disk. But if it's getting into an endless loop then it probably isn't fully crashing.

You can certainly attach the debugger to releases of the game. Just build the game as a Development Build, and it's good to include PDBs along with it. Unless you've narrowed things down a bit, attaching the debugger might only be helpful if your code is getting into an endless loop. (Attaching the debugger and pausing sometimes shows you which code it's looping in.)

You also said it's random, meaning that it happens at different times and places in your game? Any more information on that to help you narrow it down? Like, it always seems to happen after something else happens?

You can also add a bunch of logging to your code, and then look at the log file. Often that's the best way to see which code was the "last" code that did something before things fell apart.

Anyway, figuring these kinds of things out takes some insight and luck, in my opinion. Eventually you'll hopefully see a pattern, and it will direct your investigation. You can also start by looking at all "while" loops, or other looping structures in all of your code to see if any of those could possible, under weird circumstances, run forever.

 

@dgoyette Thanks for your detailed insight. It doesn't seem to ever crash in development build and in fact we just did a 12 hour overnight running test using a dev build and macro-ing to press buttons and it didn't crash.

When it does crash in a release build, the CPU usage drops to 0% in the task manager and certain threads seem to still keep running such as the background music still plays. The memory usage freezes at around 800mb.

We do occasionally see these warnings in the console, but they don't have a stack trace and we get them also in a dev build without freezes.

1. Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak
2. To Debug, enable the define: TLA_DEBUG_STACK_LEAK in ThreadsafeLinearAllocator.cpp. This will output the callstacks of the leaked allocation

This is a hard one! :/

 

That's going to be tough to pin down, then. Best I can think of (and hopefully someone else has some better ideas) I've already mentioned: trying to attach a debugger when this happens, to see if it can tell you where the code is; or adding tons of logging, so you can at least see the last thing that got logged before things locked up, and hope to think of the reason from that. That's going to feel fairly trial-and-error, though, and it might take a lot of logging before you stumble on the culprit.

In general, though, given the low CPU usage, that doesn't sound like an endless while loop. But maybe it's a Coroutine WaitWhile or WaitUntil that never reaches the condition it's expecting? Do you have any of those in your code that could possibly be failing? Usually I avoid WaitWhile and WaitUntil, instead using a custom yield instruction to wait until something happens, but to cap it at a certain max amount of wait time, to avoid this kind of problem. No idea if that's possibly what's happening in your case, though.

 

I would recommend use Il2CPP scripting backend and run player with cmdline params:
1) -debugallocator - throws exception on OOB or deleted memory memory access.
2) -systemallocator and use WPA (https://docs.unity3d.com/uploads/Ex...rformance_using_Event_Tracing_for_Windows.pdf)

With this you can figure out:
1) IL2CPP - when hang happens is it in Unity code or game code - use ProcessExplorer to see thread stack (or dump process and open in VS).
2) -debugallocator - if there is memory corruption.
3) -systemallocator - detailed analysis of allocations and activity in WPA.

I hope that the version of Unity you use allows you to use those options