Feedback "Unfortunately, we are not able to reproduce it. Are you still experiencing this problem..."

This has become a very familiar canned response:

I get one of these responses every time I submit a bug report when the Unity editor crashes. Editor crashes tend to occur out of the blue, and aren't reproducible. So, I never have specific steps to reproduce them. Therefore, pretty much the moment QA gets one of my crash reports, they respond with this canned email.

On the one hand, Unity prompts me to submit a bug report any time the editor crashes. On the other hand, every time I do that, QA tells me they can't reproduce it, and the issue is closed. This all seems like a huge waste of time for everyone.

Do these crash dumps not contain actual useful information that Unity can use to diagnose and correct the crashes? I realize that steps to reproduce issues are always better than not having them, but that's not always possible.

Could Unity either stop prompting users to submit crash reports, or change the approach to triaging/addressing the issues to not make it seem like every one of those crash reports is going directly into the trash?

 

Just file the Unity bug reporter in your mind as one of those annoying but useless pop ups that software is riddled with these days.

 

Hi Tomas,

I attached the crash dumps to 1256404 and 1256461, but I'm not looking for assistance specifically with either of these issues. I've had four separate crashes since upgrading from 2019.3 to 2019.4. None was identical, though all of them occurred after doing the same thing (returning to Unity after having just saved some code changes in Visual Studio.) Again, I'm not looking for help with any of these issues specifically, since I generally don't expect to see them occur again.

In general, I don't report bugs unless I've isolated them and created a reproducible test case in a separate project. But Editor crashes are the exception. I general report these whenever they occur, hoping that the dumps contain some useful information that will help the devs stabilize the problem. Maybe that's not the case. But it feels like unless QA can specifically reproduce it, the devs will never even see the crash dump.

I complained to QA about this in May, under the context of issue #1247271, another random editor crash. Here's the specific response I was given:

That seems like a pretty limiting policy when it comes to trying to fix crashes.

 

Having QA individually and manually decide when a crash dump is "worthy" of being sent to developers is not how it is done in modern software development, and it's incredible a software company as large as Unity uses such a primitive crash handling pipeline.

Crash dumps should be automatically parsed and entered into a database, grouped by sanitized stack trace, and viewable through a dashboard where the most common crashes can be easily found and investigated (common crashes tend to have the same call stack).

Unity doesn't even need to develop their own system for this, as there are several products and services for this exact purpose, both open source and commercial.

Game developers quickly learn the value of automated crash analytics after they publish their first games (as it's baked into pretty much all digital storefronts) , it's disappointing Unity relies on Manual processing.

 

I’m guessing the philosophy is, if a user isn’t willing to donate tons of their time to produce a foolproof repro, it’s not worth even a second of their developers’ time. That’s why they ask for bug reports with repros for everything.

 

Hi Tomas,

Thanks for looking into this. You've mentioned two thing, so I'll respond to each individually.

1) I'll almost certainly never have a reproducible test case for a random crash. That's completely the point of this thread. I'm 100% onboard when it comes to providing reproducible test cases for bugs and behavior issues. But when Unity "randomly" crashes, there's very little chance it will ever happen to me again. However, I assume that maybe it's happening across the entire user base, so it's worth reporting. So, requiring reproducible test cases for crashes is just not at all practical. If Unity can't track down crashes without reproducible test cases, then Unity should be adding more diagnostics to their code, just like game devs do when they're trying to resolve "random" crashes.

2) Your plan to send more stuff on to the developers is exactly what I'd hope to hear from you. Hopefully something comes of that.

 

I understand that, specially when a crash is caused by data corruption that happened outside of the call stack. But in my experience there is value in having these statistics and individual dumps visible to developers for proactive bug hunting in addition to QA triage.

For example, if a particular crash stack trace shows up often on the database but there is no repro attached to it, a developer can insert additional checks and logging around the suspected code to try to grab additional information. QA could also be made aware of "most wanted" crashes and engage more closely to customers whose report contain these stack traces.