Criticism on the QA team about CRASH bug reporting

I have read from Unity Blog long time ago that talks about project stripping, explaining reproduce steps, making it easier for the QA team to make it into an actual bug.

I understand, there are way too many thousands of fake bugs either from inexperienced developer getting into their own problem that you can't simply follow just mere words or clues for everyone. A reproducible project is the king that helps QA team get to work fast, getting real bug fixed.

But one sentence that I agree with very much : "If it is a CRASH bug, please always report it no matter what if you could attach reproduce steps or a project or not, or even if it occurs randomly. Because crash contains stack trace that is always automatically helpful."

I think it was a blog from some 4.x release that the feature was mainly fixes bugs that I can't find. But I love it. It reassure you that if you happen to run into some mysterious crash, at least Unity team could also be informed about them.

But nowadays that promise became invalid. The QA team became too strict about having reproduce project. And I mean "project", because simply explaining steps will most likely get a reply to ask for a project anyways. You just can't get your words pass that gate if you don't have a reproduce project. (I had reported a bug reproducible with vanilla Timeline and one audio clip, possible to do in steps in a few minutes, but I still was asked for a project with a TimelineAsset and required tracks made for them!)

You need to change how QA team works if you want to get rid of instabilities one at a time.

Crash bug should always be an exception because stack trace screams at specific location in a code. With reproduce steps? Cool. With a project? Even better. But what's different about crash bug is that the QA team should let it pass to an actual programmer responsible for that code around the stack trace. If I am the programmer, I would very much like to know all the stack traces that could occur at my asserts. Even if the source of that is unknown, often just that leads to huge discovery.

While requesting for repro project make sense mostly (resource wise, spam filtering wise), it doesn't make sense for one case : if it is a crash bug. You should always treat crash bug with high priority and doesn't even need a Quality Assurance. You can assure that the stack trace is very high quality as average coders can't intentionally fake one easily. Maybe you should just QA that the stack trace looks like a stack trace with some Regex.

For example, as I worked on Unity it randomly closes down with this on editor log :

From the stack, it seems like something with animation. And the game of course contains tons of animation. Sometimes it crash even if I am doing nothing as the animations are running. Sometimes it runs as I take action such as resizing the Game view.

But you get the point, it happens sometimes but not always. Even if I attach the project, I would have to tell "just work on the game normally" on the reproducing steps. That is not helping anything.

But because this is a crash bug, luckily I got stack trace that let me know it is something about animation in the first place. So I used the popup that appears after, with "Crash Report" selection locked, greyed out, to submit this valuable clue without the project. I just want to let the programmer around that line of code know.

This is the reply I got :



In the message I even said it occurs randomly while I work on the game, so the report is not even meant to be reproduced. Yet I still get this kind of message, it left very bad taste as if not a human processed my report, but simply see that it has no project and auto replied "I can't reproduce it".

You tell me to strip down. But this bug occurs randomly while I work on the game. After stripping down what am I supposed to work on if the game is not the same game anymore? Randomly click things? And what if the bug disappear after I stripped things down? Should I click randomly for N days before I give up? Do I have to submit a new report with full project anyways, that describe that after stripping down X then the bug is fixed? In the end I must submit a full project and I can't just hand the developer the stack trace.

Weighting with massive amount of bugs you guys receive that you need to always request a project, which I also understand. Then the best solution is :

QA team should make an exception for CRASH bug to get through. It is 100% an actual bug with no need to prove it. You can 99.99% trust that no one wrote the stack trace up by hand just to waste your time, unlike "mere words" bug reports with no steps or project. The cause is unknown but clues are always valuable to the programmer of your engine.

Please make submitting crash bug hopeful again like that blog post from long time ago said. Submitting stack trace is the most I could quickly help you and no way that a stack trace is going to be false negative, please make it count again and not disappeared into QA void. These days it is just too hopeless. Thank you.

 

i've also stopped submitting those kind of crashes, since it feels like its bothering them..

 

I complained about this last month, essentially all the same things:

https://forum.unity.com/threads/unf...xperiencing-this-problem.914135/#post-6000422

One of the Unity staff suggested they're going to try to improve this somewhat, though it sounds like they're still thinking of kind of a manual process, rather than just receiving crashes, indexing the data, and finding some way to get value from crash reports that don't also include clear steps to reproduce.

But I 100% agree, it feels like reporting crashes is a waste of time. QA will ignore just it and close the issue. I've never seen a crash report lead to a resolution or an escalation. QA's fine if I can reproduce an issue and submit a sample project. But as soon as it's a crash, QA is a brick wall.

 

Yeah I just got this after trying to explain that the crash dump will be very useful to the developers though there is no steps :



The very little thing you could do is actually just give the crash to developer, then you can close the issue, I don't care. But please let them know there is a possible problem around there.

This make the "Sometimes but not always" drop down completely irrelevant.. I believe the engine's reputation of being unstable nowadays really stems from this QA brick wall that repels actual useful submits. It piles up over and over for who knows how long.

Every version upgrade that makes big change (e.g. new prefab) make any old project more prone to migration related bugs, that will get asked for repro project, and one cannot easily go back in version to make a project that randomly break in the current version. Trying to repro in the current project doesn't work since it is a migration bug. My case I suspect is this kind of bug. Something with the animation or timeline crash the computer but newly created animation or timeline will not. Really, the most helpful a game developer could help out when reproducing bug is that exponentially hard is to give Unity the log. But if that gateway is blocked by the QA, it forever remains a bug.

 

Last time we talked about this we were told that the callstack goes to a database which helps eventually.

https://forum.unity.com/threads/unity-bug-report-requirements.747440/#post-5209448

With that said, the way QA replies feels like crash bug reports are bothering them, I know I feel discouraged from sending any.

Maybe it’s on purpose. With Unity’s whole shtick of “it’s not our fault, there’s no user submitted bug on this” (I’m sure you’ve all seen the passive aggressive dev posts about how they want to fix bugs, but they can find no bug in the database), which is an insane concept, but somehow Unity users have accepted as a fact of life, they no longer have to fix hard to find crash bugs while appearing that it’s not their fault.

 

I may as well add more QA replies that need improvements. I just received this reply after a bug where in 2019.4.4, it spams invalid GUID in the .meta file for only 1 certain file repeatedly. The bug wasn't here in 2019.4.3. Even deleting .meta file by hand, the newly generated one still triggers an error. Just one minor version difference is a really good criteria to catch the bug, I would say.

The reply gives an impression that the team has RegEx on a common Unity error string and give out a help-your-self solution because it is "probably your mistake rather than a bug". That is not a good mindset for a QA because the job is literally assure the quality of bug report. Unity error message does not mean it is 100% user's fault but you are assuming so by default. The reply screams that QA didn't even touch the repro project I managed to separate from my game.

Sorry if this isn't true, but I have to believe it is true because the QA would have know that the tool does not fix the issue if the QA try running the tool even once before sending this reply. Actually if you know the tools and you read my bug report, even deleting the .meta file by hand doesn't work, so how could an automated script that try to preserve GUID link works? With this reasoning the reply doesn't make sense in the first place.

Therefore this reply must be over-automated in some way. It is as sad as discovering that your mother was an automated robot all along by noticing irregular reply patterns 20 years later into your life. I hope that QA team could improve something in the pipeline to not discourage the bug reporters like this. Make it more human, if an amount is large then hire more personnel, etc. because I think this is absolutely worth it for an entire engine.

 

I have the same feeling right there. Every time I submit an Editor/Mobile crash even with a project and stack trace I have the same answer as "Unable to reproduce, closing the bug".

I have stopped sending bug reports since almost all of them are not reproducible always (if they were reproducible always a simple Unit test will solve the issue, and I think on that part it is solid).

Its seems for me that Unity is very unstable these days and submit bugs doesn't help at all