Search Unity

Feedback "Unfortunately, we are not able to reproduce it. Are you still experiencing this problem..."

Discussion in 'Editor & General Support' started by dgoyette, Jun 17, 2020.

  1. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,195
    This has become a very familiar canned response:

    I get one of these responses every time I submit a bug report when the Unity editor crashes. Editor crashes tend to occur out of the blue, and aren't reproducible. So, I never have specific steps to reproduce them. Therefore, pretty much the moment QA gets one of my crash reports, they respond with this canned email.

    On the one hand, Unity prompts me to submit a bug report any time the editor crashes. On the other hand, every time I do that, QA tells me they can't reproduce it, and the issue is closed. This all seems like a huge waste of time for everyone.

    Do these crash dumps not contain actual useful information that Unity can use to diagnose and correct the crashes? I realize that steps to reproduce issues are always better than not having them, but that's not always possible.

    Could Unity either stop prompting users to submit crash reports, or change the approach to triaging/addressing the issues to not make it seem like every one of those crash reports is going directly into the trash?
     
    AcidArrow and PraetorBlue like this.
  2. TomasKucinskas

    TomasKucinskas

    Unity Technologies

    Joined:
    Dec 20, 2017
    Posts:
    60
    From my experience, I could say some crashes are very hard to reproduce(hardware specific) and this is what our QA is ultimately trying to do. If they can't trigger the crash on our systems they are pretty much left with the only option to dismiss the case. Will check up with them if we can do something about that.

    However, maybe I'll be able to help. Could you post the crash dump here? Those dumps do contain very precise information where the crash occurred(Though you need to have the context to read it) and we might be able to track the issue down.
     
  3. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,790
    Just file the Unity bug reporter in your mind as one of those annoying but useless pop ups that software is riddled with these days.
     
  4. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,195
    Hi Tomas,

    I attached the crash dumps to 1256404 and 1256461, but I'm not looking for assistance specifically with either of these issues. I've had four separate crashes since upgrading from 2019.3 to 2019.4. None was identical, though all of them occurred after doing the same thing (returning to Unity after having just saved some code changes in Visual Studio.) Again, I'm not looking for help with any of these issues specifically, since I generally don't expect to see them occur again.

    In general, I don't report bugs unless I've isolated them and created a reproducible test case in a separate project. But Editor crashes are the exception. I general report these whenever they occur, hoping that the dumps contain some useful information that will help the devs stabilize the problem. Maybe that's not the case. But it feels like unless QA can specifically reproduce it, the devs will never even see the crash dump.

    I complained to QA about this in May, under the context of issue #1247271, another random editor crash. Here's the specific response I was given:

    That seems like a pretty limiting policy when it comes to trying to fix crashes.
     
    PraetorBlue likes this.
  5. TomasKucinskas

    TomasKucinskas

    Unity Technologies

    Joined:
    Dec 20, 2017
    Posts:
    60
    I've looked at your cases and found they already contain the information that I would be able to pull out of your crash dumps.

    I also asked our QA to look at the case and they said that once the get a project that reproduces the crash on your machine they will get developers to test your case. Hopefully, this will be enough to find what is causing this.

    As for bigger changes coming from this thread I've talked with one of our QA leads and they were already thinking about forwarding more crashes(even though testers cannot reproduce them) to developers. Today we are thinking about the formula which would let us evaluate which crashes are most likely to get fixed and are not going to prevent developers from being productive.

    Anyhow thanks for reaching out, I hope your current cases get resolved soon.
     
  6. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,751
    Having QA individually and manually decide when a crash dump is "worthy" of being sent to developers is not how it is done in modern software development, and it's incredible a software company as large as Unity uses such a primitive crash handling pipeline.

    Crash dumps should be automatically parsed and entered into a database, grouped by sanitized stack trace, and viewable through a dashboard where the most common crashes can be easily found and investigated (common crashes tend to have the same call stack).

    Unity doesn't even need to develop their own system for this, as there are several products and services for this exact purpose, both open source and commercial.

    Game developers quickly learn the value of automated crash analytics after they publish their first games (as it's baked into pretty much all digital storefronts) , it's disappointing Unity relies on Manual processing.
     
  7. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,790
    I’m guessing the philosophy is, if a user isn’t willing to donate tons of their time to produce a foolproof repro, it’s not worth even a second of their developers’ time. That’s why they ask for bug reports with repros for everything.
     
  8. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    4,195
    Hi Tomas,

    Thanks for looking into this. You've mentioned two thing, so I'll respond to each individually.

    1) I'll almost certainly never have a reproducible test case for a random crash. That's completely the point of this thread. I'm 100% onboard when it comes to providing reproducible test cases for bugs and behavior issues. But when Unity "randomly" crashes, there's very little chance it will ever happen to me again. However, I assume that maybe it's happening across the entire user base, so it's worth reporting. So, requiring reproducible test cases for crashes is just not at all practical. If Unity can't track down crashes without reproducible test cases, then Unity should be adding more diagnostics to their code, just like game devs do when they're trying to resolve "random" crashes.

    2) Your plan to send more stuff on to the developers is exactly what I'd hope to hear from you. Hopefully something comes of that.
     
  9. TomasKucinskas

    TomasKucinskas

    Unity Technologies

    Joined:
    Dec 20, 2017
    Posts:
    60
    KokkuHub, you are right. We do have this database but Unity being a general-purpose engine we have many ways and scenarios our tools can be used in. Thus having only crash stack and knowing how many times it occurred within reported crashes is not enough. We need QA to try and reproduce the crash otherwise our developers may take this frequent crash, see the dump, see the corrupted data, and not know what caused it or how it got there. Currently, it is generally a must for testers to reproduce the crash to be able to forward the case.

    As AcidArrow said that's why user collaboration on reproducing the issue is most of the time crucial for us to be able to come up with a fix.

    Now let's talk crashes that are the main point this thread.

    I understand. The crashes you reported in your cases happen when scripts are being recompiled so just notice which project reproduces the crash(the smaller the project, the better). When you have this project, share it with our QA and say this project did reproduce the crash. At least we will have the script collection that triggered the crash(a definite validity raiser for the case)

    dgoyette, I fully agree with you. Let's hope the wish will be fulfilled.
     
  10. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,751
    I understand that, specially when a crash is caused by data corruption that happened outside of the call stack. But in my experience there is value in having these statistics and individual dumps visible to developers for proactive bug hunting in addition to QA triage.

    For example, if a particular crash stack trace shows up often on the database but there is no repro attached to it, a developer can insert additional checks and logging around the suspected code to try to grab additional information. QA could also be made aware of "most wanted" crashes and engage more closely to customers whose report contain these stack traces.