Search Unity

FogBugz feedback loop on reported bugs?

Discussion in 'Unity 5 Pre-order Beta' started by seattlebluesky, Nov 12, 2014.

  1. seattlebluesky

    seattlebluesky

    Joined:
    Sep 2, 2014
    Posts:
    170
    In the past few months like many of you I've had the opportunity to try out Unity 4.6 Beta and as a pre-order customer Unity 5.0 Beta. Love the new game GUI API in 4.6 btw. During this time I've reported many bugs. I'm sure they are of various levels of quality w.r.t. data captured, repro' steps etc. BUT in that time I've heard nothing from the Unity Team beyond an automated response from the FogBugz system.

    A couple of proposals. My 2c.

    1) Some kind of non-automated, bug triage, response from Unity QA.
    It would be simply awesome to see some kind of 'triage' message from QA for these issues. Something as simple as, 'we're looking at this', 'forget it!', 'not enough information'. Granted in nearly 3 months it looks like the cases have increased by approx. 18,000 (assuming the case numbers are sequential) so perhaps that's not practical. Are customers using FogBugz as a form of technical support? 18,000 new issues in the past 3 months is A LOT. Certainly I'm guessing a substantial number are duplicates, cruft, already fixed, not reproducible.

    2) Ability to edit/annotate issues post submission.
    Any way a customer who reported a bug could be able to edit and annotate the issue? Some of the bugs below got 'corrupted' titles and I'd love to improve the message.

    Thanks for listening!

    CaseTitleStatusDate Opened
    646862Baking occlusion data and enabling it for occlusion cullingOpen11/10/2014 9:29 PM
    646645Unity installer does not install the Microsoft Visual C++Open11/10/2014 3:20 AM
    645661Editor during GI Iterative Bake will show many errors,Open11/5/2014 10:49 PM
    645037Crashesd while rendering scene in Editor. FWIW I hadn't seenOpen11/3/2014 8:39 PM
    644394- Unity 5.0 GI light map baking does not respect a light'sOpen10/31/2014 10:49 PM
    629614-e Typo. Should be 'masked' and not 'masedOpen8/29/2014 12:30 AM
    629580Upon load of the project the Unity Console reports 4 errorsOpen8/28/2014 8:39 PM
    628098Names of Axes for Standalone Input Module (Script) for 'Open8/23/2014 1:59 AM
    628055-e 4.6.0b17 BetaOpen8/22/2014 10:39 PM
    628044-e 4.6.0b17 BetaOpen8/22/2014 9:29 PM
    628042-e 4.6.0b17 BetaOpen8/22/2014 9:10 PM
     
    bjornrun and MrEsquire like this.
  2. or1on

    or1on

    Joined:
    Nov 22, 2013
    Posts:
    4
    Second that. I reported various bugs (at lest 10) in the last two years, and got replies for only TWO of them. All the others are still marked as "open" on fogbugz, and that's it. At least marking them as fixed/won't fix/working on.. would be nice.
     
    fschneider and MrEsquire like this.
  3. MrEsquire

    MrEsquire

    Joined:
    Nov 5, 2013
    Posts:
    2,712
    Totally agree with you guys. Not been pleased with the Unity QA for some time. I check the board each day and if you really want a bug fixed you just got keep pushing and pushing and hope that one the devs sees it from your view as a important issue and they then look into it personally.

    Thank you developers for the support on here, I think its great especially when you get involved in specific cases although your not QA sometimes its about being proud of your work and really helping others.

    Cant say much about the QA as all the run ins I had with them in forum have been negative.
     
  4. shkar-noori

    shkar-noori

    Joined:
    Jun 10, 2013
    Posts:
    833
    I've reported bugs too, but I've received manual responses from the QA Team, so they pretty much are responsive for me. I've had several responses to the bugs which seem critical.
     
  5. seattlebluesky

    seattlebluesky

    Joined:
    Sep 2, 2014
    Posts:
    170
    For sure with this one's mileage might vary. It's ok. I'm entering issues and the latest reported case was repro'd and fixed for b15. So awesome as I helped improve the product which is the point with this Beta. I think though, for what it's worth, that the feedback loop and process could be more consistent and tidied up.
     
  6. alexzzzz

    alexzzzz

    Joined:
    Nov 20, 2010
    Posts:
    1,447
    I usually get responses on the next working day after I've submitted a bug: http://fogbugz.unity3d.com/default.asp?649965_jcdk2r2bljlhek6q

    The only unanswered report there is the earliest one, when, I suspect, I forgot to save the demo scene before sending the report. I have always created a small repro project featuring the bug, and the latest reports also have a link to a small screen-cast video.
     
  7. seattlebluesky

    seattlebluesky

    Joined:
    Sep 2, 2014
    Posts:
    170
    Yep, I agree that every issue reported should carry succinct repro' steps and a project that demonstrates the issue if feasible and I've strived to do that where appropriate. But my initial point is that in my experience, and yours too apparently, the response tends to be either 'we've reproduced this issue' or no response at all.
     
  8. Marble

    Marble

    Joined:
    Aug 29, 2005
    Posts:
    1,243
    It is kind of a lonely feeling to send a bug and have it listed as "Open" for months. Half of my submitted bugs are still open even though they all contain a succinct repro that is 100% reproducible on my machine. If QA has failed to reproduce it on theirs, I'd like to know so I can help. I suspect in some cases, the bug has simply been set aside. I understand that they have to focus on critical bugs over noncritical bugs, but after a revision is released the bug is then even lower priority for belonging to an earlier version of Unity. I'd like to know once a bug report has been completely abandoned so that I can report it again if necessary.
     
    seattlebluesky likes this.
  9. Eric5h5

    Eric5h5

    Volunteer Moderator Moderator

    Joined:
    Jul 19, 2006
    Posts:
    32,317
    Tons of those are not bug reports; you get case #s when uploading assets to the store, for example. I do get emails from Unity about bug reports, such as "this was reproducible, handed over to be fixed", though not all the time.

    --Eric
     
  10. QA-for-life

    QA-for-life

    Unity Technologies

    Joined:
    Dec 23, 2011
    Posts:
    89
    I guess you are actually in luck. Many also say we reply "We couldn't reproduce, please send a stripped repro project.".

    Here's the conundrum. We can decide to do totally closed alphas and betas, which we did for most of 4.x, in whic case we closed out about 98% of all reported incidents (incident = customer reported issue. Bug = Reproduced internally). Or we can decide to go open beta and be absolutely horribly swamped in incidents that are mostly duplicates. You simply can't have both an open beta and us being able to respond to every incident.

    And to give you numbers: In 4.6 we have received 4252 incidents. Of those we have processed 40.5%, while we have also had to process feedback on 5.0 and 4.5. It is really simple. We are not going to process everything and we have a rating script to help us chose the highest chance incidents to look at. (see http://blogs.unity3d.com/2013/10/28/bug-reports-incidents-and-some-bashing/).
     
  11. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    It is certainly understandable that this whole process is very time consuming and my personal impression is that it is constantly being improved.

    Being someone who always tries to create a minimal reproduction with a short but precise description, it is quite frustrating not to get any feedback at all. Of course, I am not asking for a personalized response, but getting an automated reply with the rating of the incident would already feel a lot better. If the bug could not be reproduced or if it was assigned to a developer (though it is not clear whether or when it might be resolved), it would help the person who submitted the bug to feel more comfortable.
    At the moment the process of submitting a bug feels like a huge black box. Some of the bugs took me half a day, just to create a minimal reproduction. After several months without a reply it feels pretty weird, because I have no feedback at all. I don't even know whether it could reproduced or if anyone had a look at it. Getting an automated response whether you were able to reproduce it, whether it was assigned to a developer or even a response that it won't be fixed would help the submitter a lot.
    When I submit a bug and take the time to make it right, I would like to get some kind of automated feedback, such that I know my time wasn't wasted. Whenever I submitted one or two bugs and got no feedback at all, I tend to not submit further bugs, because it feels like wasted time.
     
    djweinbaum, Trigve, Marble and 3 others like this.
  12. MrEsquire

    MrEsquire

    Joined:
    Nov 5, 2013
    Posts:
    2,712
    Hit the nail on the head.. :)
     
  13. alexzzzz

    alexzzzz

    Joined:
    Nov 20, 2010
    Posts:
    1,447
    I've been always wondering, since I read that article a while ago, what's the difference between "very little description", "small amount of description" and "good amount of description"?

    Some bugs don't really need many words to describe, like

    What's wrong: when I do this, that thing breaks.
    To reproduce: open the demo scene, press Run, notice that this thing bounces (for examaple) instead of standing still.
    http://screencast.com/abcdefg

    Is this a very little amount, small amount or good amount?
     
    MrEsquire likes this.
  14. SpiriTx

    SpiriTx

    Graphics QA Unity Technologies

    Joined:
    Apr 12, 2012
    Posts:
    252
    We are really grateful to have users like you, kind sir <3

    I'd like to assure you that if your case gets looked by a QA, no matter the situation - even if doesn't have sufficient info, it's not reproducible, duplicate or a confirmed bug - you'll get a response.

    While in closed alpha/beta stage we tend to process around 90% of the cases, but with open betas and public releases the amount of reported cases increases tenfold and we don't manage to handle all of them.
    We do prioritise cases like yours (from newest versions and with highest ratings), but right now we have several hundreds of those top-notch cases alone.

    We are trying our best to improve our processes and tools (public issue tracker, working on new bug reporter), to increase visibility, minimise your effort with duplicate bugs and handle cases more efficiently. We are also continuously actively hiring to have more manpower reacting to rapidly increasing user base.

    Feel free to drop me a pm with your most important and critical bugs, so that I can assure they get looked at.

    Thanks and have a nice day!
     
    Arkade, the_motionblur and chrismarch like this.
  15. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    Thanks for the reply. For me it is very obvious that there were lots of improvements in the previous years on your side. In my opinion the QA Team does a fantastic job!

    Even if I am aware that someone checked all the issues that I submitted, it would motivate me a lot more to submit further bugs if I just got a small update like:
    • We were able to reproduce the issue
    • The priority of the issue was changed to ...
    • The bug was assigned to a developer
    • It is unclear whether the issue will be resolved
    • The bug should be resolved in version ... that was just released
    Getting just those little information snippets would make a huge difference for me and would motivate me a lot more, just because I could see that something is happening.
     
    shkar-noori likes this.
  16. SpiriTx

    SpiriTx

    Graphics QA Unity Technologies

    Joined:
    Apr 12, 2012
    Posts:
    252
    Probably I wasn't clear enough, if some of your cases didn't get a response - they weren't looked at, otherwise they would definitely get some kind of update.

    As I already mentioned we simply have too many cases to go through all of them, but ones with higher rating have way higher chance to get looked at.
    Please drop me a PM if something critical was missed.
     
  17. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    Good to hear. I just checked my bug reports. There is around 8 for which I never got a reply, still I took the time to make the description short and to create a small reproduction. My motivation to take any time to submit bug reports just... Well...
    I am kind of speechless. Should I spend my time to complain here in the forum instead of creating a meaningful reproduction? Sounds like a dumb strategy, but it seems to work better then mine.
     
  18. ecurtz

    ecurtz

    Joined:
    May 13, 2009
    Posts:
    640
    This is actually WORSE because it means at least 1/3 of the bugs I submitted during 2014 have never been looked at. Which also means they never make it into IssueTracker, so nobody can tell if they're submitting a duplicate bug or vote on fixes which are important for their projects.
     
    Trigve and MrEsquire like this.
  19. Devil_Inside

    Devil_Inside

    Joined:
    Nov 19, 2012
    Posts:
    1,072
    So, you just ignore some of the reports? Ehm... how do you determine the rating of a case if it was never looked at?
     
    MrEsquire likes this.
  20. MrEsquire

    MrEsquire

    Joined:
    Nov 5, 2013
    Posts:
    2,712
    Good point also they forgot to mention - any report without a repo does not get looked at , these words of the Qa manager himself in a thread i read month or two ago.

    so like someone mentioned, something that does not need a repo is not looked at..simple things and reports without i think they put them buttom..Im just shocked at this way of working and of course i would make noise and make concern about it
     
  21. QA-for-life

    QA-for-life

    Unity Technologies

    Joined:
    Dec 23, 2011
    Posts:
    89
    The rating mechanism is mentioned in the blog post I linked and is fully automated. It also shows the proof of how well that rating actually works in our hunt for reproducible bugs.

    You do realize that in order to fix a bug you need to confirm and reproduce it? Not only to be able to fix it, but also to confirm after the fix that the bug is actually fixed? So yes, we require that we can reproduce a bug before handing it over to devs.
     
  22. alexzzzz

    alexzzzz

    Joined:
    Nov 20, 2010
    Posts:
    1,447
    Does this mean that e.g. a bug report, saying that an item in the editor's main menu has a typo, will be thrown away by the rating mechanims?
     
    MrEsquire likes this.
  23. MrEsquire

    MrEsquire

    Joined:
    Nov 5, 2013
    Posts:
    2,712
    I think your not understanding me;

    So far throughout the forums it is mentioned if no repo is attached, and by repo I mean a sample of the project the user is working on then it will not be looked at - I seen this mentioned by QA members in the forums.

    I'm saying why would I need to include a repo or sample of my project if bug steps are like:

    1) Log into the editor
    2) Goto inspector
    3) Check the Graphics Level dropbox
    4) Missing Graphics "Automatic"
    Make sure you have selected Android Build type

    So please Sir tell me why would such issues not be looked at...
    Lets all add fake samples and attachment so the system can see there is "some" file attached.
     
  24. QA-for-life

    QA-for-life

    Unity Technologies

    Joined:
    Dec 23, 2011
    Posts:
    89
    No, that is not the case and it is mentioned in the blog post. Those would be rated 4 and be high in the list of incidents to be looked at. If something different has been mentioned here in some other thread, it is a misunderstanding.
     
  25. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    It proofs that it works well in the cases you were looking at. If I am trying to keep my description as short as possible, such that you are not wasting time, it is very likely that my report is not considered because it gets a low ranking. That just because I took the time to make it compact.
     
  26. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    I reported a few documentation issues before the documentation section in this forum was in place. It seems no one ever had a look at it.
     
    Trigve, ecurtz and MrEsquire like this.
  27. MrEsquire

    MrEsquire

    Joined:
    Nov 5, 2013
    Posts:
    2,712
    Okay that clears that up, btw there was no misunderstanding, I think the whole QA team need to be briefed on one story to tell users in the forums, seems right now there has been to much contradictions and therefore everyone has different thoughts on how things are done.

    Also in the future it be good if you create a MINI QA team (2-3ppl) to deal with small bugs reports that do not require deep investigation work or hardcore repo project, like Dantus mentioned stuff such as documentation and making sure basics of the Editor, screen orientation issues, build settings etc.
     
  28. QA-for-life

    QA-for-life

    Unity Technologies

    Joined:
    Dec 23, 2011
    Posts:
    89
    I think you just simply underestimate the scale at which we are operating. We have a team of 17 people whose life is to handle all these incidents. And still we are totally overloaded. What I am trying to convey here is that we are not handling everything, we will NEVER handle everything and if we were to handle everything, we would go bankrupt in salaries. Especially considering only 6.34% of all those incidents are actually valid bugs.

    Like mentioned, that was a random sample of ~2000 incidents, which is plenty to be statistically viable.
     
  29. Devil_Inside

    Devil_Inside

    Joined:
    Nov 19, 2012
    Posts:
    1,072
    I second this.
    I've reported a bug in May about "Limit velocity over lifetime" module in Shuriken being fps dependent. It took me probably two days of debugging to realize why my particles in the play mode behaved differently from what I set them up in edit mode. And the reason was that the scene/game view in edit mode is vsynced, while in play mode it depends on your quality settings. So, if you turn vsync off, you particles would behave completely different from what you've set them up in editor. At the time, when I finally realized what is the source of the issue, and narrowed it down to the "Limit velocity over lifetime" module it felt like a real achievement. Fast forward 6 months. This issue in on the first page of the issue tracker, if I filter it to show active issues. In my understanding it's a trivial fix, yet the issue i still there, and not fixed even in Unity 5.

    P.S. This rant is probably a bit offtopic, as fixing bugs is not QA job, so I think we need a small team of developers to fix "low hanging" bugs.
     
    Last edited: Nov 26, 2014
  30. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    Statistics is by definition not a proof, it just shows a tendency.

    In the current rating system people who take the time to submit compact reports are more likely not considered because of statistical reasons.

    Edit: This is really not meant to be offensive. But for someone like me who consciously took the time to write short descriptions and create small reproductions, it is pretty irritating. I most likely wasted quite some time, because I wanted to make the job of the QA team easier. Unfortunately, I made it easier than expected, because some were likely never checked at all due to statistical reasons.
     
    Trigve likes this.
  31. ecurtz

    ecurtz

    Joined:
    May 13, 2009
    Posts:
    640
    I guarantee more than 6.34% of the reports from people who care enough to post in a thread like this are valid bugs. Should I be resubmitting reproducible bugs that get no response? How long should I wait? A month? A number of beta updates? It sounds as if something that isn't looked at pretty quickly is never going to be looked at.
     
    MrEsquire and Marble like this.
  32. seattlebluesky

    seattlebluesky

    Joined:
    Sep 2, 2014
    Posts:
    170
    From my perspective I think the rating system may help separate the 'wheat from the chaff' in a chaotic bug-reporting environment that exists for general releases. In that scenario the rating system is differentiating between bug reports that have had some effort applied to filing them vs. little/less effort.

    Though not all short description issues with no attachments should be poorly rated imho.

    Take http://fogbugz.unity3d.com/default.asp?629614_ga21dnfaqvsgdhh4 that I found 2 months ago during 4.6 Beta. This concerns the typo 'mased' instead of 'masked' that exists on http://docs.unity3d.com/ScriptReference/UI.MaskableGraphic.html. The typo is a low priority item I'm sure but FWIW it still exists. The bug description is short. There's no need for repro' steps of any substantial length, and certainly not attachments. Therefore I suspect the bug was rated low and not looked at, and therefore not fixed.

    Overall bug quality I expect would decrease from those reported by Unity QA, by closed Beta Participants, by open Beta Participants, and finally by general Product Usage.

    The rating system http://blogs.unity3d.com/2013/10/28/bug-reports-incidents-and-some-bashing/ measures the quality of bugs for one of the populaces above broadly, but disregards the notion that a quality bug report is usually more a function of its bug reporter than the length of its description, number of attachments etc.

    In summary, I would think that a better rating system would be that of rating the bug reporter(s) and not rating the bugs reported.
     
    Trigve and MrEsquire like this.
  33. QA-for-life

    QA-for-life

    Unity Technologies

    Joined:
    Dec 23, 2011
    Posts:
    89
    I fully agree with this and it is something we have on a long term roadmap. It is surpringly complicated to do with the entire chain of data taken into consideration, but we have the foundation in the pipes.
     
    seattlebluesky likes this.
  34. alexzzzz

    alexzzzz

    Joined:
    Nov 20, 2010
    Posts:
    1,447
    I filed a bug report about an error in error message. Does UT give achievements for things like that?
     
  35. alexzzzz

    alexzzzz

    Joined:
    Nov 20, 2010
    Posts:
    1,447
    I think this time is definitely not wasted regardless of whether you get the feedback or not.

    1. A small project that reproduces the bug guarantees that this is actually a bug in Unity and not in your main project.

    2. When you create a repro project you investigate the bug more deeply. You may find that it reveals itself only if certain conditions are met, in other cases everything works fine. Or you may find that A is not the only thing that is broken, that B and C, related to A, also don't work as expected. Or you may realize that even though A is broken, B and C still work fine, and it gives you a temporary workaround until A is fixed.

    3. Release notes don't usually cover all the changes that have happened in the recent version of Unity. If you have a simple repro project, you can use it to quickly check new versions of Unity. They may have this bug fixed because someone else reported it, or Unity guys found it by themselves, or it was a part of a bigger bug/issue that was fixed.
     
  36. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    I agree with you in most of what you wrote.
    My main issue that still exists is that even though I made reproductions, Unity did not even have a look at the bug reports due to the mentioned automatic filtering. After I wrote here and PM'd the information about the bugs, Unity checked them and I got a bunch of feedback. Even though I invested a lot of time to create them, they would have been ignored if I wouldn't have written in this thread.
     
    shkar-noori likes this.
  37. alexzzzz

    alexzzzz

    Joined:
    Nov 20, 2010
    Posts:
    1,447
    I have reported 13 bugs since this thread was started. All of them are either fixed by now or confirmed. Right after a bug report has been sent I receive an automated reply, and then, usually on the next working day, I get a reply from the QA with the verdict. From my side, the system seems to work fine, or I'm just lucky.
     
  38. orb

    orb

    Joined:
    Nov 24, 2010
    Posts:
    2,960
    I usually get responses pretty soon after reporting something. The latest one I reported is such a widely reported problem that I suspect it'll be an exception, though, because there probably are too many to reply to :)

    But looking over my long list of reports there are still a lot of open cases. It's possible some of them can be closed, as they're mostly for old versions, but I can't see anything I can do about it.
     
  39. DanSuperGP

    DanSuperGP

    Joined:
    Apr 7, 2013
    Posts:
    408
    You probably write good quality reports.
     
  40. seattlebluesky

    seattlebluesky

    Joined:
    Sep 2, 2014
    Posts:
    170
    Certainly writing good quality bug reports ups your chances of having the bug looked at, and then reproduced ala the rating system http://blogs.unity3d.com/2013/10/28/bug-reports-incidents-and-some-bashing/. From my perspective so long as the issues, if important, are fixed and the quality of the final release is high then how the bug reports are handled I suppose is neither here nor there. In an ideal world every issue opened would be resolved in some fashion, but that's no doubt not practical.
     
  41. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
    They have an automatic system that checks the report. I doubt that this automatic detection can find out whether it is a good quality report or not.
     
  42. shkar-noori

    shkar-noori

    Joined:
    Jun 10, 2013
    Posts:
    833
    Same here..
     
  43. SpiriTx

    SpiriTx

    Graphics QA Unity Technologies

    Joined:
    Apr 12, 2012
    Posts:
    252
    That's not exactly the full story. Yes, automated rating affects the choice of cases to be looked, but there are other factors like title, area or a version of the case.

    I myself, for example, often first opened cases with meaningful titles and it worked great for me as it helped me pick actually high quality reports. I also had a practice to check all of recent user's reports if the current one I processed was good, because it usually led to other bugs.

    Overall, I can't claim that our system is anything near perfect, but we went through different workflows (we even tried going through all cases, but it proved to be highly ineffective) to get here and I'm very open to any suggestions you might have to improve it.

    In any case, huge thanks for caring for Unity and voicing your concerns!
     
    DanSuperGP and shkar-noori like this.
unityunity