FogBugz feedback loop on reported bugs?

In the past few months like many of you I've had the opportunity to try out Unity 4.6 Beta and as a pre-order customer Unity 5.0 Beta. Love the new game GUI API in 4.6 btw. During this time I've reported many bugs. I'm sure they are of various levels of quality w.r.t. data captured, repro' steps etc. BUT in that time I've heard nothing from the Unity Team beyond an automated response from the FogBugz system.

A couple of proposals. My 2c.

1) Some kind of non-automated, bug triage, response from Unity QA.
It would be simply awesome to see some kind of 'triage' message from QA for these issues. Something as simple as, 'we're looking at this', 'forget it!', 'not enough information'. Granted in nearly 3 months it looks like the cases have increased by approx. 18,000 (assuming the case numbers are sequential) so perhaps that's not practical. Are customers using FogBugz as a form of technical support? 18,000 new issues in the past 3 months is A LOT. Certainly I'm guessing a substantial number are duplicates, cruft, already fixed, not reproducible.

2) Ability to edit/annotate issues post submission.
Any way a customer who reported a bug could be able to edit and annotate the issue? Some of the bugs below got 'corrupted' titles and I'd love to improve the message.

Thanks for listening!

CaseTitleStatusDate Opened
646862Baking occlusion data and enabling it for occlusion cullingOpen11/10/2014 9:29 PM
646645Unity installer does not install the Microsoft Visual C++Open11/10/2014 3:20 AM
645661Editor during GI Iterative Bake will show many errors,Open11/5/2014 10:49 PM
645037Crashesd while rendering scene in Editor. FWIW I hadn't seenOpen11/3/2014 8:39 PM
644394- Unity 5.0 GI light map baking does not respect a light'sOpen10/31/2014 10:49 PM
629614-e Typo. Should be 'masked' and not 'masedOpen8/29/2014 12:30 AM
629580Upon load of the project the Unity Console reports 4 errorsOpen8/28/2014 8:39 PM
628098Names of Axes for Standalone Input Module (Script) for 'Open8/23/2014 1:59 AM
628055-e 4.6.0b17 BetaOpen8/22/2014 10:39 PM
628044-e 4.6.0b17 BetaOpen8/22/2014 9:29 PM
628042-e 4.6.0b17 BetaOpen8/22/2014 9:10 PM

 

Second that. I reported various bugs (at lest 10) in the last two years, and got replies for only TWO of them. All the others are still marked as "open" on fogbugz, and that's it. At least marking them as fixed/won't fix/working on.. would be nice.

 

Totally agree with you guys. Not been pleased with the Unity QA for some time. I check the board each day and if you really want a bug fixed you just got keep pushing and pushing and hope that one the devs sees it from your view as a important issue and they then look into it personally.

Thank you developers for the support on here, I think its great especially when you get involved in specific cases although your not QA sometimes its about being proud of your work and really helping others.

Cant say much about the QA as all the run ins I had with them in forum have been negative.

 

I've reported bugs too, but I've received manual responses from the QA Team, so they pretty much are responsive for me. I've had several responses to the bugs which seem critical.

 

For sure with this one's mileage might vary. It's ok. I'm entering issues and the latest reported case was repro'd and fixed for b15. So awesome as I helped improve the product which is the point with this Beta. I think though, for what it's worth, that the feedback loop and process could be more consistent and tidied up.

 

I usually get responses on the next working day after I've submitted a bug: http://fogbugz.unity3d.com/default.asp?649965_jcdk2r2bljlhek6q

The only unanswered report there is the earliest one, when, I suspect, I forgot to save the demo scene before sending the report. I have always created a small repro project featuring the bug, and the latest reports also have a link to a small screen-cast video.

 

Yep, I agree that every issue reported should carry succinct repro' steps and a project that demonstrates the issue if feasible and I've strived to do that where appropriate. But my initial point is that in my experience, and yours too apparently, the response tends to be either 'we've reproduced this issue' or no response at all.

 

It is kind of a lonely feeling to send a bug and have it listed as "Open" for months. Half of my submitted bugs are still open even though they all contain a succinct repro that is 100% reproducible on my machine. If QA has failed to reproduce it on theirs, I'd like to know so I can help. I suspect in some cases, the bug has simply been set aside. I understand that they have to focus on critical bugs over noncritical bugs, but after a revision is released the bug is then even lower priority for belonging to an earlier version of Unity. I'd like to know once a bug report has been completely abandoned so that I can report it again if necessary.

 

Tons of those are not bug reports; you get case #s when uploading assets to the store, for example. I do get emails from Unity about bug reports, such as "this was reproducible, handed over to be fixed", though not all the time.

--Eric

 

It is certainly understandable that this whole process is very time consuming and my personal impression is that it is constantly being improved.

Being someone who always tries to create a minimal reproduction with a short but precise description, it is quite frustrating not to get any feedback at all. Of course, I am not asking for a personalized response, but getting an automated reply with the rating of the incident would already feel a lot better. If the bug could not be reproduced or if it was assigned to a developer (though it is not clear whether or when it might be resolved), it would help the person who submitted the bug to feel more comfortable.
At the moment the process of submitting a bug feels like a huge black box. Some of the bugs took me half a day, just to create a minimal reproduction. After several months without a reply it feels pretty weird, because I have no feedback at all. I don't even know whether it could reproduced or if anyone had a look at it. Getting an automated response whether you were able to reproduce it, whether it was assigned to a developer or even a response that it won't be fixed would help the submitter a lot.
When I submit a bug and take the time to make it right, I would like to get some kind of automated feedback, such that I know my time wasn't wasted. Whenever I submitted one or two bugs and got no feedback at all, I tend to not submit further bugs, because it feels like wasted time.

 

Hit the nail on the head..

 

I've been always wondering, since I read that article a while ago, what's the difference between "very little description", "small amount of description" and "good amount of description"?

Some bugs don't really need many words to describe, like

What's wrong: when I do this, that thing breaks.
To reproduce: open the demo scene, press Run, notice that this thing bounces (for examaple) instead of standing still.
http://screencast.com/abcdefg

Is this a very little amount, small amount or good amount?

 

Thanks for the reply. For me it is very obvious that there were lots of improvements in the previous years on your side. In my opinion the QA Team does a fantastic job!

Even if I am aware that someone checked all the issues that I submitted, it would motivate me a lot more to submit further bugs if I just got a small update like:

• We were able to reproduce the issue
• The priority of the issue was changed to ...
  
• The bug was assigned to a developer
• It is unclear whether the issue will be resolved
• The bug should be resolved in version ... that was just released

Getting just those little information snippets would make a huge difference for me and would motivate me a lot more, just because I could see that something is happening.

 

Good to hear. I just checked my bug reports. There is around 8 for which I never got a reply, still I took the time to make the description short and to create a small reproduction. My motivation to take any time to submit bug reports just... Well...
I am kind of speechless. Should I spend my time to complain here in the forum instead of creating a meaningful reproduction? Sounds like a dumb strategy, but it seems to work better then mine.

 

This is actually WORSE because it means at least 1/3 of the bugs I submitted during 2014 have never been looked at. Which also means they never make it into IssueTracker, so nobody can tell if they're submitting a duplicate bug or vote on fixes which are important for their projects.

 

So, you just ignore some of the reports? Ehm... how do you determine the rating of a case if it was never looked at?

 

Good point also they forgot to mention - any report without a repo does not get looked at , these words of the Qa manager himself in a thread i read month or two ago.

so like someone mentioned, something that does not need a repo is not looked at..simple things and reports without i think they put them buttom..Im just shocked at this way of working and of course i would make noise and make concern about it

 

Does this mean that e.g. a bug report, saying that an item in the editor's main menu has a typo, will be thrown away by the rating mechanims?

 

I think your not understanding me;

So far throughout the forums it is mentioned if no repo is attached, and by repo I mean a sample of the project the user is working on then it will not be looked at - I seen this mentioned by QA members in the forums.

I'm saying why would I need to include a repo or sample of my project if bug steps are like:

1) Log into the editor
2) Goto inspector
3) Check the Graphics Level dropbox
4) Missing Graphics "Automatic"
Make sure you have selected Android Build type

So please Sir tell me why would such issues not be looked at...
Lets all add fake samples and attachment so the system can see there is "some" file attached.

 

It proofs that it works well in the cases you were looking at. If I am trying to keep my description as short as possible, such that you are not wasting time, it is very likely that my report is not considered because it gets a low ranking. That just because I took the time to make it compact.

 

I reported a few documentation issues before the documentation section in this forum was in place. It seems no one ever had a look at it.

 

Okay that clears that up, btw there was no misunderstanding, I think the whole QA team need to be briefed on one story to tell users in the forums, seems right now there has been to much contradictions and therefore everyone has different thoughts on how things are done.

Also in the future it be good if you create a MINI QA team (2-3ppl) to deal with small bugs reports that do not require deep investigation work or hardcore repo project, like Dantus mentioned stuff such as documentation and making sure basics of the Editor, screen orientation issues, build settings etc.

 

I second this.
I've reported a bug in May about "Limit velocity over lifetime" module in Shuriken being fps dependent. It took me probably two days of debugging to realize why my particles in the play mode behaved differently from what I set them up in edit mode. And the reason was that the scene/game view in edit mode is vsynced, while in play mode it depends on your quality settings. So, if you turn vsync off, you particles would behave completely different from what you've set them up in editor. At the time, when I finally realized what is the source of the issue, and narrowed it down to the "Limit velocity over lifetime" module it felt like a real achievement. Fast forward 6 months. This issue in on the first page of the issue tracker, if I filter it to show active issues. In my understanding it's a trivial fix, yet the issue i still there, and not fixed even in Unity 5.

P.S. This rant is probably a bit offtopic, as fixing bugs is not QA job, so I think we need a small team of developers to fix "low hanging" bugs.

 

Statistics is by definition not a proof, it just shows a tendency.

In the current rating system people who take the time to submit compact reports are more likely not considered because of statistical reasons.

Edit: This is really not meant to be offensive. But for someone like me who consciously took the time to write short descriptions and create small reproductions, it is pretty irritating. I most likely wasted quite some time, because I wanted to make the job of the QA team easier. Unfortunately, I made it easier than expected, because some were likely never checked at all due to statistical reasons.

 

I guarantee more than 6.34% of the reports from people who care enough to post in a thread like this are valid bugs. Should I be resubmitting reproducible bugs that get no response? How long should I wait? A month? A number of beta updates? It sounds as if something that isn't looked at pretty quickly is never going to be looked at.

 

From my perspective I think the rating system may help separate the 'wheat from the chaff' in a chaotic bug-reporting environment that exists for general releases. In that scenario the rating system is differentiating between bug reports that have had some effort applied to filing them vs. little/less effort.

Though not all short description issues with no attachments should be poorly rated imho.

Take http://fogbugz.unity3d.com/default.asp?629614_ga21dnfaqvsgdhh4 that I found 2 months ago during 4.6 Beta. This concerns the typo 'mased' instead of 'masked' that exists on http://docs.unity3d.com/ScriptReference/UI.MaskableGraphic.html. The typo is a low priority item I'm sure but FWIW it still exists. The bug description is short. There's no need for repro' steps of any substantial length, and certainly not attachments. Therefore I suspect the bug was rated low and not looked at, and therefore not fixed.

Overall bug quality I expect would decrease from those reported by Unity QA, by closed Beta Participants, by open Beta Participants, and finally by general Product Usage.

The rating system http://blogs.unity3d.com/2013/10/28/bug-reports-incidents-and-some-bashing/ measures the quality of bugs for one of the populaces above broadly, but disregards the notion that a quality bug report is usually more a function of its bug reporter than the length of its description, number of attachments etc.

In summary, I would think that a better rating system would be that of rating the bug reporter(s) and not rating the bugs reported.

 

I filed a bug report about an error in error message. Does UT give achievements for things like that?

 

I think this time is definitely not wasted regardless of whether you get the feedback or not.

1. A small project that reproduces the bug guarantees that this is actually a bug in Unity and not in your main project.

2. When you create a repro project you investigate the bug more deeply. You may find that it reveals itself only if certain conditions are met, in other cases everything works fine. Or you may find that A is not the only thing that is broken, that B and C, related to A, also don't work as expected. Or you may realize that even though A is broken, B and C still work fine, and it gives you a temporary workaround until A is fixed.

3. Release notes don't usually cover all the changes that have happened in the recent version of Unity. If you have a simple repro project, you can use it to quickly check new versions of Unity. They may have this bug fixed because someone else reported it, or Unity guys found it by themselves, or it was a part of a bigger bug/issue that was fixed.

 

I agree with you in most of what you wrote.
My main issue that still exists is that even though I made reproductions, Unity did not even have a look at the bug reports due to the mentioned automatic filtering. After I wrote here and PM'd the information about the bugs, Unity checked them and I got a bunch of feedback. Even though I invested a lot of time to create them, they would have been ignored if I wouldn't have written in this thread.

 

I have reported 13 bugs since this thread was started. All of them are either fixed by now or confirmed. Right after a bug report has been sent I receive an automated reply, and then, usually on the next working day, I get a reply from the QA with the verdict. From my side, the system seems to work fine, or I'm just lucky.

 

I usually get responses pretty soon after reporting something. The latest one I reported is such a widely reported problem that I suspect it'll be an exception, though, because there probably are too many to reply to

But looking over my long list of reports there are still a lot of open cases. It's possible some of them can be closed, as they're mostly for old versions, but I can't see anything I can do about it.

 

You probably write good quality reports.

 

Certainly writing good quality bug reports ups your chances of having the bug looked at, and then reproduced ala the rating system http://blogs.unity3d.com/2013/10/28/bug-reports-incidents-and-some-bashing/. From my perspective so long as the issues, if important, are fixed and the quality of the final release is high then how the bug reports are handled I suppose is neither here nor there. In an ideal world every issue opened would be resolved in some fashion, but that's no doubt not practical.

 

They have an automatic system that checks the report. I doubt that this automatic detection can find out whether it is a good quality report or not.

 

Same here..