People who don't file good bugs
Gervase Markham
A researcher working with the Bugzilla database has recently sent me
some data on which Bugzilla participants have the worst bug filing
records. That is, it is a list of Bugzilla IDs, together with the number
of bugs they have filed which are now RESOLVED, and the percentage of
those which are RESOLVED FIXED.
There is one bug reporter, for example, who has filed 101 bugs, not a
single one of which has been RESOLVED FIXED.
https://bugzilla.mozilla.org/buglist.cgi?resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=DUPLICATE&resolution=WORKSFORME&resolution=INCOMPLETE&resolution=EXPIRED&resolution=MOVED&emailreporter1=1&emailtype1=exact&email1=info%40fortytwo.eu.org
However, having reviewed the top 5 "worst" bug reporters, none of them
seems obviously totally incompetent. So I am trying to decide what to do
with this information, if anything.
One possibility is to send anyone whose FIXED ratio is below e.g. 5% an
email, politely pointing them at some resources which might help them
file better bugs. I don't know if it'll help much, but I can't see much
of a downside.
What do you all think?
Gerv
Clint Talbert
> Hi QA community,
> However, having reviewed the top 5 "worst" bug reporters, none of them
> seems obviously totally incompetent. So I am trying to decide what to do
> with this information, if anything.
>
> One possibility is to send anyone whose FIXED ratio is below e.g. 5% an
> email, politely pointing them at some resources which might help them
> file better bugs. I don't know if it'll help much, but I can't see much
> of a downside.
>
> What do you all think?
I think this is a great idea. We have to make it polite and not put
people off. It's great that they are filing bugs, but perhaps they need
to learn ways to better search for dupes or to better file a specific
type of bug etc.
We have plenty of documents on QMO and elsewhere on the wiki that has
"how to file a good bug" topics pretty well covered, so perhaps pointing
them at that will help.
Conversely, can this research help us to determine which community folks
are doing really great bug filing? Then we could invite those people to
get more involved with either test days or with our ongoing bug triage
projects?
Thanks,
Clint
Anthony Hughes
Thanks,
Anthony Hughes (:ashughes)
MozQA Intern
Hi QA community,
Gerv
_______________________________________________
dev-quality mailing list
dev-q...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-quality
Gervase Markham
> I think this is a great idea. We have to make it polite and not put
> people off.
Right.
> We have plenty of documents on QMO and elsewhere on the wiki that has
> "how to file a good bug" topics pretty well covered, so perhaps pointing
> them at that will help.
Some pointers, or even (if you feel inspired) a draft of the email would
be great!
> Conversely, can this research help us to determine which community folks
> are doing really great bug filing? Then we could invite those people to
> get more involved with either test days or with our ongoing bug triage
> projects?
Now that's a very interesting idea. We can certainly determine who is
doing good work, but how do we programatically determine that they
"aren't as involved as they could be"? :-)
Gerv
Gervase Markham
> I'm not sure I understand the correlation between FILED vs RESOLVED
> FIXED being an accurate metric for poorly written bug reports.
> Please explain.
"Poorly written" is not necessarily what I mean. My point is that if
someone has managed to file 101 bugs and not a single one of them has
ended up FIXED, then (whether they have good intentions or bad) the
amount of triage effort that people have had to spend on those bugs
means that the person has not been a net asset to the project, but a
drain. So we need to either tell them to stop filing bugs and leave us
alone, or we need to try and help them file better bugs - fewer
duplicates, for example, or get a better understanding of what is a bug
and what will end up INVALID, or make sure they test on a latest build
so their bugs don't end up as WORKSFORME.
Gerv
Boris Zbarsky
> "Poorly written" is not necessarily what I mean. My point is that if
> someone has managed to file 101 bugs and not a single one of them has
> ended up FIXED, then (whether they have good intentions or bad) the
> amount of triage effort that people have had to spend on those bugs
> means that the person has not been a net asset to the project, but a
> drain. So we need to either tell them to stop filing bugs and leave us
> alone, or we need to try and help them file better bugs - fewer
> duplicates, for example, or get a better understanding of what is a bug
> and what will end up INVALID, or make sure they test on a latest build
> so their bugs don't end up as WORKSFORME.
Or file bugs on different components, if they were focused on some
component where the module owner tended to wontfix stuff a lot?
-Boris
Ludovic Hirlimann
> On 9/3/09 3:08 AM, Gervase Markham wrote:
>> Hi QA community,
>
>> However, having reviewed the top 5 "worst" bug reporters, none of them
>> seems obviously totally incompetent. So I am trying to decide what to do
>> with this information, if anything.
[snip]
> We have plenty of documents on QMO and elsewhere on the wiki that has
> "how to file a good bug" topics pretty well covered, so perhaps pointing
> them at that will help.
How about we also ask them what is wrong in our documentation ?
Did they read it ? If not, was it because they didn't know it existed,
or is it because they couldn't find it ?
ludo
--
Ludovic Hirlimann MozillaMessaging QA lead
http://www.spreadthunderbird.com/aff/79/2
Ludovic Hirlimann
What's the ratio of bug resolved vs bugs still open for those users ?
Ludo
Nickolay Ponomarev
> However, having reviewed the top 5 "worst" bug reporters, none of them
> seems obviously totally incompetent. So I am trying to decide what to do
> with this information, if anything.
>
> One possibility is to send anyone whose FIXED ratio is below e.g. 5% an
> email, politely pointing them at some resources which might help them file
> better bugs. I don't know if it'll help much, but I can't see much of a
> downside.
>
>
We should probably not contact people who are not likely to file another bug
(e.g. inactive in the last year).
How many "active" reporters with non-trivial (>3 ?) number of bugs reported
and low FIXED rate? What portion of the total bug number did they file? This
exercise might not be worth the effort if most useless bugs are filed by new
people... Or perhaps reaching out can be done manually.
And as other people in this thread commented, due to the imperfections of
the triage/fixing process, it might be better to look at the percentage of
"definitely worthless" (INVA, INCO, DUP) bugs, rather than FIXED bugs in the
# total bugs reported.
Nickolay
Mike Shaver
> "Poorly written" is not necessarily what I mean. My point is that if someone
> has managed to file 101 bugs and not a single one of them has ended up
> FIXED, then (whether they have good intentions or bad) the amount of triage
> effort that people have had to spend on those bugs means that the person has
> not been a net asset to the project, but a drain. So we need to either tell
> them to stop filing bugs and leave us alone, or we need to try and help them
> file better bugs
Or we need to improve our tools and processes so that people who are
obviously motivated to help are able to find related bugs, easily add
their test case or STR to it, track its progress, and help deal with
them when they're finished. I have to spend a lot of time figuring
out where to file many of the bugs I report; it impacts how helpful I
am still feeling when I get to the filing-place, how much of the
problem I still remember, and indeed whether I file the bug at all.
It's possible that many of our contributors do not have as much
experience as I do with the project, or as much incentive to improve
the product.
Mike
Michael Lefevre
Sounds like a reasonable idea. I do note that the one reporter you
mentioned was given various instructions at various times, including
links to https://developer.mozilla.org/en/Bug_writing_guidelines
I must say that it's not entirely the reporters fault on lots of those
bugs - the triage of mailnews/suite bugs was not up to much at the time
of many of the bugs. Some of the WFM bugs are because the bug wasn't
looked at until someone added an "is this still a problem?" comment
months or years after the bug was filed - so they should really have
been marked as dupes, and in some cases duped to the later bug where the
fix was made.
Lack of response is certainly an issue with many bugs/reporters, and if
they are not following responses then there is also not much hope that
any of their future reports will get better. Could there be more
emphasis in the reporting process (or documentation, although I rather
doubt many people ready it) on the fact that the process often does not
end when you submit the report?
If you're going to work on a FIXED ratio, I think it should probably be
based on bugs filed in the past year, or some shorter period than the
existence of bugzilla. There's no point mailing people who filed a lot
of bad bugs 6 years ago and have only filed a couple of good ones in the
past year.
And I guess there should also be some indication that someone is filing
multiple bugs. Based on 5%, if filing 1 good bug means you're allowed 19
duds, then shouldn't new filers be allowed 19 duds before they get to
the good one? Maybe only mail people who have filed at least 5 bugs as
well as using the ratio?
It's also worth noting that bug resolutions aren't an ideal measure. If
the answer is "this actually works but the documentation is wrong, and
there's a bug about the documentation", then it's a bit harsh to mark
the bug WFM. Also "you're reporting one of two bugs but I'm not sure
which, but they're both fixed" shouldn't really be WFM. (I'm not saying
anyone should waste excessive time to improve correctness of resolutions...)
Michael
Michael Lefevre
> "Poorly written" is not necessarily what I mean.
I think it probably should be. :)
> So we need to either tell them to stop filing bugs and leave us
> alone
Seems pretty pointless - I think the number of bugs that don't get fixed
filed by the same people is tiny in proportion to the number of bugs
filed by people who file one bug and disappear. I think this would
mainly achieve starting arguments and causing bad feeling. It's also
impossible to stop them - if you close a bugzilla account they can open
another, if you get them off bugzilla they can go to newsgroups, email, etc.
> or we need to try and help them file better bugs - fewer
> duplicates, for example, or
That would be good, but I find it pretty hard to help people with
finding fewer duplicates. It seems our best effort is
https://developer.mozilla.org/en/Screening_duplicate_bugs - and, with
all due respect to the authors, some of it is out of date, there are
broken links (Netscape newsgroups?), but most obviously, it's 2700 words
long. A better way to help people would be to improve the tools...
> get a better understanding of what is a bug
> and what will end up INVALID
This is a tricky one - what will end up as INVALID/WFM depends on the
developers (and to some extent QA) in whichever product/component. I'm
sure I could file a valid bug and have it fixed quickly if I dived in
and did some testing of video support or tracemonkey. On the other
hand, I know if I file a bug about a more obscure aspect of
Thunderbird's newsgroup handling, it may well sit around for a year or
two and eventually become WFM or INVALID because the feature I reported
a bug against has been removed, rewritten or replaced. If I think I
have a genius idea for fixing some Firefox UI, I have much less ability
to predict whether the UI folks will agree it is brilliant or think it's
pointless and mark it INVALID. I'm sure if only developers were allowed
to file bugs on their own code, the proportion of fixed bugs would shoot
up, but I don't think that's the right goal...
Michael
Martin Olsson
> How about we also ask them what is wrong in our documentation ?
> Did they read it ? If not, was it because they didn't know it existed,
> or is it because they couldn't find it ?
Many bugs that never gets fixed have insufficient actionable information
in them (no repro steps, no crash dump etc). In Ubuntu they started to
very aggressively mark such bugs as INCOMPLETE which makes the bug
automatically close within 30 days if no additional info is provided.
The most important thing is that someone sets such bugs into this
INCOMPLETE state fast enough so that the bug reporters get some "feedback"
on his/her bug reporting. This way people are given a chance to learn
what is needed in a good bug report.
Similarly, whenever a bug is closed as a duplicate there should be a stock
comment included that says "your bug report was already reported, to avoid
wasting mozilla resources please try to search the bug database before
filing a new bug. thanks".
The idea that people will, on their own initiative, go look for and read some
sort of "manual" or documentation is naive, I bet very few people would do this.
Martin
Boris Zbarsky
> Similarly, whenever a bug is closed as a duplicate there should be a stock
> comment included that says "your bug report was already reported, to avoid
> wasting mozilla resources please try to search the bug database before
> filing a new bug. thanks".
This should only be the case if:
1) The default search presented to new users might have found the bug
this one is a duplicate of (not true in many cases).
2) The bug this bug is being marked a duplicate of was filed earlier
(often not true either).
-Boris
Jack
> In Ubuntu they started to very aggressively mark such bugs as
INCOMPLETE which makes the bug
> automatically close within 30 days if no additional info is provided.
> The most important thing is that someone sets such bugs into this
> INCOMPLETE state fast enough so that the bug reporters get some
"feedback"
> on his/her bug reporting. This way people are given a chance to learn
> what is needed in a good bug report.
I agree with you. I have also some bugs open were I never got any
answer, therefore I dont know if I have not added enough information, or
if this bug is completly wrong.
I would like to see a statistic with how many bugs are open where only
the reporter added comments but no one else. Maybe addressing these bugs
would also be a first step.
cheers,
Jack
Henrik Skupin
> I agree with you. I have also some bugs open were I never got any
> answer, therefore I dont know if I have not added enough information, or
> if this bug is completly wrong.
It really depends on the component the bug is filed against. As we know
Firefox/General is a bad idea. There are not so many people who watch
this component or work through those unconfirmed bugs in a regular
basis. But also other components aren't covered well enough. I have
taken ownership for search, menus, toolbars, and tabbed browsing now and
constantly work on those unconfirmed. But having a component with more
than 400 open bugs is a hard work.
Getting components cleaned-up from unconfirmed is a goal we have for a
long time but it's not so easy as you can see.
If you can point us to one of your old bugs which haven't gotten
attention until now we could help and check why no-one have seen them.
> I would like to see a statistic with how many bugs are open where only
> the reporter added comments but no one else. Maybe addressing these bugs
> would also be a first step.
No idea if this can be done via the Bugzilla web interface. But would be
interesting thought.
--
Henrik Skupin
QA Execution Engineer
Mozilla Corporation
Jack
> But having a component with more than 400 open bugs is a hard work.
> Getting components cleaned-up from unconfirmed is a goal we have for a
> long time but it's not so easy as you can see.
I know that addressing a bug takes a lot of time.
> If you can point us to one of your old bugs which haven't gotten
> attention until now we could help and check why no-one have seen them.
The problem is that I lost my whole history/data because of a complete
computer crash a few months ago, and I dont know the dummy email adress
which I used for the bugzilla account anymore :(
As far as I remember correctly had I opened these bugs a few years ago
(FF1.x) and I never got any kind of feedback.
I reported a few bugs in the UI in internal version of firefox.
Well anyway I stopped now reporting bugs (because of no feedback at all
) except when FF is crashing (but sometimes not even the crash reporter
comes up so that I can report a crash :( )
cu,
Jack.
Henrik Skupin
> The problem is that I lost my whole history/data because of a complete
> computer crash a few months ago, and I dont know the dummy email adress
> which I used for the bugzilla account anymore :(
>
> As far as I remember correctly had I opened these bugs a few years ago
> (FF1.x) and I never got any kind of feedback.
> I reported a few bugs in the UI in internal version of firefox.
If you could remember a bug summary or the topic it should be easy to
find your email address and all the other bugs.
> Well anyway I stopped now reporting bugs (because of no feedback at all
> ) except when FF is crashing (but sometimes not even the crash reporter
> comes up so that I can report a crash :( )
I hope we can get some light into your bug history, will be able to
identify why they haven't gotten any attention, and to encourage you to
file bugs again.
Gervase Markham
> We should probably not contact people who are not likely to file another bug
> (e.g. inactive in the last year).
Yes, that makes sense.
> And as other people in this thread commented, due to the imperfections of
> the triage/fixing process, it might be better to look at the percentage of
> "definitely worthless" (INVA, INCO, DUP) bugs, rather than FIXED bugs in the
> # total bugs reported.
Even those bugs are not definitely worthless; I marked some of my own
bugs INVALID the other day because the task they represented was being
done elsewhere. It's always going to be a fuzzy metric - which is why we
aren't doing things like telling the 40% people they need to make it
above 51%, or something. We're just working at the extremes - and
tweaking the metric isn't going to make too much difference out there.
Gerv
Gervase Markham
> What's the ratio of bug resolved vs bugs still open for those users ?
They are normally long-term contributors, so 70% to 100% of their bugs
are resolved. Why do you ask?
Gerv
Robert Kaiser
Right, we need to be careful with calling INVA/INCO/DUP bugs "definitely
worthless", esp. INVA might even have been a valid report at some time
but invalidated by throwing away the code it was reported on - possibly
it might even have been thrown away and replaced with other code because
of a number of bugs reported against the old implementation, that
now-INVA one being one of them. Also, DUP are sometimes valid reports
that just have the same root cause as something else and so they are
duped to the bug solving the root cause.
In both cases, it was valid to report this, and with having dupes in the
default search queries, they can be valuable in finding if your problem
has actually been reported.
So, there are cases where those bugs are definitely not completely
worthless...
Robert Kaiser
Gervase Markham
> One possibility is to send anyone whose FIXED ratio is below e.g. 5% an
> email, politely pointing them at some resources which might help them
> file better bugs. I don't know if it'll help much, but I can't see much
> of a downside.
So, summing up, it seems that most people think this is a good idea as
long as:
- We only email still active people - e.g. people who have filed at
least 3 bugs in the past year
- We say something polite, and point them at resources which might help
them, rather than being rude.
- We realise that this isn't going to solve all our problems overnight,
but we hope it might have some effect on bug quality for a fairly small
amount of effort.
If that seems right, I'll get Diederik to produce a list of only the
active ones.
I'd appreciate pointers to web resources we could mention in the email.
Which are the best beginning QA docs out there?
Gerv
Tyler Downer
> On 03/09/09 11:08, Gervase Markham wrote:
>
> > One possibility is to send anyone whose FIXED ratio is below e.g. 5% an
> > email, politely pointing them at some resources which might help them
> > file better bugs. I don't know if it'll help much, but I can't see much
> > of a downside.
>
> So, summing up, it seems that most people think this is a good idea as
> long as:
>
> - We only email still active people - e.g. people who have filed at
> least 3 bugs in the past year
>
> - We say something polite, and point them at resources which might help
> them, rather than being rude.
>
> - We realise that this isn't going to solve all our problems overnight,
> but we hope it might have some effect on bug quality for a fairly small
> amount of effort.
>
> If that seems right, I'll get Diederik to produce a list of only the
> active ones.
>
> I'd appreciate pointers to web resources we could mention in the email.
> Which are the best beginning QA docs out there?
to write a bug, covering grammar, steps to reproduce, etc. Hopefully
that will be up soon. But, for now, there is http://quality.mozilla.org/bug-writing-guidelines
Tyler
>
> Gerv
Nickolay Ponomarev
> One possibility is to send anyone whose FIXED ratio is below e.g. 5% an
> email, politely pointing them at some resources which might help them file
> better bugs. I don't know if it'll help much, but I can't see much of a
> downside.
>
So maybe I'm blind, but I can't see any "Bug writing guidelines" doc linked
from
https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox&format=guided (it
*is* linked from the non-guided page). Adding a link there might be a good
start...
Nickolay
Michael Lefevre
> So, summing up, it seems that most people think this is a good idea as
> long as:
>
> - We only email still active people - e.g. people who have filed at
> least 3 bugs in the past year
>
> - We say something polite, and point them at resources which might help
> them, rather than being rude.
>
> - We realise that this isn't going to solve all our problems overnight,
> but we hope it might have some effect on bug quality for a fairly small
> amount of effort.
I'd add: as long as it's somehow acknowledged (which can be done subtly)
in the email that the way that the recipients of the email was
determined isn't perfect (i.e. that the email might be wrong...)
> I'd appreciate pointers to web resources we could mention in the email.
> Which are the best beginning QA docs out there?
This also seems to be a bit of stumbling point. The link already posted
to http://quality.mozilla.org/bug-writing-guidelines is something, but
there is a lot of stuff there, but still some important stuff missing
(which is covered in part 1
http://quality.mozilla.org/documents-home/bugs-docs/bug-writing-guidelines
).
Those docs in turn link to even lengthier stuff. The "handy guide" to
see if there is a "duplicate bug already created" is 2800 words. That's
not "handy" (which is why it's called "screening duplicate bugs" and is
hosted on Mozilla Developer Center - it also has out of date references
to the Mozilla Suite and other historic stuff...)
Of course, there's also
https://bugzilla.mozilla.org/page.cgi?id=bug-writing.html , but that's
not brilliant either.
What's needed is a short and simple document aimed at bug filers (as
opposed to QA volunteers triaging the bugs of others) which contains
everything within it rather than linking out to huge documents. Would
be nice if that existed before the email was sent out.
Michael
Aakash Desai
The current bugzilla write-a-bug newbie form that it has is pretty dang sufficient for a casual person that has no idea what a user agent is, what component their bug might or might not belong to, how to write a proper summary for triagers who try to decipher these bugs and re-write them in a more meaningful way. The documents on quality.mozilla.org are meant to allow those who are beginning to get accustomed to the bug management process in bugzilla to have the tools (in writing) available to not be at a disadvantage when writing bugs in the future. The purpose is to the lay out the information in a way that is structured and useful, but while not completely disregarding a learning curve.
Whether you like to admit it or not, a bug management system that has over 500,000 bugs filed is going to have some sort of learning curve and cannot be incredibly simple to use. There's no natural connection to such things.
I'd suggest to think about what a good implementation of this would look like in terms of the audience of your documentation and what this doc aims to offer that the audience may be deemed to benefit from, while also paying respect to the intricacies of the system that's being used. It's definitely not as simple as trashing the things that we currently have and I hope you please stop from doing so in the future.
Thanks,
Aakash Desai
Michael
Michael Lefevre
> Michael, I'd suggest you try writing a bug writing tutorial for yourself and try to find a way to write something that makes sense to people that might be new to bugzilla, mozilla AND QA. I can tell you from experience there's some things that are just impossible to make into a 100 word documents without really losing the real impact on it.
>
> The current bugzilla write-a-bug newbie form that it has is pretty dang sufficient for a casual person that has no idea what a user agent is, what component their bug might or might not belong to, how to write a proper summary for triagers who try to decipher these bugs and re-write them in a more meaningful way. The documents on quality.mozilla.org are meant to allow those who are beginning to get accustomed to the bug management process in bugzilla to have the tools (in writing) available to not be at a disadvantage when writing bugs in the future. The purpose is to the lay out the information in a way that is structured and useful, but while not completely disregarding a learning curve.
>
> Whether you like to admit it or not, a bug management system that has over 500,000 bugs filed is going to have some sort of learning curve and cannot be incredibly simple to use. There's no natural connection to such things.
>
> I'd suggest to think about what a good implementation of this would look like in terms of the audience of your documentation and what this doc aims to offer that the audience may be deemed to benefit from, while also paying respect to the intricacies of the system that's being used. It's definitely not as simple as trashing the things that we currently have and I hope you please stop from doing so in the future.
Ok, sorry for "trashing" the existing documents. I did not mean to say
that the content of those documents is not useful for some audiences -
what I am questioning is whether it is useful for the audience that Gerv
is proposing to send emails to (given your comments, do you agree that
it is not?).
If there can be no document that is reasonably short and will help these
"people who don't file good bugs", then I have doubts if there is any
point in sending out a mass email to them.
Michael
Clint Talbert
> On 14/09/2009 21:50, Aakash Desai wrote:
>> Whether you like to admit it or not, a bug management system that has
>> over 500,000 bugs filed is going to have some sort of learning curve
>> and cannot be incredibly simple to use. There's no natural connection
>> to such things.
>>
app. If it only had a few hundred bugs in it, it would still be a
difficult app. The arcane ways that we have hacked on to the underlying
app to deal with this volume of stuff certainly haven't improved things,
but we are working with an application skewed to fairly advanced users
already.
>
> Ok, sorry for "trashing" the existing documents. I did not mean to say
> that the content of those documents is not useful for some audiences -
> what I am questioning is whether it is useful for the audience that Gerv
> is proposing to send emails to (given your comments, do you agree that
> it is not?).
versions of these documents.
>
> If there can be no document that is reasonably short and will help these
> "people who don't file good bugs", then I have doubts if there is any
> point in sending out a mass email to them.
the tone of this discussion, here is my attempt, weighing in at 1200
words: https://wiki.mozilla.org/User:Ctalbert:Filing_A_Bug
I think it must be less than 1000 words. Ideally, less than 800.
You and everyone else who is interested are welcome and encouraged to
mark up, copy, modify, and otherwise mash up my text.
Also, I have an idea in the text for an extension. Any takers?
I look forward to seeing some under-1000 word versions on this topic.
"You have your mission, should you choose to accept it." :)
Have Fun.
Clint
Gervase Markham
> Actually, I am working with Dave to make a very simple doc on who not
> to write a bug, covering grammar, steps to reproduce, etc. Hopefully
> that will be up soon. But, for now, there is
> http://quality.mozilla.org/bug-writing-guidelines
Without being rude, why do we need yet another doc like that? Can't we
improve the existing ones instead?
In fact, the Bug Writing Guidelines from a few years ago, which were
written by a usability expert and extensively field-tested, might be
better than the ones we have today. We should find the history of that
file and look at the last version edited by Eli Goldberg.
Gerv
Mike Shaver
> In fact, the Bug Writing Guidelines from a few years ago, which were written
> by a usability expert and extensively field-tested, might be better than the
> ones we have today. We should find the history of that file and look at the
> last version edited by Eli Goldberg.
I have many questions, since I didn't know the history of this Bug
Writing Guidelines document!
Where are the results of those tests? Were they tested against the
patterns and configuration (including massive component-count
overkill) of our bugzilla, or against a simpler one? Who were the
users? (Is the person a web-site usability expert, or a software
process or getting-useful-bugs-from-people expert?)
Mike
Justin Dolske
> usability expert
Now there's a title that should set off alarm bells. :-P
Justin
Gervase Markham
> Where are the results of those tests? Were they tested against the
> patterns and configuration (including massive component-count
> overkill) of our bugzilla, or against a simpler one? Who were the
> users? (Is the person a web-site usability expert, or a software
> process or getting-useful-bugs-from-people expert?)
Eli was (and presumably is; I think he's at Microsoft now) a QA engineer
and usability buff. I think he's a getting-useful-bugs-from-people
expert if ever there was. He was one of those instrumental in building
the QA community in the early days. In an email to me a few months ago
he said:
"I do have to say I am very disappointed by the gutting of the default
Bugzilla distribution bug writing guidelines that I worked quite hard
(with a principal-level tech writer) two years ago to shorten and improve.
I don't think the revised document solves the end-user problems that I
originally wrote it to solve, and I don't see landscape changes that
render those problems any less relevant."
Hence my suggestion that we at least revisit his version to see if it's
any good. Here it is, albeit in template form (I changed the content
type so it displays a bit better):
https://bug352165.bugzilla.mozilla.org/attachment.cgi?id=239278
Gerv
Heather Arthur
> And I agree on that point wholeheartedly. So in an attempt to change the
> tone of this discussion, here is my attempt, weighing in at 1200 words:
> https://wiki.mozilla.org/User:Ctalbert:Filing_A_Bug
>
>
I find Clint's version to be the most comforting and direct for regular
users with limited technical experience. It doesn't overwhelm the user with
steps, technical jargon, or external links. It would be great if we could
get them to file it in the correct component or find out if it's a memory
leak, but that could be done when we follow up with them. Confronting them
with all that stuff initially will lessen the chances that they file a bug
in the first place and will lessen the chances that they read the most
important instructions thoroughly and digest them. Also, mentioning what
happens next is essential. It's a good 'first-landing' place for bug
writing. My 2 cents.
harthur
Henrik Skupin
> leak, but that could be done when we follow up with them. Confronting them
> with all that stuff initially will lessen the chances that they file a bug
> in the first place and will lessen the chances that they read the most
> important instructions thoroughly and digest them. Also, mentioning what
Sorry for drifting a bit off-topic now but while reading this comment
from Heather I remembered a new feature of QAC which checks for already
existing bugs when entering the summary. I believe it would be really
helpful if we could encourage the Bugzilla developers to add a
autocomplete feature for the summery which returns possible matches to
existing bugs. That will probably also lessen the number of filed dupes
or invalid bugs.
Smokey Ardisson
> On Tue, Sep 15, 2009 at 1:44 AM, Clint Talbert <ctalb...@mozilla.com> wrote:
> > And I agree on that point wholeheartedly. So in an attempt to change the
> > tone of this discussion, here is my attempt, weighing in at 1200 words:
> >https://wiki.mozilla.org/User:Ctalbert:Filing_A_Bug
>
> I find Clint's version to be the most comforting and direct for regular
> users with limited technical experience. It doesn't overwhelm the user with
> steps, technical jargon, or external links.
I, too, really like the feel of Clint's document. It's really longer
than I'd like, but I can't think of anything obvious in there to cut--
and there are a few things I'd suggest adding :-) I think a bit more
length is the price you have to pay for removing dense jargon in favor
of user-friendly language and for making steps really clear for non-
technical users.
It's hard. Camino's document tries to go that route, too, although it
covers a lot less ground (and there are still lots of problems with
it; I have an entire page of notes of things to fix/change):
http://caminobrowser.org/documentation/bugzilla/
I really like how Clint's document tries to redirect the support
requests at the very beginning, too.
Specific issues:
1) The parts I wrestle with the most are the where to send the bug
bits. How strong/caught-up/attentive is the triage effort in various
individual Core components vs. the various application components? Is
a bug a user mis-files in Core component Q more likely to be noticed
and moved to the right place than a Core bug mis-filed in Firefox?
My personal thought is that it's better to mis-file a Core bug in an
app component and let app triagers make a better guess (or ask someone
who knows) when moving, but I freely admit that's influenced by the
fact we can handle the volume of bug reports in Camino and that the
reports are skewing more and more to very, very non-technical people
(searching for dupes is a difficult task for may, and trying to have
them guess between Camino and various Core components would be a
confusing, frustrating experience). I really don't have a feel for
the state of triage in Firefox and Core other than "we're behind" and
that no-one is actively triaging Core:General.
2) In Camino, we've found in many cases an (annotated) screenshot is a
useful bit of additional data, especially for issues related to non-
English websites, where the bug may not be obvious to us even with the
reporter's description. Not sure exactly how one might work
screenshots into Clint's "Marshall Your Data" and "Use Those Notes!"
sections, but that is something to consider suggesting for certain
types of issues. If you did include that, you'd probably want to have
a SUMO article that explained how to take screenshots on various
platforms.
3) I'd add about two more half-sentences to the existing text in
"What's Next" in order to explicitly state two things that are
currently only implied:
First, explain that although you'll get the comments by email, you
need to visit your bug in Bugzilla to post replies. (Hmm, Bugzilla
enh request?: don't show email addresses in bugmail, so that users
aren't tempted to reply to the commenter when replying to bugzilla-
daemon fails).
Second, add a half-sentence asking the reporters to please be
attentive to the emails/comments/requests for more information that
they do receive and to reply promptly.
4) Somewhere in the text I think it's important to emphasize "one bug/
issue per report"; that's the only significant guideline from
https://developer.mozilla.org/en/Bug_writing_guidelines that I think
Clint's draft is missing.
5) TODO #2: about:crashes is part of Toolkit, so any Moz app that
implements the crash reporter should be able to access the page,
browsers via typing the URI in their location bar, and other types of
apps via a custom button or menu item somewhere that would open the
page. philor tells me Tb currently recommends installing the "View
About" extension until they get some access point wired up in their
UI.
Smokey
--
Smokey Ardisson
Co-Lead
Triage/QA and Website & Documentation
The Camino Project
Clint Talbert
> On Sep 17, 6:28 pm, Heather Arthur<fayeart...@gmail.com> wrote:
>> On Tue, Sep 15, 2009 at 1:44 AM, Clint Talbert<ctalb...@mozilla.com> wrote:
>>> And I agree on that point wholeheartedly. So in an attempt to change the
>>> tone of this discussion, here is my attempt, weighing in at 1200 words:
>>> https://wiki.mozilla.org/User:Ctalbert:Filing_A_Bug
>>
>> I find Clint's version to be the most comforting and direct for regular
>> users with limited technical experience. It doesn't overwhelm the user with
>> steps, technical jargon, or external links.
>
> I, too, really like the feel of Clint's document. It's really longer
> than I'd like, but I can't think of anything obvious in there to cut--
> and there are a few things I'd suggest adding :-) I think a bit more
> length is the price you have to pay for removing dense jargon in favor
> of user-friendly language and for making steps really clear for non-
> technical users.
>
one, but instead it is gaining supporters. I'll take a look at it and
see if I can find something to remove.
> It's hard. Camino's document tries to go that route, too, although it
> covers a lot less ground (and there are still lots of problems with
> it; I have an entire page of notes of things to fix/change):
> http://caminobrowser.org/documentation/bugzilla/
>
I like your document. I'd never seen that one before. It is certainly
a difficult tightrope to walk between what to include and what not to
include.
>
> Specific issues:
>
> 1) The parts I wrestle with the most are the where to send the bug
> bits. How strong/caught-up/attentive is the triage effort in various
> individual Core components vs. the various application components? Is
> a bug a user mis-files in Core component Q more likely to be noticed
> and moved to the right place than a Core bug mis-filed in Firefox?
>
> My personal thought is that it's better to mis-file a Core bug in an
> app component and let app triagers make a better guess (or ask someone
> who knows) when moving, but I freely admit that's influenced by the
> fact we can handle the volume of bug reports in Camino and that the
> reports are skewing more and more to very, very non-technical people
> (searching for dupes is a difficult task for may, and trying to have
> them guess between Camino and various Core components would be a
> confusing, frustrating experience). I really don't have a feel for
> the state of triage in Firefox and Core other than "we're behind" and
> that no-one is actively triaging Core:General.
I also tend to think this way. I wouldn't suggest people put anything
in core:general, but I do think that Core:<specific component> are
decently well triaged, at least the ones I watch tend to be (not through
any fault of my own, I should add). The problem is, how do you train a
new user to add their bug to a "watched" core component over an
"unwatched" or less watched core component. That's hard. like you, I'd
rather them err on the side of filing in Application space as that has
far more eyes on it and can get moved to core as needed. Perhaps I
should update the doc to reflect that?
>
> 2) In Camino, we've found in many cases an (annotated) screenshot is a
> useful bit of additional data, especially for issues related to non-
> English websites, where the bug may not be obvious to us even with the
> reporter's description. Not sure exactly how one might work
> screenshots into Clint's "Marshall Your Data" and "Use Those Notes!"
> sections, but that is something to consider suggesting for certain
> types of issues. If you did include that, you'd probably want to have
> a SUMO article that explained how to take screenshots on various
> platforms.
>
This is a good idea and came up in another conversation I was having
regarding this topic. I'll see what I can do.
> 3) I'd add about two more half-sentences to the existing text in
> "What's Next" in order to explicitly state two things that are
> currently only implied:
>
> First, explain that although you'll get the comments by email, you
> need to visit your bug in Bugzilla to post replies. (Hmm, Bugzilla
> enh request?: don't show email addresses in bugmail, so that users
> aren't tempted to reply to the commenter when replying to bugzilla-
> daemon fails).
>
> Second, add a half-sentence asking the reporters to please be
> attentive to the emails/comments/requests for more information that
> they do receive and to reply promptly.
>
Both good ideas. I'll work them in.
> 4) Somewhere in the text I think it's important to emphasize "one bug/
> issue per report"; that's the only significant guideline from
> https://developer.mozilla.org/en/Bug_writing_guidelines that I think
> Clint's draft is missing.
>
True, I'll try to add that.
> 5) TODO #2: about:crashes is part of Toolkit, so any Moz app that
> implements the crash reporter should be able to access the page,
> browsers via typing the URI in their location bar, and other types of
> apps via a custom button or menu item somewhere that would open the
> page. philor tells me Tb currently recommends installing the "View
> About" extension until they get some access point wired up in their
> UI.
>
This is the kind of "if you need it, click on it" sort of detail that
I'd like to have in a link. Perhaps this could also be a SUMO article
that I could link to? To keep the text light, I'd like to put details
like this into other documents.
I'll try my hand at the changes and post with a new draft.
Thanks for the feedback, Smokey.
Clint