Looking for Firefox triagers to evaluate a triage tool
John Anvik
community specifically, and the Mozilla triager community in general. I
apologize if I'm in the wrong place.)
I am a PhD student at the University of British Columbia working on a
way to assist triagers. Specifically, I am looking at an approach to
recommend to whom a bug report should be assigned (i.e. which
developer(s) has/have the correct expertise for a particular bug report).
I have a working web-based tool called Sibyl for providing both
developer and component recommendations for bug reports. What I am
looking for is triagers to try it and give me feedback. Sibyl is found
at http://www.cs.ubc.ca/~janvik/sibyl. The page also has a short video
tutorial so you can get a feel for how Sibyl works and looks like.
Details about the evaluation are at
http://www.cs.ubc.ca/~janvik/sibyl-ad.htm, including that people who try
it out get a $20 gift certificate from Amazon.
Sibyl is currently set-up for to make recommendations for the Firefox
project, but if there is an interest from triagers of any other Mozilla
project, I am quite willing to set it up for them also.
Any questions,comments, or suggestions please contact me at
jan...@cs.ubc.ca.
(FYI: The Eclipse Platform project is also trying out Sibyl, so don't
think that if you tried Sibyl out you would be the only one. The more
feedback I can get the better for everyone, so please participate).
Dave Liebreich
> I am a PhD student at the University of British Columbia working on a
> way to assist triagers. Specifically, I am looking at an approach to
> recommend to whom a bug report should be assigned (i.e. which
> developer(s) has/have the correct expertise for a particular bug report).
Hmm. I thought we assigned bugs to product/component pairs rather than
to individual developers, and the developers "watched" the components to
get bug notifications.
If this is the case, and I think it is, how does your tool handle it?
Triagers with more experience than me (which is a lot of you), please
correct my understanding if it is wrong.
-Dave
--
Dave Liebreich
Test Architect, Mozilla Corporation
Martijn
> John Anvik wrote:
>
> > I am a PhD student at the University of British Columbia working on a
> > way to assist triagers. Specifically, I am looking at an approach to
> > recommend to whom a bug report should be assigned (i.e. which
> > developer(s) has/have the correct expertise for a particular bug report).
>
> Hmm. I thought we assigned bugs to product/component pairs rather than
> to individual developers, and the developers "watched" the components to
> get bug notifications.
I know some bugs added in some components (like core->style system to
dbaron and layout->view rendering to roc) are automatically assigned
to certain developers, but indeed, I think in most cases are assigned
to a product/component.
Regards,
Martijn
> If this is the case, and I think it is, how does your tool handle it?
>
> Triagers with more experience than me (which is a lot of you), please
> correct my understanding if it is wrong.
>
> -Dave
>
>
> --
> Dave Liebreich
> Test Architect, Mozilla Corporation
> _______________________________________________
> dev-quality mailing list
> dev-q...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-quality
>
John Anvik
product/component pair and let the developers self-assign. I have been
looking at developing a process for creating recommenders that
generalizes to other projects that don't necessarily follow that same model.
However, I still think that this tool is useful within the Mozilla project:
1) While the focus of the research is to recommend specific developers
within a project, Sibyl also recommends components for bug reports (and
the Eclipse people have already requested sub-component recommendation,
so that is in the works if that is of interest - just let me know).
Theoretically Sibyl can recommend products, the limitation is just
practical. To recommend products would require gathering data from
across the entire Mozilla repository and I think that is infeasible
without running software on the Mozilla servers. That is why I have
restricted the evaluation to Firefox (for the moment). If the evaluation
shows that the recommendations are good enough for
components/developers, then the effort towards making product
recommendations feasible could be justified. Having said that, if there
is enough interest fro the Mozilla community I'll look more into how
this can be done practically at the current time - perhaps you could
provide suggestions for the most confused projects so I don't have to
use the entire repository?
2) Even if in the Mozilla project developers self assign, providing
Mozilla triagers recommendations about who should be assigned to the bug
will help them to know who to ask questions to.
The philosophy at work here is that the sooner the report gets into the
hands of the right developer (or one in the right set of developers) the
sooner progress can be made on the bug. Perhaps this does not fit
exactly with the Mozilla project process, but I still think that the
project can benefit from using the tool.
John
Axel Hecht
I've been reading some on
http://www.cs.ubc.ca/labs/spl/projects/bugTriage.html, and in
particular, the "Who Should Fix This Bug?" paper.
Generally, I think that mozilla wouldn't be that fond of people
reassigning bugs other than to default assigne/QA contact. I personally
tend to beat people with a stick if they try that.
The way we get people to pay attention to a bug is to put them on CC.
That is a slightly different scenario than what you tried to do, and
comes with different problems and benefits.
Given on how our bug data base looks, we probably only need to
investigate bugs that were filed in the Firefox product, maybe even the
General Category. We do need to watch those bugs in other products,
though, in particular the Core and the Toolkit product, as the firefox
product really only covers Firefox feature problems.
A automated tool to suggest product/category combinations would be
probably valuable, with a good hint of folks to put on CC (and who
wouldn't be watching the default assignee/QA).
I'm not sure whether the category suggestion stuff only includes Firefox
categories, could you comment on that?
Could you publish the bug list that you use to train your machine
learning? It may be interesting for us to see a histogram of people you
suggested as assignees in your statistics.
I wonder your statistics look if one just randomly assigns bugs to
either gavin or mconnor (which would be for 2.0, you used 1.0 data, right?).
In general, I think that many Mozilla folks will be much more interested
in details about your approach and example bugs than Amazon vouchers to
get sucked into spending some time testing your work.
I saw that a colleague of yours is working on a tool to find dupes, that
would be probably a requirement to have first, as that's an important
part of our triage work. And we wouldn't want to fragment our
dupe-finding activities by moving bugs out of the central focus.
On a more general note, you may want to contact, I'd say, try Asa, about
getting a dump of our bugzilla db for your science. Harvesting that data
via the web is something that isn't in the best interest of both sides,
and we have done cooperations like this with scientist on a regular basis.
I'm personally not doing any triage myself, so I wouldn't be in your
target group anyway, just thought that others may have the same
questions that I do. I hope the feedback helps.
Axel
John Anvik
CC Recommendations
------------------
I have been considering the idea of extending my work to include CC
recommendations. If you are interested in contributing to that
discussion I have opened a feature request in the Sibyl repository
(http://ws.cs.ubc.ca/~janvik/bugs/show_bug.cgi?id=27)
Product/Category Recommendations
--------------------------------
Sibyl makes component (that is what you mean by categories right?)
recommendations just under the Firefox product for the practical reasons
that I mentioned previously.
I think that it is important to point out that Sibyl only makes
recommendations. The triager is free to use or not use them. Nothing is
ever automatically assigned when using Sibyl. The interface is designed
to minimize process impact (e.g. the developer recommendations are added
to the Bugzilla page as a drop down box next to the existing text box)
so a triager using Sibyl should only have to make a minor change (i.e.
look at the values in a drop down box).
It seems to me that given the Mozilla process model, the best way for
Firefox triagers to use Sibyl is to ignore the developer recommendations
and just use the component recommendations. I think this would be a big
help with the Firefox bugs in general, and specifically for those that
are filed in the Firefox General category. However, to know if that is
truly the case I need some triagers to try it out. I have gone as far
with this as I can without having actual triagers use it. That is why
I'm soliciting help from the community.
Approach Details
----------------
Pretty much all the details of the approach are in the paper that you
referenced. Any other specific questions, just ask. If Firefox triagers
don't need the Amazon incentive to use Sibyl, that is great.
John