Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Proposal for new JS bug triage tags
19 views
Skip to first unread message
Dave Mandelin
May 7, 2012, 5:07:22 PM5/7/12
to
Background:
JS team bug wrangling currently works like this: I look at all incoming bugs, triage them, record triage notes in a text file, and mark k9o+ or tracking-fxN+ if applicable. Sometimes it's hard to tell how serious a bug is when it comes in, and in that case, if no one else grabs it, I do an initial investigation, and then triage it. I also have a script that merges the triage file with Bugzilla data, producing a report showing important bugs, how urgent they are, and what next steps are needed, and I use that to follow up on bugs and keep them moving. I try to do both triage and followup daily; in practice it's usually every 1-3 days.
This has been working pretty well, but it's getting creaky. The main problem is that it's strictly 1-person: it doesn't let anyone else help triage or follow up, and it doesn't directly communicate the triage decisions to anyone. Also, sometimes I don't have time for initial investigation, so I need to get more help with that too. For example, I have 2 conference-type events to prepare for over the next month, so I'm worried that I'll get overloaded or fail to prep if I spend too much time on investigations.
Beyond the problems, I've been wanting to make things better. We do well catching the important bugs before shipping, but I'd like to get even better. I would also like to be able to better understand how many serious bugs we have outstanding, what other projects they're competing with, and whether we have enough time to fix them all. Finally, I would like to give more people the opportunity to learn and get involved with team bug wrangling.
The first step I want to take is to add triage tags to important new JS bugs--the proposal is below. That means I will have to change my own processes to use the new tags, and Naveed will be getting involved in triage. After that, I'm sure various other practices will change over time. The whole point is to do a better job catching important bugs with less effort, so as time goes on, if that's not happening, or if you have ideas to make things even better, let me know.
Proposal: New JS bug triage tags
(Note: this won't affect IonMonkey very much immediately, since you guys are focused on that and know what to do. You will start having bugs triaged under this system once you land.)
The basic triage info will be stored in the whiteboard in a format similar to that of Kilimanjaro with priority and target nightly iteration:
[js:p<m>:fx<n>]
[js:p<m>]
Example 1: [js:p1:fx15] means a bug with priority 1 that is targeted to be fixed by the time the fx15 nightly cycle finishes. Note that the target is defined differently (I think) from Kilimanjaro. It *does not* mean "fix for Firefox 15"--it means "fix by the day on which Firefox 15 merges to Aurora".
Example 2: [js:p3] means a bug with priority 3 and no target iteration.
The priority values:
p1 = "must do"
includes: serious user-facing defects, sg:critical, sg:high
will be targeted for fixing on nightly or aurora
expected to be about 20-30 bugs/release
p2 = "want to do"
includes: high-value feature work (internal or to support other
teams and projects), medium user-facing defects
expected to be mostly feature work
p3 = "may do"
includes: minor user-facing defects and spec violations
The operational meaning of the priority values is just that it's preferable to miss a target for a p2 than for a p1.
The priority values only work if they are set sensibly. And that's not always easy: there seems to be a tendency on my part (and others, for other releases) for the criteria for a high-priority bug to get looser over time. So I want everyone to feel like they can participate, talk back, and get priorities adjusted if they are wrong.
Not all bugs are going to get triage tags. About 2/3 of our bugs are internally generated (pieces of projects, refactorings, etc), and usually won't be tagged. About 1/6 are fuzz bugs, and those should continue to be fixed in the normal course of things as they are now. So only about 1/6 of bugs will be tagged.
There are a few more tags besides the basic triage tag. The most important is the investigation tag.
[js:investigate]
Means the bug needs investigation. For all we know, the bug
could be pretty bad, so this would normally be a p1. The p1
is for the investigation itself: after investigation it will
be retriaged. An example right now would be bug 750019, which
is a new topcrash.
[js:nonactionable]
Means the bug can't be worked on because we don't have enough
information. It sets expectations for anyone watching the
bug and reminds us that we don't have to bother with it unless
we get more information. An example right now would be GC
mark phase topcrashes.
[jso:dm]
[jso:nd]
Identifies who is following up on the bug by first/last
initials. This is so Naveed and I can divide up the followup
work. 'o' is for 'owner'. (This is like the Kilimanjaro
owner concept, but I'm not going to steal the assignee
field for that purpose.)
Initially, Naveed and I are going to be triaging bugs, but anyone is invited to participate.
Feedback wanted: Do the tags make sense? Do the priority values sound right? Does it sound like it will be helpful? Any questions?
Dave
JS team bug wrangling currently works like this: I look at all incoming bugs, triage them, record triage notes in a text file, and mark k9o+ or tracking-fxN+ if applicable. Sometimes it's hard to tell how serious a bug is when it comes in, and in that case, if no one else grabs it, I do an initial investigation, and then triage it. I also have a script that merges the triage file with Bugzilla data, producing a report showing important bugs, how urgent they are, and what next steps are needed, and I use that to follow up on bugs and keep them moving. I try to do both triage and followup daily; in practice it's usually every 1-3 days.
This has been working pretty well, but it's getting creaky. The main problem is that it's strictly 1-person: it doesn't let anyone else help triage or follow up, and it doesn't directly communicate the triage decisions to anyone. Also, sometimes I don't have time for initial investigation, so I need to get more help with that too. For example, I have 2 conference-type events to prepare for over the next month, so I'm worried that I'll get overloaded or fail to prep if I spend too much time on investigations.
Beyond the problems, I've been wanting to make things better. We do well catching the important bugs before shipping, but I'd like to get even better. I would also like to be able to better understand how many serious bugs we have outstanding, what other projects they're competing with, and whether we have enough time to fix them all. Finally, I would like to give more people the opportunity to learn and get involved with team bug wrangling.
The first step I want to take is to add triage tags to important new JS bugs--the proposal is below. That means I will have to change my own processes to use the new tags, and Naveed will be getting involved in triage. After that, I'm sure various other practices will change over time. The whole point is to do a better job catching important bugs with less effort, so as time goes on, if that's not happening, or if you have ideas to make things even better, let me know.
Proposal: New JS bug triage tags
(Note: this won't affect IonMonkey very much immediately, since you guys are focused on that and know what to do. You will start having bugs triaged under this system once you land.)
The basic triage info will be stored in the whiteboard in a format similar to that of Kilimanjaro with priority and target nightly iteration:
[js:p<m>:fx<n>]
[js:p<m>]
Example 1: [js:p1:fx15] means a bug with priority 1 that is targeted to be fixed by the time the fx15 nightly cycle finishes. Note that the target is defined differently (I think) from Kilimanjaro. It *does not* mean "fix for Firefox 15"--it means "fix by the day on which Firefox 15 merges to Aurora".
Example 2: [js:p3] means a bug with priority 3 and no target iteration.
The priority values:
p1 = "must do"
includes: serious user-facing defects, sg:critical, sg:high
will be targeted for fixing on nightly or aurora
expected to be about 20-30 bugs/release
p2 = "want to do"
includes: high-value feature work (internal or to support other
teams and projects), medium user-facing defects
expected to be mostly feature work
p3 = "may do"
includes: minor user-facing defects and spec violations
The operational meaning of the priority values is just that it's preferable to miss a target for a p2 than for a p1.
The priority values only work if they are set sensibly. And that's not always easy: there seems to be a tendency on my part (and others, for other releases) for the criteria for a high-priority bug to get looser over time. So I want everyone to feel like they can participate, talk back, and get priorities adjusted if they are wrong.
Not all bugs are going to get triage tags. About 2/3 of our bugs are internally generated (pieces of projects, refactorings, etc), and usually won't be tagged. About 1/6 are fuzz bugs, and those should continue to be fixed in the normal course of things as they are now. So only about 1/6 of bugs will be tagged.
There are a few more tags besides the basic triage tag. The most important is the investigation tag.
[js:investigate]
Means the bug needs investigation. For all we know, the bug
could be pretty bad, so this would normally be a p1. The p1
is for the investigation itself: after investigation it will
be retriaged. An example right now would be bug 750019, which
is a new topcrash.
[js:nonactionable]
Means the bug can't be worked on because we don't have enough
information. It sets expectations for anyone watching the
bug and reminds us that we don't have to bother with it unless
we get more information. An example right now would be GC
mark phase topcrashes.
[jso:dm]
[jso:nd]
Identifies who is following up on the bug by first/last
initials. This is so Naveed and I can divide up the followup
work. 'o' is for 'owner'. (This is like the Kilimanjaro
owner concept, but I'm not going to steal the assignee
field for that purpose.)
Initially, Naveed and I are going to be triaging bugs, but anyone is invited to participate.
Feedback wanted: Do the tags make sense? Do the priority values sound right? Does it sound like it will be helpful? Any questions?
Dave
Naveed Ihsanullah
May 8, 2012, 1:38:15 PM5/8/12
to dev-tech-js-en...@lists.mozilla.org
Dave,
This proposal looks good and in-line with our conversations so far. One
thing that is not perfectly clear is which bugs are part of this
process. Additionally, for bugs that are not part of this process how
can we know they are not part of this process or just in a pre-triaged
state? I am thinking in terms of Bugzilla queries that would allow us to
quickly: 'Find un-triaged bugs'.
Thanks,
Naveed
> _______________________________________________
> dev-tech-js-engine-internals mailing list
> dev-tech-js-en...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
This proposal looks good and in-line with our conversations so far. One
thing that is not perfectly clear is which bugs are part of this
process. Additionally, for bugs that are not part of this process how
can we know they are not part of this process or just in a pre-triaged
state? I am thinking in terms of Bugzilla queries that would allow us to
quickly: 'Find un-triaged bugs'.
Thanks,
Naveed
> dev-tech-js-engine-internals mailing list
> dev-tech-js-en...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
Jim Blandy
May 9, 2012, 1:56:40 PM5/9/12
to
Since you asked for feedback: :)
- It makes sense to me.
- I understand what the priority tags mean.
- If it covers everything your personal process has been covering, then
that means it's helpful, because you've put your state where everyone
can see it.
- It makes sense to me.
- I understand what the priority tags mean.
- If it covers everything your personal process has been covering, then
that means it's helpful, because you've put your state where everyone
can see it.
Dave Mandelin
May 9, 2012, 9:21:31 PM5/9/12
to
Since not too many people commented, I polled a couple of people individually. They basically said it was OK. Jeff did suggest some alternate mechanics, such as using the bugzilla priority field or getting some new checkboxes added. I don't want to use the priority field for now, because it can get set randomly. But over time we can see how the new flags are working and if they stabilize we can try to get them added more directly to bugzilla (pave the cowpaths).
In response to Naveed's question about finding untriaged bugs, I think for now we can use the REST API to search for bugs reported by "not us (us includes fuzzers)"--the bugs we didn't generate are the ones we really need to triage. It sure would be nice to have a tool with a queue that any triager could pick out of...maybe we can get that too someday if it's needed enough.
We'll start triaging the current bugs out of my old system and into the new. I have a meeting set up for tomorrow at 1pm. Anyone should feel free to join, but let me know so I can find you or give you dialin info.
Dave
0 new messages