I'd like to start my feedback with a request.
It'd help me to get a big-picture of the stuff that surrounds this email. Things I'd like to see is information about who's been consulted going in to this. Also, which threads about bug lifecycle got looked at.
It'd also be nice to see how this one should play into more forward-looking work.
I know that I wasn't consulted because I didn't insist. I did have the chance, though. Others might feel easier if they had similar insight.
Also, the question about "is this the problem to solve" benefit from context. This one might just be a dependency of the ideas on how to get to more heavy stuff.
And now to remember all that when I write my own heavy-weight stuff soon.
More details inline.
Bug Program Next Steps
Over the last week, I’ve asked you to step up and identify developers who will be responsible for bugs triaged into their component (in Firefox, Core, Toolkit, Fennec iOS, and Fennec Android.)
Why This Matters
Bugs are a unit of quality we can use to see how we’re doing.
We believe that the number of outstanding, actionable bugs is the best metric of code quality available, and that how this number changes over time will be a strong indicator of the evolving quality of Firefox. Actionable bugs are those with the status NEW for which the next action required - if any - is unambiguous: does the bug need immediate attention, will it be worked on for a future release, will it be worked on at all?
There are two parts to maintaining the value of that metric. First, we want to make better assertions about the quality of our releases by making clear decisions about which bugs must be fixed for each release (urgent) and actively tracking those bugs. The other is the number of good bugs filed by our community. Filing a bug report is a gateway to greater participation in the Mozilla project, and we owe it to our community to make quick and clear decisions about each bug.
Making decisions on new bugs quickly helps us avoid point releases, and gives positive feedback to people filing bugs so that they will file more good bugs.
What’s Your Role
Starting in the second quarter of this year, if you’ve taken on a component, I’m expecting you or your team to look at the bugs which landed in the component on a more frequent basis than a weekly triage.
In February, we’re starting a pilot with four groups of components where we’ll get the process changes and field tested, so that we can we can take the changes to all the affected bugzilla components for review and comment before we implement them across all of the work on Firefox.
How are those four groups chosen?
Hold on, we already have a weekly triage!
That’s fantastic, but a weekly pace means we miss bugs that affect upcoming releases. So I’m expecting you to scan that list of inbound bugs daily for the urgent ones (I’ll define urgent below) and put them into one of the states described in the next section, the others can go into your regular triage.
At Your Regular Triage
You’ll look at the bugs which landed in your component and decide on how to act on them using the states described in the next section.
We don’t have a regular triage
This is a process which you’ll need to start, and the bug program team will help with this.
This is potentially a lot of work for one person
Looking at the urgent bugs does not have to be one person’s task. You can have a rotation of people doing this. Look at the Core::Graphics triage wiki for an example of what you could be doing.
Initially, these states will be marked in bugzilla.mozilla.org using whiteboard tags during the pilot. The bugzilla team will be making further changes once we’ve settled on a process.
You’ll be looking at bugs in your component as they land, in your component. We expect most of these will be NEW bugs, but some will be in UNCONFIRMED.
There are four states you’ll need to decide to put each bug, and in your reviews between your team’s weekly triages, we want you to be on the watch for bugs with characteristics which make getting it in front of someone urgent: these are bugs with crash, topcrash, regression, or dataloss keywords; crash logs linked in comments; references to mozregression results; and others.
The bug should not remain in this state after your review of it.
You’ll need to decide which of the following states you’ll move this bug into, if you can’t you’ll need to be taking action: such as getting someone to run mozregression, need info’ing a domain expert, looking at checkins, and whatever else techniques you have to get a bug reduced.
Once you have an understanding of the bug, you should assign it to one of these states.
I'd ask you to make additions to these states for two use-cases:
As you're asking for this to be done on a daily basis, I'd expect it to be time-bound. We do have bugs that just take a day or two to figure out what the next step is.
I think this is an important option as we're learning how this works out for us.
I'm also generally concerned how UX bugs or crashes would fit
into these buckets. UX bugs, and possibly any other flavor of
ideation, have the majority of work associated with "should we do
this or not". And crashes as a single crash stack are hardly ever
Assigned to a developer, release tracking flags nominated, and set at priority `P1`. If the bug is being worked on by somebody from outside your core team, a team mentor should be assigned.
All these need to be set for the bug when you assign it to this state. This state is for bugs you find in your daily review which need a developer immediately.
If the bug is not in need of immediate attention, then your team’s process should land the bug in one of the following states.
This is a NEW bug that the team acknowledges is a bug, but is not a current priority, but will consider taking on. If the bug contains regression, crash, topcrash, or similar keywords and metadata, then the team can explain why it’s not a high priority.
Is a Bug, Not Prioritized
This is a terminal state for a NEW bug. We acknowledge the bug exists, it affects people, but it is not important enough to warrant working on it. The team will review and accept patches from the community for this bug report.
This is a terminal state for a NEW bug. After review, the bug is not something that can be worked on, or describes existing and expected behavior.
There are other states we’re looking at for bugs. These will cover a bug as it’s worked on, as ASSIGNED as is doesn’t provide useful information as to progress; flag bugs that have stalled, or needing code reviews, or sheriff attention; bugs that are on a release train; and bugs with fixes in general or ESR version.
Now: finish finding component responsible parties
February: pilot review of NEW bugs with four groups of components, draft new process
March: comment period for new process, finalize process
Q2: implement new process across all components
Q3: all newly triaged bugs following the new process
Are there good metrics to use to evaluate the success and impact of this process?
To pick up on what I said earlier, getting a category for "didn't work" could be one way to measure, or at least enable us to systematically investigate areas for improvements.