Bug Triage Practices

Dave Mandelin

unread,

Jan 9, 2013, 10:01:49 PM1/9/13

to

I've been thinking about bug triage for a long time, and since it's a new year, I thought it might be a good time to try something new. My observation is that our engineering bug triage practices as of today <blunt>mostly range from crude to nonexistent</blunt>. The release managers pick up the critical bugs, but they don't track less important bugs and bugs that are important for technical reasons, and it seems like they have to do a lot of manual work and a lot of reminding of engineering teams. Other issues stem from the lack of triage, like the thousands of old open bugs littering the system, or the fact that contributors can't use Bugzilla to find good things to work on.

So, how about we start using modern bug triage practices? Ideas welcome. Is anyone already using a system they are happy with?

To make things more concrete, I think a relatively simple system like this could meet many of our needs:

- New bugs are assigned an importance of 1, 2, 3, or N. 1 is for critical bugs, 2 is default, 3 is for bugs that probably shouldn't get fixed (because there are too many more important bugs). N is for things that don't need an importance, mostly the large number of developer-filed "I need to land a patch that does X" bugs. These states let people know what's most worth fixing.
- Working on importance 1 bugs takes priority over working on other bugs.

- Any bug can be marked non-actionable with a comment. That way, everyone will know developers won't be working on it unless something changes or more information is provided.

- An open, confirmed bug can be in one of three states: active, backlog, iced. Active bugs are being worked on now, and a fix is expected soon. Backlog bugs are not being worked on, but the backlog is reviewed regularly to pick new active bugs. Iced bugs are not being worked on, and won't be reviewed unless specially asked for. These states let people know what will be fixed soon, later, or never.

Dave

L. David Baron

unread,

Jan 9, 2013, 10:20:23 PM1/9/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Wednesday 2013-01-09 19:01 -0800, Dave Mandelin wrote:
> I've been thinking about bug triage for a long time, and since
> it's a new year, I thought it might be a good time to try
> something new. My observation is that our engineering bug triage
> practices as of today <blunt>mostly range from crude to
> nonexistent</blunt>. The release managers pick up the critical
> bugs, but they don't track less important bugs and bugs that are
> important for technical reasons, and it seems like they have to do
> a lot of manual work and a lot of reminding of engineering teams.
> Other issues stem from the lack of triage, like the thousands of
> old open bugs littering the system, or the fact that contributors
> can't use Bugzilla to find good things to work on.

I think I've basically stopped using the bug system because it's too
slow; it takes 10-30 seconds for a bug to load, and similar times
for queries to run. So I keep work to do in a plain text file and
ignore the bug system.

If we had a faster bug system, I might still find that I needed a
more complex system like what I described in
http://dbaron.org/log/20120816-bug-system for it to be useful, but I
might not. It's hard for me to know without trying it.

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla http://www.mozilla.org/ 𝄂

David E. Ross

unread,

Jan 9, 2013, 10:38:14 PM1/9/13

to

On 1/9/13 7:01 PM, Dave Mandelin wrote [in part]:

> - New bugs are assigned an importance of 1, 2, 3, or N. 1 is for
critical bugs, 2 is default, 3 is for bugs that probably shouldn't get
fixed (because there are too many more important bugs). N is for things
that don't need an importance, mostly the large number of
developer-filed "I need to land a patch that does X" bugs. These states
let people know what's most worth fixing.

In other words, you want to re-invent the existing Severity item in
Bugzilla:

Severity
This field describes the impact of a bug.
blocker Blocks development and/or testing work
critical crashes, loss of data, severe memory leak
major major loss of function
normal regular issue, some loss of functionality under specific
circumstances
minor minor loss of function, or other problem where easy
workaround is present
trivial cosmetic problem like misspelled words or misaligned text
enhancement Request for enhancement

The question is: What is wrong with Severity?

--
David E. Ross
<http://www.rossde.com/>

Are taxes too high in the U.S.? Check the bar graph
at <http://www.rossde.com/taxes/trickling.html> to see.

Nicholas Nethercote

unread,

Jan 9, 2013, 11:24:04 PM1/9/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Wed, Jan 9, 2013 at 7:01 PM, Dave Mandelin <dman...@gmail.com> wrote:
>
> So, how about we start using modern bug triage practices?

What does that mean?

----

I think the single biggest problem in bug triage for Mozilla is that
the number of open bugs grows continually.

Let me share some MemShrink experience. We use a P1, P2, P3
prioritization system for all bugs tagged with "MemShrink". The
number of P1s has held fairly steady for over a year -- we typically
have about 15--20. The number of P2s and P3s has gradually increased,
to where we have slightly over 100 of each.

We triage MemShrink-tagged bugs frequently -- that's the main purpose
of our fortnightly meetings. We try to assign bugs where possible,
but plenty of them don't get assigned -- probably more than half.

We go over the P1 bugs every few meetings, deciding if any can be
closed, or downgraded, or dup'd. We rarely go over P2 bugs, and never
go over P3 bugs in meetings. Marking a bug as P2 basically says "eh,
not clear if anyone's gonna work on that". Marking a bug as P3
basically says "probably will never happen / we don't care".

The bugs that get fixed are the ones where someone decides "I'm going
to work on this", either because they decided it themselves, or
because someone convinced them to.

I think our triaging has been both aggressive and quite successful,
and indicates what you can achieve if a small number of people really
pay attention to a small subset of bugs. And yet, the number of open
bugs keeps growing, because this is Mozilla, and the bug flow is
relentless.

I guess the summary is: It's a hard problem. Pick your battles; many
bugs are destined to rot.

Nick

L. David Baron

unread,

Jan 10, 2013, 12:25:44 AM1/10/13

to dev-pl...@lists.mozilla.org

On Wednesday 2013-01-09 19:38 -0800, David E. Ross wrote:
> On 1/9/13 7:01 PM, Dave Mandelin wrote [in part]:
> > - New bugs are assigned an importance of 1, 2, 3, or N. 1 is for
> critical bugs, 2 is default, 3 is for bugs that probably shouldn't get
> fixed (because there are too many more important bugs). N is for things
> that don't need an importance, mostly the large number of
> developer-filed "I need to land a patch that does X" bugs. These states
> let people know what's most worth fixing.
>
> In other words, you want to re-invent the existing Severity item in
> Bugzilla:
>
> Severity
> This field describes the impact of a bug.
> blocker Blocks development and/or testing work
> critical crashes, loss of data, severe memory leak
> major major loss of function
> normal regular issue, some loss of functionality under specific
> circumstances
> minor minor loss of function, or other problem where easy
> workaround is present
> trivial cosmetic problem like misspelled words or misaligned text
> enhancement Request for enhancement
>
> The question is: What is wrong with Severity?

What he's proposing is really not much like severity (which is about
the effect on the user reporting the bug, but doesn't consider how
many users are affected or how hard the bug is to fix), but quite
similar to priority.

I actually have been using priorities roughly as described in
http://dbaron.org/log/20090120-bug-priorities when I triage style
system or layout bugs; I just haven't been triaging them much. (I'd
probably write the P4 and P5 descriptions differently than I wrote
them there; today I'd probably call P4 the bugs I would encourage
people to spend time on, and P5 the bugs that I would discourage
people from spending time on.)

Mike Hommey

unread,

Jan 10, 2013, 2:48:00 AM1/10/13

to L. David Baron, Dave Mandelin, dev-pl...@lists.mozilla.org

On Wed, Jan 09, 2013 at 07:20:23PM -0800, L. David Baron wrote:
> On Wednesday 2013-01-09 19:01 -0800, Dave Mandelin wrote:

> > I've been thinking about bug triage for a long time, and since
> > it's a new year, I thought it might be a good time to try
> > something new. My observation is that our engineering bug triage
> > practices as of today <blunt>mostly range from crude to
> > nonexistent</blunt>. The release managers pick up the critical
> > bugs, but they don't track less important bugs and bugs that are
> > important for technical reasons, and it seems like they have to do
> > a lot of manual work and a lot of reminding of engineering teams.
> > Other issues stem from the lack of triage, like the thousands of
> > old open bugs littering the system, or the fact that contributors
> > can't use Bugzilla to find good things to work on.
>

> I think I've basically stopped using the bug system because it's too
> slow; it takes 10-30 seconds for a bug to load, and similar times
> for queries to run. So I keep work to do in a plain text file and
> ignore the bug system.

Log out of bugzilla, you'll be amazed. Seriously.

Mike

Nicholas Nethercote

unread,

Jan 10, 2013, 4:08:28 AM1/10/13

to L. David Baron, dev-pl...@lists.mozilla.org

On Thu, Jan 10, 2013 at 4:25 PM, L. David Baron <dba...@dbaron.org> wrote:
>
> I actually have been using priorities roughly as described in
> http://dbaron.org/log/20090120-bug-priorities when I triage style
> system or layout bugs; I just haven't been triaging them much. (I'd
> probably write the P4 and P5 descriptions differently than I wrote
> them there; today I'd probably call P4 the bugs I would encourage
> people to spend time on, and P5 the bugs that I would discourage
> people from spending time on.)

Quoting from that blog post:

P1: We can't ship an alpha/beta release (or, in particular, the
next one) with this bug.
P2: We can't ship a final release (or, in particular, the next
one) with this bug.
P3: I feel really bad about saying we're might ship a release with
this bug still in it, but we might.
P4: We should fix this as soon as we have the time.
P5: It would be nice to fix this sometime.

This is a nice scheme, but it relies on everyone following it, which
is hard in a system like b.m.o which is used by so many people for so
many different purposes. It also feels biased towards defects, which
are just one kind of "bug".

I take a more reductionist view: a P1 bug, for example, is a bug that
comes up when you search for P1 bugs. Nothing more, nothing less. (I
find this a useful way of looking at most Bugzilla flags. Every time
someone suggests adding or changing a Bugzilla flag, I think "what
searches would this facilitate and who would do those searches?)

As for assigning meanings to priorities, the real meaning comes from
who searches for them, and how often, and what they do with the search
results. This is reflected in the MemShrink priorities
(https://wiki.mozilla.org/Performance/MemShrink#Bug_Tracking):

Unprioritized MemShrink bugs. These are triaged regularly in meetings.
MemShrink:P1 bugs. These are discussed regularly in meetings.
MemShrink:P2 bugs. These are discussed occasionally in meetings.
P2 is the default priority.
MemShrink:P3 bugs. These are discussed rarely in meetings.

(In practice, "occasionally" means "rarely" and "rarely" means
"never". I.e. MemShrink:P2 means "mostly ignored" and MemShrink:P3
means "ignored".)

In your system, P1--P3 bugs would presumably be searched for by
release drivers, and they would apply more pressure to people to fix
P1s than P2s, for example. As for P4 and P5, it's less clear.

Getting back to dmandelin's original question -- I suggest thinking
about what kinds of searches are going to be done, how often, by whom,
and what will those people do with the results?

Nick

Justin Lebar

unread,

Jan 10, 2013, 4:49:55 AM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

> To make things more concrete, I think a relatively simple system like this could meet many of our needs:

The idea here is basically to add more metadata to bugs.

Bugzilla already has tons of metadata fields that goes unused.
Perhaps they're not the right fields, and perhaps better fields would
be more widely used. But I'm skeptical that any system would meet so
many teams' needs so well that it would become standard practice to to
maintain this metadata.

Instead, what seems to work is team-specific metadata. Offhand, I can
think of team-specific systems used by B2G, MemShrink, Snappy, the
security team, and release drivers. Since these systems are designed
to meet a specific need, they usually do a decent job of that. And
the fact that these systems are separate allows Snappy and MemShrink
and release drivers to attach different (and conflicting) metadata to
a bug, reflecting their differing priorities and reducing pointless
arguments.

I think it would be interesting to see you try to add some of this
metadata to JS bugs. If it's helpful, maybe others will adopt it!

-Justin

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

Ed Morley

unread,

Jan 10, 2013, 5:24:02 AM1/10/13

to L. David Baron, dev-pl...@lists.mozilla.org

On 10/01/2013 03:20, L. David Baron wrote:
> I think I've basically stopped using the bug system because it's too
> slow; it takes 10-30 seconds for a bug to load, and similar times
> for queries to run.

The speed of quicksearch queries can be improved drastically by
disabling searching bug comments (in prefs; it unfortunately defaults to
on). In the last couple of months, it is also now possible to change
this per query be adding the suffix ++comments / --comments

Bug 687261 is filed to turn off comment-searching by default, which I
think would help with the perceived slowness.

Best wishes,

Ed

Robert O'Callahan

unread,

Jan 10, 2013, 6:31:12 AM1/10/13

to L. David Baron, Dave Mandelin, dev-pl...@lists.mozilla.org

I agree that Bugzilla slowness is a big problem, and I think it's avoidable
with some (major) client-side changes.

I've been thinking about this for a while, but you inspired me to edit my
thoughts into my own blog post:
http://robert.ocallahan.org/2013/01/optimizing-bugzilla-usage.html

Summary: I think a better client could fix Bugzilla's responsiveness and
make Bugzilla much more pleasant and effective to use --- without having to
make controversial changes or require all users to do anything differently.

Rob
--
Jesus called them together and said, “You know that the rulers of the
Gentiles lord it over them, and their high officials exercise authority
over them. Not so with you. Instead, whoever wants to become great among
you must be your servant, and whoever wants to be first must be your
slave — just
as the Son of Man did not come to be served, but to serve, and to give his
life as a ransom for many.” [Matthew 20:25-28]

Frédéric Buclin

unread,

Jan 10, 2013, 7:34:06 AM1/10/13

to

Le 10. 01. 13 04:20, L. David Baron a écrit :

> I think I've basically stopped using the bug system because it's too
> slow; it takes 10-30 seconds for a bug to load, and similar times
> for queries to run. So I keep work to do in a plain text file and
> ignore the bug system.

I honestly doubt any bug on bmo takes as long as 10-30 seconds to load.
Bugs are generally displayed in less than 5 seconds. Large ones may
reach 10 seconds at most. Either you have some Firefox addons which
badly interact with bmo, or bmo has some customizations which badly
affect "power" users like you (you are probably in many bmo groups) but
doesn't affect other users like us. That's something dkl or glob could
answer: are there some bmo customizations which do extra work for
"power" users when loading a bug report which could explain such a slowness?

About queries. Bugzilla 4.4 is definitely faster than 4.0 (I have some
queries which now take a few seconds only in 4.4 instead of several
minutes in 4.2 and older; of course, the performance win depends on the
search criteria and fields being displayed in buglists). Also, I guess
that all the custom pseudo-flag fields that bmo has for Core + some
other products do not help to improve performance. Also something glob
and dkl could investigate.

It's not totally fair to say that Bugzilla is slow. There are many
clients which talk to bmo to get data, including the REST API. Using
another bug system, if some developers had this idea in mind, wouldn't
help: the load would remain the same but moved to the new bug system,
making this new bug system slow again.

LpSolit

Patrick McManus

unread,

Jan 10, 2013, 8:31:35 AM1/10/13

to Frédéric Buclin, mozilla.dev.planning group

It's not totally fair to say that Bugzilla is slow. There are many
>

mozilla.bugzilla.org has the worst responsiveness of anything I regularly
use on the Internet and its a real impediment to getting work done
efficiently.

Byron Jones

unread,

Jan 10, 2013, 9:00:28 AM1/10/13

to dev-pl...@lists.mozilla.org

hi all,

>> I think I've basically stopped using the bug system because it's too
>> slow; it takes 10-30 seconds for a bug to load, and similar times
>> for queries to run. So I keep work to do in a plain text file and
>> ignore the bug system.

i've been working with dbaron off the list to figure out what's going on
with his setup, as "10-30 seconds" response times are far outside the
norm that i've seen, had reported, and can see in the logs.

--
byron - irc:glob - bugzilla.mozilla.org team -

Byron Jones

unread,

Jan 10, 2013, 9:24:59 AM1/10/13

to Patrick McManus, mozilla.dev.planning group

Patrick McManus wrote:
> > It's not totally fair to say that Bugzilla is slow. There are many

> mozilla.bugzilla.org has the worst responsiveness of anything I regularly
> use on the Internet and its a real impediment to getting work done
> efficiently.

profiling and tuning bmo and bugzilla has been a significant focus of
both the bmo and bugzilla teams recently, with improvements landing on
both bmo and future bugzilla versions.

we're on the cusp of deploying a new bmo cluster in scl3, which has
faster webheads and database servers, along with the ability to easily
add more servers into the mix if required.
Q1 this year we'll also be undertaking a major refactoring of how we
track rapid releases, which should reap performance benefits.

yes, performance is an issue; we've working on it, please bear with us
(or work with us to fix these issues!).

Byron Jones

unread,

Jan 10, 2013, 9:59:56 AM1/10/13

to mozilla.dev.planning group

Byron Jones wrote:
> Q1 this year we'll also be undertaking a major refactoring of how we
> track rapid releases, which should reap performance benefits.

i was asked about this off-list, so i thought it would be useful to
everyone to paint a picture about what's happening here.

each "release tracking flag" on bugzilla (tracking-*, status-*,
blocking-*, etc) is currently implemented as a normal bugzilla custom field.
bugzilla implements custom bug fields by adding a column to the main
"bugs" table.
every six weeks i add another six custom fields to bmo, so the bugs
table is extremely wide.
currently there are 139 custom fields, most are varchar(64).
this impacts mysql's ability to cache rows, as wider tables consume more
memory, so fewer rows can be held in ram.
the refactoring will basically change from using columns to rows for
each field.

-glob

Benjamin Smedberg

unread,

Jan 10, 2013, 11:37:58 AM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On 1/9/2013 10:01 PM, Dave Mandelin wrote:
> I've been thinking about bug triage for a long time, and since it's a new year, I thought it might be a good time to try something new. My observation is that our engineering bug triage practices as of today <blunt>mostly range from crude to nonexistent</blunt>. The release managers pick up the critical bugs, but they don't track less important bugs and bugs that are important for technical reasons, and it seems like they have to do a lot of manual work and a lot of reminding of engineering teams. Other issues stem from the lack of triage, like the thousands of old open bugs littering the system, or the fact that contributors can't use Bugzilla to find good things to work on.

I do agree that we can and should make improvements, but I think you're
making the current process look much worse than it is. I am fairly
certain that for the well-owned components, the primary mechanism for
*incoming* bug triage is component-watching bugmail. This isn't ideal,
but it seems to catch most of the important incoming bugs.

Many of the other teams have some kind of system in place. For instance,
Release Engineering uses the priority flag in a consistent manner:
https://wiki.mozilla.org/ReleaseEngineering/Bugzilla/Triage

I am in the process of adopting a very similar system for the bugs in my
modules. I have a tool I'm working on to make reprioritizing and
tracking bugs, patches, and reviews easier across the various bugzilla
product/components. The system I am using is

P1: highly urgent things (release blockers, sg:crit). Must always have
an owner. Other people should drop other to help with these bugs if
requested.
P2: priorities for the current cycle. There shouldn't be more of these
than can actually be fixed in a release.
P3: Queue of things that I will try to find owners for
P5: Valid bugs that aren't important enough for me to track (equivalent
of your "iced")

>
> - Any bug can be marked non-actionable with a comment. That way, everyone will know developers won't be working on it unless something changes or more information is provided.

The NEEDINFO flag, perhaps combined with whiteboard notes, is usually
good enough for this.

The question of bug triage is something that our new bugmaster,
Elizabeth Henry (lizzard on IRC) is primarily interested in. I'm also
very interested in this in terms of community participation. As part of
hiring the Firefox community manager, I've been looking at the ways we
ask people to participate in the engineering process. I think we
currently focus too much on writing code, and not enough on other
aspects of the engineering process. Bug triage is a "gateway drug" to
coding, and really helps people who want to write patches to know the
project and who's working on what. One of our first community efforts
should be to build out the "pre-code" community:

* Incoming bug triage
* Manual testing
* Finding regression ranges etc
* Writing testcases

Lastly, I think it would be smart for module owners to consider finding
and delegating a specific person (employee or volunteer) to serve as
"module triage peer". Module ownership and peerdom is currently is
focused on code reviews, but I think it could be valuable to explicitly
delegate the bug prioritization tasks through a peer nomination just as
we delegate patch review and other code decisions today.

--BDS

Dave Mandelin

unread,

Jan 10, 2013, 1:41:08 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Wednesday, January 9, 2013 7:20:23 PM UTC-8, L. David Baron wrote:
> On Wednesday 2013-01-09 19:01 -0800, Dave Mandelin wrote:
>
> > I've been thinking about bug triage for a long time, and since
> > it's a new year, I thought it might be a good time to try
> > something new. My observation is that our engineering bug triage
> > practices as of today <blunt>mostly range from crude to
> > nonexistent</blunt>. The release managers pick up the critical
> > bugs, but they don't track less important bugs and bugs that are
> > important for technical reasons, and it seems like they have to do
> > a lot of manual work and a lot of reminding of engineering teams.
> > Other issues stem from the lack of triage, like the thousands of
> > old open bugs littering the system, or the fact that contributors
> > can't use Bugzilla to find good things to work on.
>
> I think I've basically stopped using the bug system because it's too
> slow; it takes 10-30 seconds for a bug to load, and similar times
> for queries to run.

It is slow for me as well, although for some reason that doesn't bother me much. It takes 4-5 seconds for a bug to come up, and the text query that I tried took about 45 seconds. I tried Another Open Source Bug Tracker and got 1-2 seconds and 3-5 seconds respectively for those actions.

> So I keep work to do in a plain text file and ignore the bug system.

It's very interesting you say this, because that's how I did JS bug triage and tracking for about a year. And I know others that are using text files or paper lists. I think it's a sign of something wrong if people are abandoning the bug system for plain text files. (Not to mention github, etc.)

> If we had a faster bug system, I might still find that I needed a
> more complex system like what I described in
>
> http://dbaron.org/log/20120816-bug-system for it to be useful, but I
>
> might not. It's hard for me to know without trying it.

I also have noticed that a lot of time gets spent reading and rereading comment streams, and I've thought of similar summary areas or categories. I don't know either how to make it really work but I think it makes sense for complex tasks.

Dave

Dave Mandelin

unread,

Jan 10, 2013, 1:52:26 PM1/10/13

to dev-pl...@lists.mozilla.org

On Wednesday, January 9, 2013 9:25:44 PM UTC-8, L. David Baron wrote:
> What he's proposing is really not much like severity (which is about
> the effect on the user reporting the bug, but doesn't consider how
> many users are affected or how hard the bug is to fix), but quite
> similar to priority.

Yes. I think it would be interesting to classify bugs by both value (how much benefit is garnered by fixing it) and cost (how much effort is required to fix it). Priority decisions could made using those two as inputs. But that's quite a bit more complex than what I described, and I think that complexity might be too much, especially to start out with.

Dave

Dave Mandelin

unread,

Jan 10, 2013, 2:14:23 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Wednesday, January 9, 2013 8:24:04 PM UTC-8, Nicholas Nethercote wrote:
> On Wed, Jan 9, 2013 at 7:01 PM, Dave Mandelin <dman...@gmail.com> wrote:
> >
> > So, how about we start using modern bug triage practices?
>
> What does that mean?

I looked a couple other bug systems for open-source bug projects, and I found that they both had priority schemes, but they were pretty simple, like an "Importance" field with 3 values. I grew up with Priority and Severity a long time ago, but I think they are too fine-grained, requiring people to make too many unnecessary decisions and making it harder to keep things consistent.

> I think the single biggest problem in bug triage for Mozilla is that
> the number of open bugs grows continually.

It is a problem, which the "iced" category was meant to address. I think the real problem with that problem is that most of those open bugs will never get addressed, but there is no signal for that.

> Let me share some MemShrink experience. We use a P1, P2, P3
> prioritization system for all bugs tagged with "MemShrink". The
> number of P1s has held fairly steady for over a year -- we typically
> have about 15--20. The number of P2s and P3s has gradually increased,
> to where we have slightly over 100 of each.
>
> We triage MemShrink-tagged bugs frequently -- that's the main purpose
> of our fortnightly meetings. We try to assign bugs where possible,
> but plenty of them don't get assigned -- probably more than half.
>
> We go over the P1 bugs every few meetings, deciding if any can be
> closed, or downgraded, or dup'd. We rarely go over P2 bugs, and never
> go over P3 bugs in meetings. Marking a bug as P2 basically says "eh,
> not clear if anyone's gonna work on that". Marking a bug as P3
> basically says "probably will never happen / we don't care".

This matches my general experience (backed up by measurements I did on JS bugs a while back) that most bugs get fixed soon after filing, or not at all. There are just too many important bugs, so only a fraction of medium or low ever get fixed.

And that means that the real decision, the one that 90% determines the fate of a bug, is "fix it now" vs. "don't fix it now (i.e., probably not ever)".

> The bugs that get fixed are the ones where someone decides "I'm going
> to work on this", either because they decided it themselves, or
> because someone convinced them to.

That strikes me as a pretty ramshackle approach to shipping quality software. Why not "Decide which bugs are truly most worth fixing, record the decision, and then fix those bugs"? We used to do some approximation of that, but that system seems to have collapsed when we went to rapid release, and has been only partially replaced by the release-tracking flags.

> I think our triaging has been both aggressive and quite successful,
> and indicates what you can achieve if a small number of people really
> pay attention to a small subset of bugs.

Yes, effort and attention are the keys, and that's most of all what I'm asking for.

> And yet, the number of open bugs keeps growing, because this is
> Mozilla, and the bug flow is relentless.

Right. But I'd rather make decisions about what bugs won't get fixed, so we have a reasonable action list, rather than a huge pile of unknown stuff. As of now, we don't always know if anyone has even looked at the bug.

Dave

Dave Mandelin

unread,

Jan 10, 2013, 2:27:05 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Thursday, January 10, 2013 1:49:55 AM UTC-8, Justin Lebar wrote:
> > To make things more concrete, I think a relatively simple system like this could meet many of our needs:
>
> The idea here is basically to add more metadata to bugs.

No. The idea is to make sure every bug report gets looked at, decisions are made about what's worth working on, and people can tell what is being worked on and what bugs will and won't get fixed.

> Bugzilla already has tons of metadata fields that goes unused.

Yes. It has too many.

> Perhaps they're not the right fields, and perhaps better fields would
> be more widely used. But I'm skeptical that any system would meet so
> many teams' needs so well that it would become standard practice to

> maintain this metadata.
>
> Instead, what seems to work is team-specific metadata. Offhand, I can
> think of team-specific systems used by B2G, MemShrink, Snappy, the
> security team, and release drivers. Since these systems are designed
> to meet a specific need, they usually do a decent job of that. And
> the fact that these systems are separate allows Snappy and MemShrink
> and release drivers to attach different (and conflicting) metadata to
> a bug, reflecting their differing priorities and reducing pointless
> arguments.

That is a reasonable point, but I'm thinking not of individual teams, but the entire developer community. Team-specific tools are fine, in fact a necessity in the current environment, but:

- Each team ends up having to build and support its own tools, which is potentially a massive duplication of effort. Also, most of the tools will be crappy because they are part-time efforts.

- Team-specific tools do nothing for cross-team coordination. Other people won't necessarily be able to find or use a team-specific tool. Requiring people with cross-cutting responsibilities to navigate 23 different team-specific bug systems does not make sense.

- If bug A is a MemShrink P1, bug B is a Snappy P1, and bug C is a sec-critical, which do you fix? If there isn't a global priority, there isn't really a priority.

> I think it would be interesting to see you try to add some of this
> metadata to JS bugs. If it's helpful, maybe others will adopt it!

The JS component has been using a similar system for a while. The data entry is painful because of Bugzilla slowness and lacking tool support generally, but I believe it is better than the text files I used to use, because at least it shares the data.

Dave

Steve Wendt

unread,

Jan 10, 2013, 2:35:23 PM1/10/13

to

On 1/10/2013 11:14 AM, Dave Mandelin wrote:

>> The bugs that get fixed are the ones where someone decides "I'm going
>> to work on this", either because they decided it themselves, or
>> because someone convinced them to.
>
> That strikes me as a pretty ramshackle approach to shipping quality
> software. Why not "Decide which bugs are truly most worth fixing,
> record the decision, and then fix those bugs"?

It also needs to have some accommodation for Joe Developer in the open
source community, who decides that bugs abc and xyz, which are off the
radar of Mozilla developers, are important to him or his paying client.

Dave Mandelin

unread,

Jan 10, 2013, 2:35:25 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Thursday, January 10, 2013 8:37:58 AM UTC-8, Benjamin Smedberg wrote:
> On 1/9/2013 10:01 PM, Dave Mandelin wrote:
>
> > I've been thinking about bug triage for a long time, and since it's a new year, I thought it might be a good time to try something new. My observation is that our engineering bug triage practices as of today <blunt>mostly range from crude to nonexistent</blunt>. The release managers pick up the critical bugs, but they don't track less important bugs and bugs that are important for technical reasons, and it seems like they have to do a lot of manual work and a lot of reminding of engineering teams. Other issues stem from the lack of triage, like the thousands of old open bugs littering the system, or the fact that contributors can't use Bugzilla to find good things to work on.
>
> I do agree that we can and should make improvements, but I think you're
> making the current process look much worse than it is. I am fairly
> certain that for the well-owned components, the primary mechanism for
> *incoming* bug triage is component-watching bugmail. This isn't ideal,
> but it seems to catch most of the important incoming bugs.

You mean for detecting critical bugs? Yes, that works, although I think an in-Bugzilla listing of untriaged bugs would be far better. But that alone doesn't provide anything for recording triage decisions.

> Many of the other teams have some kind of system in place. For instance,
> Release Engineering uses the priority flag in a consistent manner:
> https://wiki.mozilla.org/ReleaseEngineering/Bugzilla/Triage

Perhaps the actual practice is more consistent, but based on the wiki article, I would dispute the assertion "uses the priority flag in a consistent manner:

+ Individual team members handle their own bugs in different ways.
+ Most team members will switch bugs from NEW to ASSIGNED when they
+ begin working on them. Some may even set the Priority flag. If
+ they do use the Priority field, the Priority levels have these
+ generally accepted meanings:

> I am in the process of adopting a very similar system for the bugs in my
> modules. I have a tool I'm working on to make reprioritizing and
> tracking bugs, patches, and reviews easier across the various bugzilla
> product/components.

Every person and team having to build their own idiosyncratic systems is *exactly* the kind of thing that I want to stop in favor of one simple system that works everywhere and enables scalable tool building and cross-team coordination.

> The system I am using is
>
> P1: highly urgent things (release blockers, sg:crit). Must always have
> an owner. Other people should drop other to help with these bugs if
> requested.
>
> P2: priorities for the current cycle. There shouldn't be more of these
> than can actually be fixed in a release.
>
> P3: Queue of things that I will try to find owners for
>
> P5: Valid bugs that aren't important enough for me to track (equivalent
>
> of your "iced")
>
> > - Any bug can be marked non-actionable with a comment. That way, everyone will know developers won't be working on it unless something changes or more information is provided.
>
> The NEEDINFO flag, perhaps combined with whiteboard notes, is usually
> good enough for this.

Not in my experience. There are many reasons a bug can be non-actionable. And I want non-actionability to clear and obvious to all readers, not something where you have to know that 'needinfo' is a codeword for 'non-actionable' depending on context.

> The question of bug triage is something that our new bugmaster,
> Elizabeth Henry (lizzard on IRC) is primarily interested in. I'm also
> very interested in this in terms of community participation. As part of
> hiring the Firefox community manager, I've been looking at the ways we
> ask people to participate in the engineering process. I think we
> currently focus too much on writing code, and not enough on other
> aspects of the engineering process.

+1.

> Bug triage is a "gateway drug" to coding, and really helps people
> who want to write patches to know the project and who's working on
> what. One of our first community efforts
> should be to build out the "pre-code" community:
>
> * Incoming bug triage
> * Manual testing
> * Finding regression ranges etc
> * Writing testcases
>
> Lastly, I think it would be smart for module owners to consider finding
> and delegating a specific person (employee or volunteer) to serve as
> "module triage peer". Module ownership and peerdom is currently is
> focused on code reviews, but I think it could be valuable to explicitly
> delegate the bug prioritization tasks through a peer nomination just as
> we delegate patch review and other code decisions today.

Already done for SpiderMonkey. It is a good idea.

Dave

Benjamin Smedberg

unread,

Jan 10, 2013, 3:09:06 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On 1/10/2013 2:35 PM, Dave Mandelin wrote:
> You mean for detecting critical bugs? Yes, that works, although I think an in-Bugzilla listing of untriaged bugs would be far better. But that alone doesn't provide anything for recording triage decisions.

In my components, the triage decision is recorded by setting the
priority from "--" to one of the values. I think it would be reasonable
to standardize this across engineering.

>
>> Many of the other teams have some kind of system in place. For instance,
>> Release Engineering uses the priority flag in a consistent manner:
>> https://wiki.mozilla.org/ReleaseEngineering/Bugzilla/Triage
> Perhaps the actual practice is more consistent, but based on the wiki article, I would dispute the assertion "uses the priority flag in a consistent manner:

Hrm, releng used to set those priorities as a group with consistent
meaning. I wonder whether/why that changed. In any case, let's imagine
that instead of "may" that everyone uses the priority flag consistently.

>
>> I am in the process of adopting a very similar system for the bugs in my
>> modules. I have a tool I'm working on to make reprioritizing and
>> tracking bugs, patches, and reviews easier across the various bugzilla
>> product/components.
> Every person and team having to build their own idiosyncratic systems is *exactly* the kind of thing that I want to stop in favor of one simple system that works everywhere and enables scalable tool building and cross-team coordination.

I certainly think that we should share tools. I don't know of a good
tool, so I'm endeavoring to create one and hopefully other teams can use
it too.

Maybe I don't understand your original post. If you think you already
have a triage system which works, and you'd like to propose that we
adopt that system across a wider set of groups, I'd love to hear it. But
if we're still trying to figure out what the most functional triage
system is, I think it would be far better to try experiments on smaller
"willing" teams and apply that to the entire project later.

>>> - Any bug can be marked non-actionable with a comment. That way, everyone will know developers won't be working on it unless something changes or more information is provided.
>> The NEEDINFO flag, perhaps combined with whiteboard notes, is usually
>> good enough for this.
> Not in my experience. There are many reasons a bug can be non-actionable. And I want non-actionability to clear and obvious to all readers, not something where you have to know that 'needinfo' is a codeword for 'non-actionable' depending on context.

And the whiteboard doesn't work? Are you looking for a
machine-readable/searchable notation, or just something you can see at a
glance? I've never had a problem with human-readable whiteboard
notations for that sort of thing.

Or... what's the problem you're trying to solve with this notation which
isn't already solved?

>>
>> Lastly, I think it would be smart for module owners to consider finding
>> and delegating a specific person (employee or volunteer) to serve as
>> "module triage peer". Module ownership and peerdom is currently is
>> focused on code reviews, but I think it could be valuable to explicitly
>> delegate the bug prioritization tasks through a peer nomination just as
>> we delegate patch review and other code decisions today.
> Already done for SpiderMonkey. It is a good idea.

How has that worked? Who's the triage peer?

--BDS

Benjamin Smedberg

unread,

Jan 10, 2013, 3:14:51 PM1/10/13

to Steve Wendt, dev-pl...@lists.mozilla.org

On 1/10/2013 2:35 PM, Steve Wendt wrote:
>
> It also needs to have some accommodation for Joe Developer in the open
> source community, who decides that bugs abc and xyz, which are off the
> radar of Mozilla developers, are important to him or his paying client.

What kind of accommodation do you mean? I think we should tune the
Mozilla project bugzilla and engineering to express the priorities of
the project clearly. Certainly companies and individuals can have
alternate priorities, but I don't think we should make any attempt to
express those within bugzilla.

--BDS

Dave Mandelin

unread,

Jan 10, 2013, 3:31:22 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Thursday, January 10, 2013 12:09:06 PM UTC-8, Benjamin Smedberg wrote:
> On 1/10/2013 2:35 PM, Dave Mandelin wrote:
>
> > You mean for detecting critical bugs? Yes, that works, although I think an in-Bugzilla listing of untriaged bugs would be far better. But that alone doesn't provide anything for recording triage decisions.
>
> In my components, the triage decision is recorded by setting the
> priority from "--" to one of the values. I think it would be reasonable
> to standardize this across engineering.

That's one way to do it. One issue is that because anyone can set those fields, they are very noisy. So I'd want us to restrict access at first, or at least have an easy way for triage drivers to get notified of changes.

> >> Many of the other teams have some kind of system in place. For instance,
> >> Release Engineering uses the priority flag in a consistent manner:
> >> https://wiki.mozilla.org/ReleaseEngineering/Bugzilla/Triage
>
> > Perhaps the actual practice is more consistent, but based on the wiki article, I would dispute the assertion "uses the priority flag in a consistent manner:
>
> Hrm, releng used to set those priorities as a group with consistent
> meaning. I wonder whether/why that changed. In any case, let's imagine
> that instead of "may" that everyone uses the priority flag consistently.

I think that would take care of priorities.

> >> I am in the process of adopting a very similar system for the bugs in my
> >> modules. I have a tool I'm working on to make reprioritizing and
> >> tracking bugs, patches, and reviews easier across the various bugzilla
> >> product/components.
>
> > Every person and team having to build their own idiosyncratic systems is *exactly* the kind of thing that I want to stop in favor of one simple system that works everywhere and enables scalable tool building and cross-team coordination.
>
> I certainly think that we should share tools. I don't know of a good
> tool, so I'm endeavoring to create one and hopefully other teams can use
> it too.

I only ever had time to create very simple tools that weren't really usable by someone else.

> Maybe I don't understand your original post. If you think you already
> have a triage system which works, and you'd like to propose that we
> adopt that system across a wider set of groups, I'd love to hear it. But
> if we're still trying to figure out what the most functional triage
> system is, I think it would be far better to try experiments on smaller
> "willing" teams and apply that to the entire project later.

My intention was really to raise the issue and see what responses people had. I think the current triage situation is a problem, but since nothing is really being done about it, I think either people aren't aware of it as a problem, or do think it's a problem, but don't feel they have the ability to do anything about it.

I've learned that if I am too abstract in raising issues, people often have no idea what I'm talking about. So I tacked on a concrete strawman as an example of what I think a better practice would be. It is similar to what the JS component is using, and even without tool support beyond the whiteboard, it's workable if not amazing, but it may not be the right system to use with the larger group.

> >>> - Any bug can be marked non-actionable with a comment. That way, everyone will know developers won't be working on it unless something changes or more information is provided.
>
> >> The NEEDINFO flag, perhaps combined with whiteboard notes, is usually
> >> good enough for this.
>
> > Not in my experience. There are many reasons a bug can be non-actionable. And I want non-actionability to clear and obvious to all readers, not something where you have to know that 'needinfo' is a codeword for 'non-actionable' depending on context.
>
> And the whiteboard doesn't work? Are you looking for a
> machine-readable/searchable notation, or just something you can see at a
> glance? I've never had a problem with human-readable whiteboard
> notations for that sort of thing.

I've observed friction and communication problems around non-actionable bugs, so I figure whatever we have now must not be good enough. In general, I would like Bugzilla to have fewer fields, but fields that are more trustworthy and that communicate more unambiguously. I also think it would help to have an obvious vehicle for declarations of non-actionability and negotiations over it.

> Or... what's the problem you're trying to solve with this notation which
> isn't already solved?

That people still ask for fixes on non-actionable bugs. :-)

> >> Lastly, I think it would be smart for module owners to consider finding
> >> and delegating a specific person (employee or volunteer) to serve as
> >> "module triage peer". Module ownership and peerdom is currently is
> >> focused on code reviews, but I think it could be valuable to explicitly
> >> delegate the bug prioritization tasks through a peer nomination just as
> >> we delegate patch review and other code decisions today.
>
> > Already done for SpiderMonkey. It is a good idea.
>
> How has that worked? Who's the triage peer?

Naveed for most things, but dvander for IonMonkey. It's been working pretty well. It's a continuation of something I was using individually in a text file. Now the data is visible to everyone and triaging effort can be distributed.

I think they still have trouble keeping up with the flood of bugs at all times. The flood of bugs is just a fact, but we could distribute triaging responsibility more widely, or create tool support to speed up the issue, like a list of untriaged bugs with key data and an interface that allows triaging decisions to be recorded quickly.

Dave

Steve Wendt

unread,

Jan 10, 2013, 3:33:54 PM1/10/13

to

No, and I think the way Bugzilla works currently satisfies it. I was
just responding to my perception that Dave was narrowly focused on
internal groups (which of course *is* the most important).

Steve Wendt

unread,

Jan 10, 2013, 3:39:19 PM1/10/13

to

On 1/10/2013 12:31 PM, Dave Mandelin wrote:

>> Or... what's the problem you're trying to solve with this notation which
>> isn't already solved?
>
> That people still ask for fixes on non-actionable bugs. :-)

That's not a problem that can be solved; even if Bugzilla were private,
there would still be noise elsewhere (although perhaps out of sight of
anyone who wished to ignore it).

Liz Henry

unread,

Jan 10, 2013, 3:40:06 PM1/10/13

to dev-pl...@lists.mozilla.org

On 1/9/13 7:01 PM, Dave Mandelin wrote:
> I've been thinking about bug triage for a long time, and since it's a
> new year, I thought it might be a good time to try something new. My
> observation is that our engineering bug triage practices as of today
> <blunt>mostly range from crude to nonexistent</blunt>. The release
> managers pick up the critical bugs, but they don't track less
> important bugs and bugs that are important for technical reasons, and
> it seems like they have to do a lot of manual work and a lot of
> reminding of engineering teams. Other issues stem from the lack of
> triage, like the thousands of old open bugs littering the system, or
> the fact that contributors can't use Bugzilla to find good things to
> work on.

Hi Dave, and everyone! Benjamin, thanks for the introduction. I'm new at
Mozilla as of mid-December, and was hired on to the Automation Tools
team as a Bugmaster to address exactly this sort of issue. In the past I
have worked as an IT person at various universities, as a network
engineer, and as a software developer. I like to teach, speak at
conferences. I love books, writing, wikis, and hackerspaces, and am
based in the San Francisco office. Here is my wiki.mozilla.org page:
https://wiki.mozilla.org/User:Lizhenry .

It was really nice to wake up this morning and see all the discussion
here on bug triage and workflow. In the last few weeks, I've been
talking with developers individually, listening in on meetings, reading
bugzilla and mailing list threads, the Bugzilla Anthropology Project
interviews, and anything else I can find to gather up the history of
people's opinions on managing BMO. Watching Matti's triaging of incoming
Firefox bugs has also been very educational.

For Q1 2013, I'm making plans to organize and expand community
involvement in early bug triage. It seems pretty clear to me that
different modules, projects, and teams have evolved a lot of different
folksonomies ie systems by which to tag. Persuading people away from
those and into some unified system is not necessarily a good goal,
because it would disrupt what's already working for many teams. People
use various meta information fields to tag up a bug. I would love to
have writeups of some of those systems. We can think of them as useful
guidelines that work in particular situations rather than picking one as
the One True Way.

On 1/10/13 1:49 AM, Justin Lebar wrote:

> The idea here is basically to add more metadata to bugs.
>

> Bugzilla already has tons of metadata fields that goes unused.

> Perhaps they're not the right fields, and perhaps better fields would
> be more widely used. But I'm skeptical that any system would meet so
> many teams' needs so well that it would become standard practice to
> to maintain this metadata.
>
> Instead, what seems to work is team-specific metadata. Offhand, I
> can think of team-specific systems used by B2G, MemShrink, Snappy,
> the security team, and release drivers. Since these systems are
> designed to meet a specific need, they usually do a decent job of
> that. And the fact that these systems are separate allows Snappy
> and MemShrink and release drivers to attach different (and
> conflicting) metadata to a bug, reflecting their differing priorities
> and reducing pointless arguments.

I think Justin expresses this very well!

I'm looking for more ways, not fewer, to compartmentalize the bigger
buckets of information that might benefit either developers or triagers
or QA (or other BMO users). One goal would be to catch (earlier) more
critical bugs that might block releases, that cause crashes or are
security problems. Another is simply to make the community happy by
addressing their issues clearly. And another - maybe most crucially for
other goals - is to break down big information categories into more
manageable chunks so that a human can understand them.

On 1/10/13 8:37 AM, Benjamin Smedberg wrote:
> Bug triage is a "gateway drug" to coding, and really helps people who
> want to write patches to know the project and who's working on what.
> One of our first community efforts should be to build out the
> "pre-code" community:
>
> * Incoming bug triage
> * Manual testing

>* Finding regression ranges
> etc * Writing testcases

I agree with this 100% ! Part of the difficulties with the many great
suggestions from developers: In order to triage thoroughly, you need to
learn many of the skills that you'd need to actually fix the bug.
Speccing out the different aspects of early triage will help bring in
more contributors -- people with some but not all of those skills. I
picture the triage or bugmaster community as a good incubator for
developer/contributor skills.

On 1/10/13 8:37 AM, Benjamin Smedberg wrote:
> Lastly, I think it would be smart for module owners to consider
> finding and delegating a specific person (employee or volunteer) to
> serve as "module triage peer". Module ownership and peerdom is
> currently is focused on code reviews, but I think it could be
> valuable to explicitly delegate the bug prioritization tasks through
> a peer nomination just as we delegate patch review and other code
> decisions today.

I also like this idea very much. Right now I have the wiki page at
https://wiki.mozilla.org/Modules/All as a guide to module ownership.
One way we could approach early triage is that I can hook community
contributors in to a particular team, product, or component.

The wiki would be a good place to start documenting some of the many
tagging-and-searching systems that people or teams use now; not on
module owners page but a page based on it or linked from it.

Despite what I said about there not being One True Way, it may be useful
for there to be some more widely accepted demarcation point to bug
triaging that happens early in the bug life cycle. If we implement the
READY status, that might be useful to indicate that a bug has left the
early triage state and entered another state. It isn't immediately
obvious how that will help catch more un-loved bugs, but I think it is
worth trying. It is not just that developers or team leads might be
interested in what is "ready" to be worked on next, it is that early
triagers with various skillsets need to have ways to find where they can
be most helpful.

It looks like the new deployment of the BMO cluster will address some of
the performance issues people have. I also have high hopes for the
dashboard improvements the bteam is adding.

These are my initial thoughts on bugmastering, again, based on my first
impressions. I would love feedback in person or through email (or from
the list of course) -- especially from people who have done triage on
incoming bugs. In general I am very interested in everyone's ideas for
using Bugzilla, building community, and building awesome tools either in
BMO or with the BzAPI. I am enjoying Mozilla very much so far and look
forward to working with you all (and to contributing some patches of my
own at some point)

Best,

Liz

--
Liz Henry
lhe...@mozilla.com

L. David Baron

unread,

Jan 10, 2013, 4:10:50 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Wednesday 2013-01-09 19:20 -0800, L. David Baron wrote:

> On Wednesday 2013-01-09 19:01 -0800, Dave Mandelin wrote:
> > I've been thinking about bug triage for a long time, and since
> > it's a new year, I thought it might be a good time to try
> > something new. My observation is that our engineering bug triage
> > practices as of today <blunt>mostly range from crude to
> > nonexistent</blunt>. The release managers pick up the critical
> > bugs, but they don't track less important bugs and bugs that are
> > important for technical reasons, and it seems like they have to do
> > a lot of manual work and a lot of reminding of engineering teams.
> > Other issues stem from the lack of triage, like the thousands of
> > old open bugs littering the system, or the fact that contributors
> > can't use Bugzilla to find good things to work on.
>

> I think I've basically stopped using the bug system because it's too
> slow; it takes 10-30 seconds for a bug to load, and similar times

> for queries to run. So I keep work to do in a plain text file and
> ignore the bug system.

So I've been investigating the performance a bit, and I'm starting
to think this may be less Bugzilla's fault than Firefox's fault.
And I have been noticing that Firefox seems to be slowing down over
the time I've had the browser running (I typically don't restart
Firefox for days/weeks) more than it used to.

In particular, while loading a Bugzilla bug with the Gecko profiler
(which I couldn't get to work), I got this data on the error
console, which shows a 5.2 second GC pause (which I think does match
my experience lately; the jank has been pretty bad):

GC(T+488342.8) Total Time: 9171.1ms, Compartments Collected: 1192, Total Compartments: 1194, MMU (20ms): 0%, MMU (50ms): 0%, SCC Sweep Total: 209.2ms, SCC Sweep Max Pause: 209.2ms, Max Pause: 5232.6ms, Allocated: 880MB, +Chunks: 0, -Chunks: 1
Slice: 0, Pause: 5232.6 (When: 0.0ms, Reason: SET_NEW_DOCUMENT): Begin Callback: 0.1ms, Wait Background Thread: 0.2ms, Mark Discard Code: 22.4ms, Purge: 0.2ms, Mark: 67.2ms, Mark Roots: 66.4ms
Slice: 5, Pause: 43.4 (When: 5730.2ms, Reason: INTER_SLICE_GC): Mark: 40.8ms
Slice: 7, Pause: 43.0 (When: 6033.6ms, Reason: INTER_SLICE_GC): Mark: 40.3ms
Slice: 28, Pause: 44.1 (When: 7629.4ms, Reason: INTER_SLICE_GC): Mark: 42.4ms
Slice: 53, Pause: 42.4 (When: 12655.3ms, Reason: INTER_SLICE_GC): Mark: 40.0ms
Slice: 62, Pause: 42.1 (When: 13517.9ms, Reason: INTER_SLICE_GC): Mark: 40.6ms
Slice: 63, Pause: 43.1 (When: 13697.0ms, Reason: INTER_SLICE_GC): Mark: 40.3ms
Slice: 64, Pause: 43.2 (When: 13887.1ms, Reason: INTER_SLICE_GC): Mark: 40.3ms
Slice: 66, Pause: 43.0 (When: 14138.4ms, Reason: INTER_SLICE_GC): Mark: 40.2ms
Slice: 67, Pause: 43.7 (When: 14298.6ms, Reason: INTER_SLICE_GC): Mark: 40.9ms
Slice: 82, Pause: 42.5 (When: 16286.8ms, Reason: INTER_SLICE_GC): Mark: 40.5ms
Slice: 83, Pause: 42.2 (When: 16339.9ms, Reason: INTER_SLICE_GC): Mark: 40.6ms
Slice: 84, Pause: 42.3 (When: 16383.2ms, Reason: INTER_SLICE_GC): Mark: 40.9ms
Slice: 85, Pause: 43.2 (When: 16532.3ms, Reason: INTER_SLICE_GC): Mark: 40.3ms
Slice: 86, Pause: 43.3 (When: 16692.9ms, Reason: INTER_SLICE_GC): Mark: 40.3ms
Slice: 87, Pause: 43.5 (When: 16840.1ms, Reason: INTER_SLICE_GC): Mark: 40.6ms
Slice: 88, Pause: 42.6 (When: 16973.4ms, Reason: INTER_SLICE_GC): Mark: 41.0ms
Slice: 89, Pause: 43.3 (When: 17121.0ms, Reason: INTER_SLICE_GC): Mark: 40.2ms
Slice: 90, Pause: 43.7 (When: 17268.1ms, Reason: INTER_SLICE_GC): Mark: 40.8ms
Slice: 91, Pause: 43.4 (When: 17414.7ms, Reason: INTER_SLICE_GC): Mark: 40.4ms
Slice: 92, Pause: 42.3 (When: 17560.2ms, Reason: INTER_SLICE_GC): Mark: 40.6ms
Slice: 93, Pause: 43.3 (When: 17700.5ms, Reason: INTER_SLICE_GC): Mark: 40.3ms
Slice: 94, Pause: 42.0 (When: 17880.1ms, Reason: INTER_SLICE_GC): Mark: 40.4ms
Slice: 95, Pause: 42.5 (When: 18027.1ms, Reason: INTER_SLICE_GC): Mark: 40.9ms
Slice: 96, Pause: 43.3 (When: 18184.9ms, Reason: INTER_SLICE_GC): Mark: 40.5ms
Slice: 97, Pause: 43.5 (When: 18318.2ms, Reason: INTER_SLICE_GC): Mark: 40.6ms
Slice: 98, Pause: 43.3 (When: 18454.2ms, Reason: INTER_SLICE_GC): Mark: 40.4ms
Slice: 99, Pause: 43.1 (When: 18597.8ms, Reason: INTER_SLICE_GC): Mark: 40.2ms
Slice: 100, Pause: 43.3 (When: 18742.3ms, Reason: INTER_SLICE_GC): Mark: 40.4ms
Slice: 122, Pause: 43.1 (When: 20156.5ms, Reason: INTER_SLICE_GC): Mark: 40.3ms
Slice: 123, Pause: 43.2 (When: 20299.5ms, Reason: INTER_SLICE_GC): Mark: 40.3ms
Slice: 126, Pause: 1164.6 (When: 20506.3ms, Reason: CC_FORCED): Mark: 553.7ms, Mark Delayed: 7.4ms, Sweep: 579.7ms, Mark During Sweeping: 72.5ms, Mark Weak: 14.2ms, Mark Gray: 57.3ms, Mark Gray and Weak: 0.9ms, Finalize Start Callback: 2.4ms, Sweep Atoms: 19.8ms, Sweep Compartments: 190.9ms, Sweep Discard Code: 21.8ms, Sweep Tables: 84.7ms, Sweep Cross Compartment Wrappers: 3.2ms, Sweep Base Shapes: 47.5ms, Sweep Intital Shapes: 30.1ms, Sweep Type Objects: 2.2ms, Discard Analysis: 83.9ms, Discard TI: 6.5ms, Free TI Arena: 0.1ms, Sweep Types: 70.8ms, Clear Script Analysis: 6.1ms, Sweep Object: 17.8ms, Sweep String: 3.4ms, Sweep Script: 12.3ms, Sweep Shape: 219.0ms, Finalize End Callback: 28.3ms, Deallocate: 0.2ms, End Callback: 11.0ms
Totals: Begin Callback: 0.1ms, Wait Background Thread: 0.4ms, Mark Discard Code: 22.4ms, Purge: 0.2ms, Mark: 3163.1ms, Mark Roots: 66.4ms, Mark Delayed: 7.4ms, Sweep: 579.7ms, Mark During Sweeping: 72.5ms, Mark Weak: 14.2ms, Mark Gray: 57.3ms, Mark Gray and Weak: 0.9ms, Finalize Start Callback: 2.4ms, Sweep Atoms: 19.8ms, Sweep Compartments: 190.9ms, Sweep Discard Code: 21.8ms, Sweep Tables: 84.7ms, Sweep Cross Compartment Wrappers: 3.2ms, Sweep Base Shapes: 47.5ms, Sweep Intital Shapes: 30.1ms, Sweep Type Objects: 2.2ms, Discard Analysis: 83.9ms, Discard TI: 6.5ms, Free TI Arena: 0.1ms, Sweep Types: 70.8ms, Clear Script Analysis: 6.1ms, Sweep Object: 17.8ms, Sweep String: 3.4ms, Sweep Script: 12.3ms, Sweep Shape: 219.0ms, Finalize End Callback: 28.3ms, Deallocate: 0.2ms, End Callback: 11.0ms

Are there known bugs on stuff like this? (smaug pointed out
https://bugzilla.mozilla.org/show_bug.cgi?id=820602 , which could be
related.)

David Bruant

unread,

Jan 10, 2013, 4:59:58 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

Le 10/01/2013 04:01, Dave Mandelin a écrit :
> So, how about we start using modern bug triage practices? Ideas welcome. Is anyone already using a system they are happy with?

I don't know what you mean by "modern bug triage practices", but I've
been interested by the approach they take at Github. Starting slide 55
of https://speakerdeck.com/holman/how-github-uses-github-to-build-github
Let's quote some slides for the sake of it:
Slide 62:
"A priority is observed, not manufactured or assigned.
Otherwise, it's necessarily not a priority"
Slide 63:
"If it were really important, it'd already be done"
Slide 65:
"Resist adding meta-work. It gets in the way of doing real work."

I don't know to what extent it could be applied to Mozilla. Maybe it
also has a lot to do with how the organization is structured.

David

Dave Townsend

unread,

Jan 10, 2013, 5:02:29 PM1/10/13

to

On 1/9/2013 7:01 PM, Dave Mandelin wrote:
> I've been thinking about bug triage for a long time, and since it's a new year, I thought it might be a good time to try something new. My observation is that our engineering bug triage practices as of today <blunt>mostly range from crude to nonexistent</blunt>. The release managers pick up the critical bugs, but they don't track less important bugs and bugs that are important for technical reasons, and it seems like they have to do a lot of manual work and a lot of reminding of engineering teams. Other issues stem from the lack of triage, like the thousands of old open bugs littering the system, or the fact that contributors can't use Bugzilla to find good things to work on.
>

> So, how about we start using modern bug triage practices? Ideas welcome. Is anyone already using a system they are happy with?
>

> To make things more concrete, I think a relatively simple system like this could meet many of our needs:
>

> - New bugs are assigned an importance of 1, 2, 3, or N. 1 is for critical bugs, 2 is default, 3 is for bugs that probably shouldn't get fixed (because there are too many more important bugs). N is for things that don't need an importance, mostly the large number of developer-filed "I need to land a patch that does X" bugs. These states let people know what's most worth fixing.
> - Working on importance 1 bugs takes priority over working on other bugs.

We already do exactly this in the Jetpack product. Once a week we look
over all bugs with no priority and give them one or ask for any
follow-up questions that we need to make a decision.

Gavin Sharp

unread,

Jan 10, 2013, 5:34:49 PM1/10/13

to L. David Baron, Dave Mandelin, dev. planning

On Thu, Jan 10, 2013 at 1:10 PM, L. David Baron <dba...@dbaron.org> wrote:
> In particular, while loading a Bugzilla bug with the Gecko profiler
> (which I couldn't get to work), I got this data on the error
> console, which shows a 5.2 second GC pause (which I think does match
> my experience lately; the jank has been pretty bad):

Interesting data point. I used to have very bad GC/CC pauses a few
months ago, but I have not noticed any issues recently (without
changing my usage patterns much at all), and certainly haven't seen
these kind of hangs with Bugzilla.

Gavin

Nicholas Nethercote

unread,

Jan 10, 2013, 7:11:25 PM1/10/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Thu, Jan 10, 2013 at 11:14 AM, Dave Mandelin <dman...@gmail.com> wrote:
>
>> The bugs that get fixed are the ones where someone decides "I'm going
>> to work on this", either because they decided it themselves, or
>> because someone convinced them to.
>
> That strikes me as a pretty ramshackle approach to shipping quality software. Why not "Decide which bugs are truly most worth fixing, record the decision, and then fix those bugs"? We used to do some approximation of that, but that system seems to have collapsed when we went to rapid release, and has been only partially replaced by the release-tracking flags.

The question of authority is critical here. The MemShrink team
doesn't have any hard authority to force someone to fix a bug.

Size of a bug is also relevant. Generational GC is a MemShrink:P1 and
a JS:P1, but it's taking a long time because it's a huge task. So you
can't call it a blocker when we have rapid releases.

Nick

Chris Peterson

unread,

Joshua Cranmer

unread,

Jan 10, 2013, 9:12:36 PM1/10/13

to

On 1/9/2013 9:01 PM, Dave Mandelin wrote:
> So, how about we start using modern bug triage practices? Ideas welcome. Is anyone already using a system they are happy with?
>

> To make things more concrete, I think a relatively simple system like this could meet many of our needs:
>
> - New bugs are assigned an importance of 1, 2, 3, or N. 1 is for critical bugs, 2 is default, 3 is for bugs that probably shouldn't get fixed (because there are too many more important bugs). N is for things that don't need an importance, mostly the large number of developer-filed "I need to land a patch that does X" bugs. These states let people know what's most worth fixing.
> - Working on importance 1 bugs takes priority over working on other bugs.
>

> - Any bug can be marked non-actionable with a comment. That way, everyone will know developers won't be working on it unless something changes or more information is provided.
>

> - An open, confirmed bug can be in one of three states: active, backlog, iced. Active bugs are being worked on now, and a fix is expected soon. Backlog bugs are not being worked on, but the backlog is reviewed regularly to pick new active bugs. Iced bugs are not being worked on, and won't be reviewed unless specially asked for. These states let people know what will be fixed soon, later, or never.
>

The Thunderbird people had a discussion about this kind of issue last
September; a more comprehensive summary can be found here:
<https://mail.mozilla.org/pipermail/tb-planning/2012-September/002134.html>.
The simplest summary that's relevant here, I think, is a few pieces:

1. "Priority" actually boils down to four orthogonal categories: how
many are affected, how bad is it, how easy is it to reproduce, and how
hard is it to fix. Not all of these factors are immediately evident in
Bugzilla.

2. Making lists is easy; the problem is getting lists down. The
suggestion is that work lists need to be small (a 300-bug list that is
used as input to two people is unmanageable), and that there needs to be
an ability to revisit why some bugs appear to stay on the list forever.

3. Lists need visibility. This is more particularly important for
Thunderbird, as it is now driven the volunteer contributors instead of
paid staff.

Dave Mandelin

unread,

Gervase Markham

unread,

Jan 11, 2013, 8:43:27 AM1/11/13

to Dave Mandelin

On 10/01/13 18:41, Dave Mandelin wrote:
> It is slow for me as well, although for some reason that doesn't
> bother me much. It takes 4-5 seconds for a bug to come up, and the
> text query that I tried took about 45 seconds. I tried Another Open
> Source Bug Tracker and got 1-2 seconds and 3-5 seconds respectively
> for those actions.

Did the system you were testing with have 800,000 bugs, and many
simultaneous users?

The REST API hasn't helped the load problem, of course; it turns out
that when you make it much easier for people to get data out of
Bugzilla, they want more of it :-)

> It's very interesting you say this, because that's how I did JS bug
> triage and tracking for about a year. And I know others that are
> using text files or paper lists. I think it's a sign of something
> wrong if people are abandoning the bug system for plain text files.
> (Not to mention github, etc.)

I certainly agree with that.

> I also have noticed that a lot of time gets spent reading and
> rereading comment streams, and I've thought of similar summary areas
> or categories. I don't know either how to make it really work but I
> think it makes sense for complex tasks.

If people need bespoke tools, that's why we have the REST API (and
several people have built them). That doesn't mean we don't need to make
Bugzilla faster, but it does provide an "out" for people in severe pain.
Remember, tools can be browser-based shims for the main UI as well as
whole new websites.

Gerv

Gervase Markham

unread,

Jan 11, 2013, 8:45:39 AM1/11/13

to Ed Morley

On 10/01/13 10:24, Ed Morley wrote:
> Bug 687261 is filed to turn off comment-searching by default, which I
> think would help with the perceived slowness.

Looks like glob plans to fix that next week.

Gerv

unread,

Jan 11, 2013, 10:36:31 AM1/11/13

to Gervase Markham, dev-pl...@lists.mozilla.org

I don't think this is true, at least given the current behavior of
bugzilla. Taking bug 801394 as an example, we want to leave the bug in
the following state:

* crash-stats links to the bug as a known issue. It doesn't say that the
bug is resolved.
* People who search for existing bugs using quicksearch still find the bug.
* The bug status needs to clearly indicate that we are actively looking
for SUMO or a volunteer to provide more information (such as a sample of
the infected DLL)

RESOLVED INCOMPLETE NEEDINFO gets close, although I don't think there's
a way to mark NEEDINFO for "somebody in general". But it still won't
show up in default quicksearch results and crash-stats will shows the
bug as RESOLVED.

Our current policy is to leave these bugs as NEW P3 with a whiteboard
comment indicating that they aren't assignable.

--BDS

David Lawrence

unread,

Jan 11, 2013, 11:07:48 AM1/11/13

to Benjamin Smedberg, dev-pl...@lists.mozilla.org, Gervase Markham

On 01/11/2013 10:36 AM, Benjamin Smedberg wrote:
> RESOLVED INCOMPLETE NEEDINFO gets close, although I don't think there's a way to mark NEEDINFO for "somebody in general". But it
> still won't show up in default quicksearch results and crash-stats will shows the bug as RESOLVED.

NEEDINFO for "somebody in general" is doable by selecting "Anyone" from the drop down next to the Needinfo checkbox. Basically
it is setting the needinfo? flag with a blank requestee.

FYI, closing a bug with a previously set needinfo flag, the flag will be cleared automatically. We would need to redo some
things if you need the needinfo flag to remain intact when closing RESOLVED/INCOMPLETE.

FWIW, a closed bug can have the needinfo flag set after closing if any of the 'status-firefox*' tracking flags have a value of
'affected' set. This was a custom request a while back.

dkl

--
David Lawrence
d...@mozilla.com

Liz Henry

unread,

Jan 11, 2013, 2:27:49 PM1/11/13

to dev-pl...@lists.mozilla.org

On 1/10/13 6:22 PM, Dave Mandelin wrote:

> I understand the point about disruption, but I really do not think the current situation is a good one. dbaron is stuck using text files, and the JS team is limping along with whiteboard tags inside of Bugzilla. There has to be a better way.
>
> I'm wondering what you think of my point that while bug-system-per-team is good for allowing each team to try to create a system for its exact needs, it scales very poorly both in terms of the cost of building redundant systems, and in the communication and visibility barriers it creates.
>

Hi Dave! That is a great point. I do think we will find points of
commonality that will be useful to make standard, and that will scale
better. I would love to hear more about what the JS team does currently
with whiteboard tags, i.e., how you are limping along. As we collect
(and document) how people are using the meta-information within teams I
think a clearer picture will emerge.

From your original email, I would like to explore this idea,

> An open, confirmed bug can be in one of three states: active, backlog, iced. Active bugs are being worked on now, and a fix is expected soon. Backlog bugs are not being worked on, but the backlog is reviewed regularly to pick new active bugs. Iced bugs are not being worked on, and won't be reviewed unless specially asked for. These states let people know what will be fixed soon, later, or never.

I like this, but I wonder who would assign those states. I also
wonder how many bugs a person would declare they are active on at the
same time. Would we end up with some people with 300 bugs active and
some with 1 active at any given time?

If you have time next week maybe we could chat and you could show me the
nitty gritty of the system you have set up for the JS team, how it works
and why it is painful for either y'all or for whoever needs to know what
state your bugs are in.

Robert O'Callahan

unread,

Jan 11, 2013, 5:22:55 PM1/11/13

to Benjamin Smedberg, dev-pl...@lists.mozilla.org, Gervase Markham

On Sat, Jan 12, 2013 at 4:30 AM, Benjamin Smedberg <benj...@smedbergs.us>wrote:

> On 1/11/2013 8:42 AM, Gervase Markham wrote:
>

>> Pretty much every metadata field Bugzilla has is there because someone
>> said it would make things better...
>>
>

> Often without considering whether the cost represented by a field is worth
> the benefit, or whether people will use it consistently enough to make the
> field actually useful. The "OS" field is the most pointless field in
> bugzilla, although you could replace "Severity" with a few keywords and
> we'd all be better off.
>

Actually "OS" is one of the few fields I do use!

Maybe there's a lesson in this exchange...

Ehsan Akhgari

unread,

Jan 11, 2013, 8:18:17 PM1/11/13

to rob...@ocallahan.org, dev-pl...@lists.mozilla.org, Benjamin Smedberg, Gervase Markham

On 2013-01-11 5:22 PM, Robert O'Callahan wrote:
> On Sat, Jan 12, 2013 at 4:30 AM, Benjamin Smedberg <benj...@smedbergs.us>wrote:
>
>> On 1/11/2013 8:42 AM, Gervase Markham wrote:
>>

>>> Pretty much every metadata field Bugzilla has is there because someone
>>> said it would make things better...
>>>
>>

>> Often without considering whether the cost represented by a field is worth
>> the benefit, or whether people will use it consistently enough to make the
>> field actually useful. The "OS" field is the most pointless field in
>> bugzilla, although you could replace "Severity" with a few keywords and
>> we'd all be better off.
>>
>
> Actually "OS" is one of the few fields I do use!
>
> Maybe there's a lesson in this exchange...

One size doesn't fit all? :-)

Cheers,
Ehsan

Dave Mandelin

unread,

Jan 11, 2013, 9:00:52 PM1/11/13

to

On Thursday, January 10, 2013 7:31:27 PM UTC-8, David E. Ross wrote:
> On 1/10/13 6:14 PM, Dave Mandelin wrote:
>
> > On Thursday, January 10, 2013 5:04:35 PM UTC-8, David E. Ross wrote:
>
> >> As an end-user, I must ask: If a number of end-users all say they see
> >> something truly discrepant and want it fixed, why would it remain
> >> non-actionable?
>
> > If there is no reasonable path to fix it. A typical example would be
> > a crash that is only known to affect one user and cannot be reproduced by
> > developers. Another example would be a signature in the crash reporting
> > system that is common, but where developers have tried several things to
> > diagnose it, but were unable to get anywhere.
>
> Your first example involves only one user and does not include my "a
> number of end-users".
> Your second example involves multiple users. It might be a bug whose
> fix needs to be deferred because of the difficulty in determining its
> cause, but it should never be declared non-actionable.

Well, it's arguable exactly what you would want to call that bug. Maybe it's non-actionable, maybe it's "we're not going to work on it."

It's true that there's always something more you can try. But we have examples of bugs that we believe we are very unlikely to be able do diagnose, and so would be very unproductive to work on, but which we keep getting asked about. It gets tiring after a while. It's a rare case, though--it's basically one class of related bugs.

Dave

Dave Mandelin

unread,

Jan 11, 2013, 9:05:09 PM1/11/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

One example is that a big city needs to have a higher fraction of its land area devoted to transportation than a small city.

A more direct software example is that in a team of 5 people, it's not too hard to sync up on priorities and for everyone to know what everyone else is doing. On a team of 500, it is very difficult to maintain consistent priorities and for people to know what's going on, so managers, project managers, newsletters, etc. become necessary. In our case, it's currently not possible to find out what's going on either automatically or by using Bugzilla. You pretty much have to know who to talk to and then talk to those people.

Another team size example is that in sports with 5-6 people per team, like basketball, things can be more free-form, but if there are 9-11 people, usually there are more fixed roles and defined plays.

Dave

Dave Mandelin

unread,

Jan 11, 2013, 9:10:40 PM1/11/13

to Dave Mandelin

On Friday, January 11, 2013 5:43:27 AM UTC-8, Gervase Markham wrote:
> On 10/01/13 18:41, Dave Mandelin wrote:
>
> > It is slow for me as well, although for some reason that doesn't
> > bother me much. It takes 4-5 seconds for a bug to come up, and the
> > text query that I tried took about 45 seconds. I tried Another Open
> > Source Bug Tracker and got 1-2 seconds and 3-5 seconds respectively
> > for those actions.
>
> Did the system you were testing with have 800,000 bugs, and many
> simultaneous users?

No idea how many users. It had over 20,000 bugs, so fairly sizable, but not as big. I guess the comparison would depend on the db schema and implementation. If both are doing linear search, then Bugzilla compares well. I would hope that most things are more like constant or logarithmic scaling, although I realize that's not gonna happen for arbitrarily complex queries.

> The REST API hasn't helped the load problem, of course; it turns out
> that when you make it much easier for people to get data out of
> Bugzilla, they want more of it :-)
>
> > It's very interesting you say this, because that's how I did JS bug
> > triage and tracking for about a year. And I know others that are
> > using text files or paper lists. I think it's a sign of something
> > wrong if people are abandoning the bug system for plain text files.
> > (Not to mention github, etc.)
>
> I certainly agree with that.
>
> > I also have noticed that a lot of time gets spent reading and
> > rereading comment streams, and I've thought of similar summary areas
> > or categories. I don't know either how to make it really work but I
> > think it makes sense for complex tasks.
>
> If people need bespoke tools, that's why we have the REST API (and
> several people have built them). That doesn't mean we don't need to make
> Bugzilla faster, but it does provide an "out" for people in severe pain.
> Remember, tools can be browser-based shims for the main UI as well as
> whole new websites.

The REST API is very cool. I have built tools using it myself, so I certainly appreciate it. And I have used it for local caching.

But I tolerate the current speed pretty well so maybe I'm the wrong person to be thinking about it too much. :-)

Dave

Dave Mandelin

unread,

Jan 11, 2013, 9:26:11 PM1/11/13

to Dave Mandelin, dev-pl...@lists.mozilla.org, lhe...@mozilla.com, rob...@ocallahan.org

On Thursday, January 10, 2013 8:57:46 PM UTC-8, Robert O'Callahan wrote:
> I think it's an open question whether the benefits of forcing everyone to
> use the same triage setup outweigh the costs.
>
> I *do* see great potential benefit --- for triage, and for a lot of other
> activities --- in Bugzilla being a lot faster and more responsive.
>
> Anyway, I suggest try getting one team to adopt your proposal. If that
> works well, try getting another team to use it. And so on, as long as the
> benefits keep outweighing the costs. I think it would be unwise to force
> everyone to change at once. We don't want to try and then find out the
> costs outweighed the benefits.

I already implemented a similar system for the JS component, and it immediately made my life easier. (You'd have to check with Naveed to see if he still thinks it's a good idea now that he's the main operator.) More than one person on this thread has described currently using a similar system. It's standard stuff, used all over the place for years, including Mozilla, until rapid release disrupted the old routines.

As to lobbying for adoption team by team, I don't have time for it. And anyway, my primary aim isn't to require a specific triage system. My aim is to convince people that we are not doing a good job at:

- reviewing each incoming bug
- making rational decisions about what to fix
- communicating those decisions to each other and to our users

And see what changes people think might help.

Dave

Ehsan Akhgari

unread,

Jan 11, 2013, 9:34:34 PM1/11/13

to Dave Mandelin, dev-pl...@lists.mozilla.org, rob...@ocallahan.org, lhe...@mozilla.com

On 2013-01-11 9:26 PM, Dave Mandelin wrote:
> As to lobbying for adoption team by team, I don't have time for it. And anyway, my primary aim isn't to require a specific triage system. My aim is to convince people that we are not doing a good job at:
>
> - reviewing each incoming bug
> - making rational decisions about what to fix
> - communicating those decisions to each other and to our users

I think reviewing each incoming bug can be a big step to improve the
current situation. If we have a system in place to make sure that each
and every incoming bug is at least moved to the right component so that
it catches the eyes of those watching that component, do you think we'll
be in a better spot?

Cheers,
Ehsan

Dave Mandelin

unread,

Jan 11, 2013, 9:44:28 PM1/11/13

to Dave Mandelin, dev-pl...@lists.mozilla.org

On Friday, January 11, 2013 2:47:22 AM UTC-8, Justin Lebar wrote:
> > - Team-specific tools do nothing for cross-team coordination. Other people won't necessarily be able to find or use a team-specific tool. Requiring people with cross-cutting responsibilities to navigate 23 different team-specific bug systems does not make sense.
>
> > - If bug A is a MemShrink P1, bug B is a Snappy P1, and bug C is a sec-critical, which do you fix? If there isn't a global priority, there isn't really a priority.
>
> I think you're begging the question of whether there ought to be a
> global priority, and indeed whether any such total ordering of bug
> priorities could be meaningful. (Obviously since the set of open bugs
> is finite, there /can/ be a total ordering of bug priorities, so
> that's not the question.)

Maybe the wording "global priority" was confusing. I meant a single priority, assigned by the team doing the work, that clearly indicates to other teams what they are going to work on the soonest.

> > The idea is to make sure every bug report gets looked at, decisions are made about what's worth working on, and people can tell what is being worked on and what bugs will and won't get fixed.
>
> Okay, AIUI there are four separate ideas here:
>
> 1. make sure every bug gets looked at
> 2. establish a total ordering of bug priorities
> 3. indicate which bugs are actively being worked on
> 4. indicate which bugs will and won't get fixed.

Also, making rational decisions about what to do next.

> Innovation at the edges -- another definition of which is "not
> imposing decisions and process on others" -- is a key advantage to the
> way Mozilla works. And recently, small teams with their own agendas
> and priorities (e.g. MemShrink) have proven to be among the most
> effective ways we have of getting things done.
>
> With this in mind, perhaps instead of thinking of this effort as
> "create a group of Manager Types who impose a total ordering of bug
> priorities (2) and force others to do (1), (3), and (4)", we could
> think of this as "create a small team responsible for 1-4." If the
> priorities generated by this team are really awesome, then people will
> pay attention to them, and you'll accomplish 1-2.

No, I want component teams to set priorities for their own bugs. No central group would be capable of doing it well across all the components and products.

> I tend to think that 3-4 aren't a problem, because I can usually tell
> whether a bug is being worked on, and if I can't, I can ask.

That takes a ton of time if you have a lot of bugs to track. Plus it seems unnecessary. Announcing what you're going to do is a normal part of working together.

> But if they're a problem for you, I imagine that a combination of
> technical approaches (e.g. getting Bugzilla to add "bug last updated
> X weeks ago" at the top of the bug)

I have used that, and it works to detect current activity, but it doesn't tell you what's going to happen in the future.

> and non-technical approaches (e.g. "hey, this is a dmandelin:P1 --
> are you actively working on this?") would work.

But less managerial pestering is part of what I would like to achieve. :-)

Dave

Dave Mandelin

unread,

Jan 11, 2013, 9:50:31 PM1/11/13

to dev-pl...@lists.mozilla.org, lhe...@mozilla.com

On Friday, January 11, 2013 11:27:49 AM UTC-8, Liz Henry wrote:
> On 1/10/13 6:22 PM, Dave Mandelin wrote:
>
> > I understand the point about disruption, but I really do not think the current situation is a good one. dbaron is stuck using text files, and the JS team is limping along with whiteboard tags inside of Bugzilla. There has to be a better way.
>
> > I'm wondering what you think of my point that while bug-system-per-team is good for allowing each team to try to create a system for its exact needs, it scales very poorly both in terms of the cost of building redundant systems, and in the communication and visibility barriers it creates.
>
> Hi Dave! That is a great point. I do think we will find points of
> commonality that will be useful to make standard, and that will scale
> better.

That would be a very good thing. And I think there are some points of commonality already, which were in fact unknown to me before this thread.

> I would love to hear more about what the JS team does currently
> with whiteboard tags, i.e., how you are limping along.

We documented it at

https://wiki.mozilla.org/Platform/JS

> As we collect
> (and document) how people are using the meta-information within teams I
> think a clearer picture will emerge.
>
> From your original email, I would like to explore this idea,
>
> > An open, confirmed bug can be in one of three states: active, backlog, iced. Active bugs are being worked on now, and a fix is expected soon. Backlog bugs are not being worked on, but the backlog is reviewed regularly to pick new active bugs. Iced bugs are not being worked on, and won't be reviewed unless specially asked for. These states let people know what will be fixed soon, later, or never.
>
> I like this, but I wonder who would assign those states.

> Setting up additional permissions for effective triage
> is something I'd really rather avoid, since it makes it harder to
> recruit people into the role of triager. I tend to think that editbugs
> is the right barrier, along with social pressure if things are being
> mismarked.

I know, and I tend to agree that although another layer of permissions would help make triage signals more reliable, it is not necessarily the right step for the system as a whole.

> Assuming you watch the bugzilla components you're triaging. If that's
> not the case, then you might need a separate report for "priorities
> changed recently", but that's not hard to accomplish either.

That's actually the hard part, that really pushed me away from trying to revive the original fields. I haven't been able to come up with a filter for bugmail that gives me just the things I need to see, so I can't really go by bugmail. The separate report would provide the information, but then I'd need to remember to use it.

Dave

Dave Mandelin

unread,

Jan 14, 2013, 9:40:52 PM1/14/13

to Dave Mandelin, dev-pl...@lists.mozilla.org, rob...@ocallahan.org

On Saturday, January 12, 2013 5:08:14 AM UTC-8, Robert O'Callahan wrote:
> On Sat, Jan 12, 2013 at 3:05 PM, Dave Mandelin <dman...@gmail.com> wrote:
>
> > A more direct software example is that in a team of 5 people, it's not too
> > hard to sync up on priorities and for everyone to know what everyone else
> > is doing. On a team of 500, it is very difficult to maintain consistent
> > priorities and for people to know what's going on, so managers, project
> > managers, newsletters, etc. become necessary.
>
> I think some of that is necessary, but I think it's also subject to a
> cost-benefit analysis. For example, what exactly is the cost of not having
> consistent priorities? I'm quite certain that it's finite and I can easily
> imagine conditions under which it's less than the cost of ensuring we have
> consistent priorities.

Those things are of course not quantifiable in practice. I'm fairly sure the answer lies somewhere between having enough communication to detect mutually exclusive goals and having 7 managers per developer.

Dave

Gervase Markham

unread,

Jan 15, 2013, 4:14:34 AM1/15/13

to Dave Mandelin

On 15/01/13 02:37, Dave Mandelin wrote:
> That's actually the hard part, that really pushed me away from trying
> to revive the original fields. I haven't been able to come up with a
> filter for bugmail that gives me just the things I need to see, so I
> can't really go by bugmail. The separate report would provide the
> information, but then I'd need to remember to use it.

Are you aware that bugmail comes with a ton of headers indicating the
status of various fields in the bug, which allows you to do client-side
filtering (at least in Thunderbird; YMMV with other clients)?

Gerv

Byron Jones

unread,

Jan 15, 2013, 9:35:57 AM1/15/13

to dev-pl...@lists.mozilla.org, Gervase Markham

or server-side with zimbra filters :)

i have different imap folders for bugs which i am assigned, reported,
cc'd, and by products which i watch, working mostly off the
X-Bugzilla-Reason and X-Bugzilla-Product headers.

--
byron - irc:glob - bugzilla.mozilla.org team -

Daniel Veditz

unread,

Jan 20, 2013, 12:48:18 PM1/20/13

to Byron Jones, mozilla.dev.planning group

On 1/10/2013 6:59 AM, Byron Jones wrote:
> every six weeks i add another six custom fields to bmo, so the bugs
> table is extremely wide.
> currently there are 139 custom fields, most are varchar(64).
> this impacts mysql's ability to cache rows, as wider tables consume more
> memory, so fewer rows can be held in ram.
> the refactoring will basically change from using columns to rows for
> each field.

Yeah, it didn't take much foresight to predict that plan was going to
end in tears once RapidRelease started.

Does this mean we're landing "Sightings" (bug 55970) or the hopefully
simpler version Asa and I advocated in bug 336790? Something different,
I guess: the sightings bug is still open/incomplete (despite working on
landfill) and Asa's bug is still WONTFIX.

How will this affect queries?

-Dan Veditz

Byron Jones

unread,

Jan 20, 2013, 10:48:02 PM1/20/13

to Daniel Veditz, mozilla.dev.planning group

Daniel Veditz wrote:
> Does this mean we're landing "Sightings" (bug 55970) or the hopefully
> simpler version Asa and I advocated in bug 336790? Something
> different, I guess: the sightings bug is still open/incomplete
> (despite working on landfill) and Asa's bug is still WONTFIX.

correct - something different, designed for just tracking rapid releases.

what we're doing is more of a band-aid solution -- taking the current
fields and re-implementing them in a way that shouldn't cause as much
pain to the infrastructure. larger breaking changes can come later, and
separation of the rapid release flags into their own bugzilla extension
paves the way for that.

> How will this affect queries?

we're aiming for 100% compatibility with the current implementation, so
it shouldn't impact any existing queries.

-byron

Robert Kaiser

unread,

Jan 25, 2013, 2:51:23 PM1/25/13

to

L. David Baron schrieb:
> So I've been investigating the performance a bit, and I'm starting
> to think this may be less Bugzilla's fault than Firefox's fault.

Yay, here's the value of dogfooding! ;-)

If we can improve Firefox speed on Bugzilla and that helps our speed to
handle other issues, I'd see that as a good outcome as well - the value
to the rest of the Firefox users may vary depending on how much it
affects non-Bugzilla sites, of course...

Robert Kaiser

Wayne

unread,

Mar 2, 2013, 7:27:50 AM3/2/13

to

This would be a huge improvement if for no other reason than the sooner
you engage the reporter, you may have a far better chance of getting
more information or more testing, for a variety of reasons. For example
if you are touching a bug only several weeks after it was filed, the
user may simply no longer care about it on several levels.

Wayne

unread,

Mar 2, 2013, 8:06:54 AM3/2/13

to

On 1/11/2013 10:36 AM, Benjamin Smedberg wrote:
> On 1/11/2013 8:48 AM, Gervase Markham wrote:
>> On 10/01/13 19:35, Dave Mandelin wrote:
>>> Not in my experience. There are many reasons a bug can be
>>> non-actionable. And I want non-actionability to clear and obvious to
>>> all readers, not something where you have to know that 'needinfo' is
>>> a codeword for 'non-actionable' depending on context.
>> If a bug is definitely non-actionable, it should be RESOLVED
>> <something>. If it's non-actionable by a developer because someone else
>> needs to do something first, then reassign it or use NEEDINFO.
>
> I don't think this is true, at least given the current behavior of
> bugzilla. Taking bug 801394 as an example, we want to leave the bug in
> the following state:
>
> * crash-stats links to the bug as a known issue. It doesn't say that the
> bug is resolved.
> * People who search for existing bugs using quicksearch still find the bug.

To which "people" do you refer? And why is this a bad thing in this case?

> * The bug status needs to clearly indicate that we are actively looking
> for SUMO or a volunteer to provide more information (such as a sample of
> the infected DLL)

Why does this specific bug not have testcase-wanted keyword?

Or perhaps for *currently* non-actionable bugs of this type need
something more stern, like testcase-required.

Among the issues in the original postings is the question of whether a
bug is actionable or important, or some other status. And the focus so
far has been on whether specific fields are useful or not. Isn't it
possible to express this as the previously discussed READY status?

And if not, and it is desirable to have a field that does not require a
person to maintain it, can a virtual field be defined in bugzilla which
self-maintains and derives it's value from various fields in the bug?

Benjamin Smedberg

unread,

Mar 4, 2013, 9:22:29 AM3/4/13

to Wayne, dev-pl...@lists.mozilla.org

On 3/2/2013 8:06 AM, Wayne wrote:
>
>> I don't think this is true, at least given the current behavior of
>> bugzilla. Taking bug 801394 as an example, we want to leave the bug in
>> the following state:
>>
>> * crash-stats links to the bug as a known issue. It doesn't say that the
>> bug is resolved.
>> * People who search for existing bugs using quicksearch still find
>> the bug.
>
> To which "people" do you refer? And why is this a bad thing in this case?

I'm confused; what's a bad thing?

>
>
>> * The bug status needs to clearly indicate that we are actively looking
>> for SUMO or a volunteer to provide more information (such as a sample of
>> the infected DLL)
>
> Why does this specific bug not have testcase-wanted keyword?

Because this is not a case where we need somebody with the skills to
write a testcase. This is a case where we need(ed) to find somebody with
the affected binaries.

> Among the issues in the original postings is the question of whether a
> bug is actionable or important, or some other status. And the focus so
> far has been on whether specific fields are useful or not. Isn't it
> possible to express this as the previously discussed READY status?

It's certainly *possible* to express whether a bug is actionable using a
new READY state. I don't necessarily think it's the right tradeoff, though!

--BDS