Making blocking flags more useful
L. David Baron
realize that the blocking flags we use on bugs don't meet our needs very
well. The fundamental problem is that they only let the drivers or
release managers choose between two options. There are really
considerably more options. These different distinctions also lead to
confusion among developers and among platform users because the meaning
of the flags changes between releases and over the course of a single
release.
(For more background on blocking flags, see
http://www.mozilla.org/roadmap/release-status.html#flags or
http://www.mozillazine.org/articles/article2779.html . Note that I'm
talking about the bug flags and not the patch flags.)
This means that triage of bugs nominated for or marked as "blocking" has
to be repeated many times over the release process because at different
points in the process we care about different distinctions. This
repetition happens in one of two ways: either a single flag is used and
bugs are re-evaluated and shuffled back and forth between the two
states, or a new flag is created for each part of the cycle (beta,
release candidate, etc.) and the same bugs are renominated in each part
of the cycle. The use of one flag for each stage causes causes
significantly more repeated work, and the use of a single flag may cause
more confusion.
I think the states that we (branch drivers) have wanted to distinguish
between during our triage are roughly the following:
1. We will hold the next beta/release candidate for this fix
2. We will hold the final release for this fix
3. We encourage people to work on this for this release (although
patches will still be considered by weighing risk and reward).
4. We will consider a patch for this release.
5. We'd rather you didn't try to fix this for this release.
(In reality there are many gradations between these, especially between
(3) and (4).)
I'd like Bugzilla's user interface to be able to reflect these states
rather than just plus or minus. Fundamentally, the user interface
change I'm proposing is quite simple: instead of "?", "+", and "-"
being the three states of blocking flags, there would be "?" plus a
number of additional states. The set of people who are authorized to
set "?" would remain the same; the set of people who are authorized to
set states other than "?" would also remain the same.
I'm interested in feedback on:
* if people agree or disagree with this proposal,
* whether we want/need a fixed set of vague states (like "P1", "P2",
... "P5"), a fixed set of named states ("blockbeta", "blockfinal",
"want", "consider", "no"), or a configurable set of states (so that
we could, e.g., add/remove betas from the list), or something else,
* which states we need to distinguish, and which we don't, and
* any other improvements people have.
I'm especially interested in feedback from people who have been involved
in triage for the security releases (which I haven't been for quite a
while), since I'm not sure how similar their requirements are.
-David
--
L. David Baron <URL: http://dbaron.org/ >
Technical Lead, Layout & CSS, Mozilla Corporation
Robert Kaiser
> I'm interested in feedback on:
>
> * if people agree or disagree with this proposal,
The SeaMonkey Council does manage it own releases with
blocking-seamonkey* flags, and so basically runs into the same issues as
drivers for those.
For me as member of that Council group, this proposal sounds like a good
idea. With such a system, we could also replace tracking bugs of "would
be good to have for that release" bugs with bug queries for a
want/consider state.
It would also be better to tell people "we won't block on this, but we'd
like to have it" with a real bug state instead of just minusing it.
> * whether we want/need a fixed set of vague states (like "P1", "P2",
> ... "P5"), a fixed set of named states ("blockbeta", "blockfinal",
> "want", "consider", "no"), or a configurable set of states (so that
> we could, e.g., add/remove betas from the list), or something else,
I think concrete states would be better than vague Px states, as
everyone would need to look up somewhere what such vague Px states mean,
while the concrete states would be much more intuitive. Also, the states
used across different releases/flags shouldn't vary too much, so I'm not
sure how much sense a configurable set would make (and not sure how
easily those could be set up in Bugzilla).
> * which states we need to distinguish, and which we don't, and
I actually like the list you made very much ("blockbeta", "blockfinal",
"want", "consider", "no"). I think it covers the different needed stages
while not being too detailed (details should be in comments), esp. as it
makes sense to also query bugs of those different stages separately.
Robert Kaiser
Mike Schroepfer
> I think concrete states would be better than vague Px states, as
> everyone would need to look up somewhere what such vague Px states mean,
> while the concrete states would be much more intuitive. Also, the states
> used across different releases/flags shouldn't vary too much, so I'm not
> sure how much sense a configurable set would make (and not sure how
> easily those could be set up in Bugzilla).
Agreed that human-readable text states (if possible) are preferable.
>
>> * which states we need to distinguish, and which we don't, and
>
> I actually like the list you made very much ("blockbeta", "blockfinal",
> "want", "consider", "no"). I think it covers the different needed stages
> while not being too detailed (details should be in comments), esp. as it
> makes sense to also query bugs of those different stages separately.
>
I'd recommend "blocknextmilestone" as a generic one so we don't have to
change for each milestone.
I'll also note from the 1.5 Cycle when we had a "blocknextmilestone" and
"blockfinal" approximated by a 1.8.b5+ and 1.8+ people got really really
confused about the difference in general to the point that we just moved
everything to "next milestone." Also, the final release blocking list
gets very very large until you basically get up to the point where the
next milestone and final release are equivalent. So I'd almost suggest
dropping "blockfinal" and just leave it as:
blocknextmilestone
want
consider
no
Also note that the best way to approximate this without changing tooling
is to agree that:
a) +=blocking next milestone + want
b) Cleared blocking flag with comment "will consider patch" = consider
c) Minus=no
Combined with being very honest about the "wants" - i.e. there should be
very little thats is a want which isn't a blocker at this point in the
release.
Best,
Schrep
Jonas Sicking
> Hey There,
>> I think concrete states would be better than vague Px states, as
>> everyone would need to look up somewhere what such vague Px states
>> mean, while the concrete states would be much more intuitive. Also,
>> the states used across different releases/flags shouldn't vary too
>> much, so I'm not sure how much sense a configurable set would make
>> (and not sure how easily those could be set up in Bugzilla).
>
> Agreed that human-readable text states (if possible) are preferable.
Same here. These thing can be slightly touchy since people can be
disappointed that they have worked hard on a patch and then get denied
to check it in. So clarity is important.
>>> * which states we need to distinguish, and which we don't, and
>>
>> I actually like the list you made very much ("blockbeta",
>> "blockfinal", "want", "consider", "no"). I think it covers the
>> different needed stages while not being too detailed (details should
>> be in comments), esp. as it makes sense to also query bugs of those
>> different stages separately.
It seems to me that it would be better to just have a 'block' state, and
let the flag name indicate which release it's for. That way if someone
nominates a bug you can flag it 'want' for the beta but 'blocking' for
release. Or any other desired combination.
Also, it took me some thinking to realize the difference between 'want'
and 'consider' (as you say, there is more of a scale than a line between
them). So I think we should make sure to document them well.
Of course, it's always a good idea for whoever flags a bug to also add a
clarifying comment in the bug.
/ Jonas
Serge Gautherie
>>> * which states we need to distinguish, and which we don't, and
>>
>> I actually like the list you made very much ("blockbeta",
>> "blockfinal", "want", "consider", "no"). I think it covers the
>> different needed stages while not being too detailed (details should
>> be in comments), esp. as it makes sense to also query bugs of those
>> different stages separately.
>>
>
> I'd recommend "blocknextmilestone" as a generic one so we don't have to
> change for each milestone.
>
> I'll also note from the 1.5 Cycle when we had a "blocknextmilestone" and
> "blockfinal" approximated by a 1.8.b5+ and 1.8+ people got really really
> confused about the difference in general to the point that we just moved
As a (bugzilla) "user", I like the idea of "targetting" explicitely ...
either by the flag name or the flag value.
> everything to "next milestone." Also, the final release blocking list
> gets very very large until you basically get up to the point where the
> next milestone and final release are equivalent. So I'd almost suggest
> dropping "blockfinal" and just leave it as:
> blocknextmilestone
> want
> consider
> no
>
> Also note that the best way to approximate this without changing tooling
> is to agree that:
> a) +=blocking next milestone + want
> b) Cleared blocking flag with comment "will consider patch" = consider
... and doesn't like the "void".
If I don't read all the comments I might ask again, or simply don't know
why it is/was not set.
Mike Connor
>>>> * which states we need to distinguish, and which we don't, and
>>>
>>> I actually like the list you made very much ("blockbeta",
>>> "blockfinal", "want", "consider", "no"). I think it covers the
>>> different needed stages while not being too detailed (details should
>>> be in comments), esp. as it makes sense to also query bugs of those
>>> different stages separately.
>
> It seems to me that it would be better to just have a 'block' state,
> and let the flag name indicate which release it's for. That way if
> someone nominates a bug you can flag it 'want' for the beta but
> 'blocking' for release. Or any other desired combination.
>
> Also, it took me some thinking to realize the difference between
> 'want' and 'consider' (as you say, there is more of a scale than a
> line between them). So I think we should make sure to document them well.
>
> Of course, it's always a good idea for whoever flags a bug to also add
> a clarifying comment in the bug.
will continually nominate for each new flag. Using blocking+milestone,
seems to have cut down on the endless and long nomination lists. Either
a bug is something we need/want for the shipping release, or its not,
which is unlikely to change between a2 and b1, or b1 and b2.
If we can make tooling changes to have states like dbaron is suggesting
for the Fx3 cycle, the "blocking" flag becomes more of an "on radar"
flag, and the states are what indicates how much we want to fix the
bug. Nominating and plussing then really become more of a "I think this
is important" statement than a "we cannot ship without this" statement,
which is the trap we fell into this cycle, and why we're cutting things
that people wanted. blockbeta and blockrelease are things we must fix,
except the latter category implies we'll target things after beta, which
we shouldn't be doing from now on. (We can fix things, but we shouldn't
plan on it)
In feature planning for Fx2 and Fx3, we've been using P1/P2/P3, which
are roughly defined as:
P1 - Must fix, we will put the most energy and focus into these
P2 - Want to fix, we will commit resources, but not stop the release for
this.
P3 - Would like to see fixed, if someone steps up to work on it.
I'd like to make these more explicit within Bugzilla, as
mustfix/wanted/nice to have, since we've already been framing
discussions along those lines for the last six months. Other values
would be "requested" and "no" or somesuch, and I think that makes it
really clear.
-- Mike
LpSolit
L. David Baron wrote:
> The fundamental problem is that they only let the drivers or
> release managers choose between two options. There are really
> considerably more options.
I think that's the goal of the priority field to distinguish between
these options. Combined with the target milestone fields, you could
have:
Target Milestone: Fx3a2 Priority: Blocker
- or -
Target Milestone: Fx2b2 Priority: Nice to have
with the advantage of being able to do mass-change to retarget open
bugs, either to a new milestone or to a new priority (e.g., all bugs
which were "wanted" for b2 are "blocker" for b3 or rc1).
> states, or a new flag is created for each part of the cycle (beta,
> release candidate, etc.) and the same bugs are renominated in each part
> of the cycle. The use of one flag for each stage causes causes
> significantly more repeated work, and the use of a single flag may cause
> more confusion.
1. Using a single flag is bad. When a flag is set, it shouldn't change,
unless too much delay finally appears or the bug appears (not) to be a
problem (anymore).
2. Even with flags having 10 states, you would need one flag per part
of the cycle; so the amount of work is the same, e.g.:
Flag: status
=========
Fx2rc2: blocker
Fx2rc1: want
Fx2b2: nice to have
Fx2b1: no
Fx2a2: no
I'm not saying that having more states per flag is bad (at least not
yet; myk just pointed me to this thread). I only say you should also
consider the usage of the priority field, combined with the target
milestone field.
My feeling about my example above, which could probably be something we
could see in any bug:
- I'm a contributor and I see the flag not being set to "no". If I'm
able to fix this bug on time, I would estimate that my patch should be
checked in independently of whether the flag is set to "nice to have"
or "blocker", because it's not a "no". If the flag was set to "no", I
wouldn't have wasted my time to write it now, because I would probably
have to unbitrot it later.
- I'm a newbie/user with no idea how Perl/C++/XUL works, and the fact
that the flag is set to anything else than "no" doesn't tell me much
more that "+" or "-".
- I'm a module owner, manager, guru of any kind. I do care about
"blockers", I probably don't care about "no" and "nice to have", and my
feeling about "want"'ed bugs will depend on whether they have a patch
ready on time or not. What I just described here is respectively
described by the "+", "-" and "?" states. I agree, we cannot
distinguish the flag with status "?" because the bug hasn't been
triaged yet and the flag with status "?" because we are still not sure
whether it should be considered as a blocker or not (the "want" status
discussed above).
At first sight, it seems to me that you would only want *one*
additional status, which would be "pending", aka "let's see". And
"pending" flags would be retriaged later when more information are
known about these bugs, either to "+" or "-".
Frédéric Buclin
(LpSolit on IRC)
L. David Baron
> L. David Baron wrote:
> > The fundamental problem is that they only let the drivers or
> > release managers choose between two options. There are really
> > considerably more options.
>
> I think that's the goal of the priority field to distinguish between
> these options. Combined with the target milestone fields, you could
> have:
>
> Target Milestone: Fx3a2 Priority: Blocker
> - or -
> Target Milestone: Fx2b2 Priority: Nice to have
>
> with the advantage of being able to do mass-change to retarget open
> bugs, either to a new milestone or to a new priority (e.g., all bugs
> which were "wanted" for b2 are "blocker" for b3 or rc1).
We generally use those fields to indicate the priority/schedule given to
a bug by its current owner. Perhaps that's not useful anymore and just
a holdover from when non-Netscape contributors didn't want Netscape
managers touching their bugs. But I think it really is useful -- if the
release managers want to give something a higher priority than the
current owner of the bug, it often really does need a new owner.
> 2. Even with flags having 10 states, you would need one flag per part
> of the cycle; so the amount of work is the same, e.g.:
>
> Flag: status
> =========
> Fx2rc2: blocker
> Fx2rc1: want
> Fx2b2: nice to have
> Fx2b1: no
> Fx2a2: no
No, this is a huge problem, and a huge amount of extra work. The goals
of the release don't change between alpha2, beta1, beta2, etc., and
there's no need to repeat the decision about the priority of the bug for
each one of those milestones. We should have one flag per final
release -- or perhaps even one flag per unique set of goals (considering
a series of security releases as a single release).
> At first sight, it seems to me that you would only want *one*
> additional status, which would be "pending", aka "let's see". And
> "pending" flags would be retriaged later when more information are
> known about these bugs, either to "+" or "-".
No, that's totally not what we need. I'd like to fix the bug system to
reflect the way we want to organize our work, and not organize our work
around the deficiencies of the bug system.
Mike Beltzner
> I think the states that we (branch drivers) have wanted to distinguish
> between during our triage are roughly the following:
>
> 1. We will hold the next beta/release candidate for this fix
> 2. We will hold the final release for this fix
> 3. We encourage people to work on this for this release (although
> patches will still be considered by weighing risk and reward).
> 4. We will consider a patch for this release.
> 5. We'd rather you didn't try to fix this for this release.
>
> (In reality there are many gradations between these, especially
> between
> (3) and (4).)
There is also a:
6. We feel that this is too risky for this release, and will never
accept it.
It should also be noted that as the release cycle from alpha to beta,
the criteria used to make the decisions for 4-6 change.
As I see it, though, there are really three states we're talking about:
A. This fix must be included in this release.
B. This fix would be very valuable to include in this release.
C. This fix is too risky / unacceptable for this release.
Deferring to a fear of over-engineering, I suggest that we try to
change our tools minimally to support our needs, and instead work
with changing our collective understanding of what we mean when we
set certain flags.
I believe that by some conventional use of the target milestone
field, and by having one flag per version that supports these three
states (!, +, -, ?) we would get the fidelity that we need.
For example, a bug that is essential to Firefox 2 would be marked
"blocking-firefox2!" with a target milestone equivalent to the
absolute last milestone which can safely contain that fix and still
provide appropriate risk mitigation in the form of proper QA and
widespread testing in pre-release milestones (ie: beta1). Once set
blocking-firefox2!, a bug should not be removed from that list.
A bug that is wanted would be marked as "blocking-firefox2+" with an
appropriate TM, but that setting could be changed as indeed could the
blocking status (it could get cleared, it could get moved up). What
this is saying is "drivers agree that this is a highly desirable fix
for this release, and suggest that we try to get it delivered by this
date, but are willing to re-evaluate in the context of fit to
schedule and resource, changing market context over time, and
changing risk".
A bug that is thought to be inappropriate for the release (ie:
requires API changes on an API stable branch) would be minused, as
would any bug that requires deep fixes too late in the cycle.
cheers,
mike
Mike Connor
> On 25-Jul-06, at 3:19 AM, L. David Baron wrote:
>
>> I think the states that we (branch drivers) have wanted to distinguish
>> between during our triage are roughly the following:
>>
>> 1. We will hold the next beta/release candidate for this fix
>> 2. We will hold the final release for this fix
>> 3. We encourage people to work on this for this release (although
>> patches will still be considered by weighing risk and reward).
>> 4. We will consider a patch for this release.
>> 5. We'd rather you didn't try to fix this for this release.
>>
>> (In reality there are many gradations between these, especially between
>> (3) and (4).)
>
> There is also a:
> 6. We feel that this is too risky for this release, and will never
> accept it.
>
> It should also be noted that as the release cycle from alpha to beta,
> the criteria used to make the decisions for 4-6 change.
>
> As I see it, though, there are really three states we're talking about:
> A. This fix must be included in this release.
> B. This fix would be very valuable to include in this release.
> C. This fix is too risky / unacceptable for this release.
I don't think this is fine-grained enough, since based on the Pn
rankings both P2 and P3 would be in B.
> Deferring to a fear of over-engineering, I suggest that we try to
> change our tools minimally to support our needs, and instead work with
> changing our collective understanding of what we mean when we set
> certain flags.
Once we upgrade b.m.o to 2.22.1, it should be fairly trivial to
implement the word-based values. Frédéric (LpSolit) already indicated
he'd write the patch to enable a flag with text values, so I think its
worth doing things that way, instead of trying to extend the +?- setup
(especially since we'd have to hack bugzilla to only do that on certain
flags).
firefox3-radar: ( "Must Fix" / "Wanted" / "Would Take" / "Not Wanted" /
"Requested") seems really straightforward to me, the ! vs + keeps
confusing me every time I look at it.
-- Mike
Gervase Markham
> During the 1.8.1 triage and approval process I (and others) have come to
> realize that the blocking flags we use on bugs don't meet our needs very
> well.
What happened to this discussion? It seemed to peter out without a
decision being made.
I'm not sure to what extent current Bugzilla supports doing more with
flags. Could we, as some have suggested, combine the flag values with
another repurposed field (e.g. priority, or well-chosen keywords) to
give better granularity?
Gerv
Mike Beltzner
> What happened to this discussion? It seemed to peter out without a
> decision being made.
The discussion ended up migrating over to a Gran Paradiso call two
weeks ago, during which time several suggestions and follow up action
items were taken:
# States:
* blocks a1 ("next milestone")
* blocks a2 ("milestone after next, only appears when we're in
the a1 endgame")
* blocks final release
* really want but doesn't block
* would take with a patch
* would not take
* nominated
# ideally
* blocking-1.9 flag that can have any of the above states
* show flag state in column bug list
# myk and dbaron will talk to bugzilla folks to see if we can do the
above
# will start doing Gecko 1.9 triage using current system + and set
keywords
(source: http://wiki.mozilla.org/Firefox3/StatusMeetings/2006-08-02)
cheers,
mike