Need "blocking" flags for Firefox 5
Robert O'Callahan
Firefox 5.
Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
Mike Beltzner
- a multistate flag called "blocking"
- states for: ?, -, experimental+, beta+, final+
- the requirement that all blocking flags be addressed before a branch "ships" (either by merging to the next branch, or by releasing the software)
then, relatedly ...
- once a bug is fixed, we use the target milestone to track where it was fixed
- we realign the target milestones to match whatever new version numbering scheme we come up with
I'd be happy to set this up.
cheers,
mike
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Christian Legnitto
On Mar 29, 2011, at 9:27 PM, Mike Beltzner wrote:
> For the sake of simplicity, I would suggest...
> - a multistate flag called "blocking"
Fine by me.
> - states for: ?, -, experimental+, beta+, final+
Names aren't decided yet but agree with the idea.
> - the requirement that all blocking flags be addressed before a branch "ships" (either by merging to the next branch, or by releasing the software)
Agree.
> then, relatedly ...
> - once a bug is fixed, we use the target milestone to track where it was fixed
So bugs are are marked with a target milestone of "experimental" if they are fixed on the mozilla-experimental repository (and landed on mozilla-central at the same time). Fine by me.
> - we realign the target milestones to match whatever new version numbering scheme we come up with
If we go Chrome-style I have a different idea that scales with the build/version #s but agree for any other style.
> I'd be happy to set this up.
I'm down with that, though it was my understanding Sheila was tasked with organizing this (I believe it used to be mentioned on https://wiki.mozilla.org/RapidRelease but can't seem to locate it). Doubt she'll have a problem with this based on our discussions about the best way to track so far.
Mike Beltzner
cheers,
mike
Sheila Mooney
pretty simple initially. Once we go through 1 or 2 release cycles, we
will have a better idea what works and what doesn't. I had already
spoke to Beltzner about this.
Cheers,
Sheila
On Tue, Mar 29, 2011 at 9:43 PM, Christian Legnitto
<cleg...@mozilla.com> wrote:
>
> On Mar 29, 2011, at 9:27 PM, Mike Beltzner wrote:
>
>> For the sake of simplicity, I would suggest...
>> - a multistate flag called "blocking"
>
> Fine by me.
>
>> - states for: ?, -, experimental+, beta+, final+
>
> Names aren't decided yet but agree with the idea.
>
>> - the requirement that all blocking flags be addressed before a branch "ships" (either by merging to the next branch, or by releasing the software)
>
> Agree.
>
>> then, relatedly ...
>> - once a bug is fixed, we use the target milestone to track where it was fixed
>
> So bugs are are marked with a target milestone of "experimental" if they are fixed on the mozilla-experimental repository (and landed on mozilla-central at the same time). Fine by me.
>
>> - we realign the target milestones to match whatever new version numbering scheme we come up with
>
> If we go Chrome-style I have a different idea that scales with the build/version #s but agree for any other style.
>
>> I'd be happy to set this up.
>
> I'm down with that, though it was my understanding Sheila was tasked with organizing this (I believe it used to be mentioned on https://wiki.mozilla.org/RapidRelease but can't seem to locate it). Doubt she'll have a problem with this based on our discussions about the best way to track so far.
Christian Legnitto
On Mar 29, 2011, at 9:53 PM, Mike Beltzner wrote:
> I'd actually meant that we'd use target milestone to indicate what version contained a fix, and yes, had been counting on the version to also contain a reference to the branch.
Ok. In the Chrome version world the version changes nightly on each branch and can ultimately vary wildly. The way the field works in bugzilla we'd have to enumerate every possible version which would be a huge list (think 5.0.0.1, 5.0.0.1............5.15.230.245). Of course the solution is fairly easy (have multiple select dropdowns with number ranges).
> I would hope - but perhaps this is naive - that any fix on a branch would also be landed on its antecedent branches.
There is zero debate on this and it is indeed a core part of the process.
Thanks,
Christian
Mike Beltzner
cheers,
mike
Mike Beltzner
If a bug is defined to be "blocking," then the decision amongst developers and product drivers is that without the bug being addressed (via a fix to resolve the issue, a backout of a change that caused it, or a product driving decision about suitability for market) we cannot ship the affected area of code to users.
Historically we have used per-version flags to indicate which bugs block a specific release. With a rolling train model, we'll want to know which bugs are "blockers" and ensure that none exist on the codebase we are about to ship to users. This means that instead of knowing only whether or not a bug blocks, we'll also need to know on which branches the bug exists. This isn't a new challenge - we've had the same issue on support branches for a while - it just means that to understand the blocking nature of a bug we need to understand both whether or not the issue is a blocker and what branches it affects.
My proposal was predicated on the idea that any code issue fixed on a branch would also need to be resolved on its antecedent branches, and given that requirement I think that will mostly work. If we broadly categorize blocking issues into the following buckets: (a) code fix required, (b) backout or killswitch required, (c) product decision required then we see that in all cases the bug will only be resolved if the proper action is taken across all branches.
As such, I think a slight modification of the proposal is in order. I suggest the creation of a "blocking-fx" flag (see bug 646449) which has four states:
"?" means the bug is nominated for blocking status
"-" means the bug has been denied blocking status
"+" means that the bug must be resolved before the affected area is shipped as part of a product release
In parallel, I think we'll want to find a better way to indicate which branches a particular bug affects. In some cases it may only affect descendant branches because the fix already exists on antecedent branches. In other cases it may only affect antecedent branches because it has been pref'd off or backed out of a descendant branch. There have been two proposed mechanisms for doing this:
Proposal A: use keywords, such as:
affects-trunk
affects-experimental
affects-beta
affects-all
Proposal B (my preference): use a multistate flag, such as affects-branch with states:
"?" means undetermined
"trunk" means mozilla-central
"experimental" means mozilla-experimental
"beta" means mozilla-beta
"trunk+experimental" means both mozilla-central and mozilla-experimental
"experimental+beta" means both mozilla-experimental and mozilla-beta
"trunk+beta" means both mozilla-central and mozilla-beta (not sure when this would happen!)
"trunk+experimental+beta" means all
Once an issue was resolved on a particular branch, this flag would need to be updated. One the issue was resolved on all branches, the issue would be resolved as FIXED.
This combination will give us the power to query against release and shipping criteria. For example, we know that in order to get beta feedback on all issues which require beta feedback, we'd need to resolve all the beta+ blockers that exist on the beta branches, so the Bugzilla Quicksearch query would be:
blocking-fx:+ affects-branch:beta
To see what issues keep us from shipping a final release to users, the query would be:
blocking-fx:+ affects-branch:beta
To see what issues are known to be blockers but are not of immediate concern as they do not affect the code that is about to ship to users:
blocking-fx:+ -affects-branch:beta
To see what blockers we are about to merge over to the beta channel from the experimental channel when that time comes around:
blocking-fx:+ affects-branch:experimental
etc.
Comments welcome!
cheers,
mike
Jesse Ruderman
>
> Proposal A: use keywords, such as:
> affects-trunk
> affects-experimental
> affects-beta
> affects-all
>
> Proposal B (my preference): use a multistate flag, such as affects-branch with states:
> "?" means undetermined
> "trunk" means mozilla-central
> "experimental" means mozilla-experimental
> "beta" means mozilla-beta
> "trunk+experimental" means both mozilla-central and mozilla-experimental
> "experimental+beta" means both mozilla-experimental and mozilla-beta
> "trunk+beta" means both mozilla-central and mozilla-beta (not sure when this would happen!)
> "trunk+experimental+beta" means all
I'd rather use versions than streams for the "affects" keywords/flags.
Naming them after streams is likely to result in piles of bugspam and
confusion after each merge [repo-per-stream world] or stream-jump
[repo-per-version world].
Mike Beltzner
> I'd rather use versions than streams for the "affects" keywords/flags.
> Naming them after streams is likely to result in piles of bugspam and
> confusion after each merge [repo-per-stream world] or stream-jump
> [repo-per-version world].
If we can find a reasonable way to do that, then by all means we should! I was basing this proposal on the currently accepted way of doing our streams, it can and should be modified if we switch to repo-per-version.
FWIW, I don't think the bugspam would be all that harsh, but perhaps it's because I'm only forseeing a world where the affects keywords/flags really matter for blockers. I suppose they'd be pretty useful for all bugs, eh wot?
cheers,
mike
Robert Kaiser
> - the requirement that all blocking flags be addressed before a branch "ships" (either by merging to the next branch, or by releasing the software)
Do I read this correctly that only "hardblockers" are blockers in the
future? If so, a BIG YAY for that!!!
Robert Kaiser
--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community needs answers to. And most of the time,
I even appreciate irony and fun! :)
Sheila Mooney
consistently used. Even with all these flags, I worry that they will
never stay up to date unless we keep things incredibly simple. It's
impossible to be tracking every bug on every channel and reading
through the content to understand where the problem is, where it's
been fixed etc. We also need a reliable way to search. I talked to
Christian and he had a suggestion about using version numbers and
somewhat automating that so people didn't have to set this data
explicitly. Last time we talked, he was still working this out. This
might help things. I also don't expect we will get it right in the
first cycle so I suggest we just pick something and work out the kinks
as we go.
By the way, what does it mean to have a blocker on the trunk? I didn't
think we would actually block on anything to cut over to experimental,
right? We will have to keep track of what is fixed on the trunk or
what needs to be fixed (found in other channels). Once we cut over to
the experimental channel the blockers are turning stuff off, backing
stuff out or fixing some bugs (call being made by a number of
stakeholders).
On Wed, Mar 30, 2011 at 8:28 AM, Mike Beltzner <belt...@mozilla.com> wrote:
> Actually, the more Johnath and I talked about this the more I wanted to think it through with everyone ...:
>
> If a bug is defined to be "blocking," then the decision amongst developers and product drivers is that without the bug being addressed (via a fix to resolve the issue, a backout of a change that caused it, or a product driving decision about suitability for market) we cannot ship the affected area of code to users.
>
> Historically we have used per-version flags to indicate which bugs block a specific release. With a rolling train model, we'll want to know which bugs are "blockers" and ensure that none exist on the codebase we are about to ship to users. This means that instead of knowing only whether or not a bug blocks, we'll also need to know on which branches the bug exists. This isn't a new challenge - we've had the same issue on support branches for a while - it just means that to understand the blocking nature of a bug we need to understand both whether or not the issue is a blocker and what branches it affects.
>
> My proposal was predicated on the idea that any code issue fixed on a branch would also need to be resolved on its antecedent branches, and given that requirement I think that will mostly work. If we broadly categorize blocking issues into the following buckets: (a) code fix required, (b) backout or killswitch required, (c) product decision required then we see that in all cases the bug will only be resolved if the proper action is taken across all branches.
>
> As such, I think a slight modification of the proposal is in order. I suggest the creation of a "blocking-fx" flag (see bug 646449) which has four states:
>
> "?" means the bug is nominated for blocking status
> "-" means the bug has been denied blocking status
> "+" means that the bug must be resolved before the affected area is shipped as part of a product release
>
> In parallel, I think we'll want to find a better way to indicate which branches a particular bug affects. In some cases it may only affect descendant branches because the fix already exists on antecedent branches. In other cases it may only affect antecedent branches because it has been pref'd off or backed out of a descendant branch. There have been two proposed mechanisms for doing this:
>
> Proposal A: use keywords, such as:
> affects-trunk
> affects-experimental
> affects-beta
> affects-all
>
> Proposal B (my preference): use a multistate flag, such as affects-branch with states:
> "?" means undetermined
> "trunk" means mozilla-central
> "experimental" means mozilla-experimental
> "beta" means mozilla-beta
> "trunk+experimental" means both mozilla-central and mozilla-experimental
> "experimental+beta" means both mozilla-experimental and mozilla-beta
> "trunk+beta" means both mozilla-central and mozilla-beta (not sure when this would happen!)
> "trunk+experimental+beta" means all
>
> Once an issue was resolved on a particular branch, this flag would need to be updated. One the issue was resolved on all branches, the issue would be resolved as FIXED.
>
> This combination will give us the power to query against release and shipping criteria. For example, we know that in order to get beta feedback on all issues which require beta feedback, we'd need to resolve all the beta+ blockers that exist on the beta branches, so the Bugzilla Quicksearch query would be:
>
> blocking-fx:+ affects-branch:beta
>
> To see what issues keep us from shipping a final release to users, the query would be:
>
> blocking-fx:+ affects-branch:beta
>
> To see what issues are known to be blockers but are not of immediate concern as they do not affect the code that is about to ship to users:
>
> blocking-fx:+ -affects-branch:beta
>
> To see what blockers we are about to merge over to the beta channel from the experimental channel when that time comes around:
>
> blocking-fx:+ affects-branch:experimental
>
> etc.
>
> Comments welcome!
>
> cheers,
> mike
David Mandelin
1. I am deeply dissatisfied with all proposals for shoehorning release management and project management information into Bugzilla. It is seriously time we got some real, purpose-built tools.
(I realize we can't wait for this for Fx5, but I really wouldn't like it if we just continued to hobble along indefinitely without taking action to improve this.)
2. It is possible to track bugs in detail with reasonable consistency and accuracy. I did it for all the JS/JSD Fx4 blockers from January through release. The daily triage sessions did it for all blockers in the end stages.
3. Detailed tracking is *a lot* of work, and can probably only be done if assigned to a person who considers tracking release bugs to be their primary responsibility. I think it pays off big in being able to understand what's going on, detecting and resolving major issues, and being able to accelerate the process/prevent delays.
4. Better tools, *designed by those who will use them*, to assist with tool-independent processes that are well understood, would reduce the burden.
Dave
Mike Beltzner
> A few thoughts I have about this:
>
> 1. I am deeply dissatisfied with all proposals for shoehorning release management and project management information into Bugzilla. It is seriously time we got some real, purpose-built tools.
Agreed, but orthogonal, IMO.
We *are* tracking release management and product management information elsewhere as well (Deb Richardson and the product team are building those systems, expect notes from them soon!) and hope to get a DRADIS like system for at a glance views into where various initiatives sit and where resources need to be applied in order to get features into specific releases.
That said, any attempt at replacing Bugzilla as being the de-facto source for information about the status of our codebase has failed, and I don't think a huge shift over will work. These new systems should integrate with RESTful APIs in Bugzilla (and that's the plan, from use of Pulse to things like - shock! - a WikiMedia plugin that can pull live information from Bugzilla!) but still be based on that source.
> 2. It is possible to track bugs in detail with reasonable consistency and accuracy. I did it for all the JS/JSD Fx4 blockers from January through release. The daily triage sessions did it for all blockers in the end stages.
Yes, I think that daily triage needs to be part of this new rolling train model. The 11:30am PT timeslot seemed to work well. We also need to ensure that there is a broad audience of leads and delegates at those meetings to reduce bus-factor.
cheers,
mike
Mike Beltzner
> By the way, what does it mean to have a blocker on the trunk? I didn't
> think we would actually block on anything to cut over to experimental,
Suppose we had a bug where the editor crashed when someone tried to use it with IME. We might be content to let that merge over to experimental but would want to log it as a blocker to ensure we did something about it before it got to a point where it might ship to end users. I do not think that it will happen often, but I do believe we need to be able to indicate blocking issues as soon as they come up and let them track across merges.
cheers,
mike
Christian Legnitto
> Mike Beltzner schrieb:
>> - the requirement that all blocking flags be addressed before a branch "ships" (either by merging to the next branch, or by releasing the software)
>
> Do I read this correctly that only "hardblockers" are blockers in the future? If so, a BIG YAY for that!!!
Yes, this is part of the proposal. And because the schedule is consistent and immutable, if the blockers aren't resolved before the merge they are blocking we will backout/pref off by default.
Christian
Gervase Markham
> Proposal B (my preference): use a multistate flag, such as affects-branch with states:
> "?" means undetermined
> "trunk" means mozilla-central
> "experimental" means mozilla-experimental
> "beta" means mozilla-beta
> "trunk+experimental" means both mozilla-central and mozilla-experimental
> "experimental+beta" means both mozilla-experimental and mozilla-beta
> "trunk+beta" means both mozilla-central and mozilla-beta (not sure when this would happen!)
> "trunk+experimental+beta" means all
Is the "version" field suitable for this, or is there some reason it
won't work? I mean, that's what it was originally for. I agree it's
still not a multiselect, but your proposal works around that.
Gerv
Mike Beltzner
It could do, except that the last time I tried to suggest that we change the way people use version and target milestone fields, the masses revolted and claimed that those fields were used differently depending on the owner of the bug. I'd be happy to repatriate them, as it were, but in the past have just not bothered to try :)
cheers,
mike
Sheila Mooney
things on channel basis, things would be tracked from a "train" perspective.
For each train we would use the following 2 flags;
-> tracking5: "?", "+", "-"
-> status5: "unaffected", "affected", "fixed"
+ Anyone that can edit bugs can set ?, unaffected, affected, fixed
+ Release drivers can set +, -
"?" means "release drivers, please look at this bug"
"+" means "release drivers have decided "something" needs to be done about
this bug during this release"
"-" means "release drivers have decided "nothing" needs to be done about
this bug for this release"
"unaffected" means "this bug is not a problem on the Firefox 5 release
train"
"affected" means "this bug is a problem in the Firefox 5 release train"
"fixed" means "this bug was fixed in the Firefox 5 release train"
We know there are probably use cases that this does not cover. It's a
tradeoff between burdening people with lots of data to manage and keeping
things relatively light-weight. After going through a bunch of scenarios it
seems like this should handle most of the cases we really want to track. It
does not specifically easily identify that a problem was found on a
particular channel. As we worked through some use cases, there was a concern
that the flag would go out of date really quickly.
Some of the questions we thought about....
+ Product management really wants a feature to land in FF6. Do we use a flag
to track this?
+ A feature has landed in -central for FF6. Do we need to track this? Why
and how?
+ A serious crash regression has been found on -central which appeared
during the current release cycle. How do we track this?
+ A serious crash regression has been found on -aurora crash stats. How do
we track this?
+ A functionality regression (Acrobat doesn't work in European Windows) has
been found when we shipped a beta. How do we track this?
+ How do we keep the metadata correct as the 6-week cycle progresses?
Is this proposal perfect? Probably not :-). Until we have gone through a
cycle or two, it's impossible to know all the various use cases we will run
into or situations where this might break down. When we move to the aurora
channel on Tues, we need something in place to help us keep track of things.
This would be relatively easy to setup and unless there is overwhelming
opposition, we would like to try this for FF5 and see how things go. Of
course we will discuss and tweak as necessary.
Cheers,
Sheila
On Fri, Apr 1, 2011 at 10:07 AM, Mike Beltzner <belt...@mozilla.com> wrote:
> It could do, except that the last time I tried to suggest that we change
> the way people use version and target milestone fields, the masses revolted
> and claimed that those fields were used differently depending on the owner
> of the bug. I'd be happy to repatriate them, as it were, but in the past
> have just not bothered to try :)
>
> cheers,
> mike
Robert Kaiser
> + Product management really wants a feature to land in FF6. Do we use a flag
> to track this?
We're really trying hard on not having any guarantees or even something
near it for specific releases, as far as I understand, so I think we
should not have such a flag.
We could have a keyword or so for "high-value" bugs, but I would not
couple those with any release or even imply doing so.
Sheila Mooney
IMHO, this kind of thing should all be tracked in project pages - we might
be aiming for a particular cycle and that's fine but I see no reason for
this information to be kept in bugzilla. The feature might be phased somehow
and this would be detailed in the project pages. The situation in which we
would use the "trackingX" flag is when we "think we have some group of work
done that is ready to put on a particular train". Setting the tracking flag
in this case alerts release drivers that we are putting it on this train and
there is something important to track here.
Cheers,
Sheila
Robert O'Callahan
> So we have a counter proposal to try for Firefox 5. Instead of tracking
> things on channel basis, things would be tracked from a "train"
> perspective.
> For each train we would use the following 2 flags;
>
> -> tracking5: "?", "+", "-"
> -> status5: "unaffected", "affected", "fixed"
>
I prefer this proposal because I don't want the meaning of Bugzilla flags to
change over time. Flags that include the branch names will effectively
change meaning over time as those branches shift from one release to the
next.
(I think any of these proposals would work, though.)
Thanks,
Gervase Markham
> It could do, except that the last time I tried to suggest that we
> change the way people use version and target milestone fields, the
> masses revolted and claimed that those fields were used differently
> depending on the owner of the bug. I'd be happy to repatriate them,
> as it were, but in the past have just not bothered to try :)
Be bold :-)
This is what they are for, and they should be used that way by the major
project. If other people want other annotations, this is why we have the
status whiteboard and keywords.
Gerv
Gervase Markham
> So we have a counter proposal to try for Firefox 5. Instead of tracking
> things on channel basis, things would be tracked from a "train" perspective.
Can someone say which of this proposal and beltzner's we are going to be
using?
Gerv
Benjamin Smedberg
section of the various pages: we are using tracking-firefox5 and
status-firefox5 and so on per-release.
--BDS
Gervase Markham
> This has been announced several times and is already in the final
> section of the various pages: we are using tracking-firefox5 and
> status-firefox5 and so on per-release.
Sorry; thank you :-)
Gerv