Onwards and upwards from Beta1 to final release (a proposal)
Mike Schroepfer
We had a long conversation in the Gecko meeting today and I wanted to
tease apart what I believe are four separate issues we should cover
below. Please do take the time to read this - if you have specific
questions, alternative ideas, etc lets get them out in the newsgroup
asap so we can pick something and move forward:
1) How to deal with the patch landing backlog from beta1. I think Damon
has that covered in his recent post here. Short summary: component
owners continue to drive patch check-ins for those areas and use the
wiki pages (http://wiki.mozilla.org/Firefox3/Beta2CheckinQueue) to make
sure we don't clobber each other.
2) Getting the blocker lists under control for beta2 and beyond. As of
this writing there are 389 platform blockers and 305 FF blockers. We
closed on the order of 20-30 platform bugs for beta1. So all component
owners should prioritize their blocker lists by setting priority
(p1-p5). You should end up with ~10-20% of your bugs P1-P2 and the rest
P3-P5. Everyone should try get this done by am pacific time on Friday.
I know this is a pain - but making sure we have the right priorities
is one of our best weapons to not spend time on the wrong stuff. If
bugs don't have a priority set by Friday AM we'll remove them from the
blocking list.
Here's the process Stuart used for GFX to set priorities:
"a) Go through each bug trying to take as little time as possible on
each bug (20 seconds?) and pick a priority for it. If I'm not sure
leave it and move on to the next bug.
b) Go through the things I skipped over before and spend more time with
them until they are also prioritized.
-- At this point I had already duped several blockers and minused
others. Some things just simply no longer needed to be on the list.
c) Go through the P5s and ask questions in the bugs, verify they still
happen see if we can drop them off the list. In some cases we could, in
others not so much.
Next I went through and tried to assign owners to all of the bugs where
it was obvious. I tried the testcases in bugs where they existed and
tried to follow the steps to reproduce where they weren't too involved.
In doing this I was able to get rid of quite a few bugs as WFM.
All in all, I probably spent 6 hours going through about 70 bugs,
prioritizing them all and finding owners for most of them. I probably
resolved 20 of those bugs and have others now marked as P5 that are "on
the edge."
The purpose of this exercise isn't to spend a lot of time on every bug
but to try and weed out things that shouldn't be on the list and to give
yourself and everyone else a much better idea of a) what is on your bug
list and that b) the list is accurate."
3) Process for managing the beta2, beta3, etc blocking lists. We tried
using the Target Milestones + blocking flags for Beta1 and there was
mixed opinion's about the results. So I propose we go back to and old
by tried and true method: blocker flags (e.g. blocking 1.9M10) for each
milestone. The bugs with + on this are really the blockers for the
next milestone. So we'll create a blocking 1.9M10 flag right now and
use that to manage beta2.
4) Process for approving blocking bugs and patches. I propose that we
move to release-driver control over the blocking bug list. This means
anyone can nom - but release drivers are the only ones who can move bugs
to blocking. This is the process we started prior to beta 1 for both
Firefox 1.5 and Firefox 2 - so we are already overdue here. If a bug is
on the blocking list patches can land once reviewed. As always anyone
can request landing rights for a patch that is not blocking and release
drivers will approve those (through the approval flag on the patch).
Release drivers will meet at least daily and these meetings are open to
anyone to participate, comment, and decide.
In order to jump start this process (e.g. not start it with a huge
backlog) we are asking everyone through #1 above to clear out the
current backlog of patches, through #2 to try and get some ordering of
the existing blocker bugs. Once #2 is completed we'll move all P1/P2
bugs onto the M10 blocker list and adds/removes will be managed in the
daily triage meetings (e.g. do #3 and #4). At each milestone we'll
endeavor to start with 40-60 bugs on the blocker list (skimmed from the
most important blocker lists) and work those lists to zero. So at each
milestone everyone here is fixing the most important bugs until we get
to a point where we are ready to ship.
I'd like to try and lock down a process we can use from here to the end
so we don't further confuse folks with new flags, TM, priorities fields,
etc. So once we clear this week I'm suggesting we just have blocker
flags for the next milestone + release driver triage. So this is a
proposal - I want a chance for everyone to weight in, a chance for us to
clarify anything confusing, and then lock things down from here until
the end.
Comments welcomed - I'm honestly open to any specific implementation
that gets us the right trade-offs to get the release out the door. This
proposal is not perfect - but I'm not sure how to make a better set of
trade offs and it relies on what we know works (however painful).
Schrep
Q&A:
1) Why do we need release driver triage?
Our best weapon to get to a high-quality release asap is to contain
scope. In previous releases we went to a driver approval prior to b1
and stayed there until final. This process was meant to gatekeep, cut
scope and have a second group of folks who had a default “no” answer to
*any* change – causing us to spend a little time making sure the risk
reward trade-offs are the right ones. If we aren’t fighting to reject
bugs we are not pushing hard enough to get scope down to get the release
out the door. Having the same group of folks looking at *every single
bug* going into the release gives them a good integrated high-level view
of what level of changes, risk, and bugs are in the release which helps
make those tough judgment calls. We are currently asking component
owners to be both feature champions and release gatekeepers - which is
very tough to do at the same time.
2) How many bugs should we have on the M10 blocking list?
Based on historical fix rates this list should be between 20-60 bugs all
in if we want to release b2 by the end of the year. Since there will be
*at least* a beta3 this is not the last chance to get fixes in.
3) What happens if 400 of our 700 bugs are *really* P1 or P2?
If that's the case we need to reset expectations of when we can
realistically ship based on historical fix rates. This isn't a threat -
it just means we need to plan for many more than 3 betas.
4) How can release drivers understand the details of each component area?
They can't. They'll rely on the judgment of owners and will make
mistakes for sure. But the nom/minus process leaves a clear historical
trail and an opportunity for anyone to appeal any specific decision.
5) Why use flags per milestone instead of priority, TM, etc?
Basically because the blocking flags give us a clear workflow within BZ
to manage the queues. Noms are easy to spot, changes are controlled
etc. It's also how we manage dot releases so the process is well
understood.
6) Won't release drivers just approve most things?
In the beginning yes - but as time goes on the cost of delaying the
release goes up so the criteria get more strict. Even the very fact of
asking questions about risk/reward for bugs makes us make intentional
decisions.
7) Won't this slow down development?
Once a bug is on the blocker list check-ins can happen without further
approval. Triage will happen daily - so delays should never be long.
If it is truly hampering development than we can adjust.
8) Aren't we doing this too soon (e.g. there are too many bugs right now)?
Honestly a little hard to tell - but we are doing it much later than we
ever did for previous releases (this release is bigger, I know) and we
are trying to streamline it as much as possible so it doesn't get in the
way. But we *must* do #1 and #2 above to get a handle on how we are
doing first.
Gavin Sharp
> If bugs don't have a priority set by Friday AM we'll remove them from the
> blocking list.
Maybe this was just phrased oddly, but it worries me. Blockers are
bugs that have been previously triaged and evaluated to be bugs
important enough to block the release (or at least important enough to
make sure we continue tracking them and deal with them again before
the release). Surely you don't propose blindly removing bugs from that
list because the component owner (for one reason or another) hasn't
completed this second triage process by Friday?
Gavin
Mike Schroepfer
Yep I sure do! If we need more time than Friday that's fine - but I
think having each component owner or group take a least a cursory pass
over the list and confirm that everything on there is important (and set
priorities) is critical. As I said if the result really is we have 700
blockers we should be planning for 2H08 or 1H09 (or later) release - in
which case we'd do things differently *right now*.
If you think a) the priority pass is not important enough, b) this isn't
enough time, or c) there is a better way to get this done I'm all ears :-).
Really not trying to be a jerk - just trying to get us to where we need
to be.
Best,
Schrep
Mike Schroepfer
Yep I sure do! If we need more time than Friday that's fine - but I
Gavin Sharp
> If you think a) the priority pass is not important enough, b) this isn't
> enough time, or c) there is a better way to get this done I'm all ears :-).
I have no problem with asking component owners to do this triage, and
agree with you that it's important. I understand that you want to
compel the component owners to do it quickly, but I don't think
threats about losing blocker bugs is the right way to go about it :)
This whole discussion is moot assuming the triage gets done, of
course, but I think that blindly removing bugs from the list if
components owners don't act by Friday would have a negative impact
that greatly outweighs whatever motivational impact the threat may
have before then.
Gavin
Jesse Ruderman
on what I heard at this morning's meeting:
http://wiki.mozilla.org/Firefox3/StatusMeetings/2007-11-06#Beta_2_criteria.2C_proposal_2
Boris Zbarsky
> If a bug is
> on the blocking list patches can land once reviewed.
Which blocking list? 1.9, or 1.9Mnext?
-Boris
Mike Schroepfer
1.9Mnext.
Mike
Boris Zbarsky
> 1.9Mnext.
Which means that anything not on that blocker list requires explicit approval,
then? Even if it's blocking 1.9 in general?
-Boris
Mike Schroepfer
Yes. General idea here is that we are focused on the stuff of 1.9Mnext
so they get the fast path. Other things are definitely fair game but
require a patch approval. Total number of release-driver approval steps
is the same in either case (either the bug is moved onto Mnext or the
patch is approved for Mnext). Daily triage should make sure there
isn't a backlog.
Robert Kaiser
> We had a long conversation in the Gecko meeting today and I wanted to
> tease apart what I believe are four separate issues we should cover
> below. Please do take the time to read this - if you have specific
> questions, alternative ideas, etc lets get them out in the newsgroup
> asap so we can pick something and move forward:
I just hope something will be done about the suckiness we had so far wrt
not triaging or noticing flags that are set on bugs outside the
selection of components that are monitored. If we touch toolkit or core
files with a patch in e.g. the Mozilla Application Suite or Thunderbird
products, and the flags are available there, someone should at least
notice and triage them. the *1.9* flags are not supposed to be about
anything but the 1.9 release(s), so they should at least be monitored by
the radar queries.
Robert Kaiser
L. David Baron
> Boris Zbarsky wrote:
> > Mike Schroepfer wrote:
> >> 1.9Mnext.
> >
> > Which means that anything not on that blocker list requires explicit
> > approval, then? Even if it's blocking 1.9 in general?
> Yes. General idea here is that we are focused on the stuff of 1.9Mnext
> so they get the fast path. Other things are definitely fair game but
> require a patch approval. Total number of release-driver approval steps
> is the same in either case (either the bug is moved onto Mnext or the
> patch is approved for Mnext). Daily triage should make sure there
> isn't a backlog.
This disagrees with what the rules linked from tinderbox say:
http://wiki.mozilla.org/TreeStatus . And both disagree with the
state of bugzilla, where the approval1.9 flag isn't currently
requestable.
What are the rules actually?
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
L. David Baron
> This whole discussion is moot assuming the triage gets done, of
> course, but I think that blindly removing bugs from the list if
> components owners don't act by Friday would have a negative impact
> that greatly outweighs whatever motivational impact the threat may
> have before then.
Agreed. I'm astonished that threats like this are being seriously
suggested, never mind actually made.
Mike Schroepfer
The title of this was (a proposal) and I was explicitly inquiring about
changing the rules. I wanted feedback, alternatives, etc before we
changed them. I think the rules have been changed a little too often -
so I wanted a chance for us all to debate them, come up with a new rule
set and implement.
I haven't seen any specific alternative suggestions nor a suggestion
that we keep on the current path. I want to leave this thread open for
today to make sure everyone has had a chance to weigh in (really I do
want people to speak up if I'm being a idiot here) then we can figure
out the path forward.
Best,
SChrep
Mike Schroepfer
The title of this was (a proposal) and I was explicitly inquiring about
changing the rules. I wanted feedback, alternatives, etc before we
changed them. I think the rules have been changed a little too often -
so I wanted a chance for us all to debate them, come up with a new rule
set and implement.
I haven't seen any specific alternative suggestions nor a suggestion
that we keep on the current path. I want to leave this thread open for
today to make sure everyone has had a chance to weigh in (really I do
want people to speak up if I'm being a idiot here) then we can figure
out the path forward.
Best,
SChrep
Mike Schroepfer
> On Wednesday 2007-11-07 18:31 -0500, Gavin Sharp wrote:
>> This whole discussion is moot assuming the triage gets done, of
>> course, but I think that blindly removing bugs from the list if
>> components owners don't act by Friday would have a negative impact
>> that greatly outweighs whatever motivational impact the threat may
>> have before then.
>
> Agreed. I'm astonished that threats like this are being seriously
> suggested, never mind actually made.
>
> -David
>
Hey There,
I personally think that having 700 blocker bugs with absolutely no
prioritization amongst them is ludicrous and putting the release at
grave risk. All I'm asking is that we be very intentional in the
order in which we tackle issues to drive us to completion as fast as
possible.
So yea - I'm proposing extreme measures. I said this in the Gecko
meeting yesterday - and I really do mean it - I *personally* apologize
to anyone here that is offended or thinks I'm being a jerk or treating
them badly. I'm astonished at the level of effort put into this
release (peterv worked for like 15 days straight to get the CC ready for
beta, dbaron I know you've been cranking on the reflow stuff for a long
long time, gavin has pulled us through many a tough time, I can go
on...) and I can't help you code through it - but I can help make sure
the rules are clear and that we don't spend a second of time on the
wrong stuff. Jesse says we've fixed on the order of 11,500k bugs for
1.9 and I want to make sure everyone's efforts over the last 2+ years
are out in the real world as soon as possible.
So - if you think the priotiziation exercise is not necessary, that we
should do it differently, that we need more time, or any specific issue
or suggestion about how to get to the finish faster - please, please -
let's engage on the issue! I really am reasonable about this stuff and
am trying to help - everyone here (i hope) knows this by now. Forget
the threat of minusing - let's either agree that it is important and do
it - or let's come up with an alternative plan.
All the best,
Mike
Mike Schroepfer
> On Wednesday 2007-11-07 18:31 -0500, Gavin Sharp wrote:
>> This whole discussion is moot assuming the triage gets done, of
>> course, but I think that blindly removing bugs from the list if
>> components owners don't act by Friday would have a negative impact
>> that greatly outweighs whatever motivational impact the threat may
>> have before then.
>
> Agreed. I'm astonished that threats like this are being seriously
> suggested, never mind actually made.
>
> -David
>
Hey There,
Benjamin Smedberg
> I personally think that having 700 blocker bugs with absolutely no
> prioritization amongst them is ludicrous and putting the release at
> grave risk. All I'm asking is that we be very intentional in the
> order in which we tackle issues to drive us to completion as fast as
> possible.
Sure... the problem seems to be that if the area drivers somehow don't get
prioritization done, or miss a set of bugs, or something, that the *bugs*
somehow become not worth of blocking, when in fact the problem is probably
with the drivers or the process they're using. So the solution (minusing the
bugs) doesn't fit the problem.
--BDS
stu...@gmail.com
If people would spend less time arguing about it and more time
prioritizing the bugs we wouldn't we won't ever have to deal with bugs
that aren't prioritized....
stuart
Mike Schroepfer
So I proposed this in the message above:
"So - if you think the priotiziation exercise is not necessary, that we
should do it differently, that we need more time, or any specific issue
or suggestion about how to get to the finish faster - please, please -
let's engage on the issue! I really am reasonable about this stuff and
am trying to help - everyone here (i hope) knows this by now. Forget
the threat of minusing - let's either agree that it is important and do
it - or let's come up with an alternative plan."
Which you didn't quote. Shorter summary: "let's just go do the
priorization and move on or come up with an alternate plan". You are
not proposing an alternative solution ... so should we just move forward?
Benjamin Smedberg
> "So - if you think the priotiziation exercise is not necessary, that we
> should do it differently, that we need more time, or any specific issue
> or suggestion about how to get to the finish faster - please, please -
> let's engage on the issue! I really am reasonable about this stuff and
> am trying to help - everyone here (i hope) knows this by now. Forget
> the threat of minusing - let's either agree that it is important and do
> it - or let's come up with an alternative plan."
>
> Which you didn't quote. Shorter summary: "let's just go do the
> priorization and move on or come up with an alternate plan". You are
> not proposing an alternative solution ... so should we just move forward?
I propose that if there are unprioritized bugs on Friday, you should find
different people to prioritize those bugs, or have the end-game drivers do it.
--BDS
Serge Gautherie
> 2) Getting the blocker lists under control for beta2 and beyond. As of
> this writing there are 389 platform blockers and 305 FF blockers. We
> closed on the order of 20-30 platform bugs for beta1. So all component
> owners should prioritize their blocker lists by setting priority
> (p1-p5). You should end up with ~10-20% of your bugs P1-P2 and the rest
> P3-P5. Everyone should try get this done by am pacific time on Friday.
> I know this is a pain - but making sure we have the right priorities is
> one of our best weapons to not spend time on the wrong stuff. If bugs
> don't have a priority set by Friday AM we'll remove them from the
> blocking list.
Why not simply set them to P5 (for example), as a default for the time
being ?
Mike Schroepfer
Sold! Serge recommended moving them to P5 by default as well which
sounds reasonable.
As a practical note 42/401 1.9 bugs have no priority right now - so that
should be quick to get through. 257/303 Firefox bugs are in this
state so that needs much more attention.
I'd also note that we really really need help from component owners
triaging the nomination list - there are 120 noms on the 1.9 list (only
16 on FF). Can component owners plow through these today so we can
start with a fresh slate tomorrow? On that note - nothing I'm saying
here should prevent component owners and everyone from helping us get
through the triage lists. The incoming rate is really pretty high for
any one group to trudge through - so we still need everyone's help here.
Thanks again to everyone for their patience and feedback as we work all
this out.
Best,
Schrep
Damon Sicore
to use those two priorities as indication of beta blocking. There
are 47 P1s alone. I realize there are several of these that have
patches, but for the moment assume 50% don't. I know I've talked a a
few on the platform team about using just the P1s as beta 2 blockers,
and schrep just confirmed my suspicion by poking his head in my
office and suggesting we do so.
So, what do we think of using just P1s as beta 2 blockers?
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Benjamin Smedberg
> Also, we've currently 121 total P1 and P2s. We were originally going to
> use those two priorities as indication of beta blocking. There are 47
> P1s alone. I realize there are several of these that have patches, but
> for the moment assume 50% don't. I know I've talked a a few on the
> platform team about using just the P1s as beta 2 blockers, and schrep
> just confirmed my suspicion by poking his head in my office and
> suggesting we do so.
>
> So, what do we think of using just P1s as beta 2 blockers?
Can we suspend that decision for a week and see what the P1 (and P2) lists
look like at that point? I suspect that clearing the tree backup will make
the situation clearer as to what is waiting to land and what really isn't
done yet.
--BDS
Mike Connor
Yeah, I'm doing a coarse sort (b2/b3/probably cut) right now. That
roughly equates to:
b2: P1/P2
b3: P3/P4
probably cut: P5
It looks better than it seems if you look at priorities, but I will
be setting those as a second pass.
-- Mike
Damon Sicore
Benjamin,
Just clicked through every bug on the beta2 checkin queue wiki pages,
and I found only 17 P1s and P2s (about 50/50 each). Are there others
that you think will be landing in the next day or two that will
dramatically affect these numbers? Trying to keep in mind here the
fact we have four weeks, really, to try to crank out a beta 2 by the
end of the year. Waiting a week for 80-100 P1s to land seems a bit
of a stretch. Just want to make sure we're focusing on the right
bugs as soon as possible.
Sincerely,
Damon
> --BDS
Benjamin Smedberg
> Just clicked through every bug on the beta2 checkin queue wiki pages,
> and I found only 17 P1s and P2s (about 50/50 each). Are there others
> that you think will be landing in the next day or two that will
> dramatically affect these numbers? Trying to keep in mind here the fact
> we have four weeks, really, to try to crank out a beta 2 by the end of
> the year. Waiting a week for 80-100 P1s to land seems a bit of a
> stretch. Just want to make sure we're focusing on the right bugs as
> soon as possible.
I'm not pretending that 80 bugs are going to land before next Wednesday...
but I expect that perhaps 20-40% of the existing P1s and P2s might. At least
we'd have a clearer picture of the fix rate and bug counts.
I'm assuming that those who own P1s will focus on them first anyway... right?
The checkin queue wikipage is not a permanent thing, right? We're just
metering checkins until we've cleared the backlog? I suspect people are
simply waiting for the backlog to clear, rather than adding things to the
wikipage and extending the painful metering solution further.
--BDS
Damon Sicore
Assume we see 40% of P1s and P2s checked in by next thursday. That
still leaves us with 71 P1s and P2s remaining to complete in 3
weeks. Are we comfortable with that? I'm not. :-/
Damon Sicore
On Nov 8, 2007, at 3:23 PM, Damon Sicore wrote:
>
> On Nov 8, 2007, at 3:09 PM, Benjamin Smedberg wrote:
>
> Assume we see 40% of P1s and P2s checked in by next thursday. That
> still leaves us with 71 P1s and P2s remaining to complete in 3
> weeks. Are we comfortable with that? I'm not. :-/
One thing I forgot to say is that getting 47 P1s done to get beta 2
out this quarter does *indeed* seem like an achievable goal. If
we're practical and choose realistic goals, it gives us a light at
the end of the tunnel.
Mike Schroepfer
> The checkin queue wikipage is not a permanent thing, right? We're just
> metering checkins until we've cleared the backlog? I suspect people are
> simply waiting for the backlog to clear, rather than adding things to the
> wikipage and extending the painful metering solution further.
You bet. Let's kill that wiki page asap...
Mike Schroepfer
So I'm gonna get the 1.9M10 blocking flags setup and we'll switch to
them tomorrow a.m. unless there are any big objections.
People have been doing an awesome job with the priorities. If all
component owners can push through and:
1) Finish off the 1.9? lists (I'm catching all general component issues
here), either +/- the bug and set a priority whilst you are there.
2) Approve/Deny the approval1.9? lists
We can start with a good, clean slate tomorrow. w00t!
Mike
Mike Schroepfer wrote:
> Hey Folks,
>
> We had a long conversation in the Gecko meeting today and I wanted to
> tease apart what I believe are four separate issues we should cover
> below. Please do take the time to read this - if you have specific
> questions, alternative ideas, etc lets get them out in the newsgroup
> asap so we can pick something and move forward:
>
> 1) How to deal with the patch landing backlog from beta1. I think Damon
> has that covered in his recent post here. Short summary: component
> owners continue to drive patch check-ins for those areas and use the
> wiki pages (http://wiki.mozilla.org/Firefox3/Beta2CheckinQueue) to make
> sure we don't clobber each other.
>
> 2) Getting the blocker lists under control for beta2 and beyond. As of
> this writing there are 389 platform blockers and 305 FF blockers. We
> closed on the order of 20-30 platform bugs for beta1. So all component
> owners should prioritize their blocker lists by setting priority
> (p1-p5). You should end up with ~10-20% of your bugs P1-P2 and the rest
> P3-P5. Everyone should try get this done by am pacific time on Friday.
> I know this is a pain - but making sure we have the right priorities is
> one of our best weapons to not spend time on the wrong stuff. If bugs
> don't have a priority set by Friday AM we'll remove them from the
> blocking list.
>
> Here's the process Stuart used for GFX to set priorities:
>
> "a) Go through each bug trying to take as little time as possible on
> each bug (20 seconds?) and pick a priority for it. If I'm not sure
> leave it and move on to the next bug.
>
> b) Go through the things I skipped over before and spend more time with
> them until they are also prioritized.
>
> -- At this point I had already duped several blockers and minused
> others. Some things just simply no longer needed to be on the list.
>
> c) Go through the P5s and ask questions in the bugs, verify they still
> happen see if we can drop them off the list. In some cases we could, in
> others not so much.
>
> Next I went through and tried to assign owners to all of the bugs where
> it was obvious. I tried the testcases in bugs where they existed and
> tried to follow the steps to reproduce where they weren't too involved.
>
> In doing this I was able to get rid of quite a few bugs as WFM.
>
> All in all, I probably spent 6 hours going through about 70 bugs,
> prioritizing them all and finding owners for most of them. I probably
> resolved 20 of those bugs and have others now marked as P5 that are "on
> the edge."
>
> The purpose of this exercise isn't to spend a lot of time on every bug
> but to try and weed out things that shouldn't be on the list and to give
> yourself and everyone else a much better idea of a) what is on your bug
> list and that b) the list is accurate."
> 3) Process for managing the beta2, beta3, etc blocking lists. We tried
> using the Target Milestones + blocking flags for Beta1 and there was
> mixed opinion's about the results. So I propose we go back to and old
> by tried and true method: blocker flags (e.g. blocking 1.9M10) for each
> milestone. The bugs with + on this are really the blockers for the
> next milestone. So we'll create a blocking 1.9M10 flag right now and
> use that to manage beta2.
>
> 4) Process for approving blocking bugs and patches. I propose that we
> move to release-driver control over the blocking bug list. This means
> anyone can nom - but release drivers are the only ones who can move bugs
> to blocking. This is the process we started prior to beta 1 for both
> Firefox 1.5 and Firefox 2 - so we are already overdue here. If a bug is
> on the blocking list patches can land once reviewed. As always anyone
> can request landing rights for a patch that is not blocking and release
> drivers will approve those (through the approval flag on the patch).
> Release drivers will meet at least daily and these meetings are open to
> anyone to participate, comment, and decide.
>
> In order to jump start this process (e.g. not start it with a huge
> backlog) we are asking everyone through #1 above to clear out the
> current backlog of patches, through #2 to try and get some ordering of
> the existing blocker bugs. Once #2 is completed we'll move all P1/P2
> bugs onto the M10 blocker list and adds/removes will be managed in the
> daily triage meetings (e.g. do #3 and #4). At each milestone we'll
> endeavor to start with 40-60 bugs on the blocker list (skimmed from the
> most important blocker lists) and work those lists to zero. So at each
> milestone everyone here is fixing the most important bugs until we get
> to a point where we are ready to ship.
>
> I'd like to try and lock down a process we can use from here to the end
> so we don't further confuse folks with new flags, TM, priorities fields,
> etc. So once we clear this week I'm suggesting we just have blocker
> flags for the next milestone + release driver triage. So this is a
> proposal - I want a chance for everyone to weight in, a chance for us to
> clarify anything confusing, and then lock things down from here until
> the end.
>
> Comments welcomed - I'm honestly open to any specific implementation
> that gets us the right trade-offs to get the release out the door. This
> proposal is not perfect - but I'm not sure how to make a better set of
> trade offs and it relies on what we know works (however painful).
>
> Schrep
>
> Q&A:
>
> 1) Why do we need release driver triage?
>
> Our best weapon to get to a high-quality release asap is to contain
> scope. In previous releases we went to a driver approval prior to b1
> and stayed there until final. This process was meant to gatekeep, cut
> scope and have a second group of folks who had a default “no” answer to
> *any* change – causing us to spend a little time making sure the risk
> reward trade-offs are the right ones. If we aren’t fighting to reject
> bugs we are not pushing hard enough to get scope down to get the release
> out the door. Having the same group of folks looking at *every single
> bug* going into the release gives them a good integrated high-level view
> of what level of changes, risk, and bugs are in the release which helps
> make those tough judgment calls. We are currently asking component
> owners to be both feature champions and release gatekeepers - which is
> very tough to do at the same time.
>
> 2) How many bugs should we have on the M10 blocking list?
>
> Based on historical fix rates this list should be between 20-60 bugs all
> in if we want to release b2 by the end of the year. Since there will be
> *at least* a beta3 this is not the last chance to get fixes in.
>
> 3) What happens if 400 of our 700 bugs are *really* P1 or P2?
>
> If that's the case we need to reset expectations of when we can
> realistically ship based on historical fix rates. This isn't a threat -
> it just means we need to plan for many more than 3 betas.
>
> 4) How can release drivers understand the details of each component area?
>
> They can't. They'll rely on the judgment of owners and will make
> mistakes for sure. But the nom/minus process leaves a clear historical
> trail and an opportunity for anyone to appeal any specific decision.
>
> 5) Why use flags per milestone instead of priority, TM, etc?
>
> Basically because the blocking flags give us a clear workflow within BZ
> to manage the queues. Noms are easy to spot, changes are controlled
> etc. It's also how we manage dot releases so the process is well
> understood.
>
> 6) Won't release drivers just approve most things?
>
> In the beginning yes - but as time goes on the cost of delaying the
> release goes up so the criteria get more strict. Even the very fact of
> asking questions about risk/reward for bugs makes us make intentional
> decisions.
>
> 7) Won't this slow down development?
>
> Once a bug is on the blocker list check-ins can happen without further
> approval. Triage will happen daily - so delays should never be long.
> If it is truly hampering development than we can adjust.
>
> 8) Aren't we doing this too soon (e.g. there are too many bugs right now)?
>
> Honestly a little hard to tell - but we are doing it much later than we
> ever did for previous releases (this release is bigger, I know) and we
> are trying to streamline it as much as possible so it doesn't get in the
> way. But we *must* do #1 and #2 above to get a handle on how we are
> doing first.
>
>
Boris Zbarsky
> So I'm gonna get the 1.9M10 blocking flags setup and we'll switch to
> them tomorrow a.m. unless there are any big objections.
So this will be the second time that a patch that's all ready to go will
suddenly no longer be OK? And again due to long-term tree closures (weeklong
windows bustage last time, freeze this time)?
Put another way, will patches that are ready to go right now on blocker+ bugs
get grandfathered? Or will they have to request approval if they don't land
tonight (into a metered tree, fwiw)?
-Boris
P.S. I think I have some bugs for which this will be the third time seeking
approval as a result of all the thrashing... They were ready to go, then we
changed the rules, then they got approved but depended on bugs whose approvals
never got triaged in time, then we froze, now we're about to reset the clock again.
Mike Schroepfer
> Mike Schroepfer wrote:
>> So I'm gonna get the 1.9M10 blocking flags setup and we'll switch to
>> them tomorrow a.m. unless there are any big objections.
>
> So this will be the second time that a patch that's all ready to go will
> suddenly no longer be OK? And again due to long-term tree closures
> (weeklong windows bustage last time, freeze this time)?
>
> Put another way, will patches that are ready to go right now on blocker+
> bugs get grandfathered? Or will they have to request approval if they
> don't land tonight (into a metered tree, fwiw)?
Yes they get grandfathered. This is what I meant by "clear the decks" -
let's not re-visit anything already ready to go. Waste of time of
for everyone. Anything with patches with 1.9+ can land. I'm asking
above that we (== component owners) triage all patches with 1.9? tonight
to approve/not. Anything reviewed, waiting to go, and blocking+ should
land when the coast is clear.
If you are not sure use your judgment - seriously - let's not wait for a
strict interpretation of the rules here to get real work done.
Best,
Schrep
Mike Schroepfer
> land when the coast is clear.
"coast is clear" meant to cover tree closures, test failures, and not
clobbering each other with check-ins - e.g. general rules of the road.
Nothing new there or no new signal you need to wait for.
Best,
Mike
Robert Kaiser
> (note: this is being sent on my BlackBerry, so it won't make it onto the newsgroup due to encoding issues)
>
> Robert, I agree completely that these things should be part of triage. Can you give some examples of bugs that are getting missed? Our queries divide up components so that the couple of toolkit areas that aren't owned by the platform team are seen by Connor and myself, but we might be missing some. I'd like to look into it.
https://bugzilla.mozilla.org/show_bug.cgi?id=402294 was a recent
expample that had to be switched to Core just to be visible on the used
queries just because it needed approval1.9 as it touched a core file
despite being a Camino patch.
We also had such bug in SeaMonkey when doing cleanup in XPFE or
switching SeaMonkey to using even more of toolkit than before - and
we'll go on to have such bugs.
I think it would be good if platform drivers would have a query that
includes the other products like Camino, Thunderbird or Mozilla
Application Suite (SeaMonkey). This doesn't add more than a handful of
items to the query usually, as we only request approval for thing that
touch the default (FF) builds, but if the flag exists there, they also
should be triaged, IMHO.
Note that I don't expect we would ever have things like 1.9 blockers
there, but we have approval requests in those products.
> When using the queries that I put on the "Keeping an eye on blockers" post on devnews, do the bugs we're referring to show up?
I haven't looked at those queries personally yet, but from what I hear,
they don't show up because they're not in Core, Toolkit or Firefox products.
Robert Kaiser
