Aurora/Beta Approval flags (Was: Managing approvals under rapid release)
Johnathan Nightingale
> In today's Aurora meeting, we agreed that the mozilla-aurora tree (implicitly mozilla-beta as well) should default to APPROVAL REQUIRED for checkins. This brought up an obvious question:
>
> Q: How do we request/grant approvals?
>
> I offered to start a thread on determining this. I think there are actually 3 questions hidden inside it:
>
> 1) What should the bugzilla mechanism be?
> 2) Who grants approvals and how often?
> 3) What do we do in the meantime?
FOR QUESTION 1: I propose that we create an approval flag per channel (i.e., approval-aurora: ?,-,+ and approval-beta: ?,-,+), rather than creating one approval flag per version (i.e. approval5, approval6, approval7...).
Making the flag channel-specific instead of version-specific allows us to:
* easily clear all missed approvals at each merge
* triage aurora-level approvals with a different risk calculus than beta-level approvals
* build tools to track approvals which don't need updating every 6 weeks
* avoid making more noise about version numbers than necessary
* avoid creating a new disposable flag every 6 weeks
One downside of per-channel flags over per-version ones is that when merging from aurora to beta, missed approvals need to be re-assessed instead of implicitly carrying forward. I don't think this is a bad thing, certainly not one that outweighs the positives.
The principal upsides of a per-version flag, as far as I can tell, are the similarity to what we've done historically (approval2.0, &c) and the parity between approval and tracking flags. I don't personally feel that those outweigh the advantages of per-channel, but would like to hear where others come down.
Johnathan
---
Johnathan Nightingale
Director of Firefox Engineering
joh...@mozilla.com
Johnathan Nightingale
> In today's Aurora meeting, we agreed that the mozilla-aurora tree (implicitly mozilla-beta as well) should default to APPROVAL REQUIRED for checkins. This brought up an obvious question:
>
> Q: How do we request/grant approvals?
>
> I offered to start a thread on determining this. I think there are actually 3 questions hidden inside it:
>
> 1) What should the bugzilla mechanism be?
> 2) Who grants approvals and how often?
> 3) What do we do in the meantime?
FOR QUESTION 2: I propose that the answer is "done during regular aurora meetings, to be scheduled by Sheila."
I anticipate concern that triage will swamp those meetings. I don't think it will, because as I said before, I think the bar is much higher:
> THE DEFAULT ANSWER to "Can I land this on aurora/beta?" is, "No. Land it on central, wait a few weeks, get on the next train." The only things that should need aurora/beta approval are safe fixes to features we will otherwise just back out/disable. This is a much smaller set than the "opportunistic wins and non-blocking feature work" of prior releases. Typically, those bugs should NOT be nominated for approval in our new model.
J
Ehsan Akhgari
approval-beta? flags? I just want to make sure that they won't end up
in the same situation as approval2.0? (except for the Firefox 4 end-game).
Cheers,
Ehsan
Christian Legnitto
On Apr 12, 2011, at 2:59 PM, Johnathan Nightingale wrote:
> On 2011-04-12, at 5:56 PM, Johnathan Nightingale wrote:
>
>> In today's Aurora meeting, we agreed that the mozilla-aurora tree (implicitly mozilla-beta as well) should default to APPROVAL REQUIRED for checkins. This brought up an obvious question:
>>
>> Q: How do we request/grant approvals?
>>
>> I offered to start a thread on determining this. I think there are actually 3 questions hidden inside it:
>>
>> 1) What should the bugzilla mechanism be?
>> 2) Who grants approvals and how often?
>> 3) What do we do in the meantime?
>
> FOR QUESTION 1: I propose that we create an approval flag per channel (i.e., approval-aurora: ?,-,+ and approval-beta: ?,-,+), rather than creating one approval flag per version (i.e. approval5, approval6, approval7...).
>
> Making the flag channel-specific instead of version-specific allows us to:
>
> * easily clear all missed approvals at each merge
> * triage aurora-level approvals with a different risk calculus than beta-level approvals
> * build tools to track approvals which don't need updating every 6 weeks
> * avoid making more noise about version numbers than necessary
> * avoid creating a new disposable flag every 6 weeks
>
> One downside of per-channel flags over per-version ones is that when merging from aurora to beta, missed approvals need to be re-assessed instead of implicitly carrying forward. I don't think this is a bad thing, certainly not one that outweighs the positives.
>
> The principal upsides of a per-version flag, as far as I can tell, are the similarity to what we've done historically (approval2.0, &c) and the parity between approval and tracking flags. I don't personally feel that those outweigh the advantages of per-channel, but would like to hear where others come down.
>
> Johnathan
>
> ---
> Johnathan Nightingale
> Director of Firefox Engineering
> joh...@mozilla.com
>
>
>
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
JP Rosevear
> On 2011-04-12, at 5:56 PM, Johnathan Nightingale wrote:
>
> > In today's Aurora meeting, we agreed that the mozilla-aurora tree (implicitly mozilla-beta as well) should default to APPROVAL REQUIRED for checkins. This brought up an obvious question:
> >
> > Q: How do we request/grant approvals?
> >
> > I offered to start a thread on determining this. I think there are actually 3 questions hidden inside it:
> >
> > 1) What should the bugzilla mechanism be?
> > 2) Who grants approvals and how often?
> > 3) What do we do in the meantime?
>
> FOR QUESTION 1: I propose that we create an approval flag per channel (i.e., approval-aurora: ?,-,+ and approval-beta: ?,-,+), rather than creating one approval flag per version (i.e. approval5, approval6, approval7...).
>
> Making the flag channel-specific instead of version-specific allows us to:
>
> * easily clear all missed approvals at each merge
> * triage aurora-level approvals with a different risk calculus than beta-level approvals
> * build tools to track approvals which don't need updating every 6 weeks
> * avoid making more noise about version numbers than necessary
> * avoid creating a new disposable flag every 6 weeks
>
> One downside of per-channel flags over per-version ones is that when merging from aurora to beta, missed approvals need to be re-assessed instead of implicitly carrying forward. I don't think this is a bad thing, certainly not one that outweighs the positives.
>
> The principal upsides of a per-version flag, as far as I can tell, are the similarity to what we've done historically (approval2.0, &c) and the parity between approval and tracking flags. I don't personally feel that those outweigh the advantages of per-channel, but would like to hear where others come down.
I originally liked per-version more for this reason, but flag naming
should not be an absolute blocker, so I am fine with the above for now.
-JP
--
JP Rosevear <j...@mozilla.com>
Mozilla
Ted Mielczarek
feature-disabling, or are those things that will be decided on a
case-by-case basis anyway, and should just get approval as part of
that decision? Do we have a blocking-firefox5+ flag? I got lost in all
the discussion on that, but presumably issues that were considered bad
enough to block on (as in disable or backout the feature), would be
assumed approved?
-Ted
Benjamin Smedberg
> Does it make sense to grant blanket approval for backouts and
> feature-disabling, or are those things that will be decided on a
> case-by-case basis anyway, and should just get approval as part of
because various stakeholders such as marketing will be informed of the
feature set at the beginning of the aurora cycle, and any changes after
that point will need additional communication.
> that decision? Do we have a blocking-firefox5+ flag? I got lost in all
> the discussion on that, but presumably issues that were considered bad
> enough to block on (as in disable or backout the feature), would be
> assumed approved?
>
We are not going to have a blocking flag. Please use
"tracking-firefox5?" to inform release drivers of issues which may be
important enough to block on, and we'll deal with the tracking flag as a
blocking flag (for backouts or whatever) as appropriate.
--BDS
L. David Baron
> On 2011-04-12, at 5:56 PM, Johnathan Nightingale wrote:
>
> > In today's Aurora meeting, we agreed that the mozilla-aurora tree (implicitly mozilla-beta as well) should default to APPROVAL REQUIRED for checkins. This brought up an obvious question:
> >
> > Q: How do we request/grant approvals?
> >
> > I offered to start a thread on determining this. I think there are actually 3 questions hidden inside it:
> >
> > 1) What should the bugzilla mechanism be?
> > 2) Who grants approvals and how often?
> > 3) What do we do in the meantime?
>
>
> I anticipate concern that triage will swamp those meetings. I
> don't think it will, because as I said before, I think the bar is
> much higher:
I was actually worrying about the opposite concern earlier today
after today's regular aurora meeting: that the rest of the meeting
will overwhelm the triage, and I wouldn't want to sit through all
the rest of it just for the triage.
I also think that as we become more comfortable with how things work
in the new model (and come to consensus on how they do), we should
deëmphasize the use of meetings for triage (perhaps particularly
during the earlier weeks of the aurora cycle). However, I think the
meetings may be useful initially to build a set of common standards.
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
Johnathan Nightingale
There's another thread in this newsgroup about triage schedule, but I
actually think the thing you're concerned about matters less in this
version of the world to begin with.
Consider: We used to have a problem with large numbers of untriaged
approval noms going stale, bit-rotting because they couldn't land
without approval, and yet no one was giving out approvals unless you
whined in IRC.
Today, though, what bugs fit that pattern? Any old fix can land whenever
it wants, because mozilla-central is an OPEN tree, regardless of how
close to release we are; mozilla-central itself is *never* close to
release. So those fixes don't languish like they might.
The only things that should get nominated for approval are ones we think
need to be considered for aurora/beta on their way to release, but those
things should have already landed on central. So in the worst case,
where no triage ever happened, those bugs would still be picked up in
the next merge.
And the worst case for those (presumably much smaller) nom queues will
not actually happen, since those noms ARE perpetually in the release
end-game, and hence attended to.
Make sense?
J
Ehsan Akhgari
> On 4/12/2011 6:38 PM, Ehsan Akhgari wrote:
>> Do we have a plan of actively triaging the list of approval-aurora? and
>> approval-beta? flags? I just want to make sure that they won't end up in
>> the same situation as approval2.0? (except for the Firefox 4 end-game).
>
> There's another thread in this newsgroup about triage schedule, but I
> actually think the thing you're concerned about matters less in this
> version of the world to begin with.
>
> Consider: We used to have a problem with large numbers of untriaged
> approval noms going stale, bit-rotting because they couldn't land
> without approval, and yet no one was giving out approvals unless you
> whined in IRC.
This wasn't my concern...
> Today, though, what bugs fit that pattern? Any old fix can land whenever
> it wants, because mozilla-central is an OPEN tree, regardless of how
> close to release we are; mozilla-central itself is *never* close to
> release. So those fixes don't languish like they might.
>
> The only things that should get nominated for approval are ones we think
> need to be considered for aurora/beta on their way to release, but those
> things should have already landed on central. So in the worst case,
> where no triage ever happened, those bugs would still be picked up in
> the next merge.
>
> And the worst case for those (presumably much smaller) nom queues will
> not actually happen, since those noms ARE perpetually in the release
> end-game, and hence attended to.
>
> Make sense?
The thing which worries me is patches which are really important for
aurora getting ignored until beta, and then they would be deemed
(perhaps rightly so) too risky for beta. It's true that they'll get
shipped within six weeks, but still if we can fix something very trivial
for a release, we should probably do it.
Which makes me think that it might make sense to make sure that we
triage at least the subset of approval nominations which have
tracking-firefoxN+ (and maybe even explicitly clear the rest!).
Ehsan
Boris Zbarsky
> The only things that should get nominated for approval are ones we think
> need to be considered for aurora/beta on their way to release, but those
> things should have already landed on central. So in the worst case,
> where no triage ever happened, those bugs would still be picked up in
> the next merge.
I assume there is a separate process for tracking regression bugs that
need to be fixed on aurora (e.g. by backout of the problem changeset),
right?
-Boris
Sheila Mooney
Cheers,
Sheila
Boris Zbarsky
> No, regressions would follow the exact same process.
In that case the worst case scenario if we get behind on triage is that
we ship breakage that we don't want to ship.
Which probably means we need to stay on top of triage.
-Boris
Christian Legnitto
On Apr 12, 2011, at 3:01 PM, Johnathan Nightingale wrote:
> On 2011-04-12, at 5:56 PM, Johnathan Nightingale wrote:
>
>> In today's Aurora meeting, we agreed that the mozilla-aurora tree (implicitly mozilla-beta as well) should default to APPROVAL REQUIRED for checkins. This brought up an obvious question:
>>
>> Q: How do we request/grant approvals?
>>
>> I offered to start a thread on determining this. I think there are actually 3 questions hidden inside it:
>>
>> 1) What should the bugzilla mechanism be?
>> 2) Who grants approvals and how often?
>> 3) What do we do in the meantime?
>
> FOR QUESTION 2: I propose that the answer is "done during regular aurora meetings, to be scheduled by Sheila."
>
> I anticipate concern that triage will swamp those meetings. I don't think it will, because as I said before, I think the bar is much higher:
>
>> THE DEFAULT ANSWER to "Can I land this on aurora/beta?" is, "No. Land it on central, wait a few weeks, get on the next train." The only things that should need aurora/beta approval are safe fixes to features we will otherwise just back out/disable. This is a much smaller set than the "opportunistic wins and non-blocking feature work" of prior releases. Typically, those bugs should NOT be nominated for approval in our new model.
Summary
---------------------
* We will have triage and driver status meetings, focused on channels/repos. Developers should feel free to only attend meetings they care about
* Focusing on mozilla-central is the regular platform meeting. This meeting is every Tuesday 11:00 am-noon PDT (no change)
* Focusing on mozilla-aurora is a new bi-weekly meeting. This meeting is every Tuesday/Thursday 2:00-3:00 pm PDT
* mozilla-aurora also has a weekly triage-only meeting, every Wednesday 11:30 am-12:00noon PDT
** Please note for the next couple of weeks this Wednesday meeting will be both triage and project level items until we get a handle on the overall process!
* Focusing on mozilla-beta is a new weekly meeting. This meeting is every Monday, 2:00-3:00 pm PDT
* There is no meeting for mozilla-release as it is handled in the mozilla-beta meetings
* Once we stop supporting < Firefox 4 the normal M/W/F branch triage meetings will be folded into the meetings above
Details
---------------------
Because each repo/channel may have different interested parties and different action items, it makes sense to have the triage and driver status meetings align with them. Dividing PM work and developer meeting attendance by channel/repo also makes a lot of sense.
--
mozilla-central / Nightly
* We'll use the current Tuesday platform meeting, meeting every Tuesday 11:00 am-noon PDT
* The meeting will focus on upcoming work, dependencies that need to shake out before mozilla-central landings, mozilla-central health, etc
* This meeting should hopefully become more useful and will feel fairly different than the Wednesday product meeting (currently they feel a bit similar)
--
mozilla-aurora / Aurora
* We'll have a new bi-weekly meeting, meeting every Tuesday/Thursday 2:00-3:00 pm PDT
* This is bi-weekly as there will be a decent amount of churn on Aurora
* Meeting on Tuesday means this meeting will align with merge days. We can choose to merge during the meeting if desired or merge in the morning and have this be a status meeting
* We will triage all Aurora approval requests ("approval-mozilla-aurora:?"), look at nominations for bugs we need to track ("tracking-firefoxN:?"), check the bugs that need to be fixed before we merge to beta ("tracking-firefoxN:+"), etc
* We will determine deal with the mechanics of backing out / preffing off, keep appraised of that status, and drive aurora-specific items to completion
* There will also be a weekly triage-only meeting, every Wednesday 11:30 am-12:00noon PDT. This should make sure bug triage is a focus even if it is glossed over in the bi-weekly meetings. This will be a standing meeting that we can automatically cancel if no triage queries have actionable items
** Please note for the next couple of weeks this Wednesday meeting will be both triage and project level items until we get a handle on the overall process!
--
mozilla-beta / Firefox Beta
* We'll have a new weekly meeting, meeting every Monday, 2:00-3:00 pm PDT
* This only needs to be weekly as beta shouldn't have too much churn. We may up it to bi-weekly if weekly isn't often enough
* Meeting on Monday means this meeting will happen the day before the mozilla-beta -> mozilla-release point
* We will triage all Beta approval requests ("approval-mozilla-beta:?"), look at nominations for bugs we need to track ("tracking-firefoxN:?"), check the bugs that need to be fixed before we go to release ("tracking-firefoxN:+"), etc
JP Rosevear
> Specific triage and driver meeting proposal below....
>
> On Apr 12, 2011, at 3:01 PM, Johnathan Nightingale wrote:
>
> > On 2011-04-12, at 5:56 PM, Johnathan Nightingale wrote:
> >
> >> In today's Aurora meeting, we agreed that the mozilla-aurora tree (implicitly mozilla-beta as well) should default to APPROVAL REQUIRED for checkins. This brought up an obvious question:
> >>
> >> Q: How do we request/grant approvals?
> >>
> >> I offered to start a thread on determining this. I think there are actually 3 questions hidden inside it:
> >>
> >> 1) What should the bugzilla mechanism be?
> >> 2) Who grants approvals and how often?
> >> 3) What do we do in the meantime?
> >
> > FOR QUESTION 2: I propose that the answer is "done during regular aurora meetings, to be scheduled by Sheila."
> >
> > I anticipate concern that triage will swamp those meetings. I don't think it will, because as I said before, I think the bar is much higher:
> >
> >> THE DEFAULT ANSWER to "Can I land this on aurora/beta?" is, "No. Land it on central, wait a few weeks, get on the next train." The only things that should need aurora/beta approval are safe fixes to features we will otherwise just back out/disable. This is a much smaller set than the "opportunistic wins and non-blocking feature work" of prior releases. Typically, those bugs should NOT be nominated for approval in our new model.
>
>
> Summary
> ---------------------
>
> * We will have triage and driver status meetings, focused on channels/repos. Developers should feel free to only attend meetings they care about
>
> * Focusing on mozilla-central is the regular platform meeting. This meeting is every Tuesday 11:00 am-noon PDT (no change)
>
> * Focusing on mozilla-aurora is a new bi-weekly meeting. This meeting is every Tuesday/Thursday 2:00-3:00 pm PDT
>
> * mozilla-aurora also has a weekly triage-only meeting, every Wednesday 11:30 am-12:00noon PDT
> ** Please note for the next couple of weeks this Wednesday meeting will be both triage and project level items until we get a handle on the overall process!
>
> * Focusing on mozilla-beta is a new weekly meeting. This meeting is every Monday, 2:00-3:00 pm PDT
After some testing, these times are not great for me (at least
personally - 2-3 hours later is actually way better) on the east coast,
nor are they amenable for Europe and most of Asia. I'm wondering if
they can be moved up 1-2 hours, or even more radically 4-5 hours.
Alternately having one of the meetings move might be good so then all
timezones can easily participate in at least one per week.
John O'Duinn
+1. Monday afternoons (PDT) are not great for me either, no matter how
much I hope otherwise. Are there any possible alternate times?
> -JP
tc
John.
=====
Sheila Mooney
was a bit late but it's hard to get that sought-after 10:00am-noon
timeframe. I would like to here what Christian proposes but my first
thoughts are....
+ Aurora: We could try and move the Thursday meeting to 11:00am. Considering
we have the platform and planning meetings at that time, clearly it works
for a larger group. In general I like the idea of having one of the meetings
that works better for those in Europe and on the East Coast. Moving the Tues
meeting might be doable. I am not sure that moving it to 1:00pm would have
much impact time-wise - JP? Another option could be 10:00am on Tues. Right
now those meetings have been long simply because there are lots of growing
pains getting through this new cycle. I expect this to be short-lived and
they will be much quicker and streamlined.
+ Beta: Mondays are probably bad in general. We could try Wed. Here again a
morning slot will be challenging. We could potentially drop the Aurora
Triage (I think it will go away eventually anyhow) and use the 11:30am spot.
The concern here would be that the planning meeting spills over and the Beta
meeting ends up going really long. Are we trying to line this up with the
day we will do the build? I suspect the thought was that we would meet
Monday and go to build maybe Tues?
Other thoughts, suggestions?
Cheers,
Sheila
Christian Legnitto
On Apr 25, 2011, at 5:29 PM, Sheila Mooney wrote:
> JP, John - thanks for the feedback. I was also thinking that the 2:00pm time was a bit late but it's hard to get that sought-after 10:00am-noon timeframe. I would like to here what Christian proposes but my first thoughts are....
Yeah, I picked the time because I knew of no cross-functional standing meetings.
> + Aurora: We could try and move the Thursday meeting to 11:00am. Considering we have the platform and planning meetings at that time, clearly it works for a larger group. In general I like the idea of having one of the meetings that works better for those in Europe and on the East Coast. Moving the Tues meeting might be doable. I am not sure that moving it to 1:00pm would have much impact time-wise - JP? Another option could be 10:00am on Tues. Right now those meetings have been long simply because there are lots of growing pains getting through this new cycle. I expect this to be short-lived and they will be much quicker and streamlined.
I'm fine with this. The times were chosen somewhat arbitrarily and I knew having them both at the same time could cause issues...though Tu/Th at the same time, same place doing the same thing gets us into a rhythm in these early stages. I also think after 3 or 4 releases it will probably just be the release team showing up (as it basically is for branches now).
> + Beta: Mondays are probably bad in general. We could try Wed. Here again a morning slot will be challenging. We could potentially drop the Aurora Triage (I think it will go away eventually anyhow) and use the 11:30am spot. The concern here would be that the planning meeting spills over and the Beta meeting ends up going really long. Are we trying to line this up with the day we will do the build? I suspect the thought was that we would meet Monday and go to build maybe Tues?
Yep, It was explicitly chosen to happen on Monday so we always have a meeting the day before a release is supposed to go out (which is always Tuesdays!). I think it is valuable to have that for mechanics, last minute freak outs, etc. We could have yet another meeting but seems silly to add even more, especially because the beta meeting's charter is to deal with beta issues!
Thanks,
Christian