Proposal: mark mozilla-central APPROVAL_REQUIRED for the next week

[Johnathan Nightingale]

Howdy folks,

The new Fennec is getting close to beta quality. We're taking down the remaining beta blockers[1] and hope to push it in the next few weeks. As with major desktop releases in the pre-train days, we feel a lot of pressure to lock this one down and get it out the door. Part of this pressure is internal, because we are getting close and we can feel it and we don't want any surprises. Part of it is external because it's really much better than the XUL fennec currently in the market, and we want to get these improvements out to users.

Fennec is based on Gecko 14, which is still on mozilla-central for another 9 days. I'd like to ask that we move mozilla-central to APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk of a late-landing change breaking Fennec. I'd propose that desktop-only changes get swift approval, and that we should have daily triage for all other requests to ensure we don't introduce too much drag.

I know it's a bit weird, since the last year of train-based releases has seen mozilla-central open almost continuously. I think this is a pretty rare event, though, and that Fennec needs the help reducing risk wherever possible. Once it is out the door, we'll get Fennec back on the trains so that we don't have this kind of dependency.

We discussed a few other options on release-drivers (migrate to Aurora early, put mobile development on other branches, do nothing) but each seemed like a worse outcome to me than using the APPROVAL_REQUIRED hook we already have. We can get the flag added today, but first I want to know whether there are any deal-breaker concerns with this proposal.

Thanks,

Johnathan

[1] https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocking-fennec1.0:beta

---
Johnathan Nightingale
Sr. Director of Firefox Engineering
@johnath

[Ehsan Akhgari]

What will the approval flag triage process look like? I remember than for
a while during the Firefox 4 development, the approval flag was not triaged
actively because of the sheer volume unless one pinged somebody on the
release drivers team. Is there going to be daily triages so that
developers can be sure that every patch they mark for approval will be
looked at within, let's say, a day?

Thanks!
--
Ehsan
<http://ehsanakhgari.org/>

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
>

[Ehsan Akhgari]

Well, I need to work on my reading comprehension skills. Or my attention
deficit disorder. Or both!

"I'd propose that desktop-only changes get swift approval, and that we
should have daily triage for all other requests to ensure we don't
introduce too much drag."

--
Ehsan
<http://ehsanakhgari.org/>

[Jonathan Kew]

Could you clarify how you expect trees that regularly merge to m-c (in
particular, mozilla-inbound) to operate during this period? Should we
assume such merges will be suspended (although individual approved
patches might be transplanted), or should inbound *also* be made
APPROVAL_REQUIRED, or what?

Thanks,

JK

[Dave Townsend]

On 04/16/12 07:56, Johnathan Nightingale wrote:
> Howdy folks,
>
> The new Fennec is getting close to beta quality. We're taking down the remaining beta blockers[1] and hope to push it in the next few weeks. As with major desktop releases in the pre-train days, we feel a lot of pressure to lock this one down and get it out the door. Part of this pressure is internal, because we are getting close and we can feel it and we don't want any surprises. Part of it is external because it's really much better than the XUL fennec currently in the market, and we want to get these improvements out to users.
>
> Fennec is based on Gecko 14, which is still on mozilla-central for another 9 days. I'd like to ask that we move mozilla-central to APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk of a late-landing change breaking Fennec. I'd propose that desktop-only changes get swift approval, and that we should have daily triage for all other requests to ensure we don't introduce too much drag.

Unless we're anticipating breaking changes (do we know of stuff landing
that is likely to have impact?) this seems like added bureaucracy for
the majority of people. I'd instead suggest that we just become even
more vigilant with the backout stick for the next 9 days and pull stuff
out of the build at the first signs of trouble.

[Johnathan Nightingale]

On Apr 16, 2012, at 12:25 PM, Dave Townsend wrote:
> Unless we're anticipating breaking changes (do we know of stuff landing that is likely to have impact?) this seems like added bureaucracy for the majority of people. I'd instead suggest that we just become even more vigilant with the backout stick for the next 9 days and pull stuff out of the build at the first signs of trouble.

*nod* I thought about something similar and mooted it on rel-drivers over the weekend, basically a "Do nothing, but ask people to tread gently" option. My concern, though, is that we have broken Fennec more than once in the past in ways that have taken a while to detect, given Fennec's much (much!) smaller user base. So the back out stick might not be invoked as swiftly as it ought. That is actually what led me to support more preemptive filtering via approvals.

J

[Kartikaya Gupta]

On 12-04-16 10:56 , Johnathan Nightingale wrote:
> Fennec is based on Gecko 14, which is still on mozilla-central for another 9 days. I'd like to ask that we move mozilla-central to APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk of a late-landing change breaking Fennec. I'd propose that desktop-only changes get swift approval, and that we should have daily triage for all other requests to ensure we don't introduce too much drag.

How would the triage team determine whether or not a change should land?
Could this process be applied retroactively on changes that land on m-c,
and back out any changes which don't meet the criteria? (i.e. use a
post-landing approval process rather than a pre-landing approval
process). That might reduce the drag if you anticipate most of the
changes being approved.

kats

[Kartikaya Gupta]

On 12-04-16 10:56 , Johnathan Nightingale wrote:

> Fennec is based on Gecko 14, which is still on mozilla-central for another 9 days. I'd like to ask that we move mozilla-central to APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk of a late-landing change breaking Fennec. I'd propose that desktop-only changes get swift approval, and that we should have daily triage for all other requests to ensure we don't introduce too much drag.

[Johnathan Nightingale]

On Apr 16, 2012, at 12:19 PM, Jonathan Kew wrote:

>> We discussed a few other options on release-drivers (migrate to
>> Aurora early, put mobile development on other branches, do nothing)
>> but each seemed like a worse outcome to me than using the
>> APPROVAL_REQUIRED hook we already have. We can get the flag added
>> today, but first I want to know whether there are any deal-breaker
>> concerns with this proposal.
>
> Could you clarify how you expect trees that regularly merge to m-c (in particular, mozilla-inbound) to operate during this period? Should we assume such merges will be suspended (although individual approved patches might be transplanted), or should inbound *also* be made APPROVAL_REQUIRED, or what?

I'd be interested in thoughts, really. The approval required hook will bounce pushes without a= (whether on each changeset or just the topmost, I don't remember) so by default, I think mozilla-inbound would be de facto APPROVAL_REQUIRED as well. On the other hand, we could let inbound queue up backlog to merge post-migration and let those with approved bugs land directly. We could also press another tree into service for the holding tank, and have an APPROVAL_REQUIRED inbound. So long as the goal of "only approved patches this week, thanks" is satisfied, I yield to the vikings on the best approach, there.

[Ehsan Akhgari]

On Mon, Apr 16, 2012 at 1:20 PM, Johnathan Nightingale
<joh...@mozilla.com>wrote:

We could temporarily maintain another version of inbound on the birch
branch which will be where people would land their a-minused patches, and
we can do daily merges from mozilla-central to birch to ensure proper
integration, and then we can merge birch to central once the migration
happens. This will be more work for the merge vikings, but will make
things much simpler for the rest of us. I volunteer to own birch if nobody
else wants to.

--
Ehsan
<http://ehsanakhgari.org/>

[Ms2ger]

Hi Johnathan, all,

On 04/16/2012 04:56 PM, Johnathan Nightingale wrote:
> Howdy folks,
>
> The new Fennec is getting close to beta quality. We're taking down
> the remaining beta blockers[1] and hope to push it in the next few
> weeks. As with major desktop releases in the pre-train days, we feel
> a lot of pressure to lock this one down and get it out the door. Part
> of this pressure is internal, because we are getting close and we can
> feel it and we don't want any surprises. Part of it is external
> because it's really much better than the XUL fennec currently in the
> market, and we want to get these improvements out to users.
>
> Fennec is based on Gecko 14, which is still on mozilla-central for
> another 9 days. I'd like to ask that we move mozilla-central to
> APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk
> of a late-landing change breaking Fennec.

I don't think this is a good idea. One of the reasons we moved to the
train model (AIUI) is that we realized that this "pressure to lock this
one down and get it out the door" did not, in fact, improve "this one".
Indeed, the patches for those "blocker" bugs would often be rushed in
right before shipping the release, more complex and less well tested
than would be deemed responsible for other bugs. This is the last week
of the cycle; *nobody* should be pushing high-risk patches at this
point. Not to "desktop" code, but not to mobile code either.

Secondly, if we have 26 bugs that absolutely block a good user
experience for Fennec, maybe we should think again about how ready it
*actually* is. Clearly, Fennec is not ready *right now*, or we wouldn't
need to introduce this bureaucracy. What, then, makes us believe that
next week, we will be ready? We don't know what state we will be in by
then; the proposal is to land, every day, several probably-intrusive
patches, whose interactions have not been tested (AFAIK).

Why should we believe that those patches will improve the situation
enough to make us happy to ship Fennec? Will we not be landing lots of
new code to aurora, too, to fix regressions from the blockers (as those
are unavoidable), or to further improve UX (because we're human—our code
is never good enough)?

Finally, it is also not clear to me why the risk of Fennec-breaking
changes is, apparently, deemed greater in the normal (platform and
desktop-specific) patches, than in the patches for those Fennec-specific
blockers.

I suggest that we leave the trees open, and remind those who push to
mozilla-central or any of its integration repositories (all people we
trust to be responsible enough not to break a product used by several
hundred million users, after all), that the last week of a cycle is
*never* the time to land high-risk patches.

HTH
Ms2ger

[Dave Mandelin]

In the past, we've briefly frozen a JS-specific tree for this kind of situation, e.g., freezing the |tracemonkey| for a day or two to merge JM. But it seems incredibly disruptive to do a 9-day freeze on landings project-wide. I think there's a high risk patches will back up, stalling followon work, and possibly causing some patches to get lost. Try could back up too.

Why not just fork off a new tree or branch for Fennec? No surprise landings + no disruption to non-Fennec development. I think the only things you'd actually want from non-Fennec development anyway are fixes for bugs that are biting you, and it should not be too hard to track them down.

Or did someone already suggest that and I didn't see it?

Dave

[Dave Mandelin]

On Monday, April 16, 2012 7:56:03 AM UTC-7, Johnathan Nightingale wrote:

[Ehsan Akhgari]

Would that still be a concern if the people who have non-approved patches
land them on Birch, and we merge Birch to mozilla-central once the uplift
to Aurora happens next week?

--
Ehsan
<http://ehsanakhgari.org/>

[Jeff Muizelaar]

On 2012-04-16, at 10:56 AM, Johnathan Nightingale wrote:

> Howdy folks,
>
> The new Fennec is getting close to beta quality. We're taking down the remaining beta blockers[1] and hope to push it in the next few weeks. As with major desktop releases in the pre-train days, we feel a lot of pressure to lock this one down and get it out the door. Part of this pressure is internal, because we are getting close and we can feel it and we don't want any surprises. Part of it is external because it's really much better than the XUL fennec currently in the market, and we want to get these improvements out to users.
>
> Fennec is based on Gecko 14, which is still on mozilla-central for another 9 days. I'd like to ask that we move mozilla-central to APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk of a late-landing change breaking Fennec. I'd propose that desktop-only changes get swift approval, and that we should have daily triage for all other requests to ensure we don't introduce too much drag.

What about Fennec changes? It would suck to introduce additional drag into getting Fennec done.

-Jeff

[Daniel Cater]

This might be a stupid question, but here it goes: why not ship the beta once the code reaches mozilla-beta? That way you have the whole of the Aurora window to fix any issues which appear due to any mozilla-central landings which break Fennec between now and the next migration.

[Daniel Cater]

[Johnathan Nightingale]

On Apr 16, 2012, at 4:08 PM, Daniel Cater wrote:

> This might be a stupid question, but here it goes: why not ship the beta once the code reaches mozilla-beta? That way you have the whole of the Aurora window to fix any issues which appear due to any mozilla-central landings which break Fennec between now and the next migration.

It's not a stupid question - in fact it's sort of the default assumption most of the time. But Fennec on Android is the lead piece of our push into mobile, paves the way for a solid HTML5 apps experience and, frankly, is overdue. Shipping also has a different set of risk trade offs than desktop Firefox (fewer users, more early adopters, different update experience). Given all this, we don't really want to just wait and watch the calendars roll by - we will ship beta when it's ready, and final when it's ready, and then so help me we will get it back on the trains ASAP.

J

[Dave Mandelin]

So Birch is a temporary mozilla-inbound but doesn't merge to m-c for the duration? Seems like that should work OK. I think there's some friction from needing to pull Birch trees, etc., but that's not too bad. It still seems a little cleaner and less disruptive to devs to put Fennec on Birch (or wherever) and leave the normal stuff normal. Are there some other operational reasons that make it easier to freeze m-c and use birch or 'normal' landings?

[Axel Hecht]

That was suggested on release-drivers, and was determined to be not-worthy:

- it puts the strain of the fork on the mobile team, which is stressed
out enough
- it forks the testing audience of the mobile product, which is small enough
- it's disrupting our already-hand-waving l10n efforts

Axel

[Johnathan Nightingale]

On Apr 16, 2012, at 3:26 PM, Dave Mandelin wrote:

> Why not just fork off a new tree or branch for Fennec? No surprise landings + no disruption to non-Fennec development. I think the only things you'd actually want from non-Fennec development anyway are fixes for bugs that are biting you, and it should not be too hard to track them down.
>
> Or did someone already suggest that and I didn't see it?

It was suggested in the rel-drivers thread that preceded this one. I replied by saying:

> ... I'd cut [the option to branch for mobile]. It makes things easy for the rest of m-c's dependents by moving all the pain onto mobile and rel-eng. Given that those teams are already under heavy pressure around this release, though, we should avoid options that add strain and take us off automation paths - we'll make more mistakes.


It's a pragmatic argument, I know, but it feels like taking an already high velocity team off the well-worn path, particularly around l10n and releng, risks making things worse.

J

[Dave Mandelin]

On Monday, April 16, 2012 12:39:35 PM UTC-7, Ehsan Akhgari wrote:

[Daniel Cater]

So you plan on shipping Gecko 14 before it's been through the usual Aurora and Beta periods (and before it ships on desktop)?

That doesn't sound good... Or have I missed something? Which version of Firefox Mobile (build on which version of Gecko) are we talking about here? Is there a roadmap for this on the wiki? I've been confused ever since Firefox Mobile released version 10.0.3 at the same time Firefox released version 11.

[Dave Mandelin]

Well, if it's an emergency, then you gotta do what you gotta do. Was anything decided yet with mozilla-inbound? Ehsan's proposal to still let anyone land to m-i, but merge that to the temp m-c daily (birch?) actually seems to avoid almost all of the problems.

One further question, though: is this kind of need likely to arise again in the future? If so, we should strive to be able to do it better (more automation, less disruption, whatever) next time around.

Dave

[Dave Mandelin]

On Monday, April 16, 2012 1:50:13 PM UTC-7, Johnathan Nightingale wrote:

[Mike Connor]

On 2012-04-16, at 4:03 PM, Jeff Muizelaar wrote:

>
> On 2012-04-16, at 10:56 AM, Johnathan Nightingale wrote:
>
>> Howdy folks,
>>
>> The new Fennec is getting close to beta quality. We're taking down the remaining beta blockers[1] and hope to push it in the next few weeks. As with major desktop releases in the pre-train days, we feel a lot of pressure to lock this one down and get it out the door. Part of this pressure is internal, because we are getting close and we can feel it and we don't want any surprises. Part of it is external because it's really much better than the XUL fennec currently in the market, and we want to get these improvements out to users.
>>
>> Fennec is based on Gecko 14, which is still on mozilla-central for another 9 days. I'd like to ask that we move mozilla-central to APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk of a late-landing change breaking Fennec. I'd propose that desktop-only changes get swift approval, and that we should have daily triage for all other requests to ensure we don't introduce too much drag.
>

> What about Fennec changes? It would suck to introduce additional drag into getting Fennec done.

One motivating factor here is that even within mobile-specific areas we're getting into an endgame, and we should be managing risk in all areas. Just because something is mobile-specific doesn't mean the risk calculus is correct to grant blanket approval.

-- Mike

[Mike Connor]

On 2012-04-16, at 4:58 PM, Dave Mandelin wrote:

> One further question, though: is this kind of need likely to arise again in the future? If so, we should strive to be able to do it better (more automation, less disruption, whatever) next time around.

At some point, I wouldn't rule it out. But whether that's "every other train" or "once a year" or "once in 30 months" is unknowable, and I'd hate to invest substantial resources in making this easier/better for a one-off.

I think the project branch approach mitigates most of this pain, what else would we really do?

-- Mike

[Alex Keybl]

> So you plan on shipping Gecko 14 before it's been through the usual Aurora and Beta periods (and before it ships on desktop)?

The final ship date hasn't yet been announced since it's blocker driven, but I can confirm that we're targeting pushing Fennec Native 14 to the beta product on the Google Play store prior to FF14's uplift to mozilla-beta.

> I've been confused ever since Firefox Mobile released version 10.0.3 at the same time Firefox released version 11.

We currently have XUL Fennec building off of the ESR10 branch so that Firefox Mobile users benefit from all of the security fixes we take there, but don't suffer from any new regressions introduced by newer Gecko versions. This allow our mobile team to focus on the future instead of the past.

-Alex

[Ehsan Akhgari]

On Mon, Apr 16, 2012 at 4:58 PM, Dave Mandelin <dman...@mozilla.com>wrote:

> On Monday, April 16, 2012 1:50:13 PM UTC-7, Johnathan Nightingale wrote:
> > On Apr 16, 2012, at 3:26 PM, Dave Mandelin wrote:
> >
> > > Why not just fork off a new tree or branch for Fennec? No surprise
> landings + no disruption to non-Fennec development. I think the only things
> you'd actually want from non-Fennec development anyway are fixes for bugs
> that are biting you, and it should not be too hard to track them down.
> > >
> > > Or did someone already suggest that and I didn't see it?
> >
> >
> > It was suggested in the rel-drivers thread that preceded this one. I
> replied by saying:
> >
> > > ... I'd cut [the option to branch for mobile]. It makes things easy
> for the rest of m-c's dependents by moving all the pain onto mobile and
> rel-eng. Given that those teams are already under heavy pressure around
> this release, though, we should avoid options that add strain and take us
> off automation paths - we'll make more mistakes.
> >
> >
> > It's a pragmatic argument, I know, but it feels like taking an already
> high velocity team off the well-worn path, particularly around l10n and
> releng, risks making things worse.
>
> Well, if it's an emergency, then you gotta do what you gotta do. Was
> anything decided yet with mozilla-inbound? Ehsan's proposal to still let
> anyone land to m-i, but merge that to the temp m-c daily (birch?) actually
> seems to avoid almost all of the problems.
>

That was not exactly my proposal. Here's what I'm proposing:

* mozilla-central and mozilla-inbound will be APPROVAL REQUIRED, and closed
to any non-NPOTB[1] patches without an approval flag.
* Regular daily two-way merges between mozilla-inbound and mozilla-central
will be performed as usual.
* birch will be open to all patches.
* Regular daily one-way merges from mozilla-central to birch will be
performed.
* Birch will be closed to check-ins as soon as the Aurora uplift has been
finished.
* At some point soon after the Aurora uplift, birch will be merged to
mozilla-central.

People whose patches get a- (or do not want to go through the trouble of
requesting approval, etc.) will land on birch. People who get their
patches approved will land on inbound/central as usual.

Cheers,
--
Ehsan
<http://ehsanakhgari.org/>

[Daniel Cater]

On Monday, April 16, 2012 10:10:12 PM UTC+1, Alex Keybl wrote:
> > So you plan on shipping Gecko 14 before it's been through the usual Aurora and Beta periods (and before it ships on desktop)?
>
> The final ship date hasn't yet been announced since it's blocker driven, but I can confirm that we're targeting pushing Fennec Native 14 to the beta product on the Google Play store prior to FF14's uplift to mozilla-beta.

OK, but more importantly, are you targeting releasing the final version (to non-Beta users) prior to the Firefox 14 desktop release? That could have web-compatabilty issues if a Gecko feature is turned off on mozilla-beta after Firefox 14 has been released on mobile with it enabled. Given that mobile's pre-release audience is much smaller than desktop's the chances of a problem being found are much less, even more-so if the testing time is reduced.

[Daniel Cater]

On Monday, April 16, 2012 10:10:12 PM UTC+1, Alex Keybl wrote:

> > So you plan on shipping Gecko 14 before it's been through the usual Aurora and Beta periods (and before it ships on desktop)?
>
> The final ship date hasn't yet been announced since it's blocker driven, but I can confirm that we're targeting pushing Fennec Native 14 to the beta product on the Google Play store prior to FF14's uplift to mozilla-beta.

[Ryan VanderMeulen]

On 4/16/2012 10:56 AM, Johnathan Nightingale wrote:
> Howdy folks,
>
> The new Fennec is getting close to beta quality. We're taking down the remaining beta blockers[1] and hope to push it in the next few weeks. As with major desktop releases in the pre-train days, we feel a lot of pressure to lock this one down and get it out the door. Part of this pressure is internal, because we are getting close and we can feel it and we don't want any surprises. Part of it is external because it's really much better than the XUL fennec currently in the market, and we want to get these improvements out to users.
>
> Fennec is based on Gecko 14, which is still on mozilla-central for another 9 days. I'd like to ask that we move mozilla-central to APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk of a late-landing change breaking Fennec. I'd propose that desktop-only changes get swift approval, and that we should have daily triage for all other requests to ensure we don't introduce too much drag.
>

> I know it's a bit weird, since the last year of train-based releases has seen mozilla-central open almost continuously. I think this is a pretty rare event, though, and that Fennec needs the help reducing risk wherever possible. Once it is out the door, we'll get Fennec back on the trains so that we don't have this kind of dependency.
>
> We discussed a few other options on release-drivers (migrate to Aurora early, put mobile development on other branches, do nothing) but each seemed like a worse outcome to me than using the APPROVAL_REQUIRED hook we already have. We can get the flag added today, but first I want to know whether there are any deal-breaker concerns with this proposal.
>

> Thanks,
>
> Johnathan
>
> [1] https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocking-fennec1.0:beta

>
> ---
> Johnathan Nightingale
> Sr. Director of Firefox Engineering
> @johnath
>

Presumably, the volume of patches getting approval is going to be fairly
small. At which point, what purpose is inbound even serving? The whole
idea of requiring some patches to land on birch and others on inbound
seems needlessly complicated.

Why not just land a+ patches directly on mozilla-central? Then, you can
leave inbound as-is (doing daily one-way m-c-->inbound merges). Bonus
points for switching desktop nightly users over to inbound builds so we
don't go a week and a half without regression spotting in the mean time.

-Ryan

[Dave Townsend]

I'm hoping that the Fennec changes would be the ones getting the most
attention in this process, since it is Fennec that we're trying to keep
from breaking.

[Damon Sicore]

On Apr 16, 2012, at 10:33 AM, Ms2ger wrote:

> Hi Johnathan, all,
>
> On 04/16/2012 04:56 PM, Johnathan Nightingale wrote:
>> Howdy folks,
>>
>> The new Fennec is getting close to beta quality. We're taking down
>> the remaining beta blockers[1] and hope to push it in the next few
>> weeks. As with major desktop releases in the pre-train days, we feel
>> a lot of pressure to lock this one down and get it out the door. Part
>> of this pressure is internal, because we are getting close and we can
>> feel it and we don't want any surprises. Part of it is external
>> because it's really much better than the XUL fennec currently in the
>> market, and we want to get these improvements out to users.
>>
>> Fennec is based on Gecko 14, which is still on mozilla-central for
>> another 9 days. I'd like to ask that we move mozilla-central to
>> APPROVAL_REQUIRED for the next 9 days so that we can reduce the risk
>> of a late-landing change breaking Fennec.
>
> I don't think this is a good idea. One of the reasons we moved to the train model (AIUI) is that we realized that this "pressure to lock this one down and get it out the door" did not, in fact, improve "this one".

Actually it did. All of our pre-train releases were created using similar tactics. What's different now is that we have trains running on time with a different product that is trying ship it's first release. There's less time and therefore more urgency for making sure we don't introduce regressions.

> Indeed, the patches for those "blocker" bugs would often be rushed in right before shipping the release, more complex and less well tested than would be deemed responsible for other bugs. This is the last week of the cycle; *nobody* should be pushing high-risk patches at this point. Not to "desktop" code, but not to mobile code either.

Exactly. Which is why we are suggesting approvals.

>
> Secondly, if we have 26 bugs that absolutely block a good user experience for Fennec, maybe we should think again about how ready it *actually* is. Clearly, Fennec is not ready *right now*, or we wouldn't need to introduce this bureaucracy. What, then, makes us believe that next week, we will be ready? We don't know what state we will be in by then; the proposal is to land, every day, several probably-intrusive patches, whose interactions have not been tested (AFAIK)


>

> Why should we believe that those patches will improve the situation enough to make us happy to ship Fennec?
> Will we not be landing lots of new code to aurora, too, to fix regressions from the blockers (as those are unavoidable), or to further improve UX (because we're human—our code is never good enough)?
>
> Finally, it is also not clear to me why the risk of Fennec-breaking changes is, apparently, deemed greater in the normal (platform and desktop-specific) patches, than in the patches for those Fennec-specific blockers.
>
> I suggest that we leave the trees open, and remind those who push to mozilla-central or any of its integration repositories (all people we trust to be responsible enough not to break a product used by several hundred million users, after all), that the last week of a cycle is *never* the time to land high-risk patches.

The approval is the reminder. It's simple and effective. (NOTE: I've developed *the* perfect shipping tactic, but it's too large to write in the margin of this email).

We need to minimize our path to finishing Firefox for Android. To do this, we must make sure we are mitigating risk of regressions no matter where they are coming from. Approvals is a very good technique for reducing risk.

Sincerely,

Damon

>
> HTH
> Ms2ger

[Marco Bonardo]

On 16/04/2012 19:30, Ehsan Akhgari wrote:
> We could temporarily maintain another version of inbound on the birch
> branch which will be where people would land their a-minused patches, and
> we can do daily merges from mozilla-central to birch to ensure proper
> integration, and then we can merge birch to central once the migration
> happens.

Honestly, don't understand all of this complication, can't we just let
people land on inbound and freeze m-c?
Inbound was created also to allow freezing m-c without blocking everyone
and this is exactly that case.
So I suggest we just freeze central, anyone can land on inbound, only
approved patches can land in central. Daily we merge FROM central TO
inbound, but we don't do the opposite merge till the freeze is over.

-m

[Ehsan Akhgari]

That _would_ work if people would not land on inbound with the expectation
that they will get merged to central. Not everybody may be reading this
thread, which is why I suggested closing down inbound (so that pushes
without a= in the commit message fail) and use a third integration branch
for the non-approved patches.)

--
Ehsan
<http://ehsanakhgari.org/>

[Marco Bonardo]

What's the difference? if their patches will be merged to birch are not
we still breaking their expectation to be merged to central?
What bad things may happen then? I suppose someone will sneak into
developers and ask, or check the tree status and figure out the reason.
A blog post + tree status + #developers topic can cover most of the
notification needs.

Another fact is that some devs are not even using inbound to limit the
number of cloned branches around, your proposal is now also adding Birch
to the plot.
I just don't think we need all of this complication, and it may
complicate tracking (devs would have to start tracking what's on
inbound, what's on Birch and what's on central, rather than just pushed
VS merged).
We lived for years with a single branch, the world won't come to an end
for 9 days back to the old situation :)

-m

[Johnathan Nightingale]

On Apr 17, 2012, at 10:21 AM, Marco Bonardo wrote:

> We lived for years with a single branch, the world won't come to an end for 9 days back to the old situation :)

I think Ehsan's recognizing that many devs prefer the inbound model because it means not watching the tree the same way one does landing directly on central. Having said that, I think we should:

- Mark mozilla-central and mozilla-inbound as APPROVAL_REQUIRED. Patches will bounce off those without a=, but for approved patches the workflow is unchanged.
- If you want to land unapproved stuff for future integration into FF15, Ehsan has offered to keep birch in a healthy state, but if that is annoying to you, then sit tight and land next week in the usual ways.

Akeybl has offered to manage the mechanics of getting the bug added and spinning up daily triage - I think we should make that happen today. Thanks for the discussion and the flexibility and the offers to help.

J

[Marco Bonardo]

On 17/04/2012 16:34, Johnathan Nightingale wrote:
> I think Ehsan's recognizing that many devs prefer the inbound model because it means not watching the tree the same way one does landing directly on central. Having said that, I think we should:

Right, I was trying to keep those niceties by having inbound stay open
and not forcing anyone to clone again another tree.
PErsonally I'm fine with any plan we take, just wanted to point out a
couple downsides of the Birch plan.

Cheers,
Marco

[Ehsan Akhgari]

As far as I can tell, the only downside is having to clone another
repository, and I hope people can live with that (otherwise as Johnathan
mentioned, they can hold off on their patches for a week).

[Boris Zbarsky]

On 4/17/12 11:09 AM, Ehsan Akhgari wrote:
> As far as I can tell, the only downside is having to clone another
> repository

Which is a huge downside for some contributors, actually.

> and I hope people can live with that (otherwise as Johnathan
> mentioned, they can hold off on their patches for a week).

Which is increasing the set of things people have to remember; precisely
what inbound is supposed to avoid.

I think Marco's right: we should just freeze central. If someone has
patches that _need_ to land for 14, they should be asking approval.
Worst-case after the aurora merge, once they realize they missed the
merge. If the patch is important enough that it would get approved
anyway, it would get approved the day after merge too.

And that keeps several hundred people from having to spend a week
juggling patches that they could just land and move on.

-Boris

[Mike Hommey]

On Tue, Apr 17, 2012 at 11:09:39AM -0400, Ehsan Akhgari wrote:

> As far as I can tell, the only downside is having to clone another

> repository.

If you know mercurial enough, you don't need to really clone another
repository. But it's very easy to mess up with such setups.

Mike

[Chris Peterson]

On 4/17/12 9:04 AM, Boris Zbarsky wrote:
> I think Marco's right: we should just freeze central. If someone has
> patches that _need_ to land for 14, they should be asking approval.

Could we uplift central 14 to aurora now (one week early)? Uplifting is
an existing process that does not block anyone. aurora sounds like the
more-stable-than-central, approval-required branch people are looking for.

We would not need to uplift all the channels, though aurora 13 would be
in limbo between aurora and beta until next week's uplift. Is there a
precedent of uplifting one channel before others?


chris

[Axel Hecht]

Uplifting now would cut into the aurora phase of 13, in particular, the
localization of Desktop 13.

Axel

[Kevin Brosnan]

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

This was one of the initial set of suggestions and dismissed due to
downstream partners among other reasons.

Kevin

[Alex Keybl]

We've just gone Approval Required on both mozilla-inbound and mozilla-central. The tbpl header currently reads:

> APPROVAL REQUIRED. Tree Rules. You should use inbound instead (but probably won't). Bugs marked with blocking-fennec1.0+, sg:high, sg:crit, or tracking-firefoxN+ can land on m-i or m-c without explicit approval (include a=blocking-fennec, a=security, or a=tracking-firefox in your commit comment). Any other desired FF14 landings must nominate their change for approval (currently blocked on bug 746210). Please see this dev-planning thread for more info.

Once the approval-mozilla-central flag is ready and we have details about daily triage, I'll email again on this thread. If you've got any questions about the merge mechanics between m-i and m-c, or how to land on birch, please get in touch with Ehsan.

-Alex

On Apr 17, 2012, at 7:34 AM, Johnathan Nightingale wrote:

> On Apr 17, 2012, at 10:21 AM, Marco Bonardo wrote:
>

>> We lived for years with a single branch, the world won't come to an end for 9 days back to the old situation :)
>
>

> I think Ehsan's recognizing that many devs prefer the inbound model because it means not watching the tree the same way one does landing directly on central. Having said that, I think we should:
>

[Ryan VanderMeulen]

On 4/17/2012 1:50 PM, Alex Keybl wrote:
> We've just gone Approval Required on both mozilla-inbound and mozilla-central. The tbpl header currently reads:
>
>> APPROVAL REQUIRED. Tree Rules. You should use inbound instead (but probably won't). Bugs marked with blocking-fennec1.0+, sg:high, sg:crit, or tracking-firefoxN+ can land on m-i or m-c without explicit approval (include a=blocking-fennec, a=security, or a=tracking-firefox in your commit comment). Any other desired FF14 landings must nominate their change for approval (currently blocked on bug 746210). Please see this dev-planning thread for more info.
>
> Once the approval-mozilla-central flag is ready and we have details about daily triage, I'll email again on this thread. If you've got any questions about the merge mechanics between m-i and m-c, or how to land on birch, please get in touch with Ehsan.
>
> -Alex
>

I'm very concerned about a lack of nightly testing for a week and a
half. It seems very unlikely that people are going to switch over to
Birch builds. Is there any plan to mitigate this?

[Boris Zbarsky]

On 4/17/12 5:44 PM, Ryan VanderMeulen wrote:
> I'm very concerned about a lack of nightly testing for a week and a
> half.

You're not the only one.

Are the birch nightlies going to stick around for the same amount of
time as m-c and inbound nightlies do, by the way, so we can at least try
to hunt down the regressions later?

-Boris

[Alex Keybl]

The approval-mozilla-central flag is now available for approval nominations. Rather than having daily triage meetings, we've decided to more continuously perform triage with the mobile team using the branch-triage list (see [1]). If you'd like to follow along, please subscribe to that mailing list. We plan to send a couple of rollups of new nominations a day for discussion, and will also follow up on any open questions by CC'ing that list.

-Alex

[1] https://mail.mozilla.org/listinfo/branch-triage

On Apr 17, 2012, at 10:50 AM, Alex Keybl wrote:

> We've just gone Approval Required on both mozilla-inbound and mozilla-central. The tbpl header currently reads:
>
>> APPROVAL REQUIRED. Tree Rules. You should use inbound instead (but probably won't). Bugs marked with blocking-fennec1.0+, sg:high, sg:crit, or tracking-firefoxN+ can land on m-i or m-c without explicit approval (include a=blocking-fennec, a=security, or a=tracking-firefox in your commit comment). Any other desired FF14 landings must nominate their change for approval (currently blocked on bug 746210). Please see this dev-planning thread for more info.
>
> Once the approval-mozilla-central flag is ready and we have details about daily triage, I'll email again on this thread. If you've got any questions about the merge mechanics between m-i and m-c, or how to land on birch, please get in touch with Ehsan.
>
> -Alex
>

> On Apr 17, 2012, at 7:34 AM, Johnathan Nightingale wrote:
>
>> On Apr 17, 2012, at 10:21 AM, Marco Bonardo wrote:
>>

>>> We lived for years with a single branch, the world won't come to an end for 9 days back to the old situation :)
>>
>>

>> I think Ehsan's recognizing that many devs prefer the inbound model because it means not watching the tree the same way one does landing directly on central. Having said that, I think we should:
>>
>> - Mark mozilla-central and mozilla-inbound as APPROVAL_REQUIRED. Patches will bounce off those without a=, but for approved patches the workflow is unchanged.
>> - If you want to land unapproved stuff for future integration into FF15, Ehsan has offered to keep birch in a healthy state, but if that is annoying to you, then sit tight and land next week in the usual ways.
>>
>> Akeybl has offered to manage the mechanics of getting the bug added and spinning up daily triage - I think we should make that happen today. Thanks for the discussion and the flexibility and the offers to help.
>>
>> J
>>
>> ---
>> Johnathan Nightingale
>> Sr. Director of Firefox Engineering
>> @johnath
>>
>

[Justin Dolske]

On 4/16/12 1:58 PM, Dave Mandelin wrote:

>> It's a pragmatic argument, I know, but it feels like taking an already high velocity team off the well-worn path, particularly around l10n and releng, risks making things worse.
>
> Well, if it's an emergency, then you gotta do what you gotta do.

This kind of sums up my feeling -- it's a change for 9 days. Even if it
was a crazy complete closure I think we'd still manage to get by with
minimal disruption. Not idea in the grand scheme, but neither is a
disaster. I suspect the cost of hand-wringing about it is more than the
cost of actually doing it. ;-)

Let's just not get in the habit of doing this. :)

Justin

[Kyle Huey]

On Tue, Apr 17, 2012 at 12:56 AM, Johnathan Nightingale <joh...@mozilla.com

> wrote:

> Howdy folks,
>
> The new Fennec is getting close to beta quality. We're taking down the
> remaining beta blockers[1] and hope to push it in the next few weeks. As
> with major desktop releases in the pre-train days, we feel a lot of
> pressure to lock this one down and get it out the door. Part of this
> pressure is internal, because we are getting close and we can feel it and
> we don't want any surprises. Part of it is external because it's really
> much better than the XUL fennec currently in the market, and we want to get
> these improvements out to users.
>
> Fennec is based on Gecko 14, which is still on mozilla-central for another
> 9 days. I'd like to ask that we move mozilla-central to APPROVAL_REQUIRED
> for the next 9 days so that we can reduce the risk of a late-landing change

> breaking Fennec. I'd propose that desktop-only changes get swift approval,
> and that we should have daily triage for all other requests to ensure we
> don't introduce too much drag.
>

If the Fennec blockers in [1] are not done by Tuesday, will the tree reopen
at the Aurora migration for 14?

- Kyle

[Boris Zbarsky]

On 4/17/12 7:13 PM, Justin Dolske wrote:
> This kind of sums up my feeling -- it's a change for 9 days. Even if it
> was a crazy complete closure I think we'd still manage to get by with
> minimal disruption.

Fwiw, I think a complete closure with metered checkins on reopening
would be better than the current plan of crash-landing 9 days of
development work on the trunk after the closure....

> Let's just not get in the habit of doing this. :)

We've managed to avoid doing it for almost a year.... Maybe we can
manage that again. ;)

-Boris

[Daniel Cater]

On Monday, April 16, 2012 9:42:31 PM UTC+1, Johnathan Nightingale wrote:
> On Apr 16, 2012, at 4:08 PM, Daniel Cater wrote:
>
> > This might be a stupid question, but here it goes: why not ship the beta once the code reaches mozilla-beta? That way you have the whole of the Aurora window to fix any issues which appear due to any mozilla-central landings which break Fennec between now and the next migration.
>
> It's not a stupid question - in fact it's sort of the default assumption most of the time. But Fennec on Android is the lead piece of our push into mobile, paves the way for a solid HTML5 apps experience and, frankly, is overdue. Shipping also has a different set of risk trade offs than desktop Firefox (fewer users, more early adopters, different update experience). Given all this, we don't really want to just wait and watch the calendars roll by - we will ship beta when it's ready, and final when it's ready, and then so help me we will get it back on the trains ASAP.
>

> J
>
> ---
> Johnathan Nightingale
> Sr. Director of Firefox Engineering
> @johnath

I don't really see the reasoning outweighing the downsides here. Firefox Beta on Google Play is currently at version 12, so before releasing Firefox 14 on mobile the Firefox 13 changes and the Firefox 14 changes have to be tested. This proposal only brings the amount of mozilla-central changes to be tested down from 12 weeks worth to 11 weeks worth.

[Alex Keybl]

> If the Fennec blockers in [1] are not done by Tuesday, will the tree reopen
> at the Aurora migration for 14?

Yes - our plan is to ship a beta based off of Gecko 14 (so the mobile team's focus will remain there). We'll reopen m-c on 4/24 regardless.

-Alex

[Alex Keybl]

Hi Ryan,

Builds will be available for developers to test on a regular basis - we do not have plans to move any of our nightly testing population to the birch branch. For sake of regression window hunting off of m-c, we will likely land birch changes in chunks over a few days. More details about that strategy will be available prior to 4/24.

-Alex

[Alex Keybl]

> Are the birch nightlies going to stick around for the same amount of time as m-c and inbound nightlies do, by the way, so we can at least try to hunt down the regressions later?

This is a good suggestion - I'll follow up with RelEng to make sure that we keep these builds around.

-Alex

[Alex Keybl]

Nick Thomas let me know that RelEng will be keeping birch nightlies around indefinitely.

-Alex

On Apr 17, 2012, at 4:50 PM, Alex Keybl wrote:

>> Are the birch nightlies going to stick around for the same amount of time as m-c and inbound nightlies do, by the way, so we can at least try to hunt down the regressions later?
>

> This is a good suggestion - I'll follow up with RelEng to make sure that we keep these builds around.
>

> -Alex
>

> On Apr 17, 2012, at 2:57 PM, Boris Zbarsky wrote:
>

[Ehsan Akhgari]

FWIW, if at the end of this period there are many changes on the birch
branch (there are none at the time of this writing), I can merge in the
changes in multiple chunks, over multiple nightlies, for easier regression
detection later on.

--
Ehsan
<http://ehsanakhgari.org/>

> ______________________________**_________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-planning<https://lists.mozilla.org/listinfo/dev-planning>
>

[Ms2ger]

On 04/17/2012 03:06 AM, Damon Sicore wrote:
>> Indeed, the patches for those "blocker" bugs would often be rushed
>> in right before shipping the release, more complex and less well
>> tested than would be deemed responsible for other bugs. This is the
>> last week of the cycle; *nobody* should be pushing high-risk
>> patches at this point. Not to "desktop" code, but not to mobile
>> code either.
>
> Exactly. Which is why we are suggesting approvals.

I guess you're correct in assuming our L3 committers aren't trustworthy.
Of course, you will be instating this policy for the last week of every
cycle?

Sincerely
Ms2ger

[Justin Dolske]

On 4/18/12 8:22 AM, Ms2ger wrote:

> I guess you're correct in assuming our L3 committers aren't trustworthy.
> Of course, you will be instating this policy for the last week of every
> cycle?

I don't think it's fair or helpful to say things like that. :( No one is
saying either of those things.

Justin