State of mozilla-central, consider reopening the tree?

[Ehsan Akhgari]

I'd like to know if we have an estimate on what needs to be done and how
long it's going to take before we can reopen mozilla-central? We now
have a mozilla-2.1 repository which I think is going to get used for
Fennec 4.0 release. Is there anything else holding us from reopening
the tree?

Also, may I propose that we need to meter the landings for the first few
days after mozilla-central gets reopened? I'm willing to help
sheriffing if needed.

Cheers,
--
Ehsan Akhgari
http://ehsanakhgari.org/

[Mike Beltzner]

I think that until we understand how we're going to control things as per Sayrer's latest document, the tree should remain closed to anything other than patches with explicit approval.

cheers,
mike

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Ehsan Akhgari]

On 11-03-21 4:17 PM, Mike Beltzner wrote:
> I think that until we understand how we're going to control things as per Sayrer's latest document, the tree should remain closed to anything other than patches with explicit approval.

I don't think that we should leave this as an open-ended matter. I
think we should have an expected date for mozilla-central reopening, and
we should address how things would look like with the aforementioned
changes by that date.

The number of changes that people would suddenly be pushing in huge
chunks makes me worried.

[L. David Baron]

On Monday 2011-03-21 16:30 -0400, Ehsan Akhgari wrote:
> On 11-03-21 4:17 PM, Mike Beltzner wrote:
> >I think that until we understand how we're going to control things as per Sayrer's latest document, the tree should remain closed to anything other than patches with explicit approval.
>
> I don't think that we should leave this as an open-ended matter. I
> think we should have an expected date for mozilla-central reopening,
> and we should address how things would look like with the
> aforementioned changes by that date.

Given the delay, I also think we're in danger of delaying the
reopening to past the point when we want to get our first pull from
mozilla-central to firefox-experimental.

In order to get into the rhythm of shipping regular releases, we
should really be aiming to do the first of such regular releases
based on the normal amount (i.e., six weeks) of mozilla-central
development. I think this is important to get a sense of how the
process is going to feel and how much we expect to ship in each
release, and to get into the rhythm of the new process.

In the proposed model at
http://people.mozilla.com/~sayrer/2011/temp/process.html , we'd have
six weeks of development time for each cycle (without any freeze
preceding it for people to buffer up work). If we guess that the
average developer has been done with Firefox 4 work and working on
post-Firefox 4 work for about a month, that means that first pull
from mozilla-central to firefox-experimental should be in about two
weeks.

I think trying to put more work than that (i.e., more than what
people can do by, say, early April) into Firefox 5 will put at risk
our ability to move it to firefox-experimental, firefox-beta, and to
ship on the schedule that we'd like.

-David

--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/

[Chris Hofmann]

This is definitely the way to think about it.

It's the front end of these release cycles, and the volume and
complexity of changes that are landed, that will determine if the back
end of the schedule proceeds in an orderly and reliable way.

avoiding half baked changes, and keeping them away from mozilla-central
is really a more important part of the plan, than if we have 2, 3, 4, or
10 development channels.

I might even suggest that between the dependencies falling off
Bug 610267 ( tracking patches waiting for m-c to reopen after Gecko 2.0)
and other work hanging around in developer repositories there might
already be a
full release of work completed.

The key questions to ask about all the changes that are piling up are:

-Is all the work, and and any possible dependency work, completed?
-Are test cases ready for all the landings?
-How much time has been invested in pre-landing testing?
-Is there a back-out strategy for every change that is already to go?

If these questions don't get asked, and answered for every change that
goes in once the tree is open there is a lot of risk going into hitting
that first release date.

-chofmann

[Phil Ringnalda]

On 3/21/11 2:05 PM, L. David Baron wrote:

> In the proposed model at
> http://people.mozilla.com/~sayrer/2011/temp/process.html , we'd have
> six weeks of development time for each cycle (without any freeze
> preceding it for people to buffer up work). If we guess that the
> average developer has been done with Firefox 4 work and working on
> post-Firefox 4 work for about a month, that means that first pull
> from mozilla-central to firefox-experimental should be in about two
> weeks.

And keep in mind that that's only the *average* developer - we have a
lot of stuff that got a- or a-ignored as long ago as last August, and
for people who didn't realize for a while that we weren't going to ever
have a trunk again, like timeless, the backlog is more like from last June.

[Ehsan Akhgari]

On 11-03-21 5:33 PM, Chris Hofmann wrote:
> The key questions to ask about all the changes that are piling up are:
>
> -Is all the work, and and any possible dependency work, completed?
> -Are test cases ready for all the landings?
> -How much time has been invested in pre-landing testing?
> -Is there a back-out strategy for every change that is already to go?

I totally agree. What worries me is that we don't have a vehicle for
asking these questions, and getting stuff which pass the critera landed
on mozilla-central right now. We don't even have an approval flag which
we can set for drivers to evaluate each change.

We should also have a very well defined process for asking for a change
to be evaluated, and getting *timely* answers from drivers. We used to
have the approavl2.0? flag for Gecko 2.0, but that effectively was
ignored, unless somebody asked a driver to approve a patch using other
media. We probably need a better mechanism if mozilla-central is to be
kept open only to approved patches all the time.

Another possible strategy would be to be more lenient on approvals, and
more aggressive on backouts in case regressions are found. Each
approach has its pros and cons, so we may need to choose a mixture of
those depending on the corresponding project/component/module...

[Ehsan Akhgari]

Which makes me even more scared to keep mozilla-central closed down any
further.

[Mike Connor]

Oof. That's a pretty scary situation. I wonder if we want to allocate one of the temp project branches to "land all this random crap" to keep it away from m-c, do our best to shake out the issues, and then merge. Hardest part might be finding someone to own such a mass-catchup of work...

-- Mike

[Mike Connor]

On 2011-03-21, at 2:48 PM, Ehsan Akhgari wrote:

>> And keep in mind that that's only the *average* developer - we have a
>> lot of stuff that got a- or a-ignored as long ago as last August, and
>> for people who didn't realize for a while that we weren't going to ever
>> have a trunk again, like timeless, the backlog is more like from last June.
>

> Which makes me even more scared to keep mozilla-central closed down any further.

I actually think keeping it closed until we figure out a path for this massive backlog might be even better. m-c is not the place to unravel all of that, IMO.

-- Mike

[Ehsan Akhgari]

On 11-03-21 5:48 PM, Mike Connor wrote:
>
> On 2011-03-21, at 2:40 PM, Phil Ringnalda wrote:
>

> Oof. That's a pretty scary situation. I wonder if we want to allocate one of the temp project branches to "land all this random crap" to keep it away from m-c, do our best to shake out the issues, and then merge. Hardest part might be finding someone to own such a mass-catchup of work...

I think this can only be useful if we have a good plan to QA that
branch, so that the changes there are not simply forgotten over time,
otherwise we'd just end up ignoring them.

I understand that we have this backlog that we need to deal with
somehow, but I think once that has been dealt with, the best way to
address this in the future is to never let a backlog to form again. I
think the new proposed process is well capable of achieving that.

[Mike Connor]

On 2011-03-21, at 2:54 PM, Ehsan Akhgari wrote:

> On 11-03-21 5:48 PM, Mike Connor wrote:
>>
>> On 2011-03-21, at 2:40 PM, Phil Ringnalda wrote:
>>

>> Oof. That's a pretty scary situation. I wonder if we want to allocate one of the temp project branches to "land all this random crap" to keep it away from m-c, do our best to shake out the issues, and then merge. Hardest part might be finding someone to own such a mass-catchup of work...
>
> I think this can only be useful if we have a good plan to QA that branch, so that the changes there are not simply forgotten over time, otherwise we'd just end up ignoring them.

+1. Someone has to own it, and drive it like a project with a plan to identify/resolve issues and plan a merge.

> I understand that we have this backlog that we need to deal with somehow, but I think once that has been dealt with, the best way to address this in the future is to never let a backlog to form again. I think the new proposed process is well capable of achieving that.

+1000

[Ehsan Akhgari]

On 11-03-21 6:01 PM, Mike Connor wrote:
>
> On 2011-03-21, at 2:54 PM, Ehsan Akhgari wrote:
>
>> On 11-03-21 5:48 PM, Mike Connor wrote:
>>>
>>> On 2011-03-21, at 2:40 PM, Phil Ringnalda wrote:
>>>

>>> Oof. That's a pretty scary situation. I wonder if we want to allocate one of the temp project branches to "land all this random crap" to keep it away from m-c, do our best to shake out the issues, and then merge. Hardest part might be finding someone to own such a mass-catchup of work...
>>
>> I think this can only be useful if we have a good plan to QA that branch, so that the changes there are not simply forgotten over time, otherwise we'd just end up ignoring them.
>
> +1. Someone has to own it, and drive it like a project with a plan to identify/resolve issues and plan a merge.

I may volunteer to own the project if we can focus QA resources to it.

I think this very important, especially since that many of those patches
are at the risk of being bitrotted, by changes on trunk, and many of
them come from the volunteers in the community, so leaving them out in
the cold makes them obsolete, and may have a bad impression on community
members.

Cheers,

[Christian Legnitto]

On Mar 21, 2011, at 3:08 PM, Ehsan Akhgari wrote:

> On 11-03-21 6:01 PM, Mike Connor wrote:
>>
>> On 2011-03-21, at 2:54 PM, Ehsan Akhgari wrote:
>>
>>> On 11-03-21 5:48 PM, Mike Connor wrote:
>>>>
>>>> On 2011-03-21, at 2:40 PM, Phil Ringnalda wrote:
>>>>

>>>> Oof. That's a pretty scary situation. I wonder if we want to allocate one of the temp project branches to "land all this random crap" to keep it away from m-c, do our best to shake out the issues, and then merge. Hardest part might be finding someone to own such a mass-catchup of work...
>>>
>>> I think this can only be useful if we have a good plan to QA that branch, so that the changes there are not simply forgotten over time, otherwise we'd just end up ignoring them.
>>
>> +1. Someone has to own it, and drive it like a project with a plan to identify/resolve issues and plan a merge.
>
> I may volunteer to own the project if we can focus QA resources to it.
>
> I think this very important, especially since that many of those patches are at the risk of being bitrotted, by changes on trunk, and many of them come from the volunteers in the community, so leaving them out in the cold makes them obsolete, and may have a bad impression on community members.

I'd be fine with Ehsan managing the details and I am sure we can rustle up some QA resources.

FWIW, before this thread sprung up Shelia and I were talking with bsmedberg of the proper way to manage this. Because this discussion has now generally superseded ours I'm fine to hand it off.

Christian

[chris hofmann]

On 3/21/11 2:46 PM, Ehsan Akhgari wrote:
> On 11-03-21 5:33 PM, Chris Hofmann wrote:
>> The key questions to ask about all the changes that are piling up are:
>>
>> -Is all the work, and and any possible dependency work, completed?
>> -Are test cases ready for all the landings?
>> -How much time has been invested in pre-landing testing?
>> -Is there a back-out strategy for every change that is already to go?
>
> I totally agree. What worries me is that we don't have a vehicle for
> asking these questions, and getting stuff which pass the critera
> landed on mozilla-central right now. We don't even have an approval
> flag which we can set for drivers to evaluate each change.
>

Well, there are a whole range of options for managing this.

At a minimum each developer can as the questions of themselves for each
of the changes that they plan to land. It can be self-policing.

A decade ago we used to have something called "landing planning
meetings" where at least the big changes could be discussed, decided
on, and coordinated.

The key is also not to get too many big landings to happen at the same
time, or in the same release. That makes sorting out regressions a lot
more time consuming and puts schedules at risk.

https://wiki.mozilla.org/User_talk:ChrisHofmann/QuarterlyReleases
documents a bit of how that worked.

-chofmann

[Ehsan Akhgari]

On 11-03-21 6:25 PM, Christian Legnitto wrote:
> On Mar 21, 2011, at 3:08 PM, Ehsan Akhgari wrote:
>
>> On 11-03-21 6:01 PM, Mike Connor wrote:
>>>
>>> On 2011-03-21, at 2:54 PM, Ehsan Akhgari wrote:
>>>
>>>> On 11-03-21 5:48 PM, Mike Connor wrote:
>>>>>
>>>>> On 2011-03-21, at 2:40 PM, Phil Ringnalda wrote:
>>>>>

>>>>> Oof. That's a pretty scary situation. I wonder if we want to allocate one of the temp project branches to "land all this random crap" to keep it away from m-c, do our best to shake out the issues, and then merge. Hardest part might be finding someone to own such a mass-catchup of work...
>>>>
>>>> I think this can only be useful if we have a good plan to QA that branch, so that the changes there are not simply forgotten over time, otherwise we'd just end up ignoring them.
>>>
>>> +1. Someone has to own it, and drive it like a project with a plan to identify/resolve issues and plan a merge.
>>
>> I may volunteer to own the project if we can focus QA resources to it.
>>
>> I think this very important, especially since that many of those patches are at the risk of being bitrotted, by changes on trunk, and many of them come from the volunteers in the community, so leaving them out in the cold makes them obsolete, and may have a bad impression on community members.
>
> I'd be fine with Ehsan managing the details and I am sure we can rustle up some QA resources.
>
> FWIW, before this thread sprung up Shelia and I were talking with bsmedberg of the proper way to manage this. Because this discussion has now generally superseded ours I'm fine to hand it off.

OK, I booked the cedar twig for this project
<https://bugzilla.mozilla.org/show_bug.cgi?id=643603>. I'll manage the
check-ins, and making sure that the tests pass, etc. I will also
coordinate with Matt about the QA once the patches are there on the twig.

Please contact me if you're interested in helping!

[Ehsan Akhgari]

On 11-03-21 8:38 PM, Matt Evans wrote:
> Ehsan,
>
> Let's touch base later tomorrow. Do you have bug query for this set of
> patches?

http://mzl.la/fOF72A

Currently 278 bugs in that list. (Query courtesy of mconnor and bz)

We can have a chat over the phone or on IRC. I was thinking about right
after the platform call. Would that work for those interested?

Ehsan

[Benjamin Smedberg]

On 3/21/2011 3:42 PM, Ehsan Akhgari wrote:
> I'd like to know if we have an estimate on what needs to be done and
> how long it's going to take before we can reopen mozilla-central? We
> now have a mozilla-2.1 repository which I think is going to get used
> for Fennec 4.0 release. Is there anything else holding us from
> reopening the tree?
>
> Also, may I propose that we need to meter the landings for the first
> few days after mozilla-central gets reopened? I'm willing to help
> sheriffing if needed.

Here's my proposal, I floated this via private email earlier but I can
repeat it publicly:

* Reopen mozilla-central tomorrow morning Eastery (Wednesday), to avoid
conflict with release activities today.
* I will sheriff tomorrow. We will need our most experienced sheriffs
for the next week or so, so I may rearrange the schedule a little bit.
* Because this first cycle for Firefox 5 has a small cycle, sheriffs
should heavily prioritize those checkins which have been identified as
product priorities for Firefox 5. Because product management haven't
really announced this list yet, this will involve a lot of judgement
from sheriffs.
* Of course everything must be backoutable/disableable as originally
described in sayrer's post.
* When there is no sheriff actively on duty, the tree will be closed, at
least for the first week. We'll try to round-robin the tree when
possible so that there is timezone coverage around the world.

We should discuss this plan today at the engineering meeting.

--BDS

[Ehsan Akhgari]

On 11-03-22 9:49 AM, Ehsan Akhgari wrote:
> On 11-03-21 8:38 PM, Matt Evans wrote:
>> Ehsan,
>>
>> Let's touch base later tomorrow. Do you have bug query for this set of
>> patches?
>
> http://mzl.la/fOF72A

Better query: http://mzl.la/h23LLW

This one excludes bugs outside of Core, Toolkit, or Firefox components,
and also those which have "fixed-in" in their status whiteboard.

I'll start going through the list, and will mark the bugs that I land as
fixed-in-cider. I've also shared the query under the name of "Bugs to
land on cider".

Cheers,
Ehsan

[Ehsan Akhgari]

s/cider/cedar/, of course. :)

Ehsan

[L. David Baron]

Sounds like a good plan.

One additional point: we may want to generate some additional
nightly builds over the course of the day so we have them around for
bisecting regressions.

(And of course I'm willing to help with the sheriffing.)

[Boris Zbarsky]

On 3/22/11 10:13 AM, Benjamin Smedberg wrote:
> * Because this first cycle for Firefox 5 has a small cycle, sheriffs
> should heavily prioritize those checkins which have been identified as
> product priorities for Firefox 5. Because product management haven't
> really announced this list yet, this will involve a lot of judgement
> from sheriffs.

How will this interact with the cedar approach? In particular, should I
be landing my backlog of performance fixes on cedar, or trying to land
it on m-c, or what?

-Boris

P.S. Note that at least some of said performance fixes have been landed
on cedar already.

[Chris Hofmann]

On 3/21/11 3:29 PM, chris hofmann wrote:
> On 3/21/11 2:46 PM, Ehsan Akhgari wrote:
>> On 11-03-21 5:33 PM, Chris Hofmann wrote:
>>> The key questions to ask about all the changes that are piling up are:
>>>
>>> -Is all the work, and and any possible dependency work, completed?
>>> -Are test cases ready for all the landings?
>>> -How much time has been invested in pre-landing testing?
>>> -Is there a back-out strategy for every change that is already to go?
>>
>> I totally agree. What worries me is that we don't have a vehicle for
>> asking these questions, and getting stuff which pass the critera
>> landed on mozilla-central right now. We don't even have an approval
>> flag which we can set for drivers to evaluate each change.
>>
>
> Well, there are a whole range of options for managing this.
>
> At a minimum each developer can as the questions of themselves for
> each of the changes that they plan to land. It can be self-policing.

I made a page on the wiki to record ideas about things to think about
before landing, and added some more ideas about security and localization.

Feel free to add items or more detail

https://wiki.mozilla.org/RapidRelease/LandingPlan

-chofmann

>
> A decade ago we used to have something called "landing planning
> meetings" where at least the big changes could be discussed, decided
> on, and coordinated.
>
> The key is also not to get too many big landings to happen at the same
> time, or in the same release. That makes sorting out regressions a
> lot more time consuming and puts schedules at risk.
>
> https://wiki.mozilla.org/User_talk:ChrisHofmann/QuarterlyReleases
> documents a bit of how that worked.
>
> -chofmann
>
>
>> We should also have a very well defined process for asking for a
>> change to be evaluated, and getting *timely* answers from drivers.
>> We used to have the approavl2.0? flag for Gecko 2.0, but that
>> effectively was ignored, unless somebody asked a driver to approve a
>> patch using other media. We probably need a better mechanism if
>> mozilla-central is to be kept open only to approved patches all the
>> time.
>>
>> Another possible strategy would be to be more lenient on approvals,
>> and more aggressive on backouts in case regressions are found. Each
>> approach has its pros and cons, so we may need to choose a mixture of
>> those depending on the corresponding project/component/module...
>>
>

[Mark Banner]

On 22/03/2011 14:13, Benjamin Smedberg wrote:
> * Reopen mozilla-central tomorrow morning Eastery (Wednesday), to avoid
> conflict with release activities today.

Will the version numbers (at least the Gecko number) be bumped by then?
We can work around it, but it'd be far easier for the comm-central app
plan if they were changed.

Thanks
Standard8

[Christian Legnitto]

It sounds like we should bump today. Do we have a bug # or is that generally just done by RelEng or release drivers?

Christian

[Axel Hecht]

I also plan to do some l10n dashboard tree shuffling today, which we
should be part of the party at large.

Axel

[Boris Zbarsky]

On 3/22/11 9:49 AM, Ehsan Akhgari wrote:
> Currently 278 bugs in that list.

The list is down to 181 bugs at this point (some were removed because
the patch wasn't actually reviewed or whatnot, some were landed on
cedar, some were landed on tm).

Quite honestly, 200 patches is a few days of normal development for us
at this point. I do think landing these on cedar and then merging is a
good approach, but think it's reasonable to plan on getting these into
Firefox 5. Some of the bugs on the list we may want to hold off on
because they have higher web compat risks, but we can decide on a
case-by-case basis; we've avoided pushing such patches to cedar so far.

-Boris