Post-FF4 branch organization

56 views
Skip to first unread message

Robert O'Callahan

unread,
Dec 13, 2010, 6:45:00 PM12/13/10
to mozilla.dev.planning group
We want to have more frequent Firefox releases in 2011 than we have had this
year. I think for that to work, by the time we put FF4 on a branch we should
have worked out a plan for managing the tree post-FF4. If we wait until FF4
is shipped, it'll be too late because arbitrary work will have already
landed on trunk.

This week we have a lot of people in Mountain View so it might be worth
trying to come up with a plan now. It's particularly important to figure out
what will be allowed to land on trunk once it opens.

Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]

Paul Biggar

unread,
Dec 14, 2010, 2:30:46 PM12/14/10
to rob...@ocallahan.org, mozilla.dev.planning group
Is there a discussion scheduled for this?

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
>

--
Paul Biggar
Compiler Geek
pbi...@mozilla.com

Robert O'Callahan

unread,
Dec 14, 2010, 2:36:39 PM12/14/10
to Paul Biggar, mozilla.dev.planning group
On Wed, Dec 15, 2010 at 8:30 AM, Paul Biggar <pbi...@mozilla.com> wrote:

> Is there a discussion scheduled for this?
>

Not yet.

There's a Google presentation at 2pm on Thursday about their experiences
with the Chrome release cycle.

Maybe after that? Say 4pm? Bring a plan.

Mike Beltzner

unread,
Dec 14, 2010, 2:36:38 PM12/14/10
to dev-pl...@lists.mozilla.org
On 12/13/2010 6:45 PM, Robert O'Callahan wrote:
> This week we have a lot of people in Mountain View so it might be worth
> trying to come up with a plan now. It's particularly important to figure out
> what will be allowed to land on trunk once it opens.

My initial suggestion is: let's not branch.

Instead, let's create project branches with owners, each of whom can set
the criteria as deemed appropriate for their various projects. I would
further suggest that each project be clearly defined with goals and
metrics used to judge outcomes, but let's not go too far down that
rabbit hole at present.

As for mozilla-central, it can be open to:

- NPOTB patches
- test fixes (and honestly, if in the early days all we do is WOO
work, that's a win)
- patches slated for a security/stability release
- patches with explicit driver approval

I suspect that in order to release quickly we'll always need to keep the
trunk in a "near-ready-to-ship" state. A lot of this discussion will be,
I fear, putting the cart before the horse, so the strategy I'm proposing
means to give us the most flexibility and agility for once plans are set.

(Also note that hg is like a time machine; it can be arduous to do
merges afterwards, but we can always go back in time to snip a branch if
needs be)

If there is to be a discussion on this during the work week, I recommend
it take place AFTER all the retriaging work is done and better estimates
of Firefox 4 schedule can be made. I would expect that to be afternoon
PDT on Thursday.

cheers,
mike

Paul Biggar

unread,
Dec 14, 2010, 2:45:51 PM12/14/10
to Mike Beltzner, dev-pl...@lists.mozilla.org
On Tue, Dec 14, 2010 at 11:36 AM, Mike Beltzner <belt...@mozilla.com> wrote:
> My initial suggestion is: let's not branch.
>
> Instead, let's create project branches with owners, each of whom can set the
> criteria as deemed appropriate for their various projects. I would further
> suggest that each project be clearly defined with goals and metrics used to
> judge outcomes, but let's not go too far down that rabbit hole at present.

We were discussing in the JS-pit how well this was working for us. We
have tracemonkey and jaegermonkey trees, and a manager who takes
responsibility for merging tracemonkey to mozilla-central (which means
it's significantly easier to get work done).

One thing that needs to get fixed first is the cost of creating
project branches. It took us a month to get the branches we have (and
the trees need to fit into things like tryserver), so we've been
reusing the branches to avoid this.


Paul

Phil Ringnalda

unread,
Dec 14, 2010, 3:03:49 PM12/14/10
to
On 12/14/2010 11:36 AM, Mike Beltzner wrote:
> My initial suggestion is: let's not branch.
>
> Instead, let's create project branches with owners

Be sure to include releng in the planning session, since we're already
short on slaves for what we're doing now, really short on slaves for
adding in opt and debug WinXP, and the rev3 minis are no longer
available, so if you're planning on adding multiple trees, you're
probably talking about replacing the entire build farm with new minis.

And don't forget about the contributors of random little patches, who
have been waiting for there to be a trunk again since last July, who are
going to find it even more baffling to have "Getting your patch in the
tree" include steps like "figure out whether this should land on some
project branch, in which case you need review plus approval from some
particular person, or whether there is no project for it, in which case
you need review plus approval from the wind."

Robert O'Callahan

unread,
Dec 14, 2010, 5:42:10 PM12/14/10
to dev-pl...@lists.mozilla.org
I think we also need to consider whether doing releases as frequently as
once a quarter requires we default to mandatory (silent) updates across
major versions. If so, are we going to do that and when is it going to
happen?

Robert O'Callahan

unread,
Dec 14, 2010, 5:55:45 PM12/14/10
to Phil Ringnalda, dev-pl...@lists.mozilla.org
On Wed, Dec 15, 2010 at 9:03 AM, Phil Ringnalda <philri...@gmail.com>wrote:

> Be sure to include releng in the planning session, since we're already
> short on slaves for what we're doing now, really short on slaves for adding
> in opt and debug WinXP, and the rev3 minis are no longer available, so if
> you're planning on adding multiple trees, you're probably talking about
> replacing the entire build farm with new minis.
>

Not all branches need continuous build and testing.

But releng should certainly be involved.

And don't forget about the contributors of random little patches, who have
> been waiting for there to be a trunk again since last July, who are going to
> find it even more baffling to have "Getting your patch in the tree" include
> steps like "figure out whether this should land on some project branch, in
> which case you need review plus approval from some particular person, or
> whether there is no project for it, in which case you need review plus
> approval from the wind."


I hope most small patches could just land on trunk, with driver approval.

Robert O'Callahan

unread,
Dec 14, 2010, 5:56:23 PM12/14/10
to Paul Biggar, mozilla.dev.planning group
On Wed, Dec 15, 2010 at 8:36 AM, Robert O'Callahan <rob...@ocallahan.org>wrote:

> Maybe after that? Say 4pm? Bring a plan.


I have booked 10-Forward for 4pm Thursday.

Mike Beltzner

unread,
Dec 14, 2010, 6:08:02 PM12/14/10
to dev-pl...@lists.mozilla.org
On 12/14/2010 5:42 PM, Robert O'Callahan wrote:
> I think we also need to consider whether doing releases as frequently as
> once a quarter requires we default to mandatory (silent) updates across
> major versions. If so, are we going to do that and when is it going to
> happen?

Yes, we need to consider it. Obviously to make that decision we'll need
to understand the costs and the required infrastructure changes to
support it. We'll also need to reevaluate some assumptions that I think
are commonly held and determine which we're willing to change in order
to support the core goal of shipping faster.

I'm not at all sure that this decision needs to be reached this week, or
even in order for us to talk and propose ideas for various branching
approaches, though.

(FWIW, I wouldn't equate mandatory with silent - there are ways of doing
automatic updates that are not silent, and I find that silent ends up
putting people on tilt a bit)

cheers,
mike

Steve Wendt

unread,
Dec 14, 2010, 7:01:27 PM12/14/10
to
On 12/14/2010 3:08 PM, Mike Beltzner wrote:

> (FWIW, I wouldn't equate mandatory with silent - there are ways of doing
> automatic updates that are not silent, and I find that silent ends up
> putting people on tilt a bit)

In a similar vein, automatic updates need to work in more situations;
for example, Windows 7 users cannot update without entering an
administrator password, which is impossible in a corporate setting.
Presumably, the same is true on Linux and OSX.

Armen Zambrano Gasparnian

unread,
Dec 14, 2010, 7:21:25 PM12/14/10
to
On 10-12-14 11:45 AM, Paul Biggar wrote:
> One thing that needs to get fixed first is the cost of creating
> project branches. It took us a month to get the branches we have (and
> the trees need to fit into things like tryserver), so we've been
> reusing the branches to avoid this.
>
From releng side we are thinking on how to lower the cost of doing this.

L. David Baron

unread,
Dec 14, 2010, 7:22:09 PM12/14/10
to Phil Ringnalda, dev-pl...@lists.mozilla.org
On Tuesday 2010-12-14 12:03 -0800, Phil Ringnalda wrote:
> On 12/14/2010 11:36 AM, Mike Beltzner wrote:
> >My initial suggestion is: let's not branch.
> >
> >Instead, let's create project branches with owners
>
> Be sure to include releng in the planning session, since we're
> already short on slaves for what we're doing now, really short on
> slaves for adding in opt and debug WinXP, and the rev3 minis are no
> longer available, so if you're planning on adding multiple trees,
> you're probably talking about replacing the entire build farm with
> new minis.

I don't think we're talking about an increase in changesets or
pushes. We're just talking about distributing the same number of
changesets and pushes across more branches. And the load on the
build machines largely comes from the number of pushes, not the
number of branches.

> And don't forget about the contributors of random little patches,
> who have been waiting for there to be a trunk again since last July,
> who are going to find it even more baffling to have "Getting your
> patch in the tree" include steps like "figure out whether this
> should land on some project branch, in which case you need review
> plus approval from some particular person, or whether there is no
> project for it, in which case you need review plus approval from the
> wind."

I think we do need a place to iterate on small patches that fix
problems discovered by Web authors and standards compliance
testsuites, and get them pushed to nightly users sooner rather than
later. Unless we've decided we're now comfortable taking such
patches in a security release without a beta cycle, I think we do
still need a trunk that's open to such things.

-David

--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/

Armen Zambrano Gasparnian

unread,
Dec 14, 2010, 7:26:22 PM12/14/10
to
On 10-12-14 2:55 PM, Robert O'Callahan wrote:
> On Wed, Dec 15, 2010 at 9:03 AM, Phil Ringnalda<philri...@gmail.com>wrote:
>
>> Be sure to include releng in the planning session, since we're already
>> short on slaves for what we're doing now, really short on slaves for adding
>> in opt and debug WinXP, and the rev3 minis are no longer available, so if
>> you're planning on adding multiple trees, you're probably talking about
>> replacing the entire build farm with new minis.
>>
> Not all branches need continuous build and testing.
Which branches are you referring to?
What level of testing do you think is valuable for those branches?
We could add full coverage for those branches on self-serve basis
(without having to poke a releng to trigger it for the branch manager).

>
> But releng should certainly be involved.
>
> Rob

Armen

Nicholas Nethercote

unread,
Dec 14, 2010, 7:28:49 PM12/14/10
to L. David Baron, dev-pl...@lists.mozilla.org, Phil Ringnalda
On Wed, Dec 15, 2010 at 11:22 AM, L. David Baron <dba...@dbaron.org> wrote:
>>
>> Be sure to include releng in the planning session, since we're
>> already short on slaves for what we're doing now, really short on
>> slaves for adding in opt and debug WinXP, and the rev3 minis are no
>> longer available, so if you're planning on adding multiple trees,
>> you're probably talking about replacing the entire build farm with
>> new minis.
>
> I don't think we're talking about an increase in changesets or
> pushes.  We're just talking about distributing the same number of
> changesets and pushes across more branches.  And the load on the
> build machines largely comes from the number of pushes, not the
> number of branches.

More branches may even reduce the load on the try machines.

As a JS hacker I get to land my patches on the tracemonkey branch, and
because breakage on that branch doesn't affect too many people, I'm
willing to land most patches without doing a try server run. If I had
to land patches on mozilla-central, I'd be much more conservative and
use the try server more. This would increase try machine load and
slow down my development speed.

Nick

Robert O'Callahan

unread,
Dec 14, 2010, 8:10:10 PM12/14/10
to Armen Zambrano Gasparnian, dev-pl...@lists.mozilla.org
On Wed, Dec 15, 2010 at 1:26 PM, Armen Zambrano Gasparnian <
arm...@mozilla.com> wrote:

> Which branches are you referring to?
>

Suppose I want to do a medium-sized graphics project, say implement a new
graphics API to speed up canvas drawing. It'll be quite a bit of work and
need contributions from multiple developers. It should be on a branch, but
it doesn't need to be tested until we think it's done.


What level of testing do you think is valuable for those branches?
>

Occasional tryserver testing.

We could add full coverage for those branches on self-serve basis (without
> having to poke a releng to trigger it for the branch manager)


That'd be pretty cool, but is there any downside to just pushing the branch
to tryserver?

Joshua Cranmer

unread,
Dec 14, 2010, 9:08:16 PM12/14/10
to
On 12/14/2010 08:10 PM, Robert O'Callahan wrote:
> Suppose I want to do a medium-sized graphics project, say implement a new
> graphics API to speed up canvas drawing. It'll be quite a bit of work and
> need contributions from multiple developers. It should be on a branch, but
> it doesn't need to be tested until we think it's done.

I may be wrong, but wouldn't one of the disposable branches be good for
this?
<https://wiki.mozilla.org/ReleaseEngineering/DisposableProjectBranches>

Armen Zambrano Gasparnian

unread,
Dec 14, 2010, 9:10:21 PM12/14/10
to
On 10-12-14 5:10 PM, Robert O'Callahan wrote:
> On Wed, Dec 15, 2010 at 1:26 PM, Armen Zambrano Gasparnian<
> arm...@mozilla.com> wrote:
>
>> Which branches are you referring to?
>>
>
> Suppose I want to do a medium-sized graphics project, say implement a new
> graphics API to speed up canvas drawing. It'll be quite a bit of work and
> need contributions from multiple developers. It should be on a branch, but
> it doesn't need to be tested until we think it's done.
>
OK. I thought you were referring to current existing branches.
Coverage can currently be tailored.

>
> We could add full coverage for those branches on self-serve basis (without
>> having to poke a releng to trigger it for the branch manager)
>
> That'd be pretty cool, but is there any downside to just pushing the branch
> to tryserver?
>
> Rob

No downside on regards of full coverage.
Maybe prioritization of project branch jobs over tryserver jobs.
In the mid-future, branch managers could bump priority of jobs related
to a changeset if need be to override the current prioritization rules.

Armen

Armen Zambrano Gasparnian

unread,
Dec 14, 2010, 9:11:31 PM12/14/10
to
Yes, you are right.

Armen

Henri Sivonen

unread,
Dec 15, 2010, 12:35:15 AM12/15/10
to mozilla.dev.planning group

If most development happens on branches instead of the trunk, surely more than three concurrent project branches will be needed. Will this mechanism scale up? (Having a fixed number of disposable branches suggests that the mechanism doesn't scale arbitrarily at present.)

--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/


Ehsan Akhgari

unread,
Dec 15, 2010, 1:53:07 AM12/15/10
to Joshua Cranmer, dev-pl...@lists.mozilla.org
On 10-12-14 6:08 PM, Joshua Cranmer wrote:

Or even a user repo? If a project branch doesn't require continuous
integration testing, I don't think there's any reason for us to use
anything more complicated than a user repo, which takes about 5 minutes
to set up.

Ehsan

Robert O'Callahan

unread,
Dec 15, 2010, 2:06:18 AM12/15/10
to Ehsan Akhgari, Joshua Cranmer, dev-pl...@lists.mozilla.org
On Wed, Dec 15, 2010 at 7:53 PM, Ehsan Akhgari <ehsan....@gmail.com>wrote:

> Or even a user repo? If a project branch doesn't require continuous
> integration testing, I don't think there's any reason for us to use
> anything more complicated than a user repo, which takes about 5 minutes
> to set up.
>

Yes. I think a user repo plus pushes to try when desired should work quite
well.

Ben Hearsum

unread,
Dec 15, 2010, 4:03:12 AM12/15/10
to Henri Sivonen, mozilla.dev.planning group

Surely. Branches are relatively cheap these days, and if the names of
them are not of concern, can be en-masse ahead of time. There's probably
lots we can do in the future to make branch development even easier, and
I have no doubt that once we figure out firmer requirements that we can
make some of those things happen!

Ben Hearsum

unread,
Dec 15, 2010, 4:03:12 AM12/15/10
to Henri Sivonen, mozilla.dev.planning group
On 12/14/10 09:35 PM, Henri Sivonen wrote:

Surely. Branches are relatively cheap these days, and if the names of

Kyle Huey

unread,
Dec 15, 2010, 6:06:13 AM12/15/10
to Phil Ringnalda, dev-pl...@lists.mozilla.org
On Tue, Dec 14, 2010 at 3:03 PM, Phil Ringnalda <philri...@gmail.com>wrote:

> And don't forget about the contributors of random little patches, who have
> been waiting for there to be a trunk again since last July, who are going to
> find it even more baffling to have "Getting your patch in the tree" include
> steps like "figure out whether this should land on some project branch, in
> which case you need review plus approval from some particular person, or
> whether there is no project for it, in which case you need review plus
> approval from the wind."
>

IMHO this is really important to keep in mind. I have patches that were
a2.0-'d in August and while I can live with that I can't imagine that having
to wait several months for your patches to land is encouraging to new
contributors.

- Kyle

Mark Banner

unread,
Dec 15, 2010, 9:36:04 AM12/15/10
to
On 14/12/2010 19:36, Mike Beltzner wrote:
> My initial suggestion is: let's not branch.

Certainly an interesting idea. Presumably you would still create a
separate repo for security releases for 4.0 at some stage?

> As for mozilla-central, it can be open to:
>
> - NPOTB patches
> - test fixes (and honestly, if in the early days all we do is WOO work,
> that's a win)
> - patches slated for a security/stability release
> - patches with explicit driver approval
>
> I suspect that in order to release quickly we'll always need to keep the
> trunk in a "near-ready-to-ship" state.

I agree with keeping trunk in a "near-ready-to-ship" state, that is good.

What I don't get about your proposal is what does the explicit driver
approval get us?

If we take an example of a patch that I submit in future. I've tested it
myself, made sure its got the appropriate unit tests, pushed to try
server, and it has been reviewed.

What is there that would cause a driver to deny the patch? I've done
everything required to get the review. The only thing could be an
assessment of risk, but if the reviewer thinks it is too risky,
shouldn't the reviewer be denying it or asking for more work?

If an approver is making the assessment of a patch being too risky, and
they deny it, what mechanism do we then use for patches that are
slightly risky but no-other way of doing them? The only way we have to
get those patches out is to land them for the nightly builds. Although
project branches could have nightlies, you're surely not going to get
the same kind of coverage.

And then aren't we actually turning the approval request back into what
super-review used to be - more of a it is ok to land this patch?

This also will increase the load on approvers. We've already seen that
at times some approvals have been delayed for multiple months, I would
argue, that if we're having this additional step, then it needs to be
1-2 days maximum.


That discussion is a bit longer than I was intending. I'm certainly open
to the possibility of having approvals-only, but at the moment I fail to
see the benefits over what the existing review mechanisms provide.

Standard8

Chris Hofmann

unread,
Dec 15, 2010, 10:30:56 AM12/15/10
to Mike Beltzner, dev-pl...@lists.mozilla.org
On 12/14/10 11:36 AM, Mike Beltzner wrote:
>
> I suspect that in order to release quickly we'll always need to keep
> the trunk in a "near-ready-to-ship" state.

Yeah, but in addition we need to step up the pace of betas.

More betas help to stay in that ready-to-ship state, and help us to
understand the impact on all our users.

More betas have the effect of helping a large number people inside and
outside the project understand the work that is going on at Mozilla and
in increasing our impact and influence.

More betas help us reach out to and engage with browser enthusiasts
while not constantly disrupting the other 390 Million users with updates
that might not be fully baked yet.

Let continue to build the beta program and go from 1 million beta
testers to 10 million beta testers in the next year.

Lets continue to build the feedback channels that empower and engage
with this beta testing audience.

Lets refine our beta process so we can get it moving at a faster pace
and a 3 or 4 week cycle. We can by doing things like front loading big
changes from project branches *only* during the first week of the beta
cycle and and staying away from the position of waiting for large
landings and recovery efforts to happen at the end of beta development
cycles. That's the key area where we need better discipline that will
help us to ship both betas and final releases faster.

-chofmann

Mike Beltzner

unread,
Dec 15, 2010, 10:55:08 AM12/15/10
to Chris Hofmann, dev-pl...@lists.mozilla.org
On 2010-12-15, at 10:30 AM, Chris Hofmann wrote:

> On 12/14/10 11:36 AM, Mike Beltzner wrote:
>>
>> I suspect that in order to release quickly we'll always need to keep the trunk in a "near-ready-to-ship" state.
>
> Yeah, but in addition we need to step up the pace of betas.

Chris,

I totally agree that we need to determine how we can accomplish all the things you assert as being provided by our current use of betas. Whether that's by "shipping betas" or promoting a "dev channel" a la Chrome or some other fashion, I think, we should leave open for various proposals. It also has less to do with the specifics of branching, I think, except for in how it requires that we keep some tree of our to-be-shipped product line in a near-shippable state.

> Lets refine our beta process so we can get it moving at a faster pace and a 3 or 4 week cycle. We can by doing things like front loading big changes from project branches *only* during the first week of the beta cycle and and staying away from the position of waiting for large landings and recovery efforts to happen at the end of beta development cycles. That's the key area where we need better discipline that will help us to ship both betas and final releases faster.

Again, I see this as a process decision that's somewhat related to branching, but more to do with how we plan the development of features. I agree that features should be planned more fully in advance (in terms of what their goals are, how we'll know we've hit those goals, what the project milestones are and metrics to measure against them, how we'll know we haven't had unplanned deleterious effects, how we'll release and share those changes amongst our various feedback audiences and how we'll gather the feedback, etc) and then if they aren't ready by some deadline, they don't make the next development milestone and skip over to the following one. The implication for branching strategy is keeping the main development branch in ready-to-ship state, though, mostly.

cheers,
mike

Benjamin Smedberg

unread,
Dec 15, 2010, 11:54:05 AM12/15/10
to
On 12/14/10 10:53 PM, Ehsan Akhgari wrote:

> Or even a user repo? If a project branch doesn't require continuous
> integration testing, I don't think there's any reason for us to use
> anything more complicated than a user repo, which takes about 5 minutes
> to set up.

The primary advantages of a project repo over a try repo are:

* organized nightly build with updates
* Talos integration which provides historical charts

--BDS

Axel Hecht

unread,
Dec 15, 2010, 12:49:59 PM12/15/10
to
On 14.12.10 16:22, L. David Baron wrote:
> On Tuesday 2010-12-14 12:03 -0800, Phil Ringnalda wrote:
>> On 12/14/2010 11:36 AM, Mike Beltzner wrote:
>>> My initial suggestion is: let's not branch.
>>>
>>> Instead, let's create project branches with owners
>>
>> Be sure to include releng in the planning session, since we're
>> already short on slaves for what we're doing now, really short on
>> slaves for adding in opt and debug WinXP, and the rev3 minis are no
>> longer available, so if you're planning on adding multiple trees,
>> you're probably talking about replacing the entire build farm with
>> new minis.
>
> I don't think we're talking about an increase in changesets or
> pushes. We're just talking about distributing the same number of
> changesets and pushes across more branches. And the load on the
> build machines largely comes from the number of pushes, not the
> number of branches.

There would be an increase of nightly builds if we want those, which is
mentioned elsewhere in this thread.

We might be able to cut down on those to some degree by doing those only
if the project branch actually changed. That somewhat redefines
"nightly", but I think that "sparse nightlies" are much closer to what
we want anyway. Could be an interesting option for our stable release
branches, too.

Axel

Robert O'Callahan

unread,
Dec 15, 2010, 2:00:33 PM12/15/10
to Mike Beltzner, dev-pl...@lists.mozilla.org, Chris Hofmann
On Thu, Dec 16, 2010 at 4:55 AM, Mike Beltzner <belt...@mozilla.com> wrote:

> I totally agree that we need to determine how we can accomplish all the
> things you assert as being provided by our current use of betas. Whether
> that's by "shipping betas" or promoting a "dev channel" a la Chrome or some
> other fashion, I think, we should leave open for various proposals. It also
> has less to do with the specifics of branching, I think, except for in how
> it requires that we keep some tree of our to-be-shipped product line in a
> near-shippable state.
>

I think we'll need a way to expose multiple large projects to a large number
of users before we know whether those projects can ship in the next release.
It's the only way to find Web-wide regressions in those projects, and the
volume of those regressions will determine when those projects are "done".

We won't want to split our "beta users" among multiple large projects, so
that suggests we need an integration point that can receive multiple large
landings, that we can ship "betas" or "dev channel" from, but which is
separate from release branches.

So I think these issues are pretty deeply intertwined.

Robert O'Callahan

unread,
Dec 15, 2010, 2:07:18 PM12/15/10
to Mike Beltzner, dev-pl...@lists.mozilla.org, Chris Hofmann

Paul Biggar

unread,
Dec 15, 2010, 2:39:31 PM12/15/10
to rob...@ocallahan.org, dev-pl...@lists.mozilla.org, Chris Hofmann
On Wed, Dec 15, 2010 at 11:00 AM, Robert O'Callahan
<rob...@ocallahan.org> wrote:
> I think we'll need a way to expose multiple large projects to a large number
>
> We won't want to split our "beta users" among multiple large projects


This seems compatible with the previous goals. To be concrete (based
on the conversation so far):

- clone 'FF5' from m-c at some point before the FF4 release
- FF5 is the source of betas, release candidates, etc
- FF5 is the integration point for other project repos
- small changes do not go straight to FF5, but rather have to come in
via another source (we might keep mozilla-central around for this
purpose, if we made committing to it cheaper)
- each branch/tree has an owner, like sayrer owns the tracemonkey
tree. They are responsible for integration, they are responsible for
backing out, they negotiate integration timelines with the FF5 branch
owner.


The major advantages:

- the FF5 branch is never broken (each repo will be fully tested
before integration)
- the FF5 branch is permanently ready to release something


And some minor ones:

- cost of committing is significantly lowered
- there are never patches that can't land _somewhere_
- if something breaks, it only breaks for a small number of people


Downsides:
- the more "upstream" parts of the codebase have less testing from
their "clients" before integration.
- developers are more likely to be dogfooding their own repos, rather
than the FF5 repo.


--
Paul Biggar
Compiler Geek
pbi...@mozilla.com

Robert O'Callahan

unread,
Dec 15, 2010, 6:08:17 PM12/15/10
to Paul Biggar, dev-pl...@lists.mozilla.org, Chris Hofmann
On Thu, Dec 16, 2010 at 8:39 AM, Paul Biggar <pbi...@mozilla.com> wrote:

> - clone 'FF5' from m-c at some point before the FF4 release
> - FF5 is the source of betas, release candidates, etc
> - FF5 is the integration point for other project repos
> - small changes do not go straight to FF5, but rather have to come in
> via another source (we might keep mozilla-central around for this
> purpose, if we made committing to it cheaper)
> - each branch/tree has an owner, like sayrer owns the tracemonkey
> tree. They are responsible for integration, they are responsible for
> backing out, they negotiate integration timelines with the FF5 branch
> owner.
>

As far as I can tell, this doesn't give us a place to land big projects for
testing by hundreds of thousands of users *before* we commit to shipping
them in a release.

Mike Shaver

unread,
Dec 15, 2010, 6:13:10 PM12/15/10
to rob...@ocallahan.org, Paul Biggar, dev-pl...@lists.mozilla.org, Chris Hofmann
On Wed, Dec 15, 2010 at 3:08 PM, Robert O'Callahan <rob...@ocallahan.org> wrote:
> As far as I can tell, this doesn't give us a place to land big projects for
> testing by hundreds of thousands of users *before* we commit to shipping
> them in a release.

Gotta be able to pref big stuff off, or otherwise revert. That will
let us reach even more than our beta audience with opt-in testing, as
well.

(I actually wanted to do this with OOPP on Windows in 3.6, where we'd
ship it available-but-off in one update, then get the word out through
lifehacker and SUMO and suchlike, then use that enhanced data to
refine better before shipping. I didn't really push on it, though.)

Mike

Robert O'Callahan

unread,
Dec 15, 2010, 6:22:58 PM12/15/10
to Mike Shaver, Paul Biggar, dev-pl...@lists.mozilla.org, Chris Hofmann
On Thu, Dec 16, 2010 at 12:13 PM, Mike Shaver <mike....@gmail.com> wrote:

> Gotta be able to pref big stuff off, or otherwise revert. That will
> let us reach even more than our beta audience with opt-in testing, as
> well.
>

Unfortunately, for some architectural changes that can be arbitrarily
difficult.

This will be a good question for discussion :-).

Steven Roussey

unread,
Dec 15, 2010, 9:21:30 PM12/15/10
to Paul Biggar, dev-pl...@lists.mozilla.org, Chris Hofmann
> As far as I can tell, this doesn't give us a place to land big projects for
> testing by hundreds of thousands of users *before* we commit to shipping
> them in a release.

You can ship multiple variants simultaneously (and switch between them):

http://screencast.com/t/zLamCm4S

L. David Baron

unread,
Dec 16, 2010, 2:50:44 AM12/16/10
to dev-pl...@lists.mozilla.org
On Thursday 2010-12-16 12:08 +1300, Robert O'Callahan wrote:
> As far as I can tell, this doesn't give us a place to land big projects for
> testing by hundreds of thousands of users *before* we commit to shipping
> them in a release.

One thought I had about this, although it's may be more applicable
to testing stuff for a nightly audience rather than a beta audience:

If someone thinks they're ready to merge a project branch into
mozilla-central, we could flip one day's worth of nightlies over to
that project branch and see what feedback that nightly testing
yields. (Presumably mozilla-central would get merged into that
project branch right before so they wouldn't go backwards in terms
of mozilla-central commits either.)

Or, more generally, we don't necessarily need to treat our nightly
or beta update channels as things that require each revision to be a
version control graph descendant of the previous revision. We could
move them to the branch that needs the most testing, as long as it's
believed to meet our standards for what we'd send to nightly or beta
users.

Axel Hecht

unread,
Dec 16, 2010, 9:34:28 AM12/16/10
to

We haven't started talking about how to handle l10n for the project
branches yet. I following this thread closely and start to narrow down
on that as soon as the main code discussion gets a direction.

On this point though, we should probably make our standards for
nightlies include expectations across all locales. That might be "none,
really" for some locales, down to "we may not need builds until we reach
beta for that train", but pretty high standards on a handful or so.

Axel

Mike Beltzner

unread,
Dec 16, 2010, 11:30:12 AM12/16/10
to Axel Hecht, dev-pl...@lists.mozilla.org
On 2010-12-16, at 9:34 AM, Axel Hecht wrote:

> On this point though, we should probably make our standards for nightlies include expectations across all locales. That might be "none, really" for some locales, down to "we may not need builds until we reach beta for that train", but pretty high standards on a handful or so.

I really like the idea of l10n-merge for locales, so that those who like to track through development can benefit from doing so, and those who don't can get as much translated as has been done to date, and default (en-US) strings otherwise. Seems like that provides the most thorough coverage for nightly testing, and keeps the "trunk" in closest-to-shipping state.

cheers,
mike

Phil Ringnalda

unread,
Dec 16, 2010, 11:47:45 AM12/16/10
to
On 12/14/10 11:36 AM, Mike Beltzner wrote:
> Instead, let's create project branches with owners

Two last thoughts about what those owners are taking on, that they will
not actually be able to manage well enough:

We will be fragmenting the management of randomorange. There's no way
I'll be watching a dozen trees at once, and as we've just seen with the
Places branch merge, even someone who does as good a job of tree
watching as sdwilsh did will still have some "oh, yeah, I do remember
seeing that intermittently once or twice but I didn't ever file it"
things, and probably some more "I don't remember seeing that at all, but
maybe it did happen" and maybe some "I don't think I caused that, but
maybe if you looked at everything we starred, it will turn out that we
misstarred it as something old when really it was something new we caused."

Starring is also difficult to get right when you aren't always up to
date - I try not to stick "hey, look what happened with the code from
last week!" comments from TraceMonkey failures with a week-old
mozilla-central parent that is missing three orange fixes and two
expanded logging patches into bugs, but it's really tough to remember
what has and hasn't changed even when you are cc'ed on most orange bugs.

The other thing we will be fragmenting is checkin-needed: TraceMonkey
has struggled all along to not ignore its checkin-needed (and has mostly
failed), even though it's fairly rare for a random contributor without
push access to contribute patches there, and nobody who does
checkin-needed has been willing to maintain a TraceMonkey tree to push from.

As a project branch owner, you really only have a couple of responsible
choices:

- decide that all pushes to your project's code have to land on your
project branch, and take personal responsibility for handling
checkin-needed in a timely manner (and you'll still leave some
contributions on the floor, because people willing to write a patch for
you won't always be willing to clone your tree and write a patch against
the stuff you haven't merged into the tree they have yet), or

- decide to deal with the merge-hell of always having changes to your
code that you have to merge back from mozilla-central, making the merges
always a slow and time-consuming and error-prone thing, so you'll put
them off and make your orange situation even worse.

Rob Campbell

unread,
Dec 16, 2010, 12:08:44 PM12/16/10
to dev-pl...@lists.mozilla.org

On 2010-12-16, at 08:47, Phil Ringnalda wrote:

> On 12/14/10 11:36 AM, Mike Beltzner wrote:
>> Instead, let's create project branches with owners
>
> Two last thoughts about what those owners are taking on, that they will not actually be able to manage well enough:
>
> We will be fragmenting the management of randomorange. There's no way I'll be watching a dozen trees at once, and as we've just seen with the Places branch merge, even someone who does as good a job of tree watching as sdwilsh did will still have some "oh, yeah, I do remember seeing that intermittently once or twice but I didn't ever file it" things, and probably some more "I don't remember seeing that at all, but maybe it did happen" and maybe some "I don't think I caused that, but maybe if you looked at everything we starred, it will turn out that we misstarred it as something old when really it was something new we caused."
>
> Starring is also difficult to get right when you aren't always up to date - I try not to stick "hey, look what happened with the code from last week!" comments from TraceMonkey failures with a week-old mozilla-central parent that is missing three orange fixes and two expanded logging patches into bugs, but it's really tough to remember what has and hasn't changed even when you are cc'ed on most orange bugs.

This is an interesting problem, but not one that I really think we should dictate our release and branching structure around.

I'm not sure having humans deal with oranges is really the best way to manage our branches anymore, tbh. The failures we're seeing are statistical functions and we should be able to auto-star known oranges automatically.

When we see a new orange or a leak or an obvious talos regression, we could get an alert (in #developers? via pulse? email, cable and/or text message) and dispatch the swat teams. Otherwise, TBPL could be augmented to do the starring for us.

The amount of time we spend playing Tinderbox Pong with random oranges is wasteful. I know it usually kills a chunk of my day if I check something in and have to keep monitoring the tree to check off random failures. I would like to have that part of my day back.

> The other thing we will be fragmenting is checkin-needed: TraceMonkey has struggled all along to not ignore its checkin-needed (and has mostly failed), even though it's fairly rare for a random contributor without push access to contribute patches there, and nobody who does checkin-needed has been willing to maintain a TraceMonkey tree to push from.

That's a different problem. We need people to do this. Like, it should be a rotating task for someone to grab and checkin ready bugs. We used to have Reed do that for us and it was pretty valuable.

Of course, this gets more awkward with project branches, which I guess is your real concern.

> As a project branch owner, you really only have a couple of responsible choices:
>
> - decide that all pushes to your project's code have to land on your project branch, and take personal responsibility for handling checkin-needed in a timely manner (and you'll still leave some contributions on the floor, because people willing to write a patch for you won't always be willing to clone your tree and write a patch against the stuff you haven't merged into the tree they have yet), or
>
> - decide to deal with the merge-hell of always having changes to your code that you have to merge back from mozilla-central, making the merges always a slow and time-consuming and error-prone thing, so you'll put them off and make your orange situation even worse.

This is potentially scary.

The tracemonkey team is fortunate enough to have total control over their codebase (i.e., the js code). In higher-up, shared sections, we're going to have to deal with merges more frequently. That'll probably mean lots of potentially risky manual rebasing at merge time. Oranges on mozilla-central will often (should always?) be a result of mismerges.

Anyway, fun stuff. Can we have our devtools repo now? ;)

~ rob

Christian Legnitto

unread,
Dec 16, 2010, 1:11:09 PM12/16/10
to Rob Campbell, dev-pl...@lists.mozilla.org
I'm going to stick some thoughts here after experience with other complex branch organizations and some initial discussions with Mozilla people. Also, as a release manager I likely have a different perspective than others. I intend to finish up some pretty pictures and walk through some scenarios, hopefully by the meeting this afternoon. Feel free to pick this apart.

I'm coming at this with these in mind:

#1 - All changes need to align with overall product goals
#2 - The release schedule is set in stone. We would cut features / fixes rather than slip each quarter
#3 - Doing all development together makes a release dependent on the hardest / worst managed / most invasive project
#4 - Stop acting like we're developing in CVS

Also, I am going to stop calling them project "branches", because they aren't branches, they are repos.

So, this is what I am thinking...

* Everything is done in project repos
** There will be small exceptions, but should be < 5% and should need explicit approval
* mozilla-central becomes an integration repo, not a development repo like today, and is pretty much always in a consistent release state
* Project repos are pretty much self serve. You go to a web page, fill out some details, get a new project repo with RelEng-backed goodness
** The idea is these are extremely lightweight and developers shouldn't hesitate to create one, even for a handful of changes
* You need to fill out a *repo owner*. This person will be on the hook for anything to do with the repo
* You need to fill out a *product management driver*, who advises on the feature/changes, helps set schedule, makes sure it stays on message, writes announcements, etc
* You need to fill out a *QA lead*. They will be responsible for a test plan, ongoing testing, and any regression fallout
* You need to set estimates of best/likely/worst case for the project work to be completed
* Working with release and product management, you need to specify what FF release the project is targeted to land for
** This isn't a contract in pretty much all cases though
* You specify which other project repos this new repo relies on
* You can choose to opt in to hourly/daily/weekly/monthly auto-merges from your targeted release and project repo dependencies. This is one way (into the project repo). If the merges fail, emails are sent
* You and your QA lead need to come with specific metrics of "success" that need to be fulfilled before being nominated for merging to m-c
** The places branch has been a good model of this


What does development look like?

* Decide on a feature
* Get a repo
* Manually or automatically pull from another integration or project repo to start
* Develop in your project repo, either manually refreshing from the dependencies or via automation
* Builds will be created nightly if new checkins/merges have happened. The channel will be nightly-[reponame]
* Developers (and maybe their QA lead) should be the only ones living on their project build nightlies
* Development happens as normal in the project repo
** This means that the project won't pick up implicit dependencies due to landing on top of other big projects / fixes
** try would also *go away*, as it is not needed anymore
* When the feature is in a good state, auto-merging into the project repo from dependencies will be turned on for a couple of days
* If there are product-level and tough UI decisions needed for the project, we can do A/B testing via updates off the project repo with our nightly/beta audience to collect more data
* After a waiting period (mainly for oranges and any missed integration work), project management will move a subset of nightly testers over on nightly-[reponame]. If we want to call attention to it, we can pop up an info bar after updates that tell people to go to an about page to learn more about the new feature, etc
** This means most integration and regression fallout will happen *off m-c*
** Crashes/ADU may not help too much here. It would be good to switch to a crashes/pageload model to get finer-grained data
* If the QA plan has been executed and nothing major has come up, the project repo is nominated to merge to m-c (the integration repo)
** One of the nomination criteria is to point out any known (non-blocking) issues, though there should be basically none of them
* If the project is a new UI feature, it lands preffed off and at least one release needs to go out with it like that. We will do blog posts and tell people how to pref it on to try it out (like about:labs)
* When the project merges to m-c, we move the users on any build of that project repo back into the m-c pool

Note that major / one-off security fixes fixes can be handled in a designated project repo as well.

How does this enable us to release quarterly?

* If features aren't ready within the quarter they we targeted to ship there is no backout that needs to happen
* Better visibility into interdependencies and better risk management of those
* Changes "flow" through dependencies and the integration work / fallout is spread out over time
* If needed, we can turn off auto-merging from m-c, roll back multiple features off m-c, release, roll forward, and turn back auto-merging without any development disruption. This is a HUGE win
* Surfaces underlying assumptions and sets clear success metrics via a QA plan up front
* Targeted testing of fixes via moving nightly/beta audience improves the audience's value. Crash spikes / new signatures will be extremely easy to see, feedback can be routed a lot easier
* We could get to biweekly releases and this structure would still hold

How much work is it to do this?

Not too much. The building blocks are there. As an intermediate step, instead of on-demand repos we can use a pool of them like we have now. This is mainly a process / integration change with some beefing up of existing tools (like AUS).

I know I didn't describe this as well as I would like, feel free to come over and chat with me in person (I'll likely be in Warp Core in an hour or so). This isn't radically different than what was proposed earlier in this thread but I am sure there are some controversial bits strewn about).

Thanks,
Christian

PS: Not sure about mailing list etiquette. This isn't in response to specifics in the current thread but is close enough I didn't want to start another thread. I replied but deleted all previous response text. Feel free to email me directly and tell me if you would have preferred something else.

Daniel Holbert

unread,
Dec 16, 2010, 1:59:56 PM12/16/10
to dev-pl...@lists.mozilla.org, Christian Legnitto
On 12/16/2010 10:11 AM, Christian Legnitto wrote:
> ** try would also *go away*, as it is not needed anymore

I think try would still be quite helpful for a number of things,
actually -- here are a few items off the top of my head, none of which
would really be appropriate for pushing to project branches:
- generating one-off builds with "does this fix it?" patches for
community members to test, for bugs that we can't reproduce locally.
(This has been helpful in the past for bugs with obscure printer
drivers, among other things.)
- getting a full set of test runs before landing in a project branch
- trying out a possible workaround/fix for randomorange bugs

~Daniel

Boris Zbarsky

unread,
Dec 16, 2010, 2:06:37 PM12/16/10
to
On 12/16/10 10:11 AM, Christian Legnitto wrote:
> ** The idea is these are extremely lightweight and developers shouldn't hesitate to create one, even for a handful of changes
....

> * You need to fill out a *repo owner*. This person will be on the hook for anything to do with the repo
> * You need to fill out a *product management driver*, who advises on the feature/changes, helps set schedule, makes sure it stays on message, writes announcements, etc
> * You need to fill out a *QA lead*. They will be responsible for a test plan, ongoing testing, and any regression fallout
> * You need to set estimates of best/likely/worst case for the project work to be completed
> * Working with release and product management, you need to specify what FF release the project is targeted to land for
> ** This isn't a contract in pretty much all cases though
> * You specify which other project repos this new repo relies on
> * You can choose to opt in to hourly/daily/weekly/monthly auto-merges from your targeted release and project repo dependencies. This is one way (into the project repo). If the merges fail, emails are sent
> * You and your QA lead need to come with specific metrics of "success" that need to be fulfilled before being nominated for merging to m-c

This seems like it would work well for new-feature kind of projects. It
doesn't seem at all lightweight for your basic bugfixes (for which you'd
first have to figure out what "project" they belong to!).

> What does development look like?
>
> * Decide on a feature

What if you aren't working on a feature?

-Boris

Shawn Wilsher

unread,
Dec 16, 2010, 2:08:00 PM12/16/10
to dev-pl...@lists.mozilla.org
On 12/16/2010 10:59 AM, Daniel Holbert wrote:
> - getting a full set of test runs before landing in a project branch
Is that really needed though? If you break your branch, you can back
out, but you only hurt the people working with you (likely small).

Cheers,

Shawn

L. David Baron

unread,
Dec 16, 2010, 2:17:17 PM12/16/10
to Christian Legnitto, dev-pl...@lists.mozilla.org
On Thursday 2010-12-16 10:11 -0800, Christian Legnitto wrote:
> I'm coming at this with these in mind:
>
> #1 - All changes need to align with overall product goals

I'm curious what you mean by overall product goals. We should
generally strive to maintain or improve our speed, memory use,
stability, standards compliance, security, usability for end users
and for developers, and visual polish, among other things. While
some improvements in these areas are large projects, many of the
most effective (in terms of cost vs. benefit) are small. And I
worry that a statement like #1 might make it more difficult to work
on small and effective things (as do a significant number of the
policies you list below that follow from it).

> #2 - The release schedule is set in stone. We would cut features / fixes rather than slip each quarter
> #3 - Doing all development together makes a release dependent on the hardest / worst managed / most invasive project

I definitely agree on these.

> #4 - Stop acting like we're developing in CVS

I agree here as well, but we also need to think about what we need
from user testing and how to focus that user testing appropriately.

> * Development happens as normal in the project repo
> ** This means that the project won't pick up implicit dependencies due to landing on top of other big projects / fixes

There's a big danger here on the opposite side, which is that
projects end up being very hard to merge with each other.

Christian Legnitto

unread,
Dec 16, 2010, 2:16:03 PM12/16/10
to Daniel Holbert, dev-pl...@lists.mozilla.org
On Dec 16, 2010, at 10:59 AM, Daniel Holbert wrote:

> On 12/16/2010 10:11 AM, Christian Legnitto wrote:
>> ** try would also *go away*, as it is not needed anymore
>
> I think try would still be quite helpful for a number of things,
> actually -- here are a few items off the top of my head, none of which
> would really be appropriate for pushing to project branches:

I struggled with this bit too. Initially I thought every project branch should have a shadowing try repo, but the more I think about it the more I think try in general try should go away.

> - generating one-off builds with "does this fix it?" patches for
> community members to test, for bugs that we can't reproduce locally.
> (This has been helpful in the past for bugs with obscure printer
> drivers, among other things.)

Private repos? Local builds? Spinning up a "printer-experiments" repo?

> - getting a full set of test runs before landing in a project branch

Not sure this is needed? The point of doing so currently is to not block other m-c development and not subject nightly users to possible bustage. In this model those are not a concern and the test run should happen in the repo where it is planning to end up.

> - trying out a possible workaround/fix for randomorange bugs

Fixing random oranges should likely be in their own "randomorange" repo. While good, they aren't tied to a specific project. Additionally, there is inherent risk and cost to the fixes, even though we don't treat them like that now. Those fixes and landings should be managed, not ad-hoc. This is a proposal for managing change with a consistent framework.

Christian

Christian Legnitto

unread,
Dec 16, 2010, 2:32:35 PM12/16/10
to Boris Zbarsky, dev-pl...@lists.mozilla.org

I think that's the point. If they don't fit in with a project, they don't make it in the tree. A project does not only mean feature though, just a structured way of thinking about and managing some class of fixes.

The underlying issue is we can't take random fixes and expect to ship every 3 months. We can't even do that with betas (oh yeah, I went there ha). This is likely where my release management perspective is wildly different than the development perspective.

>
>> What does development look like?
>>
>> * Decide on a feature
>
> What if you aren't working on a feature?

Have a couple of patches to make AIX work? Create a "AIX-suport" or more general "platform-support" repo, put them there, nominate it with justification. We could then decide "increased platform support" is a "feature" of a coming release and manage expectations and the landing. Have some little perf wins? Let's create a "perf-parpercuts" repo, land them there, and nominate for integration once we've aggregated a bunch. Have bug fixes that are only needed to support a feature? Fix those in the feature repo. Do multiple features depend on some fixes? Break those fixes out into their own repo, it's now a sub-feature. Have the dependent feature repos depend/merge from it while it merges from m-c/integration branches and its dependents.

Again, there will be exceptions that will land straight on the integration trains, but I'm not sure I know what those look like.

Every fix has inherent QA cost, testing cost, and schedule risk...we can't ship quarterly w/o managing those.

Christian

Ted Mielczarek

unread,
Dec 16, 2010, 2:38:59 PM12/16/10
to Christian Legnitto, dev-pl...@lists.mozilla.org
On Thu, Dec 16, 2010 at 2:32 PM, Christian Legnitto
<cleg...@mozilla.com> wrote:
> Have a couple of patches to make AIX work? Create a "AIX-suport" or more general "platform-support" repo, put them there, nominate it with justification. We could then decide "increased platform support" is a "feature" of a coming release and manage expectations and the landing. Have some little perf wins? Let's create a "perf-parpercuts" repo, land them there, and nominate for integration once we've aggregated a bunch. Have bug fixes that are only needed to support a feature? Fix those in the feature repo. Do multiple features depend on some fixes? Break those fixes out into their own repo, it's now a sub-feature. Have the dependent feature repos depend/merge from it while it merges from m-c/integration branches and its dependents.

This sounds appealing on some level, but our level of process overhead
is already a problem for outside contributors. Adding this additional
level in will without a doubt be damaging to community contributors.
We have lots of people submitting patches for whom development is just
a hobby, and forcing additional process on them will just make them
stop contributing. We lose lots of people already simply by ignoring
their patches or bugs.

-Ted

L. David Baron

unread,
Dec 16, 2010, 2:54:37 PM12/16/10
to Christian Legnitto, Boris Zbarsky, dev-pl...@lists.mozilla.org
On Thursday 2010-12-16 11:32 -0800, Christian Legnitto wrote:
> The underlying issue is we can't take random fixes and expect to
> ship every 3 months. We can't even do that with betas (oh yeah, I
> went there ha). This is likely where my release management
> perspective is wildly different than the development perspective.

Do you have evidence for that? It seems to me that most of the
blockers for Firefox 4 are either regressions from large features or
things needed to make large features good enough to ship.

I think the fundamental change we need to make in order to ship
every three months is to *forbid* people from saying "we can't ship
until feature X is ready".

Boris Zbarsky

unread,
Dec 16, 2010, 3:00:57 PM12/16/10
to
On 12/16/10 11:32 AM, Christian Legnitto wrote:
> I think that's the point. If they don't fit in with a project, they don't make it in the tree.

That involves creating some pretty vacuous "projects" like "fix web spec
compliance issues", no?

Maybe we do just need a "project branch" called "misc fixes" or
something....

> A project does not only mean feature though, just a structured way of thinking about and managing some class of fixes.

And all I'm saying is that the overhead of classifying small but
high-impact fixes would be high enough they just wouldn't happen, or
we'd have bogus classifications to work around the system.

> The underlying issue is we can't take random fixes and expect to ship every 3 months.

Well, I agree in general. But I think we're throwing the baby out with
the bathwater here. We have plenty of near-zero-risk "random fixes"
that we can take without slipping schedules. We also have plenty of
nonzero risk stuff that we should indeed merge in a more structured way.
I think we need to trust, say, our module owners to tell the two
apart. If we don't, we've already lost because they'll just work around
by creating bogus "project" branches for all the unrelated fixes in
their module. Except that will raise the bar on getting patches in for
people who don't work in one particular small area or who don't have
push access, since they or whoever deigns to look for checkin-needed
bugs will have to figure out which repo to push to (and then pull that
repo, etc; a pretty time-consuming operation).

> Have some little perf wins? Let's create a "perf-parpercuts" repo, land them there, and nominate for integration once we've aggregated a bunch.

Why do we need the aggregation? What if I just have one safe perf win?
Why should it need to wait on other things which may or may not ever
happen?

This isn't a hypothetical situation; our typical workflow for
non-feature perf fixes is someone files a bug about a particular page
being slow, then we find some one place in our code where we're doing
something stupid, we fix it. Who knows when the next such bug will come
along.

> Have bug fixes that are only needed to support a feature? Fix those in the feature repo.

Sure; this case is easy. Again, your setup works well for _features_.
I think you're underestimating the amount of non-feature work that's
done, especially by volunteers.

> Every fix has inherent QA cost, testing cost, and schedule risk...we can't ship quarterly w/o managing those.

While I agree, that cost for some fixes is very very low. I worry that
the additional process we're trying to impose on every patch would
prevent us from taking patches we really want to take and alienate
contributors.

-Boris

Mike Shaver

unread,
Dec 16, 2010, 3:09:03 PM12/16/10
to Christian Legnitto, Boris Zbarsky, dev-pl...@lists.mozilla.org
> The underlying issue is we can't take random fixes and expect to ship every 3 months. We can't even do that with betas (oh yeah, I went there ha). This is likely where my release management perspective is wildly different than the development perspective.

I don't think that taking "random fixes" is a major source of our
delays here, since they tend to be isolated and easily reversible. I
like data, though, so if there is data to be had on this stuff I'm all
ears.

I think our delays are from:
1) bad dependency management (compartments vs JM)
2) commitments to features-in-a-given-beta-number rather than
labelling once things have landed and stuck (often caused by 1)
3) taking big landings late in the beta (often caused by 2)

I think we should figure out how to better isolate the larger
features, and make sure that we don't land them on day N-1 of an N-day
beta cycle, and then revisit where we are.

If we imagine 3-week milestone cycles, of "ABB|ABB|ABB" pattern, then
in addition to a reason to change our hold music to Dancing Queen, we
would also be able to:

- schedule feature landings in only A weeks
- use B1 to decide if the feature is getting fixed or pulled for the milestone
- use B2 to finish the cleanup or pulling
- avoid day-for-day slips into the end of the milestone as larger
patches bounce: if you miss A, you're in the next A
- get QA started earlier on integration testing of larger features,
without as much risk of needing retest due to disruptive changes right
before milestone ship
- still give marketing and devrel and partners 1-2 weeks of lead time
for what will be in a given milestone
- watch crash stats for new features emerge over a week and still have
time to fix or pull cleanly

(I have a proposal for this that I'm working on in more detail, but
it's not taking shape quickly for a number of reasons.)

>> What if you aren't working on a feature?
>

> Have a couple of patches to make AIX work? Create a "AIX-suport" or more general "platform-support" repo, put them there, nominate it with justification.

I think we should separate this from managing expectations about
specific coming releases; that decoupling will be very important to
still be able to make quality threshold decisions for individual
features.

> Have some little perf wins? Let's create a "perf-parpercuts" repo, land them there, and nominate for integration once we've aggregated a bunch.

This reduces the value of our nightly testing, by clumping more
changes into a given nightly. For regression hunting (explicit or
analytic through crash reporting or similar), we want things spread
out more widely.

I don't think we need one ring to rule them all. If we get our major
work sorted out better, we'll be able to tell whether we need to do
something more structured for smaller changes as well, and learn
incrementally about what characteristics of changes need what sort of
attention.

Mike

Benjamin Smedberg

unread,
Dec 16, 2010, 3:08:33 PM12/16/10
to
On 12/16/10 11:32 AM, Christian Legnitto wrote:

>> This seems like it would work well for new-feature kind of projects.
>> It doesn't seem at all lightweight for your basic bugfixes (for which
>> you'd first have to figure out what "project" they belong to!).
>
> I think that's the point. If they don't fit in with a project, they don't
> make it in the tree. A project does not only mean feature though, just a
> structured way of thinking about and managing some class of fixes.

From what you've described, it seems like there will be more friction. How
are we going to manage that friction so that we don't frustrate people who
are trying to fix bugs?

I'd say that normally, 60-70% of patches that land in -central fall into the
"very small" category. People don't think of them as projects with goals:
they may be part of a larger project such as "UI polish" or "AIX build
compatibility" or "clang build compatibility" or "making the DOM faster".
And perhaps more importantly, they typically don't land as entire projects.
The review cycle means that all AIX or UI polish bugs are not done at the
same time and then merge. Instead, the patches and reviews and landings
trickle in.

For many of these small changes which are typically low-risk, I think we'd
do better with a small set of continuous-integration branches modeled after
the current TraceMonkey tree, each one managed by a release driver. So the
process of getting a small, low-risk patch landed would be to ask an
appropriate release driver to pull it into their tree, and those trees would
merge at appropriate safe points as TM does now.

Larger projects which have higher risk profile could still use the
project-branch model you have described. I think that the model with test
plans and QA leads and the heavyweight process you've outlined would help
these larger/riskier projects significantly.

But my main concern is not with the landing structure, but how we manage
testing and regressions. Every large feature has regressions, and a test
plan that doesn't involve our 30k nightly users or 1M beta users isn't going
to deal with the inevitable regressions. Unless we have a strong plan to get
the nightly users on the project branch *before* it lands, or alternatively
have a strong plan for dealing with regressions or backing out project
branches when the regressions become "too serious", I'm not sure that the
project-branch model will reduce our schedule risk.

--BDS

Mike Shaver

unread,
Dec 16, 2010, 3:15:36 PM12/16/10
to L. David Baron, Boris Zbarsky, Christian Legnitto, dev-pl...@lists.mozilla.org
On Thu, Dec 16, 2010 at 11:54 AM, L. David Baron <dba...@dbaron.org> wrote:
> I think the fundamental change we need to make in order to ship
> every three months is to *forbid* people from saying "we can't ship
> until feature X is ready".

Yes, we need to be willing to kick any feature down the road. This
means that we need to be able to back out or disable pretty much
anything. That, in turn, requires better dependency understanding
than we have now, for larger stuff at least, and it probably requires
that we don't defer cleanup or niggly details for very long after an
initial feature lands.

About "feature"or "larger stuff" here: right now we have two levels of
information about FF4, roughly speaking.

1 bit: Firefox 4 is not yet ready to ship
600 bits: individual blockers

We're starting (Sheila and others) to roll up the outstanding work in
terms of features (panorama, html5 forms inputs, webgl, indexeddb,
3rd-party-add-on-stuff, jetpack, etc. etc.) so that we can reason
about 20-25 different things, and track their dependencies at that
granularity. I think this will help us reason about how close we are
to shipping (and what we would be shipping in that case), and better
focus our efforts.

There are some other scheduling and work-allocation changes that would
improve our effectiveness with these larger features (and I think also
our satisfaction with doing the work!), as well. I'm going to try to
get some of this written up over the next two weeks, to keep blood
flowing in my brain.

Mike

Axel Hecht

unread,
Dec 16, 2010, 3:21:05 PM12/16/10
to
The thing that I'm clueless about is how to get l10n-coverage for this.

I don't think we should expect our project branches to land properly
localizable code, so getting some coverage is essential. How to get
that, no ideas I'm somewhat happy with. Like, this is not so much about
having machines that merge strings, but really about humans looking at
the code that lands in a timely manner and make sure the problems of
localizability are discovered early.

Axel

Henri Sivonen

unread,
Dec 16, 2010, 3:24:48 PM12/16/10
to mozilla.dev.planning group
On Dec 16, 2010, at 11:16, Christian Legnitto wrote:

>> - getting a full set of test runs before landing in a project branch
>
> Not sure this is needed? The point of doing so currently is to not block other m-c development and not subject nightly users to possible bustage. In this model those are not a concern and the test run should happen in the repo where it is planning to end up.

I think it's useful to be able to run unit tests "in the cloud" for stuff that's not "done" enough to show up in any long-term repo. Running unit tests "in the cloud" is useful, because doesn't lock up the developer's computer and block the developer from working. It is useful to be able to run unit tests without polluting the eternal history and without having to get review.

(I routinely send stuff that's not ready for review to try to get a full unit test run on one platform which is the same one as my development platform. I'm not sure if that's frowned upon as far as resource use goes, but it seems to me that it would be worse resource use for me to sit and watch while my local machine runs through all the tests.)

We'd still require review to land in a project repo, right? If we don't, the eventual merge will be so big that no one will have the time / skill / attention span to review it properly.

--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/


Robert O'Callahan

unread,
Dec 16, 2010, 3:27:45 PM12/16/10
to Benjamin Smedberg, dev-pl...@lists.mozilla.org
On Fri, Dec 17, 2010 at 9:08 AM, Benjamin Smedberg <benj...@smedbergs.us>wrote:

> For many of these small changes which are typically low-risk, I think we'd
> do better with a small set of continuous-integration branches modeled after
> the current TraceMonkey tree, each one managed by a release driver. So the
> process of getting a small, low-risk patch landed would be to ask an
> appropriate release driver to pull it into their tree, and those trees would
> merge at appropriate safe points as TM does now.
>

One single staging area for changes that aren't project-specific might be
good.

Echoing what other people have said, I think it's essential to have a
minimal-overhead way of getting small changes into the tree. It's essential
not only to encourage contributors, but also for our own developers to make
opportunistic improvements. If I spot a simple bug in someone else's code we
need to be able to get that patch landed without much effort, or people just
aren't going to do it and opportunities to improve the code will be lost.

Christian Legnitto

unread,
Dec 16, 2010, 3:31:36 PM12/16/10
to L. David Baron, mozilla.dev.planning group

On Dec 16, 2010, at 11:54 AM, L. David Baron wrote:

> On Thursday 2010-12-16 11:32 -0800, Christian Legnitto wrote:

>> The underlying issue is we can't take random fixes and expect to
>> ship every 3 months. We can't even do that with betas (oh yeah, I
>> went there ha). This is likely where my release management
>> perspective is wildly different than the development perspective.
>

> Do you have evidence for that?

Anecdotal, sure.

Security releases are on a ~1.5 month schedule. We've had to respin at least 3 times in the last year for a bad issue. We have had to roll back changes to keep schedules decent, something that would be a lot harder to do on m-c. Fairly large regressions have come from changes considered so "safe" they had automatic approval to land. And this is with all changes metered, approved, and targeted. Perhaps this is only due to less attention on the branches though.

Beta 8 was waiting on sync changes, then crashers (which came in from changes landed waiting on sync), then feature work, then RelEng issues, etc. It's been whack-a-mole. Beta 7 felt the same but I admittedly wasn't as plugged in.

> It seems to me that most of the
> blockers for Firefox 4 are either regressions from large features or
> things needed to make large features good enough to ship.

Right. If these were being developed and integrated elsewhere we would be in a position to ship what we had now as FF4 and come back a couple months later with a FF5 once things have firmed up. Because all that work has been layered it's a huge release management and development nightmare when we go to cut things to get a release out the door.

> I think the fundamental change we need to make in order to ship
> every three months is to *forbid* people from saying "we can't ship
> until feature X is ready".

And we can't do that if the feature lands on top of other features on top of opportunity fixes. We also don't want to slow down development, so that has to overflow somewhere and the merging / integration for a release likely has to be deliberate.

Perhaps my experience has been colored by working with 100's of projects across 100's of engineering groups elsewhere though. In that case random one-off fixes are clearly unsustainable but they might not be as worrisome here. But it's my job to worry :-)

Christian

Nicholas Nethercote

unread,
Dec 16, 2010, 4:17:24 PM12/16/10
to Shawn Wilsher, dev-pl...@lists.mozilla.org
On Fri, Dec 17, 2010 at 6:08 AM, Shawn Wilsher <sdw...@mozilla.com> wrote:

> On 12/16/2010 10:59 AM, Daniel Holbert wrote:
>>
>>  - getting a full set of test runs before landing in a project branch
>
> Is that really needed though?  If you break your branch, you can back out,
> but you only hurt the people working with you (likely small).

My experience with the TraceMonkey repo is that yes, it is useful for
large patches. Even though fewer people are affected if you break a
project repo, you still want to avoid it.

Nick

Shawn Wilsher

unread,
Dec 16, 2010, 4:47:24 PM12/16/10