Post-FF4 branch organization
Robert O'Callahan
year. I think for that to work, by the time we put FF4 on a branch we should
have worked out a plan for managing the tree post-FF4. If we wait until FF4
is shipped, it'll be too late because arbitrary work will have already
landed on trunk.
This week we have a lot of people in Mountain View so it might be worth
trying to come up with a plan now. It's particularly important to figure out
what will be allowed to land on trunk once it opens.
Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
Paul Biggar
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
>
--
Paul Biggar
Compiler Geek
pbi...@mozilla.com
Robert O'Callahan
> Is there a discussion scheduled for this?
>
Not yet.
There's a Google presentation at 2pm on Thursday about their experiences
with the Chrome release cycle.
Maybe after that? Say 4pm? Bring a plan.
Mike Beltzner
> This week we have a lot of people in Mountain View so it might be worth
> trying to come up with a plan now. It's particularly important to figure out
> what will be allowed to land on trunk once it opens.
My initial suggestion is: let's not branch.
Instead, let's create project branches with owners, each of whom can set
the criteria as deemed appropriate for their various projects. I would
further suggest that each project be clearly defined with goals and
metrics used to judge outcomes, but let's not go too far down that
rabbit hole at present.
As for mozilla-central, it can be open to:
- NPOTB patches
- test fixes (and honestly, if in the early days all we do is WOO
work, that's a win)
- patches slated for a security/stability release
- patches with explicit driver approval
I suspect that in order to release quickly we'll always need to keep the
trunk in a "near-ready-to-ship" state. A lot of this discussion will be,
I fear, putting the cart before the horse, so the strategy I'm proposing
means to give us the most flexibility and agility for once plans are set.
(Also note that hg is like a time machine; it can be arduous to do
merges afterwards, but we can always go back in time to snip a branch if
needs be)
If there is to be a discussion on this during the work week, I recommend
it take place AFTER all the retriaging work is done and better estimates
of Firefox 4 schedule can be made. I would expect that to be afternoon
PDT on Thursday.
cheers,
mike
Paul Biggar
> My initial suggestion is: let's not branch.
>
> Instead, let's create project branches with owners, each of whom can set the
> criteria as deemed appropriate for their various projects. I would further
> suggest that each project be clearly defined with goals and metrics used to
> judge outcomes, but let's not go too far down that rabbit hole at present.
We were discussing in the JS-pit how well this was working for us. We
have tracemonkey and jaegermonkey trees, and a manager who takes
responsibility for merging tracemonkey to mozilla-central (which means
it's significantly easier to get work done).
One thing that needs to get fixed first is the cost of creating
project branches. It took us a month to get the branches we have (and
the trees need to fit into things like tryserver), so we've been
reusing the branches to avoid this.
Paul
Phil Ringnalda
> My initial suggestion is: let's not branch.
>
> Instead, let's create project branches with owners
Be sure to include releng in the planning session, since we're already
short on slaves for what we're doing now, really short on slaves for
adding in opt and debug WinXP, and the rev3 minis are no longer
available, so if you're planning on adding multiple trees, you're
probably talking about replacing the entire build farm with new minis.
And don't forget about the contributors of random little patches, who
have been waiting for there to be a trunk again since last July, who are
going to find it even more baffling to have "Getting your patch in the
tree" include steps like "figure out whether this should land on some
project branch, in which case you need review plus approval from some
particular person, or whether there is no project for it, in which case
you need review plus approval from the wind."
Robert O'Callahan
once a quarter requires we default to mandatory (silent) updates across
major versions. If so, are we going to do that and when is it going to
happen?
Robert O'Callahan
> Be sure to include releng in the planning session, since we're already
> short on slaves for what we're doing now, really short on slaves for adding
> in opt and debug WinXP, and the rev3 minis are no longer available, so if
> you're planning on adding multiple trees, you're probably talking about
> replacing the entire build farm with new minis.
>
Not all branches need continuous build and testing.
But releng should certainly be involved.
And don't forget about the contributors of random little patches, who have
> been waiting for there to be a trunk again since last July, who are going to
> find it even more baffling to have "Getting your patch in the tree" include
> steps like "figure out whether this should land on some project branch, in
> which case you need review plus approval from some particular person, or
> whether there is no project for it, in which case you need review plus
> approval from the wind."
I hope most small patches could just land on trunk, with driver approval.
Robert O'Callahan
> Maybe after that? Say 4pm? Bring a plan.
I have booked 10-Forward for 4pm Thursday.
Mike Beltzner
> I think we also need to consider whether doing releases as frequently as
> once a quarter requires we default to mandatory (silent) updates across
> major versions. If so, are we going to do that and when is it going to
> happen?
Yes, we need to consider it. Obviously to make that decision we'll need
to understand the costs and the required infrastructure changes to
support it. We'll also need to reevaluate some assumptions that I think
are commonly held and determine which we're willing to change in order
to support the core goal of shipping faster.
I'm not at all sure that this decision needs to be reached this week, or
even in order for us to talk and propose ideas for various branching
approaches, though.
(FWIW, I wouldn't equate mandatory with silent - there are ways of doing
automatic updates that are not silent, and I find that silent ends up
putting people on tilt a bit)
cheers,
mike
Steve Wendt
> (FWIW, I wouldn't equate mandatory with silent - there are ways of doing
> automatic updates that are not silent, and I find that silent ends up
> putting people on tilt a bit)
In a similar vein, automatic updates need to work in more situations;
for example, Windows 7 users cannot update without entering an
administrator password, which is impossible in a corporate setting.
Presumably, the same is true on Linux and OSX.
Armen Zambrano Gasparnian
> One thing that needs to get fixed first is the cost of creating
> project branches. It took us a month to get the branches we have (and
> the trees need to fit into things like tryserver), so we've been
> reusing the branches to avoid this.
>
L. David Baron
> On 12/14/2010 11:36 AM, Mike Beltzner wrote:
> >My initial suggestion is: let's not branch.
> >
> >Instead, let's create project branches with owners
>
> Be sure to include releng in the planning session, since we're
> already short on slaves for what we're doing now, really short on
> slaves for adding in opt and debug WinXP, and the rev3 minis are no
> longer available, so if you're planning on adding multiple trees,
> you're probably talking about replacing the entire build farm with
> new minis.
I don't think we're talking about an increase in changesets or
pushes. We're just talking about distributing the same number of
changesets and pushes across more branches. And the load on the
build machines largely comes from the number of pushes, not the
number of branches.
> And don't forget about the contributors of random little patches,
> who have been waiting for there to be a trunk again since last July,
> who are going to find it even more baffling to have "Getting your
> patch in the tree" include steps like "figure out whether this
> should land on some project branch, in which case you need review
> plus approval from some particular person, or whether there is no
> project for it, in which case you need review plus approval from the
> wind."
I think we do need a place to iterate on small patches that fix
problems discovered by Web authors and standards compliance
testsuites, and get them pushed to nightly users sooner rather than
later. Unless we've decided we're now comfortable taking such
patches in a security release without a beta cycle, I think we do
still need a trunk that's open to such things.
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
Armen Zambrano Gasparnian
> On Wed, Dec 15, 2010 at 9:03 AM, Phil Ringnalda<philri...@gmail.com>wrote:
>
>> Be sure to include releng in the planning session, since we're already
>> short on slaves for what we're doing now, really short on slaves for adding
>> in opt and debug WinXP, and the rev3 minis are no longer available, so if
>> you're planning on adding multiple trees, you're probably talking about
>> replacing the entire build farm with new minis.
>>
> Not all branches need continuous build and testing.
What level of testing do you think is valuable for those branches?
We could add full coverage for those branches on self-serve basis
(without having to poke a releng to trigger it for the branch manager).
>
> But releng should certainly be involved.
>
Armen
Nicholas Nethercote
>>
>> Be sure to include releng in the planning session, since we're
>> already short on slaves for what we're doing now, really short on
>> slaves for adding in opt and debug WinXP, and the rev3 minis are no
>> longer available, so if you're planning on adding multiple trees,
>> you're probably talking about replacing the entire build farm with
>> new minis.
>
> pushes. We're just talking about distributing the same number of
> changesets and pushes across more branches. And the load on the
> build machines largely comes from the number of pushes, not the
> number of branches.
More branches may even reduce the load on the try machines.
As a JS hacker I get to land my patches on the tracemonkey branch, and
because breakage on that branch doesn't affect too many people, I'm
willing to land most patches without doing a try server run. If I had
to land patches on mozilla-central, I'd be much more conservative and
use the try server more. This would increase try machine load and
slow down my development speed.
Nick
Robert O'Callahan
arm...@mozilla.com> wrote:
> Which branches are you referring to?
>
Suppose I want to do a medium-sized graphics project, say implement a new
graphics API to speed up canvas drawing. It'll be quite a bit of work and
need contributions from multiple developers. It should be on a branch, but
it doesn't need to be tested until we think it's done.
What level of testing do you think is valuable for those branches?
>
Occasional tryserver testing.
We could add full coverage for those branches on self-serve basis (without
> having to poke a releng to trigger it for the branch manager)
That'd be pretty cool, but is there any downside to just pushing the branch
to tryserver?
Joshua Cranmer
> Suppose I want to do a medium-sized graphics project, say implement a new
> graphics API to speed up canvas drawing. It'll be quite a bit of work and
> need contributions from multiple developers. It should be on a branch, but
> it doesn't need to be tested until we think it's done.
I may be wrong, but wouldn't one of the disposable branches be good for
this?
<https://wiki.mozilla.org/ReleaseEngineering/DisposableProjectBranches>
Armen Zambrano Gasparnian
> On Wed, Dec 15, 2010 at 1:26 PM, Armen Zambrano Gasparnian<
> arm...@mozilla.com> wrote:
>
>> Which branches are you referring to?
>>
>
> Suppose I want to do a medium-sized graphics project, say implement a new
> graphics API to speed up canvas drawing. It'll be quite a bit of work and
> need contributions from multiple developers. It should be on a branch, but
> it doesn't need to be tested until we think it's done.
>
Coverage can currently be tailored.
>
> We could add full coverage for those branches on self-serve basis (without
>> having to poke a releng to trigger it for the branch manager)
>
> That'd be pretty cool, but is there any downside to just pushing the branch
> to tryserver?
>
> Rob
No downside on regards of full coverage.
Maybe prioritization of project branch jobs over tryserver jobs.
In the mid-future, branch managers could bump priority of jobs related
to a changeset if need be to override the current prioritization rules.
Armen
Armen Zambrano Gasparnian
Armen
Henri Sivonen
If most development happens on branches instead of the trunk, surely more than three concurrent project branches will be needed. Will this mechanism scale up? (Having a fixed number of disposable branches suggests that the mechanism doesn't scale arbitrarily at present.)
--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/
Ehsan Akhgari
Or even a user repo? If a project branch doesn't require continuous
integration testing, I don't think there's any reason for us to use
anything more complicated than a user repo, which takes about 5 minutes
to set up.
Ehsan
Robert O'Callahan
> Or even a user repo? If a project branch doesn't require continuous
> integration testing, I don't think there's any reason for us to use
> anything more complicated than a user repo, which takes about 5 minutes
> to set up.
>
Yes. I think a user repo plus pushes to try when desired should work quite
well.
Ben Hearsum
Surely. Branches are relatively cheap these days, and if the names of
them are not of concern, can be en-masse ahead of time. There's probably
lots we can do in the future to make branch development even easier, and
I have no doubt that once we figure out firmer requirements that we can
make some of those things happen!
Ben Hearsum
Surely. Branches are relatively cheap these days, and if the names of
Kyle Huey
> And don't forget about the contributors of random little patches, who have
> been waiting for there to be a trunk again since last July, who are going to
> find it even more baffling to have "Getting your patch in the tree" include
> steps like "figure out whether this should land on some project branch, in
> which case you need review plus approval from some particular person, or
> whether there is no project for it, in which case you need review plus
> approval from the wind."
>
IMHO this is really important to keep in mind. I have patches that were
a2.0-'d in August and while I can live with that I can't imagine that having
to wait several months for your patches to land is encouraging to new
contributors.
- Kyle
Mark Banner
> My initial suggestion is: let's not branch.
Certainly an interesting idea. Presumably you would still create a
separate repo for security releases for 4.0 at some stage?
> As for mozilla-central, it can be open to:
>
> - NPOTB patches
> - test fixes (and honestly, if in the early days all we do is WOO work,
> that's a win)
> - patches slated for a security/stability release
> - patches with explicit driver approval
>
> I suspect that in order to release quickly we'll always need to keep the
> trunk in a "near-ready-to-ship" state.
I agree with keeping trunk in a "near-ready-to-ship" state, that is good.
What I don't get about your proposal is what does the explicit driver
approval get us?
If we take an example of a patch that I submit in future. I've tested it
myself, made sure its got the appropriate unit tests, pushed to try
server, and it has been reviewed.
What is there that would cause a driver to deny the patch? I've done
everything required to get the review. The only thing could be an
assessment of risk, but if the reviewer thinks it is too risky,
shouldn't the reviewer be denying it or asking for more work?
If an approver is making the assessment of a patch being too risky, and
they deny it, what mechanism do we then use for patches that are
slightly risky but no-other way of doing them? The only way we have to
get those patches out is to land them for the nightly builds. Although
project branches could have nightlies, you're surely not going to get
the same kind of coverage.
And then aren't we actually turning the approval request back into what
super-review used to be - more of a it is ok to land this patch?
This also will increase the load on approvers. We've already seen that
at times some approvals have been delayed for multiple months, I would
argue, that if we're having this additional step, then it needs to be
1-2 days maximum.
That discussion is a bit longer than I was intending. I'm certainly open
to the possibility of having approvals-only, but at the moment I fail to
see the benefits over what the existing review mechanisms provide.
Standard8
Chris Hofmann
>
> I suspect that in order to release quickly we'll always need to keep
> the trunk in a "near-ready-to-ship" state.
Yeah, but in addition we need to step up the pace of betas.
More betas help to stay in that ready-to-ship state, and help us to
understand the impact on all our users.
More betas have the effect of helping a large number people inside and
outside the project understand the work that is going on at Mozilla and
in increasing our impact and influence.
More betas help us reach out to and engage with browser enthusiasts
while not constantly disrupting the other 390 Million users with updates
that might not be fully baked yet.
Let continue to build the beta program and go from 1 million beta
testers to 10 million beta testers in the next year.
Lets continue to build the feedback channels that empower and engage
with this beta testing audience.
Lets refine our beta process so we can get it moving at a faster pace
and a 3 or 4 week cycle. We can by doing things like front loading big
changes from project branches *only* during the first week of the beta
cycle and and staying away from the position of waiting for large
landings and recovery efforts to happen at the end of beta development
cycles. That's the key area where we need better discipline that will
help us to ship both betas and final releases faster.
-chofmann
Mike Beltzner
> On 12/14/10 11:36 AM, Mike Beltzner wrote:
>>
>> I suspect that in order to release quickly we'll always need to keep the trunk in a "near-ready-to-ship" state.
>
> Yeah, but in addition we need to step up the pace of betas.
Chris,
I totally agree that we need to determine how we can accomplish all the things you assert as being provided by our current use of betas. Whether that's by "shipping betas" or promoting a "dev channel" a la Chrome or some other fashion, I think, we should leave open for various proposals. It also has less to do with the specifics of branching, I think, except for in how it requires that we keep some tree of our to-be-shipped product line in a near-shippable state.
> Lets refine our beta process so we can get it moving at a faster pace and a 3 or 4 week cycle. We can by doing things like front loading big changes from project branches *only* during the first week of the beta cycle and and staying away from the position of waiting for large landings and recovery efforts to happen at the end of beta development cycles. That's the key area where we need better discipline that will help us to ship both betas and final releases faster.
Again, I see this as a process decision that's somewhat related to branching, but more to do with how we plan the development of features. I agree that features should be planned more fully in advance (in terms of what their goals are, how we'll know we've hit those goals, what the project milestones are and metrics to measure against them, how we'll know we haven't had unplanned deleterious effects, how we'll release and share those changes amongst our various feedback audiences and how we'll gather the feedback, etc) and then if they aren't ready by some deadline, they don't make the next development milestone and skip over to the following one. The implication for branching strategy is keeping the main development branch in ready-to-ship state, though, mostly.
cheers,
mike
Benjamin Smedberg
> Or even a user repo? If a project branch doesn't require continuous
> integration testing, I don't think there's any reason for us to use
> anything more complicated than a user repo, which takes about 5 minutes
> to set up.
The primary advantages of a project repo over a try repo are:
* organized nightly build with updates
* Talos integration which provides historical charts
--BDS
Axel Hecht
> On Tuesday 2010-12-14 12:03 -0800, Phil Ringnalda wrote:
>> On 12/14/2010 11:36 AM, Mike Beltzner wrote:
>>> My initial suggestion is: let's not branch.
>>>
>>> Instead, let's create project branches with owners
>>
>> Be sure to include releng in the planning session, since we're
>> already short on slaves for what we're doing now, really short on
>> slaves for adding in opt and debug WinXP, and the rev3 minis are no
>> longer available, so if you're planning on adding multiple trees,
>> you're probably talking about replacing the entire build farm with
>> new minis.
>
> I don't think we're talking about an increase in changesets or
> pushes. We're just talking about distributing the same number of
> changesets and pushes across more branches. And the load on the
> build machines largely comes from the number of pushes, not the
> number of branches.
There would be an increase of nightly builds if we want those, which is
mentioned elsewhere in this thread.
We might be able to cut down on those to some degree by doing those only
if the project branch actually changed. That somewhat redefines
"nightly", but I think that "sparse nightlies" are much closer to what
we want anyway. Could be an interesting option for our stable release
branches, too.
Axel
Robert O'Callahan
> I totally agree that we need to determine how we can accomplish all the
> things you assert as being provided by our current use of betas. Whether
> that's by "shipping betas" or promoting a "dev channel" a la Chrome or some
> other fashion, I think, we should leave open for various proposals. It also
> has less to do with the specifics of branching, I think, except for in how
> it requires that we keep some tree of our to-be-shipped product line in a
> near-shippable state.
>
I think we'll need a way to expose multiple large projects to a large number
of users before we know whether those projects can ship in the next release.
It's the only way to find Web-wide regressions in those projects, and the
volume of those regressions will determine when those projects are "done".
We won't want to split our "beta users" among multiple large projects, so
that suggests we need an integration point that can receive multiple large
landings, that we can ship "betas" or "dev channel" from, but which is
separate from release branches.
So I think these issues are pretty deeply intertwined.
Robert O'Callahan
Paul Biggar
<rob...@ocallahan.org> wrote:
> I think we'll need a way to expose multiple large projects to a large number
>
This seems compatible with the previous goals. To be concrete (based
on the conversation so far):
- clone 'FF5' from m-c at some point before the FF4 release
- FF5 is the source of betas, release candidates, etc
- FF5 is the integration point for other project repos
- small changes do not go straight to FF5, but rather have to come in
via another source (we might keep mozilla-central around for this
purpose, if we made committing to it cheaper)
- each branch/tree has an owner, like sayrer owns the tracemonkey
tree. They are responsible for integration, they are responsible for
backing out, they negotiate integration timelines with the FF5 branch
owner.
The major advantages:
- the FF5 branch is never broken (each repo will be fully tested
before integration)
- the FF5 branch is permanently ready to release something
And some minor ones:
- cost of committing is significantly lowered
- there are never patches that can't land _somewhere_
- if something breaks, it only breaks for a small number of people
Downsides:
- the more "upstream" parts of the codebase have less testing from
their "clients" before integration.
- developers are more likely to be dogfooding their own repos, rather
than the FF5 repo.
--
Paul Biggar
Compiler Geek
pbi...@mozilla.com
Robert O'Callahan
> - clone 'FF5' from m-c at some point before the FF4 release
> - FF5 is the source of betas, release candidates, etc
> - FF5 is the integration point for other project repos
> - small changes do not go straight to FF5, but rather have to come in
> via another source (we might keep mozilla-central around for this
> purpose, if we made committing to it cheaper)
> - each branch/tree has an owner, like sayrer owns the tracemonkey
> tree. They are responsible for integration, they are responsible for
> backing out, they negotiate integration timelines with the FF5 branch
> owner.
>
As far as I can tell, this doesn't give us a place to land big projects for
testing by hundreds of thousands of users *before* we commit to shipping
them in a release.
Mike Shaver
> As far as I can tell, this doesn't give us a place to land big projects for
> testing by hundreds of thousands of users *before* we commit to shipping
> them in a release.
Gotta be able to pref big stuff off, or otherwise revert. That will
let us reach even more than our beta audience with opt-in testing, as
well.
(I actually wanted to do this with OOPP on Windows in 3.6, where we'd
ship it available-but-off in one update, then get the word out through
lifehacker and SUMO and suchlike, then use that enhanced data to
refine better before shipping. I didn't really push on it, though.)
Mike
Robert O'Callahan
> Gotta be able to pref big stuff off, or otherwise revert. That will
> let us reach even more than our beta audience with opt-in testing, as
> well.
>
Unfortunately, for some architectural changes that can be arbitrarily
difficult.
This will be a good question for discussion :-).
Steven Roussey
> testing by hundreds of thousands of users *before* we commit to shipping
> them in a release.
You can ship multiple variants simultaneously (and switch between them):
L. David Baron
> As far as I can tell, this doesn't give us a place to land big projects for
> testing by hundreds of thousands of users *before* we commit to shipping
> them in a release.
One thought I had about this, although it's may be more applicable
to testing stuff for a nightly audience rather than a beta audience:
If someone thinks they're ready to merge a project branch into
mozilla-central, we could flip one day's worth of nightlies over to
that project branch and see what feedback that nightly testing
yields. (Presumably mozilla-central would get merged into that
project branch right before so they wouldn't go backwards in terms
of mozilla-central commits either.)
Or, more generally, we don't necessarily need to treat our nightly
or beta update channels as things that require each revision to be a
version control graph descendant of the previous revision. We could
move them to the branch that needs the most testing, as long as it's
believed to meet our standards for what we'd send to nightly or beta
users.
Axel Hecht
We haven't started talking about how to handle l10n for the project
branches yet. I following this thread closely and start to narrow down
on that as soon as the main code discussion gets a direction.
On this point though, we should probably make our standards for
nightlies include expectations across all locales. That might be "none,
really" for some locales, down to "we may not need builds until we reach
beta for that train", but pretty high standards on a handful or so.
Axel
Mike Beltzner
> On this point though, we should probably make our standards for nightlies include expectations across all locales. That might be "none, really" for some locales, down to "we may not need builds until we reach beta for that train", but pretty high standards on a handful or so.
I really like the idea of l10n-merge for locales, so that those who like to track through development can benefit from doing so, and those who don't can get as much translated as has been done to date, and default (en-US) strings otherwise. Seems like that provides the most thorough coverage for nightly testing, and keeps the "trunk" in closest-to-shipping state.
cheers,
mike
Phil Ringnalda
> Instead, let's create project branches with owners
Two last thoughts about what those owners are taking on, that they will
not actually be able to manage well enough:
We will be fragmenting the management of randomorange. There's no way
I'll be watching a dozen trees at once, and as we've just seen with the
Places branch merge, even someone who does as good a job of tree
watching as sdwilsh did will still have some "oh, yeah, I do remember
seeing that intermittently once or twice but I didn't ever file it"
things, and probably some more "I don't remember seeing that at all, but
maybe it did happen" and maybe some "I don't think I caused that, but
maybe if you looked at everything we starred, it will turn out that we
misstarred it as something old when really it was something new we caused."
Starring is also difficult to get right when you aren't always up to
date - I try not to stick "hey, look what happened with the code from
last week!" comments from TraceMonkey failures with a week-old
mozilla-central parent that is missing three orange fixes and two
expanded logging patches into bugs, but it's really tough to remember
what has and hasn't changed even when you are cc'ed on most orange bugs.
The other thing we will be fragmenting is checkin-needed: TraceMonkey
has struggled all along to not ignore its checkin-needed (and has mostly
failed), even though it's fairly rare for a random contributor without
push access to contribute patches there, and nobody who does
checkin-needed has been willing to maintain a TraceMonkey tree to push from.
As a project branch owner, you really only have a couple of responsible
choices:
- decide that all pushes to your project's code have to land on your
project branch, and take personal responsibility for handling
checkin-needed in a timely manner (and you'll still leave some
contributions on the floor, because people willing to write a patch for
you won't always be willing to clone your tree and write a patch against
the stuff you haven't merged into the tree they have yet), or
- decide to deal with the merge-hell of always having changes to your
code that you have to merge back from mozilla-central, making the merges
always a slow and time-consuming and error-prone thing, so you'll put
them off and make your orange situation even worse.
Rob Campbell
On 2010-12-16, at 08:47, Phil Ringnalda wrote:
> On 12/14/10 11:36 AM, Mike Beltzner wrote:
>> Instead, let's create project branches with owners
>
> Two last thoughts about what those owners are taking on, that they will not actually be able to manage well enough:
>
> We will be fragmenting the management of randomorange. There's no way I'll be watching a dozen trees at once, and as we've just seen with the Places branch merge, even someone who does as good a job of tree watching as sdwilsh did will still have some "oh, yeah, I do remember seeing that intermittently once or twice but I didn't ever file it" things, and probably some more "I don't remember seeing that at all, but maybe it did happen" and maybe some "I don't think I caused that, but maybe if you looked at everything we starred, it will turn out that we misstarred it as something old when really it was something new we caused."
>
> Starring is also difficult to get right when you aren't always up to date - I try not to stick "hey, look what happened with the code from last week!" comments from TraceMonkey failures with a week-old mozilla-central parent that is missing three orange fixes and two expanded logging patches into bugs, but it's really tough to remember what has and hasn't changed even when you are cc'ed on most orange bugs.
This is an interesting problem, but not one that I really think we should dictate our release and branching structure around.
I'm not sure having humans deal with oranges is really the best way to manage our branches anymore, tbh. The failures we're seeing are statistical functions and we should be able to auto-star known oranges automatically.
When we see a new orange or a leak or an obvious talos regression, we could get an alert (in #developers? via pulse? email, cable and/or text message) and dispatch the swat teams. Otherwise, TBPL could be augmented to do the starring for us.
The amount of time we spend playing Tinderbox Pong with random oranges is wasteful. I know it usually kills a chunk of my day if I check something in and have to keep monitoring the tree to check off random failures. I would like to have that part of my day back.
> The other thing we will be fragmenting is checkin-needed: TraceMonkey has struggled all along to not ignore its checkin-needed (and has mostly failed), even though it's fairly rare for a random contributor without push access to contribute patches there, and nobody who does checkin-needed has been willing to maintain a TraceMonkey tree to push from.
That's a different problem. We need people to do this. Like, it should be a rotating task for someone to grab and checkin ready bugs. We used to have Reed do that for us and it was pretty valuable.
Of course, this gets more awkward with project branches, which I guess is your real concern.
> As a project branch owner, you really only have a couple of responsible choices:
>
> - decide that all pushes to your project's code have to land on your project branch, and take personal responsibility for handling checkin-needed in a timely manner (and you'll still leave some contributions on the floor, because people willing to write a patch for you won't always be willing to clone your tree and write a patch against the stuff you haven't merged into the tree they have yet), or
>
> - decide to deal with the merge-hell of always having changes to your code that you have to merge back from mozilla-central, making the merges always a slow and time-consuming and error-prone thing, so you'll put them off and make your orange situation even worse.
This is potentially scary.
The tracemonkey team is fortunate enough to have total control over their codebase (i.e., the js code). In higher-up, shared sections, we're going to have to deal with merges more frequently. That'll probably mean lots of potentially risky manual rebasing at merge time. Oranges on mozilla-central will often (should always?) be a result of mismerges.
Anyway, fun stuff. Can we have our devtools repo now? ;)
~ rob
Christian Legnitto
I'm coming at this with these in mind:
#1 - All changes need to align with overall product goals
#2 - The release schedule is set in stone. We would cut features / fixes rather than slip each quarter
#3 - Doing all development together makes a release dependent on the hardest / worst managed / most invasive project
#4 - Stop acting like we're developing in CVS
Also, I am going to stop calling them project "branches", because they aren't branches, they are repos.
So, this is what I am thinking...
* Everything is done in project repos
** There will be small exceptions, but should be < 5% and should need explicit approval
* mozilla-central becomes an integration repo, not a development repo like today, and is pretty much always in a consistent release state
* Project repos are pretty much self serve. You go to a web page, fill out some details, get a new project repo with RelEng-backed goodness
** The idea is these are extremely lightweight and developers shouldn't hesitate to create one, even for a handful of changes
* You need to fill out a *repo owner*. This person will be on the hook for anything to do with the repo
* You need to fill out a *product management driver*, who advises on the feature/changes, helps set schedule, makes sure it stays on message, writes announcements, etc
* You need to fill out a *QA lead*. They will be responsible for a test plan, ongoing testing, and any regression fallout
* You need to set estimates of best/likely/worst case for the project work to be completed
* Working with release and product management, you need to specify what FF release the project is targeted to land for
** This isn't a contract in pretty much all cases though
* You specify which other project repos this new repo relies on
* You can choose to opt in to hourly/daily/weekly/monthly auto-merges from your targeted release and project repo dependencies. This is one way (into the project repo). If the merges fail, emails are sent
* You and your QA lead need to come with specific metrics of "success" that need to be fulfilled before being nominated for merging to m-c
** The places branch has been a good model of this
What does development look like?
* Decide on a feature
* Get a repo
* Manually or automatically pull from another integration or project repo to start
* Develop in your project repo, either manually refreshing from the dependencies or via automation
* Builds will be created nightly if new checkins/merges have happened. The channel will be nightly-[reponame]
* Developers (and maybe their QA lead) should be the only ones living on their project build nightlies
* Development happens as normal in the project repo
** This means that the project won't pick up implicit dependencies due to landing on top of other big projects / fixes
** try would also *go away*, as it is not needed anymore
* When the feature is in a good state, auto-merging into the project repo from dependencies will be turned on for a couple of days
* If there are product-level and tough UI decisions needed for the project, we can do A/B testing via updates off the project repo with our nightly/beta audience to collect more data
* After a waiting period (mainly for oranges and any missed integration work), project management will move a subset of nightly testers over on nightly-[reponame]. If we want to call attention to it, we can pop up an info bar after updates that tell people to go to an about page to learn more about the new feature, etc
** This means most integration and regression fallout will happen *off m-c*
** Crashes/ADU may not help too much here. It would be good to switch to a crashes/pageload model to get finer-grained data
* If the QA plan has been executed and nothing major has come up, the project repo is nominated to merge to m-c (the integration repo)
** One of the nomination criteria is to point out any known (non-blocking) issues, though there should be basically none of them
* If the project is a new UI feature, it lands preffed off and at least one release needs to go out with it like that. We will do blog posts and tell people how to pref it on to try it out (like about:labs)
* When the project merges to m-c, we move the users on any build of that project repo back into the m-c pool
Note that major / one-off security fixes fixes can be handled in a designated project repo as well.
How does this enable us to release quarterly?
* If features aren't ready within the quarter they we targeted to ship there is no backout that needs to happen
* Better visibility into interdependencies and better risk management of those
* Changes "flow" through dependencies and the integration work / fallout is spread out over time
* If needed, we can turn off auto-merging from m-c, roll back multiple features off m-c, release, roll forward, and turn back auto-merging without any development disruption. This is a HUGE win
* Surfaces underlying assumptions and sets clear success metrics via a QA plan up front
* Targeted testing of fixes via moving nightly/beta audience improves the audience's value. Crash spikes / new signatures will be extremely easy to see, feedback can be routed a lot easier
* We could get to biweekly releases and this structure would still hold
How much work is it to do this?
Not too much. The building blocks are there. As an intermediate step, instead of on-demand repos we can use a pool of them like we have now. This is mainly a process / integration change with some beefing up of existing tools (like AUS).
I know I didn't describe this as well as I would like, feel free to come over and chat with me in person (I'll likely be in Warp Core in an hour or so). This isn't radically different than what was proposed earlier in this thread but I am sure there are some controversial bits strewn about).
Thanks,
Christian
PS: Not sure about mailing list etiquette. This isn't in response to specifics in the current thread but is close enough I didn't want to start another thread. I replied but deleted all previous response text. Feel free to email me directly and tell me if you would have preferred something else.
Daniel Holbert
> ** try would also *go away*, as it is not needed anymore
I think try would still be quite helpful for a number of things,
actually -- here are a few items off the top of my head, none of which
would really be appropriate for pushing to project branches:
- generating one-off builds with "does this fix it?" patches for
community members to test, for bugs that we can't reproduce locally.
(This has been helpful in the past for bugs with obscure printer
drivers, among other things.)
- getting a full set of test runs before landing in a project branch
- trying out a possible workaround/fix for randomorange bugs
~Daniel
Boris Zbarsky
> ** The idea is these are extremely lightweight and developers shouldn't hesitate to create one, even for a handful of changes
> * You need to fill out a *repo owner*. This person will be on the hook for anything to do with the repo
> * You need to fill out a *product management driver*, who advises on the feature/changes, helps set schedule, makes sure it stays on message, writes announcements, etc
> * You need to fill out a *QA lead*. They will be responsible for a test plan, ongoing testing, and any regression fallout
> * You need to set estimates of best/likely/worst case for the project work to be completed
> * Working with release and product management, you need to specify what FF release the project is targeted to land for
> ** This isn't a contract in pretty much all cases though
> * You specify which other project repos this new repo relies on
> * You can choose to opt in to hourly/daily/weekly/monthly auto-merges from your targeted release and project repo dependencies. This is one way (into the project repo). If the merges fail, emails are sent
> * You and your QA lead need to come with specific metrics of "success" that need to be fulfilled before being nominated for merging to m-c
This seems like it would work well for new-feature kind of projects. It
doesn't seem at all lightweight for your basic bugfixes (for which you'd
first have to figure out what "project" they belong to!).
> What does development look like?
>
> * Decide on a feature
What if you aren't working on a feature?
-Boris
Shawn Wilsher
> - getting a full set of test runs before landing in a project branch
out, but you only hurt the people working with you (likely small).
Cheers,
Shawn
L. David Baron
> I'm coming at this with these in mind:
>
> #1 - All changes need to align with overall product goals
I'm curious what you mean by overall product goals. We should
generally strive to maintain or improve our speed, memory use,
stability, standards compliance, security, usability for end users
and for developers, and visual polish, among other things. While
some improvements in these areas are large projects, many of the
most effective (in terms of cost vs. benefit) are small. And I
worry that a statement like #1 might make it more difficult to work
on small and effective things (as do a significant number of the
policies you list below that follow from it).
> #2 - The release schedule is set in stone. We would cut features / fixes rather than slip each quarter
> #3 - Doing all development together makes a release dependent on the hardest / worst managed / most invasive project
I definitely agree on these.
> #4 - Stop acting like we're developing in CVS
I agree here as well, but we also need to think about what we need
from user testing and how to focus that user testing appropriately.
> * Development happens as normal in the project repo
> ** This means that the project won't pick up implicit dependencies due to landing on top of other big projects / fixes
There's a big danger here on the opposite side, which is that
projects end up being very hard to merge with each other.
Christian Legnitto
> On 12/16/2010 10:11 AM, Christian Legnitto wrote:
>> ** try would also *go away*, as it is not needed anymore
>
> I think try would still be quite helpful for a number of things,
> actually -- here are a few items off the top of my head, none of which
> would really be appropriate for pushing to project branches:
I struggled with this bit too. Initially I thought every project branch should have a shadowing try repo, but the more I think about it the more I think try in general try should go away.
> - generating one-off builds with "does this fix it?" patches for
> community members to test, for bugs that we can't reproduce locally.
> (This has been helpful in the past for bugs with obscure printer
> drivers, among other things.)
Private repos? Local builds? Spinning up a "printer-experiments" repo?
> - getting a full set of test runs before landing in a project branch
Not sure this is needed? The point of doing so currently is to not block other m-c development and not subject nightly users to possible bustage. In this model those are not a concern and the test run should happen in the repo where it is planning to end up.
> - trying out a possible workaround/fix for randomorange bugs
Fixing random oranges should likely be in their own "randomorange" repo. While good, they aren't tied to a specific project. Additionally, there is inherent risk and cost to the fixes, even though we don't treat them like that now. Those fixes and landings should be managed, not ad-hoc. This is a proposal for managing change with a consistent framework.
Christian
Christian Legnitto
I think that's the point. If they don't fit in with a project, they don't make it in the tree. A project does not only mean feature though, just a structured way of thinking about and managing some class of fixes.
The underlying issue is we can't take random fixes and expect to ship every 3 months. We can't even do that with betas (oh yeah, I went there ha). This is likely where my release management perspective is wildly different than the development perspective.
>
>> What does development look like?
>>
>> * Decide on a feature
>
> What if you aren't working on a feature?
Have a couple of patches to make AIX work? Create a "AIX-suport" or more general "platform-support" repo, put them there, nominate it with justification. We could then decide "increased platform support" is a "feature" of a coming release and manage expectations and the landing. Have some little perf wins? Let's create a "perf-parpercuts" repo, land them there, and nominate for integration once we've aggregated a bunch. Have bug fixes that are only needed to support a feature? Fix those in the feature repo. Do multiple features depend on some fixes? Break those fixes out into their own repo, it's now a sub-feature. Have the dependent feature repos depend/merge from it while it merges from m-c/integration branches and its dependents.
Again, there will be exceptions that will land straight on the integration trains, but I'm not sure I know what those look like.
Every fix has inherent QA cost, testing cost, and schedule risk...we can't ship quarterly w/o managing those.
Christian
Ted Mielczarek
<cleg...@mozilla.com> wrote:
> Have a couple of patches to make AIX work? Create a "AIX-suport" or more general "platform-support" repo, put them there, nominate it with justification. We could then decide "increased platform support" is a "feature" of a coming release and manage expectations and the landing. Have some little perf wins? Let's create a "perf-parpercuts" repo, land them there, and nominate for integration once we've aggregated a bunch. Have bug fixes that are only needed to support a feature? Fix those in the feature repo. Do multiple features depend on some fixes? Break those fixes out into their own repo, it's now a sub-feature. Have the dependent feature repos depend/merge from it while it merges from m-c/integration branches and its dependents.
This sounds appealing on some level, but our level of process overhead
is already a problem for outside contributors. Adding this additional
level in will without a doubt be damaging to community contributors.
We have lots of people submitting patches for whom development is just
a hobby, and forcing additional process on them will just make them
stop contributing. We lose lots of people already simply by ignoring
their patches or bugs.
-Ted
L. David Baron
> The underlying issue is we can't take random fixes and expect to
> ship every 3 months. We can't even do that with betas (oh yeah, I
> went there ha). This is likely where my release management
> perspective is wildly different than the development perspective.
Do you have evidence for that? It seems to me that most of the
blockers for Firefox 4 are either regressions from large features or
things needed to make large features good enough to ship.
I think the fundamental change we need to make in order to ship
every three months is to *forbid* people from saying "we can't ship
until feature X is ready".
Boris Zbarsky
> I think that's the point. If they don't fit in with a project, they don't make it in the tree.
That involves creating some pretty vacuous "projects" like "fix web spec
compliance issues", no?
Maybe we do just need a "project branch" called "misc fixes" or
something....
> A project does not only mean feature though, just a structured way of thinking about and managing some class of fixes.
And all I'm saying is that the overhead of classifying small but
high-impact fixes would be high enough they just wouldn't happen, or
we'd have bogus classifications to work around the system.
> The underlying issue is we can't take random fixes and expect to ship every 3 months.
Well, I agree in general. But I think we're throwing the baby out with
the bathwater here. We have plenty of near-zero-risk "random fixes"
that we can take without slipping schedules. We also have plenty of
nonzero risk stuff that we should indeed merge in a more structured way.
I think we need to trust, say, our module owners to tell the two
apart. If we don't, we've already lost because they'll just work around
by creating bogus "project" branches for all the unrelated fixes in
their module. Except that will raise the bar on getting patches in for
people who don't work in one particular small area or who don't have
push access, since they or whoever deigns to look for checkin-needed
bugs will have to figure out which repo to push to (and then pull that
repo, etc; a pretty time-consuming operation).
> Have some little perf wins? Let's create a "perf-parpercuts" repo, land them there, and nominate for integration once we've aggregated a bunch.
Why do we need the aggregation? What if I just have one safe perf win?
Why should it need to wait on other things which may or may not ever
happen?
This isn't a hypothetical situation; our typical workflow for
non-feature perf fixes is someone files a bug about a particular page
being slow, then we find some one place in our code where we're doing
something stupid, we fix it. Who knows when the next such bug will come
along.
> Have bug fixes that are only needed to support a feature? Fix those in the feature repo.
Sure; this case is easy. Again, your setup works well for _features_.
I think you're underestimating the amount of non-feature work that's
done, especially by volunteers.
> Every fix has inherent QA cost, testing cost, and schedule risk...we can't ship quarterly w/o managing those.
While I agree, that cost for some fixes is very very low. I worry that
the additional process we're trying to impose on every patch would
prevent us from taking patches we really want to take and alienate
contributors.
-Boris
Mike Shaver
I don't think that taking "random fixes" is a major source of our
delays here, since they tend to be isolated and easily reversible. I
like data, though, so if there is data to be had on this stuff I'm all
ears.
I think our delays are from:
1) bad dependency management (compartments vs JM)
2) commitments to features-in-a-given-beta-number rather than
labelling once things have landed and stuck (often caused by 1)
3) taking big landings late in the beta (often caused by 2)
I think we should figure out how to better isolate the larger
features, and make sure that we don't land them on day N-1 of an N-day
beta cycle, and then revisit where we are.
If we imagine 3-week milestone cycles, of "ABB|ABB|ABB" pattern, then
in addition to a reason to change our hold music to Dancing Queen, we
would also be able to:
- schedule feature landings in only A weeks
- use B1 to decide if the feature is getting fixed or pulled for the milestone
- use B2 to finish the cleanup or pulling
- avoid day-for-day slips into the end of the milestone as larger
patches bounce: if you miss A, you're in the next A
- get QA started earlier on integration testing of larger features,
without as much risk of needing retest due to disruptive changes right
before milestone ship
- still give marketing and devrel and partners 1-2 weeks of lead time
for what will be in a given milestone
- watch crash stats for new features emerge over a week and still have
time to fix or pull cleanly
(I have a proposal for this that I'm working on in more detail, but
it's not taking shape quickly for a number of reasons.)
>> What if you aren't working on a feature?
>
> Have a couple of patches to make AIX work? Create a "AIX-suport" or more general "platform-support" repo, put them there, nominate it with justification.
I think we should separate this from managing expectations about
specific coming releases; that decoupling will be very important to
still be able to make quality threshold decisions for individual
features.
> Have some little perf wins? Let's create a "perf-parpercuts" repo, land them there, and nominate for integration once we've aggregated a bunch.
This reduces the value of our nightly testing, by clumping more
changes into a given nightly. For regression hunting (explicit or
analytic through crash reporting or similar), we want things spread
out more widely.
I don't think we need one ring to rule them all. If we get our major
work sorted out better, we'll be able to tell whether we need to do
something more structured for smaller changes as well, and learn
incrementally about what characteristics of changes need what sort of
attention.
Mike
Benjamin Smedberg
>> This seems like it would work well for new-feature kind of projects.
>> It doesn't seem at all lightweight for your basic bugfixes (for which
>> you'd first have to figure out what "project" they belong to!).
>
> I think that's the point. If they don't fit in with a project, they don't
> make it in the tree. A project does not only mean feature though, just a
> structured way of thinking about and managing some class of fixes.
From what you've described, it seems like there will be more friction. How
are we going to manage that friction so that we don't frustrate people who
are trying to fix bugs?
I'd say that normally, 60-70% of patches that land in -central fall into the
"very small" category. People don't think of them as projects with goals:
they may be part of a larger project such as "UI polish" or "AIX build
compatibility" or "clang build compatibility" or "making the DOM faster".
And perhaps more importantly, they typically don't land as entire projects.
The review cycle means that all AIX or UI polish bugs are not done at the
same time and then merge. Instead, the patches and reviews and landings
trickle in.
For many of these small changes which are typically low-risk, I think we'd
do better with a small set of continuous-integration branches modeled after
the current TraceMonkey tree, each one managed by a release driver. So the
process of getting a small, low-risk patch landed would be to ask an
appropriate release driver to pull it into their tree, and those trees would
merge at appropriate safe points as TM does now.
Larger projects which have higher risk profile could still use the
project-branch model you have described. I think that the model with test
plans and QA leads and the heavyweight process you've outlined would help
these larger/riskier projects significantly.
But my main concern is not with the landing structure, but how we manage
testing and regressions. Every large feature has regressions, and a test
plan that doesn't involve our 30k nightly users or 1M beta users isn't going
to deal with the inevitable regressions. Unless we have a strong plan to get
the nightly users on the project branch *before* it lands, or alternatively
have a strong plan for dealing with regressions or backing out project
branches when the regressions become "too serious", I'm not sure that the
project-branch model will reduce our schedule risk.
--BDS
Mike Shaver
> I think the fundamental change we need to make in order to ship
> every three months is to *forbid* people from saying "we can't ship
> until feature X is ready".
Yes, we need to be willing to kick any feature down the road. This
means that we need to be able to back out or disable pretty much
anything. That, in turn, requires better dependency understanding
than we have now, for larger stuff at least, and it probably requires
that we don't defer cleanup or niggly details for very long after an
initial feature lands.
About "feature"or "larger stuff" here: right now we have two levels of
information about FF4, roughly speaking.
1 bit: Firefox 4 is not yet ready to ship
600 bits: individual blockers
We're starting (Sheila and others) to roll up the outstanding work in
terms of features (panorama, html5 forms inputs, webgl, indexeddb,
3rd-party-add-on-stuff, jetpack, etc. etc.) so that we can reason
about 20-25 different things, and track their dependencies at that
granularity. I think this will help us reason about how close we are
to shipping (and what we would be shipping in that case), and better
focus our efforts.
There are some other scheduling and work-allocation changes that would
improve our effectiveness with these larger features (and I think also
our satisfaction with doing the work!), as well. I'm going to try to
get some of this written up over the next two weeks, to keep blood
flowing in my brain.
Mike
Axel Hecht
I don't think we should expect our project branches to land properly
localizable code, so getting some coverage is essential. How to get
that, no ideas I'm somewhat happy with. Like, this is not so much about
having machines that merge strings, but really about humans looking at
the code that lands in a timely manner and make sure the problems of
localizability are discovered early.
Axel
Henri Sivonen
>> - getting a full set of test runs before landing in a project branch
>
> Not sure this is needed? The point of doing so currently is to not block other m-c development and not subject nightly users to possible bustage. In this model those are not a concern and the test run should happen in the repo where it is planning to end up.
I think it's useful to be able to run unit tests "in the cloud" for stuff that's not "done" enough to show up in any long-term repo. Running unit tests "in the cloud" is useful, because doesn't lock up the developer's computer and block the developer from working. It is useful to be able to run unit tests without polluting the eternal history and without having to get review.
(I routinely send stuff that's not ready for review to try to get a full unit test run on one platform which is the same one as my development platform. I'm not sure if that's frowned upon as far as resource use goes, but it seems to me that it would be worse resource use for me to sit and watch while my local machine runs through all the tests.)
We'd still require review to land in a project repo, right? If we don't, the eventual merge will be so big that no one will have the time / skill / attention span to review it properly.
--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/
Robert O'Callahan
> For many of these small changes which are typically low-risk, I think we'd
> do better with a small set of continuous-integration branches modeled after
> the current TraceMonkey tree, each one managed by a release driver. So the
> process of getting a small, low-risk patch landed would be to ask an
> appropriate release driver to pull it into their tree, and those trees would
> merge at appropriate safe points as TM does now.
>
One single staging area for changes that aren't project-specific might be
good.
Echoing what other people have said, I think it's essential to have a
minimal-overhead way of getting small changes into the tree. It's essential
not only to encourage contributors, but also for our own developers to make
opportunistic improvements. If I spot a simple bug in someone else's code we
need to be able to get that patch landed without much effort, or people just
aren't going to do it and opportunities to improve the code will be lost.
Christian Legnitto
On Dec 16, 2010, at 11:54 AM, L. David Baron wrote:
> On Thursday 2010-12-16 11:32 -0800, Christian Legnitto wrote:
>> The underlying issue is we can't take random fixes and expect to
>> ship every 3 months. We can't even do that with betas (oh yeah, I
>> went there ha). This is likely where my release management
>> perspective is wildly different than the development perspective.
>
> Do you have evidence for that?
Anecdotal, sure.
Security releases are on a ~1.5 month schedule. We've had to respin at least 3 times in the last year for a bad issue. We have had to roll back changes to keep schedules decent, something that would be a lot harder to do on m-c. Fairly large regressions have come from changes considered so "safe" they had automatic approval to land. And this is with all changes metered, approved, and targeted. Perhaps this is only due to less attention on the branches though.
Beta 8 was waiting on sync changes, then crashers (which came in from changes landed waiting on sync), then feature work, then RelEng issues, etc. It's been whack-a-mole. Beta 7 felt the same but I admittedly wasn't as plugged in.
> It seems to me that most of the
> blockers for Firefox 4 are either regressions from large features or
> things needed to make large features good enough to ship.
Right. If these were being developed and integrated elsewhere we would be in a position to ship what we had now as FF4 and come back a couple months later with a FF5 once things have firmed up. Because all that work has been layered it's a huge release management and development nightmare when we go to cut things to get a release out the door.
> I think the fundamental change we need to make in order to ship
> every three months is to *forbid* people from saying "we can't ship
> until feature X is ready".
And we can't do that if the feature lands on top of other features on top of opportunity fixes. We also don't want to slow down development, so that has to overflow somewhere and the merging / integration for a release likely has to be deliberate.
Perhaps my experience has been colored by working with 100's of projects across 100's of engineering groups elsewhere though. In that case random one-off fixes are clearly unsustainable but they might not be as worrisome here. But it's my job to worry :-)
Christian
Nicholas Nethercote
> On 12/16/2010 10:59 AM, Daniel Holbert wrote:
>>
>> - getting a full set of test runs before landing in a project branch
>
> but you only hurt the people working with you (likely small).
My experience with the TraceMonkey repo is that yes, it is useful for
large patches. Even though fewer people are affected if you break a
project repo, you still want to avoid it.
Nick
Shawn Wilsher
> (I routinely send stuff that's not ready for review to try to get a full unit test run on one platform which is the same one as my development platform. I'm not sure if that's frowned upon as far as resource use goes, but it seems to me that it would be worse resource use for me to sit and watch while my local machine runs through all the tests.)
platform and the tests you want, I don't see how that's a bad thing.
Cheers,
Shawn
Paul Biggar
It depends on the project. Projects in the early stages, or being
prototyped, won't need them until they start to shape up. I suspect
there will be tons of projects in this category, and fewer in the
Tracemonkey category (at least I hope that's the case).
--
Paul Biggar
Compiler Geek
pbi...@mozilla.com
Steve Fink
> On 12/16/2010 10:59 AM, Daniel Holbert wrote:
>> - getting a full set of test runs before landing in a project branch
> Is that really needed though? If you break your branch, you can back
> out, but you only hurt the people working with you (likely small).
Custom mozconfigs.
Performance statistics gathering.
All kinds of experimentation with things you'd never want to actually
commit, but you still want information on that's hard or inefficient to
gather on your local machine. Examples: how much is fsync() costing us?
Let's just comment them all out and see. Or: what tests break when I
change this fundamental thing?
Try is awesome, and its awesomeness is not limited by always being on
the path towards eventual m-c landings.
I especially like the way that I can maintain my own (half dozen or so)
private repos and push from any of them without worrying about their
ancestry.
Chris Hofmann
> On Thursday 2010-12-16 11:32 -0800, Christian Legnitto wrote:
>> The underlying issue is we can't take random fixes and expect to
>> ship every 3 months. We can't even do that with betas (oh yeah, I
>> went there ha). This is likely where my release management
>> perspective is wildly different than the development perspective.
> blockers for Firefox 4 are either regressions from large features or
> things needed to make large features good enough to ship.
>
The difference is when you look at the beta blocking list.
In the current beta blocking list we have weeded out a lot of
the problems from "random fixes" over the past 7 betas n
the last 7 months, and most people have been working
on new features.
I bet if we looked at the composition of the beta blocking
list from last summer we see more blockers that were
associated with "random fixes" that piled up on mozilla-central
over the previous 6 month.
> I think the fundamental change we need to make in order to ship
> every three months is to *forbid* people from saying "we can't ship
> until feature X is ready".
>
Yes we need that, but we also need more betas. They help us to find
regressions from both "random fixes" and from "feature work."
Basically, they help us to get feedback on what ever we are working on.
> -David
>
Joshua Cranmer
> * Everything is done in project repos
If everything were done in a project repo, probably most of the QA would
end up being done by the developers; from experience, that will not
catch a lot of regressions. I know there have been at least a few bugs
found in JS only after tracemonkey merged to mozilla-central. Developers
catching others' bugs will take even longer: there is the distance
between the patch lands on the project, followed by the merge to
mozilla-central followed by the merge from mozilla-central to the
project repo.
Testing resources seem stretched enough to me, both in terms of
automation and in terms of human testing. I fear that requiring
everything be done in project repos would stretch this to the breaking
point.
> * After a waiting period (mainly for oranges and any missed integration work), project management will move a subset of nightly testers over on nightly-[reponame]. If we want to call attention to it, we can pop up an info bar after updates that tell people to go to an about page to learn more about the new feature, etc
> ** This means most integration and regression fallout will happen *off m-c*
Some back of the envelope calculations. Assume we have 40,000 total
nightly users. For the number of projects, my limited exposure to gecko
suggests we would have at least layout, content, sync, tracemonkey,
graphics, places, accessibility, and electrolysis repos. That comes out
to a whopping... 5,000 nightly users. For some of the projects which are
intimately related to platform differences, I would wager that number is
too low to catch most regressions.
More realistically, we would have some users on mozilla-central (since
it's possible that the merges cause regressions due to feature
confluence); given the likelihood of people taking the path of least
resistance, you would end up with barely 1,000 people on any project
branch--certainly too low to catch regressions. So I think that thinking
you'll pull the regression fallout off of mozilla-central by doing
everything in project repos is fantasy.
In addition, you'd have the disadvantage that fixes come in
poor-granularity "clumps" onto mozilla-central.
> * If needed, we can turn off auto-merging from m-c, roll back multiple features off m-c, release, roll forward, and turn back auto-merging without any development disruption. This is a HUGE win
Rolling back old features is not something we're good at right now, I
think, and I don't think separate repositories will make rolling back
any easier.
In short, I think this idea is overly optimistic as to its practicability.
Chris Hofmann
> Assume we have 40,000 total nightly users.
For planning purposes you might be able to double that number or more.
We've seen some pretty amazing growth in mozilla-central users in
the last 6 months going from around 20k to over 95k.
But this growth depends a lot on feeding these users a lot of interesting
stuff to maintain interest. Its hard to say what might happen if we made
and attempt at splitting these users off into project repo's
-chofmann
date crashes_on_all_mozilla-central_4.0b8pre
active_daily_users_on_mozilla-central-4.0b8pre
2010-12-14 5869 95036
2010-12-13 4053 93966
2010-12-12 3003 74314
2010-12-11 2802 73422
2010-12-10 3321 83412
2010-12-09 3203 85003
2010-12-08 3599 82839
2010-12-07 3755 83545
2010-12-06 3827 83761
2010-12-05 3283 68564
2010-12-04 2733 67455
2010-12-03 3323 77832
2010-12-02 3143 79306
Philip Chee
>>> * Decide on a feature
>> What if you aren't working on a feature?
>
> Have a couple of patches to make AIX work? Create a "AIX-suport" or
> more general "platform-support" repo, put them there, nominate it
> with justification. We could then decide "increased platform support"
> is a "feature" of a coming release and manage expectations and the
> landing. Have some little perf wins? Let's create a "perf-parpercuts"
> repo, land them there, and nominate for integration once we've
> aggregated a bunch. Have bug fixes that are only needed to support a
> feature? Fix those in the feature repo. Do multiple features depend
> on some fixes? Break those fixes out into their own repo, it's now a
> sub-feature. Have the dependent feature repos depend/merge from it
> while it merges from m-c/integration branches and its dependents.
So just say I'm a random outside contributor, I write small bite size
bug fixes. Today I write a tabbrowser patch, yesterday I wrote a
pageInfo patch, tomorrow I'll write a download manager patch. Then next
week I'm thinking of fix a toolbar customization bug. Pretty soon I have
a bazillion repos. The bar to outside contributors isn't as low as it
should be already. Making it harder seems sub-optimal.
Also what you are proposing appears to be a wholesale change in the way
we work. More like the linux/git style pull-requests. How would that fit
with our current bugzilla workflow?
Phil
--
Philip Chee <phi...@aleytys.pc.my>, <phili...@gmail.com>
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.
Christian Legnitto
On Dec 16, 2010, at 6:09 PM, Philip Chee <phili...@gmail.com> wrote:
> On Thu, 16 Dec 2010 11:32:35 -0800, Christian Legnitto wrote:
>
>>>> * Decide on a feature
>>> What if you aren't working on a feature?
>>
>> Have a couple of patches to make AIX work? Create a "AIX-suport" or
>> more general "platform-support" repo, put them there, nominate it
>> with justification. We could then decide "increased platform support"
>> is a "feature" of a coming release and manage expectations and the
>> landing. Have some little perf wins? Let's create a "perf-parpercuts"
>> repo, land them there, and nominate for integration once we've
>> aggregated a bunch. Have bug fixes that are only needed to support a
>> feature? Fix those in the feature repo. Do multiple features depend
>> on some fixes? Break those fixes out into their own repo, it's now a
>> sub-feature. Have the dependent feature repos depend/merge from it
>> while it merges from m-c/integration branches and its dependents.
>
> So just say I'm a random outside contributor, I write small bite size
> bug fixes. Today I write a tabbrowser patch, yesterday I wrote a
> pageInfo patch, tomorrow I'll write a download manager patch. Then next
> week I'm thinking of fix a toolbar customization bug. Pretty soon I have
> a bazillion repos.
Clearly I need to think through the scenario a bit more. But I can't help but wonder if it is optimizing for a case that rarely exists and attaching patches to bugs should cover.
Also, on my machine I have 35 cloned git repos, 7 hg repos, and 5 svn repos. I would imagine most active coder's machines look the same. Not bazillions, but I've never considered it as a concern.
> The bar to outside contributors isn't as low as it
> should be already. Making it harder seems sub-optimal.
Wholeheartedly agree.
>
> Also what you are proposing appears to be a wholesale change in the way
> we work.
Yes. We have agreed to a wholesale change to the way we release and the way we view the product. The change is prompted by the market and our current deficiencies. In my opinion if we expect to develop similarly or with small variations on the current process we are virtually guaranteed to run into the same issues we do today.
> More like the linux/git style pull-requests.
Is this a bad thing? Isn't this a benefit of a DVCS? This is an advantage we have over competitors as they are (as far as I know) mostly standardized on svn.
There may be a reason why a ton of open source development (including most other Mozilla development) has moved to GitHub and the like.
I know personally I was involved in OSS but never wrote a patch until GitHub & BitBucket made it trivial to. I don't care where it gets released, how it is merged, or how the maintainer manages their branches or releases...I just want them to have my fix and hopefully release it at some point so I see the benefit and others will as well.
> How would that fit
> with our current bugzilla workflow?
I purposely left out talking about bug processes.
timeless
<cleg...@mozilla.com> wrote:
> Clearly I need to think through the scenario a bit more. But I can't help but wonder if it is optimizing for a case that rarely exists and attaching patches to bugs should cover.
It isn't that rare. I'm in this bucket, so is BenB, so are a bunch of others.
> Also, on my machine I have 35 cloned git repos, 7 hg repos, and 5 svn repos. I would imagine most active coder's machines look the same. Not bazillions, but I've never considered it as a concern.
Something like this, swap git for hg.
>> The bar to outside contributors isn't as low as it
>> should be already. Making it harder seems sub-optimal.
+1
Robert Kaiser
> So, this is what I am thinking...
I'll just throw some random thoughts in here, don't let them scare you,
as I guess all those will resolve themselves in practice in some way.
I'm really glad I don't usually generate patches against the
mozilla-central tree or any of its clones. All I ever did there was
small patches, slight improvements of the odd thing, and already now
watching the tree for oranges after the checkin is more work than almost
any patch I ever made against this tree. With the structure you seem to
propose, I wouldn't even have a reasonable chance any more to get the
patch in myself, it seems. I'm happy we have the checkin-needed flag, as
I don't think I want to care about checking in myself any more to that
collection of clones. I think most casual contributors feel similarly.
This process gears very strongly towards heavy-contribution people and
teams, and makes the bar for casual volunteer contributors too high to
care themselves about landing. That might be good or bed, depending on
where you stand.
I also won't be able to follow day-to-day pushes as much any more. I
currently read or glance over pushes to mozilla-central every day to
know what's going on in our "main tree" and what could affect the areas
I'm working on, be it in the project I still manage for now, any
personal side projects, or anything else in Mozilla land I care about
(like bringing the open web forward through Firefox). If landing happens
in multiple different places and gets merged in large chunks to the
"main" tree, I fear reading pushlog will be more work - and it's already
a bit much right now. Though, maybe it would be good to use the time I
use to follow the checkin stream for other tasks instead. Over the
years, I have heard from a number of people who liked following those
checkin streams though to know things that happen and might affect their
own work, I wonder how this will/would affect them.
As said, those are just thoughts that come up in my head and both those
areas probably have good ways of solving them, I just wanted to throw
them in here to see if there are things I'm not seeing there or good
solutions already exist in the plans - if not, maybe we find a way to
include those.
Robert Kaiser
--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community needs answers to. And most of the time,
I even appreciate irony and fun! :)
Robert Kaiser
> Clearly I need to think through the scenario a bit more. But I can't help but wonder if it is optimizing for a case that rarely exists and attaching patches to bugs should cover.
For one thing, I think it's not _that_ rare, for the other, it's one
very critical way in defining us as an open (source) project and for
gaining new people to work with us on our success of the open browser
and web world.
That doesn't mean this case should be able to slow us down or make us
slip schedules or such, but it means we should ensure it works. And most
of those patches are small in scope and very easy to back out if they
cause problems in any way. They are low risk and low damage if they slip
into the next release by themselves, as long as we can ensure that they
can flow in.
Robert Kaiser
> If everything were done in a project repo, probably most of the QA would
> end up being done by the developers; from experience, that will not
> catch a lot of regressions.
Sounds like Linux kernel development to me. ;-)
Robert Kaiser
> I think the fundamental change we need to make in order to ship
> every three months is to *forbid* people from saying "we can't ship
> until feature X is ready".
I'm fully with you on that. It's better to ship with only 2/3 of what we
wanted finished (and those polished enough) than to delay the release
for a longer period.
Robert Kaiser
> We're starting (Sheila and others) to roll up the outstanding work in
> terms of features (panorama, html5 forms inputs, webgl, indexeddb,
> 3rd-party-add-on-stuff, jetpack, etc. etc.) so that we can reason
> about 20-25 different things, and track their dependencies at that
> granularity.
That sounds like a very great thing to have. We definitely need to get
into more modularity in the release decision process.
And it's surely not easy to balance refining the processes with
historically grown processes, openness, overall growth and being able to
act efficiently and with enough speed (i.e. not running into the
"process overhead kills swift action" dilemma). But if we work together,
I'm confident we'll find good solutions.
Robert Kaiser
> Yes we need that, but we also need more betas. They help us to find
> regressions from both "random fixes" and from "feature work."
I think we basically need to have a constant flow of development
milestones. We should not have the first Alpha a few months after having
stuff landing and producing only nightlies, I think we could start the
two-week milestone cycle (or something like it) as soon as the stable
series has branched away from the development repo and go for the first
alpha in just two-three weeks after "the tree has been opened for
development", keeping those fast alpha/beta milestone cycles throughout
the release cycle.
We need to get more testers on our code earlier and update them more
often than in past release cycles (what we started with a good portion
of the Firefox 4 beta cycles was a very good idea). And nowadays we have
a great build/release infrastructure that allows us to do that, so let's
use it. :)
Robert O'Callahan
<cleg...@mozilla.com>wrote:
> On Dec 16, 2010, at 6:09 PM, Philip Chee <phili...@gmail.com> wrote:
> > So just say I'm a random outside contributor, I write small bite size
> > bug fixes. Today I write a tabbrowser patch, yesterday I wrote a
> > pageInfo patch, tomorrow I'll write a download manager patch. Then next
> > week I'm thinking of fix a toolbar customization bug. Pretty soon I have
> > a bazillion repos.
>
> Clearly I need to think through the scenario a bit more. But I can't help
> but wonder if it is optimizing for a case that rarely exists and attaching
> patches to bugs should cover.
>
It's not that rare. Even our own developers do this. Also, many of our
strongest contributors, including people we've hired, started off creating
small patches to a few different components. We need to keep the bar to that
low. I think we can.
In my opinion if we expect to develop similarly or with small variations on
the current process we are virtually guaranteed to run into the same issues
we do today.
Maybe so but I think we will need to be specific about what those "same
issues" are. The argument "this is too similar to what we already do,
therefore it can't work" would not be specific enough, if anyone were to
make it.
Ehsan Akhgari
branches is an extremely bad idea, because of two reasons:
1. Moving everything into scattered project branches means that the much
valuable testing that we get from nightly users will be split across
different projects, which means that some project will get much less
testing than needed, and this will cause pain when the project is
finally merged into mozilla-central.
2. Products beside Firefox do not get any kind of testing from the
changes in the product branch until the changes are merged to
mozilla-central. While this may be OK for a project branch focusing on
Firefox specific UI changes for example, it's harmful for almost every
Core change, for example. I'm specially worried about Fennec, which
currently has very little (if any) automated testing to the best of my
knowledge, and is relying on manual testing from nightly users. (And
yes, I'm speaking from experience!)
To expand on (1) above, I think we're underestimating the amount of
people who are willing to turn an experimental feature on and provide
test data. I can think of at least two very successful project which
landed their feature preffed off by default, and relied on testers to
turn it on manually and provide feedback and bug reports: the HTML5
parser, and the hardware acceleration work in graphics. And that has
all happened with the horrible UI we have for turning features on
(about:config). I would imagine that a larger number of nightly testers
would be willing to try this if we provided a simple to use UI which
would let them choose which experimental features they want to turn on.
And doing things this way has also the advantage of enabling us to just
turn a feature off if it's not ready to make it into a release.
To summarize, I don't think that we should start using project branches
for everything just because we have good tools which enables us to do
that (DVCS, etc.) I think whether or not any work is better done in a
project branch or on mozilla-central preffed off by default should be
evaluated on a case by case basis, considering the benefits and harms of
each possible option.
Cheers,
Ehsan
Robert O'Callahan
people seem to agree are essential to reliably ship time-based releases on
schedule:
1) No feature can ever block a release. Self-explanatory.
2) Everything that lands on a branch that leads to a release needs an exit
strategy: a way to remove or disable it if it causes regressions that block
the release.
The exit strategy could be compile-time flags, or run-time flags, a full or
partial backout, or a disabling patch. Backout is the default strategy ---
backouts are always possible, they're just arbitrarily painful. From
thinking about it more, and talking to other developers, I think I was wrong
in my earlier reply to Shaver; even for refactorings a reasonable exit
strategy will usually be available if we plan for it in advance.
smaug
> On Fri, Dec 17, 2010 at 9:08 AM, Benjamin Smedberg<benj...@smedbergs.us>wrote:
>
>> For many of these small changes which are typically low-risk, I think we'd
>> do better with a small set of continuous-integration branches modeled after
>> the current TraceMonkey tree, each one managed by a release driver. So the
>> process of getting a small, low-risk patch landed would be to ask an
>> appropriate release driver to pull it into their tree, and those trees would
>> merge at appropriate safe points as TM does now.
>>
>
> One single staging area for changes that aren't project-specific might be
> good.
>
> Echoing what other people have said, I think it's essential to have a
> minimal-overhead way of getting small changes into the tree. It's essential
> not only to encourage contributors, but also for our own developers to make
> opportunistic improvements. If I spot a simple bug in someone else's code we
> need to be able to get that patch landed without much effort, or people just
> aren't going to do it and opportunities to improve the code will be lost.
>
Totally agree with this. Many or perhaps even most of my patches are
small changes in different places in the the code base.
For any other, even just a bit larger changes, using a separate repo
would be ok.
I assume I'll be using a repo for DOM 3 Events related work, and E10s
will happen in its own repo.
-Olli
> Rob
Robert O'Callahan
> Totally agree with this. Many or perhaps even most of my patches are
> small changes in different places in the the code base.
> For any other, even just a bit larger changes, using a separate repo
> would be ok.
> I assume I'll be using a repo for DOM 3 Events related work, and E10s
> will happen in its own repo.
>
Even if the work is done in a repo, we need an exit strategy in case we need
to back it out after it's landed on trunk.
"Back it all out" is might be the strategy. Very painful, but that's an
incentive to minimize the possibility of having to do it :-) (by doing more
testing, or by providing a switch).
Steve Fink
wanted to give my input on some of the concerns that I believe are
resolvable. So I've made
https://wiki.mozilla.org/Multiple_Project_Repositories to archive the
conversation around using more project-specific repositories feeding
into a central repo.
Depending on what you want to say, please consider updating that page.
(The discussion is probably still better kept here, but it'd be nice to
have a better record of the outcome in the wiki.)
timeless
add a NPOV tag to your article.
We aren't just talking about one off contributors. it was repeatedly
noted that we're talking about some people who have been involved with
mozilla for extended periods of time.
You don't have the word 'regression' anywhere in the page (assuming
Safari can be trusted).
You've also made some assumptions about how project repos will work.
afaiu, 'Places' merges fairly rarely. If I were a downstream (oh, I
am), I wouldn't want to pull from m-c when it had lots of random
orange, in which case I wouldn't be able to pull for months at a time
(oh, that's about right).
Nicholas Nethercote
Steve made a decent start at summarizing this thread. I'll politely
suggest that we don't crap on him for his efforts. It's a wiki page,
anyone else can edit it if they are unhappy with it.
Nick
Dan Mosedale
and not something that a lot of people are willing to invest the time in
doing. Thanks muchly, Steve!
Dan
Steve Fink
> Sorry. While I'm not a fan of WP, I think I'd like to request that you
> add a NPOV tag to your article.
but all I see are various POV tags. Are you saying that the article is
or is not neutral? (Do we even use those sorts of tags on the moz wiki?
I didn't think so.)
I tried to be fairly neutral, though it wasn't an overriding concern.
People are free to add their opinions (preferably in the form of factual
statements).
> We aren't just talking about one off contributors. it was repeatedly
> noted that we're talking about some people who have been involved with
> mozilla for extended periods of time.
Fair point. But I don't understand what this implies other than lots of
repos on your local machine. Perhaps you disagree with my opinion that
this isn't a big deal, since you "ought" to have a repo per type of
change anyway? My argument probably doesn't hold up on Windows, since I
don't think hg does any kind of hard-linking equivalent.
Whatever it is, can you describe your specific concern? The only thing I
see in the thread is Philip Chee's worry about having "a bazillion
repos". But like I said, it's gotten too hard to trawl through the whole
thread for these sorts of things. Is the repo explosion your concern, or
something else? I'd like to get it into the wiki.
> You don't have the word 'regression' anywhere in the page (assuming
> Safari can be trusted).
The positive side is somewhat implicit in the "experimental changes"
bullet point -- regressions that show up in a project repo are easier to
spot and track down. Then again, some regressions will result from
merges and those will be harder than now. But you're right that
regressions should be explicitly discussed. Do you want to take a stab
at it? If not, I'll do it, and you'll probably disagree. :-)
> You've also made some assumptions about how project repos will work.
> afaiu, 'Places' merges fairly rarely. If I were a downstream (oh, I
> am), I wouldn't want to pull from m-c when it had lots of random
> orange, in which case I wouldn't be able to pull for months at a time
> (oh, that's about right).
Absolutely right. I started out with no description at all, just the
"issues" section, and right now it's still intentionally vague. I don't
think we all agree at all on what project repos would look like; there
are a bunch of different models in use out there, and I suspect
different people are assuming different ones. We'd also want to figure
out how/how fast to migrate.
I don't have the brain cells to take a stab at it right now, but some of
the axes of choice are:
- Where does dev happen? All on a project repo, all in personal repos
that push but never/sometimes merge into the project repo, all on
mozilla-central, split across the various possible repos: project repos,
project-specific "urgent" repos (these changes go straight to m-c), m-c,
etc.
- Does the above change after a beta branches?
- How frequently does a project repo merge into a (one of the?) central
repo(s)?
- How frequently does a project repo merge changes back from a central
or other repo?
- Does everything happen between project repos and a central repo, or
are we going to have a deeper hierarchy? (If you add personal repos to
the picture, it'll already be multi-level.)
- Who does the merging? (When it goes right, and then when it goes
wrong?) Any automated merging going on?
- How does the bugzilla-related workflow change?
That's a lot of questions, and I'm sure there are more, but I don't
think we're actually considering very many possibilities for the main
configurations. Anyone up for putting together a concrete proposal or three?
Steve Fink
> Steve made a decent start at summarizing this thread. I'll politely
> suggest that we don't crap on him for his efforts. It's a wiki page,
> anyone else can edit it if they are unhappy with it.
>
> Nick
mentioned it. I guess my skin has been thickened by a history of open
source development.
No worries. I don't think timeless was intending to be rude.
Chris Hofmann
> From the discussions yesterday I think we have two key principles most
> people seem to agree are essential to reliably ship time-based releases on
> schedule:
>
> 1) No feature can ever block a release. Self-explanatory.
> 2) Everything that lands on a branch that leads to a release needs an exit
> strategy: a way to remove or disable it if it causes regressions that block
> the release.
>
> The exit strategy could be compile-time flags, or run-time flags, a full or
> partial backout, or a disabling patch. Backout is the default strategy ---
> backouts are always possible, they're just arbitrarily painful. From
> thinking about it more, and talking to other developers, I think I was wrong
> in my earlier reply to Shaver; even for refactorings a reasonable exit
> strategy will usually be available if we plan for it in advance.
>
> Rob
Yeah, we have some history of being able to do this.
I put some analysis of how it work a long time ago here
https://wiki.mozilla.org/User_talk:ChrisHofmann/QuarterlyReleases
bi-di hardly caused a hickup in the monthly release cycles of 2001
coming in from well baked from a project branch.
libpr0n might have been a bit harder, but didn't disrupt a faster pace
of releases that much.
There can be a whole series of things that can come well baked off branches.
-chofmann