Proposed post-Fx4 rapid release process
Robert Sayre
I've been working over the past few weeks on a high-level plan to get
Mozilla Firefox on a more rapid release schedule. It is currently in the
draft stage.
http://people.mozilla.com/~sayrer/2011/temp/process.html
That means nothing is final, and we'll be updating the document as we
gain experience.
The project has been talking about switching to a schedule-driven
release process for a while, and this document is the first stage of
that transition. Let's work together to iron out any bugs in the document.
- Rob
Dave Townsend
This looks really great and the latest draft answers basically all the
questions I was going to ask but there is one aspect that concerns me a
little.
I'm a little unclear on how updates are served to users on the
experimental and beta channels. Will they also pull nightly updates or
will we just trigger updates when we think necessary? My main concern is
that as you've mentioned the first few days after new code is pulled
into a channel will be quite choppy as things that are clearly not ready
are turned off. I don't think it is a good idea to actually push these
changes out to the users of those channels until at least a first pass
has been made through this to avoid too much disruption.
I guess this also raises another concern. As well as being able to
easily turn off features we should consider what the result is of a user
running with the feature one day and then not the next, maybe reverting
to an older version. For example a change to the places DB schema
introduced by a feature might have to be rolled back when the feature is
turned off or the user may lose data. Should we be thinking about
including requirements for downgrading data like this and do we need
criteria for what sorts of dataloss are allowable across the channels
for cases like this?
Chris Jones
> Let's work together to iron out any bugs in the document.
- Does "extension compatibility" (whatever that may end specifically
meaning) include binary extensions? Would we disable a new feature if
some extension(s) weren't made compatible with it in time for scheduled
release?
- How does fennec fit in?
Cheers,
Chris
Benjamin Smedberg
> On 03/16/2011 02:06 PM, Robert Sayre wrote:
>> Let's work together to iron out any bugs in the document.
>
> - Does "extension compatibility" (whatever that may end specifically
> meaning) include binary extensions? Would we disable a new feature if
> some extension(s) weren't made compatible with it in time for
> scheduled release?
release. In particular, I expect that we will revise the XPCOM binary
component version number for pretty much every release.
--BDS
Justin Dolske
> Should we be thinking about
> including requirements for downgrading data like this
My assumption is that's an implicit requirement of having features that
can be disabled/reverted after landing. Probably worth calling out
explicitly, though.
> and do we need
> criteria for what sorts of dataloss are allowable across the channels
> for cases like this?
That's an interesting question. It's probably hard to create formal
criteria, but IMO the general guideline should be no loss of data that
has end-user impact. EG, we clearly can't ever lose bookmarks or
passwords, but transient stuff (like cache) would be fine, and other
things on a case-by-case basis.
The specific channels don't matter... If something lands on
mozilla-central, it should have the expectation of being able able to
make it to a final release as-is (at least w.r.t. data formats, schemas,
and upgrade/downgrade plans). EG, it would be unacceptable to land a
Places change on mozilla-central that was unable to roll-back to the
previous beta.
Justin
Sheppy
> Hi Everybody,
>
> I've been working over the past few weeks on a high-level plan to get
> Mozilla Firefox on a more rapid release schedule. It is currently in the
> draft stage.
>
> http://people.mozilla.com/~sayrer/2011/temp/process.html
>
> That means nothing is final, and we'll be updating the document as we
> gain experience.
My primary concern is that this rapid and more fluid development
process is going to make it much harder to have thorough developer
documentation complete by release time, given that we won't
necessarily know which features will arrive in which releases. There
will almost certainly be gaps in the documentation at release time.
We're going to need a lot of added cooperation from developers to
ensure that we can get as much documentation as possible done for each
release:
* Some developers are better than others at remembering to use
comments well in their code; this needs to become more universal, to
cut down the amount of time it takes to write things up.
* There should be more consistency in design documentation. Having
good and well-maintained design documents and notes makes documenting
new features and APIs go much more quickly.
* We're going to need good communication as to the likelihood of given
features of shipping in specific releases, in order to prioritize
documentation work.
Basically, what it comes down to is this: anything that can be done to
reduce the amount of time writers spend doing research is going to
make a huge difference in terms of documentation quality and
completeness at the time of each release.
Eric Shepherd
Developer Documentation Lead
Mozilla
http://www.bitstampede.com/
Christian Legnitto
On Mar 16, 2011, at 1:04 PM, Dave Townsend wrote:
> On 3/16/11 12:06, Robert Sayre wrote:
>> Hi Everybody,
>>
>> I've been working over the past few weeks on a high-level plan to get
>> Mozilla Firefox on a more rapid release schedule. It is currently in the
>> draft stage.
>>
>> http://people.mozilla.com/~sayrer/2011/temp/process.html
>>
>> That means nothing is final, and we'll be updating the document as we
>> gain experience.
>>
>> The project has been talking about switching to a schedule-driven
>> release process for a while, and this document is the first stage of
>> that transition. Let's work together to iron out any bugs in the document.
>
> This looks really great and the latest draft answers basically all the questions I was going to ask but there is one aspect that concerns me a little.
>
> I'm a little unclear on how updates are served to users on the experimental and beta channels. Will they also pull nightly updates or will we just trigger updates when we think necessary? My main concern is that as you've mentioned the first few days after new code is pulled into a channel will be quite choppy as things that are clearly not ready are turned off. I don't think it is a good idea to actually push these changes out to the users of those channels until at least a first pass has been made through this to avoid too much disruption.
The specifics are being determined. The current plan is to have users getting nightly builds if there are any changes to the repository. Of course, this is very much dependent on silent updates (which are part of the plan).
Additionally we are planning to be able to freeze and unfreeze nightlies easily, as the preffing off and backing out happens not on mozilla-central but the channel repos and as you indicate the first couple of days might be choppy. We will likely automate the freezing/unfreezing of nightlies during the pref off period as the clone from mozilla-central to experimental will always happen on schedule. We likely don't need to worry from experimental to beta.
We intend to manage features / work as projects better, so it will generally be clear which are of sufficient quality to stay on for additional testing and which are preffed off. Any doubt in that discussion leads to a pref off / backout.
Christian
Dave Townsend
> On Mar 16, 3:06 pm, Robert Sayre<say...@gmail.com> wrote:
>> Hi Everybody,
>>
>> I've been working over the past few weeks on a high-level plan to get
>> Mozilla Firefox on a more rapid release schedule. It is currently in the
>> draft stage.
>>
>> http://people.mozilla.com/~sayrer/2011/temp/process.html
>>
>> That means nothing is final, and we'll be updating the document as we
>> gain experience.
>
> My primary concern is that this rapid and more fluid development
> process is going to make it much harder to have thorough developer
> documentation complete by release time, given that we won't
> necessarily know which features will arrive in which releases. There
> will almost certainly be gaps in the documentation at release time.
> We're going to need a lot of added cooperation from developers to
> ensure that we can get as much documentation as possible done for each
> release:
I wonder if MDC needs a feature where you can easily "turn off"
documentation for a particular feature as it gets turned off on the
channels until it finally ships
Dave Townsend
That means though that often as well as writing the feature and making
it possible to turn it off you'll have to develop code to roll back the
data, presumably that code will get turned on if the feature is turned
off which seems a little odd to me. Either that or the new feature needs
to write its data in a way the old code can understand at least until
late in its beta cycle but that seems like an ever worse idea.
smaug
"That means each release happens regardless of whether a given feature
is ready, and releases are not delayed to wait for a feature to stabilize."
That sounds like releases could contain features which need still
stabilization. Perhaps there could be some rewording, since
later it is said "features that aren't quite ready are disabled."
But anyway, looks great!
-Olli
Sheppy
We have always documented things from time to time long in advance of
shipping; as long as stuff is marked as to which version it applies
to, having it around isn't a problem. The actual problem I'm talking
about is making sure we choose to document the stuff most likely to
ship, so that we're not stuck having spent all our time writing about
things that won't ship until a future release while not getting
written documentation that covers features shipping earlier.
Robert O'Callahan
"there is a choice to ship a general Firefox release at week 16 and every
six weeks thereafter."
Is our goal to release every six weeks? Would we intentionally drive a set
of changes through fx-exp and fx-beta not expecting to release them?
"The code on mozilla-central is pulled to the firefox-experimental channel."
I assume this means that the new firefox-experimental tip is exactly what's
in mozilla-central when we pull --- is that correct? In other words, changes
in firefox-experimental are dropped at the next pull?
Thanks,
Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
Mark Finkle
> - How does fennec fit in?
I think we need to work under the assumption that Fennec's "mobile"
folder lives in mozilla-central. I don't know if it makes sense for
Fennec and Firefox to share the same "experimental" and "beta"
repositories, but keeping the trunk code together will simplify a lot
of current issues.
Henri Sivonen
I like the plan!
I have questions about what happens before code lands in m-c:
Assuming that I'm working with a project repo, would review occur before landing to the project repo?
Would approval happen when merging a project repo into m-c? If so, how will requesting the approval (in practice for multiple changesets in the project repo history) work?
At last All Hands, the idea of having a merge bot that'd clone a project repo, merge m-c into it and push it back to the project repo if the automerge was clean or email the project repo users otherwise was mentioned. Is that kind of a merge bot part of the short-term plan of tooling to support the new release schedule?
For refactorings, the policy of having to be able to disable a feature is rather inconvenient and one might plan to use backout as the contingency plan. To lower the probability of having to back out after landing code in m-c, it would be nice to be able to do small-scale testing with builds from a project repo over a period spanning multiple builds. I'd love to be able to recruit a handful (say, 10) volunteers who'd run nightly-to-weekly builds of a parser refactoring repo (such repo doesn't exist yet) so that the volunteers would get an update channel that makes the builds self-update to new builds from the repo. Are there short-term plans to enable anything like this? (I recall an earlier idea of occasionally feeding the nightly channel from a project repo for testing.)
Oh, and one question about what happens after landing to m-c:
Suppose a refactoring (that can't be just turned on and off with a switch) has landed on m-c but at the experimental or beta stage, it's decided to undo it. My understanding is that backing out the merge node that merged the project branch into m-c would leave a dangling head and those have been previously frowned upon. Would they be more acceptable in the experimental and beta repos? If now, how would backing something out from experimental or beta work?
--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/
Robert O'Callahan
> Assuming that I'm working with a project repo, would review occur before
> landing to the project repo?
>
I hope the answer is yes!
Mark Banner
> http://people.mozilla.com/~sayrer/2011/temp/process.html
I'm confused about FF 4.0 - will there be security updates or not? The
document implies there could be, but from what I've been hearing on this
newsgroup, there probably won't be, but there may be just one 4.0.1 to
address some issues? A clear statement would be useful (even if its a we
don't actually know yet).
I also wonder about L10n and security releases. We already know that
L10n can miss releases (e.g. FF 4). So if we're in this new cycle of
releases, and for whatever reason L10n misses a release which also
contains security updates, what are we going to do about getting those
security updates to people in that locale? Do the people on that locale
have to download English, or wait six weeks?
Standard8
Jason
> To lower the probability of having to back out after landing code in m-c, it
> would be nice to be able to do small-scale testing with builds from a project
> repo over a period spanning multiple builds. I'd love to be able to recruit a
> handful (say, 10) volunteers who'd run nightly-to-weekly builds of a parser
> refactoring repo (such repo doesn't exist yet) so that the volunteers would
> get an update channel that makes the builds self-update to new builds from the
> repo.
I think allowing project repositories to have some slice of the
nightly (or even experimental) user base is potentially a very good
idea. Some of our code (such as networking) is hard to fully test
with automated testing, and often needs real world use to shake out
all bugs. Getting such user coverage *before* landing in m-c has a
number of benefits. Of course, it would also have costs in terms of
complexity. Here's some thoughts on the tradeoffs:
BENEFITS
-----------------
1) Fewer backouts.
When user coverage reveals a bug with a patch to a project repo build,
it doesn't need to be backed out of m-c. There's also potentially a
much larger window on the project repo to implement a "fix rather back
out" solution, especially if users can be migrated back to the regular
nightlies in the meantime. (The size of this window would vary by
repo: busy projects like jaegermonkey would probably back out almost
as quickly as m-c: repos with fewer developers could take more time
to fix rather than backout.)
2) Makes decision-making easier for drivers.
Developers on project repos would no longer need approval to land on m-
c in order to get user coverage. Fewer decisions for drivers to make
== big win.
When developers think their project repo is ready to merge to m-c,
drivers would actually have some user coverage history to base their
approval decision on. Also a big win.
3) Allow "land any time and get user coverage."
It seems likely that toward the end of the 6 week m-c cycle, approvals
to land on m-c will become harder to get, especially for big or risky
changes (unless I'm misinterpreting the plan). This is likely to set
up a situation where such changes can only land 50-60% of the time,
toward the start of each cycle. With user coverage for project
repos, developers can get test coverage whenever they want, instead of
waiting for 2-3 weeks.
This has synergy with #2, in that a patch that lands on a project repo
in week 5 and has a week of bug-free user coverage would be a *lot*
more likely to be taken by drivers on m-c in week 6.
COSTS
--------------
1) Additional complexity of having multiple nightly builds.
I'm no expert on our nightly build infrastructure, so I don't know how
hard it would be to automate auto-updating project repo builds.
2) Work to avoid risking user profiles/data
We'd obviously want to make sure that moving nightly users to a
project repo (and then eventually back to regular nightlies) would not
break anything like bookmarks, db schemas, etc. And we'd rather not
mess up any add-ons or extensions they're using, and I'm not sure what
sort of versioning/compat scheme for them would make sense.
Jason
Robert Kaiser
> On 16/03/2011 19:06, Robert Sayre wrote:
>> http://people.mozilla.com/~sayrer/2011/temp/process.html
>
> I'm confused about FF 4.0 - will there be security updates or not? The
> document implies there could be, but from what I've been hearing on this
> newsgroup, there probably won't be, but there may be just one 4.0.1 to
> address some issues? A clear statement would be useful (even if its a we
> don't actually know yet).
I agree, and from the data I've seen so far, esp. quite high-profile Mac
and Linux crashes, I'd really hope there will be a 4.0.1 in a few weeks
that roll in the fixes we have on our "rc2-ride-along" list but
obviously stay off right now due to not doing an RC2. I understand that
everything non-Windows is too low in volume to trigger a respin for RC,
but we shouldn't leave the other OSes too much in the cold there.
I'm also fully for releasing what we have, but we can't treat FF4 like
we plan to treat later releases, as this one has a different development
cycle behind itself and didn't have a few weeks on a super-stable
release-ready branch with 10 million testers as we hope to have in the
future.
Also, we are doing security updates for 3.6 and 3.5, and I wonder if we
need to do a 4.0.1 at the same time anyhow to make sure that 4.0 users
get the same set of security fixes - or do we have all of them in 4.0
proper already?
In addition to that, we know that we'd gain an interesting amount of
additional first-language coverage if we ship the vi locale for 4.0, and
other locales have string improvements that would be good to be shipped
as well.
All in all, I think doing a mostly "regular" 4.0.1 would be a good thing
for all those reasons above - but then, it's a decision the release
drivers need to make, I'm just adding my 2c here.
And, I'm all for not doing any more 4.0.x updates once FF5 is here - but
I think that's still a few weeks off right now. ;-)
Robert Kaiser
Benjamin Smedberg
> I think allowing project repositories to have some slice of the
> nightly (or even experimental) user base is potentially a very good
> idea. Some of our code (such as networking) is hard to fully test
> with automated testing, and often needs real world use to shake out
> all bugs. Getting such user coverage *before* landing in m-c has a
> number of benefits.
attempt this. We will obviously continue to have nightly builds of
project branches for regression-tracking purposes, but trying to segment
the nightly population will sow significant confusion and dilute the
testing we want on the primary integration branch (mozilla-central).
--BDS
Henri Sivonen
> project branches for regression-tracking purposes,
Will these builds be on a per-project branch update channel, or will people have to update branch builds manually?
Ted Mielczarek
<benj...@smedbergs.us> wrote:
> On 3/17/2011 9:01 AM, Jason wrote:
>>
>> nightly (or even experimental) user base is potentially a very good
>> idea. Some of our code (such as networking) is hard to fully test
>> with automated testing, and often needs real world use to shake out
>> all bugs. Getting such user coverage *before* landing in m-c has a
>> number of benefits.
>
> branches for regression-tracking purposes, but trying to segment the nightly
> population will sow significant confusion and dilute the testing we want on
> the primary integration branch (mozilla-central).
I agree that fragmenting the nightly userbase is not desirable. What
about allowing project branches to take over the nightly channel
entirely for short periods of time? This would give wider distribution
for their changes, but if we find blocking issues we can simply not
merge the project branch instead of having to figure out how to back
out something scary.
The downside is that we lose mozilla-central nightly tester coverage
during that period of time. This could be mitigated by requiring the
project branch to take daily merges from m-c while it's getting the
nightly exposure.
-Ted
Mike Beltzner
> I agree that fragmenting the nightly userbase is not desirable. What
> about allowing project branches to take over the nightly channel
> entirely for short periods of time? This would give wider distribution
> for their changes, but if we find blocking issues we can simply not
> merge the project branch instead of having to figure out how to back
> out something scary.
It's more than just fragmentation of the nightly userbase, its fragmentation of the code. Project branches are a great way to allow small, focused teams to kickstart and make large, sweeping iterations of their approaches and designs, but the goal should be to arrive at "chunks" that can be integrated with the main codebase (behind compile-time or runtime switches and prefs at first!) so that the nightly audience can test not just the changes from that project, but how those changes interact with the entire codebase.
The more I think about this, the less I think we want to enable our testing audience to move over to project branches. Those builds should be for people who consider themselves (as developers, testers, or interested community members) to be part of a specific project team. People who are interested in helping develop the product as a whole should stay on nightly builds. This is what we've done in the past with project branches like JagerMonkey, and it's worked very well.
cheers,
mike
Boris Zbarsky
> The more I think about this, the less I think we want to enable our testing audience to move over to project branches. Those builds should be for people who consider themselves (as developers, testers, or interested community members) to be part of a specific project team. People who are interested in helping develop the product as a whole should stay on nightly builds. This is what we've done in the past with project branches like JagerMonkey, and it's worked very well.
I think Jagermonkey worked ok. I'm not convinced about "very well". In
particular, precisely the kinds of major restructuring we are talking
about here (e.g. compartments) didn't work out that well in the JM model.
That said, I think we should start with Henri's question: will we have
separate update channels for project branches, so that if someone wants
to track the project branch nightlies they can at least easily do that?
-Boris
Benjamin Smedberg
> That said, I think we should start with Henri's question: will we have
> separate update channels for project branches, so that if someone
> wants to track the project branch nightlies they can at least easily
> do that?
doubt that they will be mentioned anywhere such as the new update-stream
chooser in the prefpane.
--BDS
Boris Zbarsky
> We currently do, and I don't think that's going to change.
Ah, excellent.
> I very much doubt that they will be mentioned anywhere such as the new update-stream
> chooser in the prefpane.
That's probably fine. When a project branch closes down, are we able to
migrate users on that update channel to m-c?
-Boris
Mike Beltzner
> That's probably fine. When a project branch closes down, are we able to migrate users on that update channel to m-c?
Yes, we control that particular horizontal and vertical :)
(For those that don't know, there are three bits to updating: the client, which checks for updates, the update server which issues update "snippets" that tell the client where to fetch update files, and the updater which applies them. The clients use a predictable URI structure when checking for updates that includes information about what their current version is, what their buildID is, what their architecture is, etc. The server can be programmed to return snippets based on the URI that is being asked for, and those snippets can tell the client to get updates from any location and either apply them automatically or ask the user if they should be applied. The updater then applies the update that it sees.)
cheers,
mike
Benjamin Smedberg
--BDS
Boris Zbarsky
> On 3/17/11 11:28 AM, Mike Beltzner wrote:
>> The more I think about this, the less I think we want to enable our
>> testing audience to move over to project branches. Those builds should
>> be for people who consider themselves (as developers, testers, or
>> interested community members) to be part of a specific project team.
>> People who are interested in helping develop the product as a whole
>> should stay on nightly builds. This is what we've done in the past
>> with project branches like JagerMonkey, and it's worked very well.
>
> I think Jagermonkey worked ok. I'm not convinced about "very well".
I was a little confused here between JM and TM and stuff. ;)
The JM model worked great. But JM was an additive change, not a
restructuring of existing code. Therefore it was quite possible to put
it behind a pref, with some work. That was the main saving grace, by
the way: JM as it landed on m-c could not possibly ship, at least partly
due to lack of testing on the JM branch.
I think for code restructuring we'll end up needing to "manually"
recruit testers and rely on the completeness of our automated testing....
-Boris
Henri Sivonen
> On 3/17/2011 11:49 AM, Boris Zbarsky wrote:
> > That said, I think we should start with Henri's question: will we
> > have
> > separate update channels for project branches, so that if someone
> > wants to track the project branch nightlies they can at least easily
> > do that?
Great!
Are there minimum project size requirements for getting a project branch?
Mike Beltzner wrote:
> The more I think about this, the less I think we want to enable our
> testing audience to move over to project branches. Those builds should
> be for people who consider themselves (as developers, testers, or
> interested community members) to be part of a specific project team.
I'm not thinking of a feature with a team with QA staff or multiple developers. Just one developer (me) rewriting stuff around parsers and a few (maybe 10) interested people to run builds to get slightly more real-world usage coverage than my own dogfooding before merging onto m-c.
Christian Legnitto
On Mar 17, 2011, at 9:43 AM, Henri Sivonen <hsiv...@iki.fi> wrote:
> Benjamin Smedberg wrote:
>> On 3/17/2011 11:49 AM, Boris Zbarsky wrote:
>>> That said, I think we should start with Henri's question: will we
>>> have
>>> separate update channels for project branches, so that if someone
>>> wants to track the project branch nightlies they can at least easily
>>> do that?
>> We currently do, and I don't think that's going to change.
>
> Great!
>
> Are there minimum project size requirements for getting a project branch?
No. RelEng also recognizes that projects repos are cheap, and we'd rather have more work on project repos than less.
Also, there will still be the three rentable repos for anyone to grab as they see fit.
>
> Mike Beltzner wrote:
>> The more I think about this, the less I think we want to enable our
>> testing audience to move over to project branches. Those builds should
>> be for people who consider themselves (as developers, testers, or
>> interested community members) to be part of a specific project team.
>
> I'm not thinking of a feature with a team with QA staff or multiple developers. Just one developer (me) rewriting stuff around parsers and a few (maybe 10) interested people to run builds to get slightly more real-world usage coverage than my own dogfooding before merging onto m-c.
>
> --
> Henri Sivonen
> hsiv...@iki.fi
> http://hsivonen.iki.fi/
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Ian Gilman
bug fixes that land on the more stable channels? Presumably there will
be fixes in mozilla-experimental, for instance. Do those double land
on mozilla-experimental and mozilla-central? Do fixes for mozilla-beta
triple land? Or does everything get pushed back up the chain
periodically?
Dave Townsend
I'd hope that in general (ignoring security stuff perhaps) fixes would
land on mozilla-central and bake there for a time before going to
experimental and then beta, pretty much the same as our branch policy.
Not possible in all cases but the safest approach.
Asa Dotzler
> Mark Banner schrieb:
>> On 16/03/2011 19:06, Robert Sayre wrote:
>>> http://people.mozilla.com/~sayrer/2011/temp/process.html
>
> I agree, and from the data I've seen so far, esp. quite high-profile Mac
> and Linux crashes
Mac is already more stable than Windows so I don't think we add a lot of
benefit by focusing on Mac-specific crashers.
But to the main topic of this sub-thread, yes, I believe we're very
likely going to have a 4.0.1 some short number of weeks after we ship
Firefox 4. We've never, in my long history of Mozilla releases, not
needed a follow-up to fix something that will only surface with many
millions of "regular" users. I think it's useful to start to question
the need for quick follow-up stability and security releases in the
upcoming new world of a more rapid major release process, but we haven't
achieved that new world yet and I so I suspect there will be a 4.0.1
stability and security update.
- A
Asa Dotzler
Yes, this is the expected normal process. A change will land on
mozilla-central and eventually be uplifted to the experimental branch.
If it doesn't get reverted on that branch because of some shortcoming,
then it will be uplifted to the beta branch and then on to the release
channel.
Asa Dotzler
If the channels, builds, updates, and snippets exist for project
branches, then it should be fairly easy for someone to build an
extension that adds the full available channel list to the channel
chooser prefpane, right? Then some set of dedicated nightly testers can
bounce around between project branches.
- A
Asa Dotzler
> The JM model worked great. But JM was an additive change, not a
> restructuring of existing code. Therefore it was quite possible to put
> it behind a pref, with some work. That was the main saving grace, by the
> way: JM as it landed on m-c could not possibly ship, at least partly due
> to lack of testing on the JM branch.
>
> I think for code restructuring we'll end up needing to "manually"
> recruit testers and rely on the completeness of our automated testing....
>
> -Boris
>
Perhaps a channel chooser extension that adds project channels to the
built in chooser would be a good tool to help in that recruiting effort.
Maybe the first project branch that feels it really needs the extra
testing can build that extension.
- A
Asa Dotzler
This shouldn't be difficult at all. A blog post begging for some testers
with a link to the the branch builds can probably get you a dozen testers.
- A
Christian Legnitto
Project repos and what that world looks like are out of scope for this proposal and will be tackled once the need to do so is demonstrated. Development can generally be thought of like so:
1 - project repos, user repos, GitHub, Bitbucket, svn, user repos, patch queues, whatever -> 2 - mozilla-central -> 3 - experimental -> 4 - beta -> 5 - firefox/release
This proposal only covers base conditions to go between #1 and #2 (pref off or easy-ish backout) and how to shepherd fixes from #2 to #5.
Once we get into a release cadence and different groups have tried different processes in #1 we'll loop back around and get some formal processes, tools, and best practices.
Thanks,
Christian
Robert Kaiser
> On 3/17/2011 7:53 AM, Robert Kaiser wrote:
>> Mark Banner schrieb:
>>> On 16/03/2011 19:06, Robert Sayre wrote:
>>>> http://people.mozilla.com/~sayrer/2011/temp/process.html
>
>>
>> I agree, and from the data I've seen so far, esp. quite high-profile Mac
>> and Linux crashes
>
> Mac is already more stable than Windows so I don't think we add a lot of
> benefit by focusing on Mac-specific crashers.
It might be pretty stable, but that DrawPlugin crash (that IIRC has a
fix landed on trunk and marked as "rc-ride-along" even though it's hard
to reproduce) is pretty high in topcrashers for a Mac crash.
> But to the main topic of this sub-thread, yes, I believe we're very
> likely going to have a 4.0.1 some short number of weeks after we ship
> Firefox 4. We've never, in my long history of Mozilla releases, not
> needed a follow-up to fix something that will only surface with many
> millions of "regular" users.
I know, I have been around for long enough to have lived through a
number of those, and I sincerely hope that the new development process
changes this (it surely has the potential to do so, IMHO). We're not
there with 4.0 yet, I think, though from what I'm seeing, it could very
well come out as being more stable than any other version once it goes
to the masses (we of course have a different sample of people and usage
on early adopters and testers, so it's hard to guess what what numbers
we end up when we go full-speed).
All in all, I think we're actually very much at the same page here. :)
Robert Kaiser
--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community needs answers to. And most of the time,
I even appreciate irony and fun! :)
Christian Legnitto
Let's focus on the proposal for FF5+ rather than FF4.
Thanks,
Christian
Asa Dotzler
Yes, you are correct. We're very much on the same page here.
- A
Chris AtLee
>
>
> On Mar 17, 2011, at 9:43 AM, Henri Sivonen<hsiv...@iki.fi> wrote:
>
>> Benjamin Smedberg wrote:
>>> On 3/17/2011 11:49 AM, Boris Zbarsky wrote:
>>>> That said, I think we should start with Henri's question: will we
>>>> have
>>>> separate update channels for project branches, so that if someone
>>>> wants to track the project branch nightlies they can at least easily
>>>> do that?
>>> We currently do, and I don't think that's going to change.
>>
>> Great!
>>
>> Are there minimum project size requirements for getting a project branch?
>
> No. RelEng also recognizes that projects repos are cheap, and we'd rather have more work on project repos than less.
>
> Also, there will still be the three rentable repos for anyone to grab as they see fit.
And we can spin up more fairly quickly if needed. There hasn't been much
contention for the 3 we have now.
Jonathan Watt
I think Ian's question was about what Robert's document calls "spot fixes".
Should such a spot fix that's written for fx-beta land in fx-exp as well as m-c,
or just in m-c?
Dave Townsend
I think we'd want as much testing as possible on any fix landing on the
fx-beta channel out of the normal cycle so I'd say yes.
Chris Jones
> On 3/16/2011 4:08 PM, Chris Jones wrote:
>> On 03/16/2011 02:06 PM, Robert Sayre wrote:
>>> Let's work together to iron out any bugs in the document.
>>
>> - Does "extension compatibility" (whatever that may end specifically
>> meaning) include binary extensions? Would we disable a new feature if
>> some extension(s) weren't made compatible with it in time for
>> scheduled release?
> We should expect that extensions may need to be updated for each
> release. In particular, I expect that we will revise the XPCOM binary
> component version number for pretty much every release.
We're post-FF4 and it's been about a week, so I'll gently press for a
bit more information.
Here's what I'm worried about: First, any semblance of binary-extension
compatibility implies a vtable freeze at some point in the dev cycle.
This adds another dimension of complexity to escape plans: features must
be backout-or pref-off-able, but those changes must also not change
vtables. I think this might send us to a hell of MOZILLA_N_BRANCH or
FEATURE_X_BRANCH or lots of dummy impls. Maybe there are good ideas on
how to manage this. We could avoid this issue entirely by declaring
that binary extensions are on their own and, like core developers, might
need to deal with features being disabled (viz. vtable changes) all the
way up to release. (I thought this had been planned for a while,
actually, but maybe not.)
Second, if a feature in a new release breaking any extension is cause
enough to disable the feature, another dimension of complexity will be
added to the development of major features (here I have e10s in mind).
We will need a first step, in release N, in which we provide the new
interfaces for the new feature, but with a "trivial" impl, alongside the
old/deprecated and encourage extensions to migrate to the new interface
(here I'm thinking, e.g., of in-process messager). Hopefully at release
N+1, but maybe N+k, a quorum of extensions are upgraded, and we swap out
the trivial impl for the real new feature. This isn't a theoretical
problem, but rather one we're going to face soon. I don't have any
answers but I think a policy is direly needed.
Cheers,
Chris
Benjamin Smedberg
> Here's what I'm worried about: First, any semblance of
> binary-extension compatibility implies a vtable freeze at some point
> in the dev cycle.
push experimental to beta.
> This adds another dimension of complexity to escape plans: features
> must be backout-or pref-off-able, but those changes must also not
> change vtables.
We expect that features which are disabled will be disabled primarily
when we move mozilla-central to -experimental, so I really hope that we
don't have this problem by the time we get to beta.
--BDS
Ian Gilman
a good thing for new work to have a kill switch while it's still
unstable, those kill switches should be removed once that code area
stabilizes. Otherwise we'll end up a byzantine morass of conditional
switches. Somehow the "kill switch end of life" needs to be part of
the process.
Robert Kaiser
Good thought! I guess it should be something like "remove the kill
switch on trunk as soon as what it enables has been shipped in a
release" (or should it be when it has been merged to beta?).