Hey There,

We've had a number of good discussions in the recent Mozilla 2 meetings
about what we want to accomplish, and the work on ActionMonkey is
progressing well: http://wiki.mozilla.org/JavaScript:ActionMonkey.

I wanted to get some thoughts out on how to plan the next major Gecko
release.  Some of this is re-hash from the Moz2 meetings - so bear with
me.  Short version: I think we should work towards a Moz2 release as the
next release.  We will also  continue on a variety of lower-risk tasks
in parallel and aim one way or another to release another major version
~1 year after Firefox3.

Longer Explanation:

Definition: Moz2 means we can break API compatibility (hence
incrementing the major version number). That's it.  It is implied that
some of the bigger changes below may either require us to break API
compat or may be substantially easier to do if we break compat.

Overall Goals (not in a particular order and stolen from Moz2 meeting):
     * Improve developer productivity
     * Performance
     * Footprint
     * Security
     * Innovative web features
     * Improved embedding and mobile support
     * Building features in XUL needed by apps
     * Better Support multicore systems
     * Continuing to improve our UE

Goals of our Process:

We are building Mozilla for the long-run.  Our primary goal is to push
the web forward.  Firefox is the current instrument of that mission and
we should seek to maximize long-term development/product productivity.
We don't have any quarter-end, trade-show, or other artificial deadlines
which force us to make short term decisions that are detrimental to the
long-term.  On the other hand, most of us are at Mozilla to ship
software that real people use.  So I'd like us to ship as frequently as
possible but make decisions that ensure a long-term continuous stream of
high quality products from Mozilla. Shooting for the best product
delievery and engineering efficiency over a span of years is the goal
here.

Release Date:

Firefox 1.5 shipped just over 1 year from Firefox 1.0
Firefox 2.0 shipped about 11 months after Firefox 1.5
Firefox 3.0 (if ships in 1H08) will ship 17-30 months after Firefox 2.0
(but was under development for closer to 2-3 years)

Given our goals, past experience, it feels right to set a target for the
follow-on to Firefox 3 to ship about 1 year from Firefox 3.  That should
give us enough time to make progress on the big long-term stuff but
isn't too long to compound risk.  This is an interpolation between the
times of Firefox 3 and Firefox 2.

Tasks:

I'd like us to think of our tasks in three rough buckets.  These buckets
are meant to categorize things mostly by level of risk to the overall
release.  I've likely got the specific tasks in bucket B/C wrong (since
we haven't done this yet and this is a swag) but we can address later
here.  Thinking of things in these buckets is more important this second
that the specific tasks.

A) Big, invasive, destabilizing changes.  These are the sort of tasks
that don't respond well to compression (e.g. throwing more people at it)
and touch large pieces of the code and/or can destabilize other parts of
the development.  These are the sort of changes that once landed into
the mainline development tree can become the long-pole for release and
are difficult to unwind.  From the Moz2 meetings it appears the
following are in this category:
        1) mmGC (including XPCOMGC) *already underway*
        2) Exception support *already underway for tooling at least*
        3) deCOM
        4) Taramin JIT replacing SM *already underway*
        5) Better threading support in the core layout engine
        6) Centralized security checks
        7) Removal of XPConnect
        8) Major GFX pipeline changes
        9) Protected mode

B) Medium sized, significant in impact changes.  These are non-trival,
but better isolated.  Easier to turn off or backout.   Things such as:
        1) XBL2
        2) Frame construction cleanup  
        3) Inline re-flow
        4) XUL box Model
        5) Treaded parsing
        6) Threaded image decoding
        7) Fewer restarts (e.g. protected extension installs, etc)
        8) Features needed for UE
        9) UE improvements

C) Smaller, isolated changes.  These are smaller, easy to turn off, or
don't impact other critical paths:
        1) Worker threads
        2) Video
        3) Minor CSS improvements
        4) Jail tag
        5) SVG animation
        6) SVG fonts
        7) Tabs between windows
        8) UE improvements

Plan:

We should pick 2-3 of the most important large tasks and prioritize them
as a #1 priority and resource them for success.  This means as folks
peel off of Gecko 1.9 they should first try to help with these 2-3 large
tasks.  Because we have a huge and awesome community, and because many
of these tasks don't parallelize well we'll have plenty of resources
left over to crank on items from buckets B and C.

We organize into teams so that each area has at least 2 folks working on
  it (to prevent blocking on reviews, etc) and no more than 5-6 folks on
it (just too much communication overhead).   Each feature team takes an
area from start to finish - e.g. don't start new tasks until this one is
done (to prevent end-game context switching and release risk).   Ideally
no one person should be working on more than 1 (or maybe 2) features at
once (again to prevent context switching).

Each team (depending on size of task) will be working on a separate Hg
branch.  We'll keep hg-central overall very high quality and
ready-to-ship.  Merges to hg-central will not be allowed without
regression, performance, functionality parity.  No more landing then
optimizing.   We'll organize the tasks to prevent merge conflicts
wherever possible.  This is why we can't undertake too many large items
at once.  Easier Hg merging, the try server, and other tools will make
it much easier for folks to test out changes without first having to
merge them to mainline.

We ship alpha's from hg-central early on in the process on a 6-8 week
cadence.  We'd strongly hope that 1-3 of the big (A) tasks is ready for
ship during the schedule.  However, since we are working on lots of
smaller stuff in parallel if the big stuff takes longer than expected we
still have a stable base and a set of improvements to go into a release.

This gives us a couple of things:
a) It ensures that the big long-pole items of high impact are never
starved and are delivered as soon as possible
b) We don't take on *too* many of them at once and have to deal with
with challenges of compound regressions, performance issues, etc.
c) Gives us the ability to ship more easily if some of the big tasks
take too long
d) Tries to reduce the context-switch overhead that was hard on many
folks during the Firefox 3 release
e) The constant quality of the mainline makes it easier to isolate
changes, regressions,and be ready to ship.

So in short I think we need to courage to take on a few big tasks and
resource them for success, and the wisdom to not block all development
behind these tasks.

Thoughts?  There are obviously many many details to nail down in terms
of prioritizing and ordering tasks, etc. and that will take some time.
But, I'd love to enter 2008 with a rough agreement at least on the 1.9.1
vs Moz2 issue.

Best,

Schrep

On Dec 18, 5:14 pm, Mike Schroepfer <sch...@mozilla.com> wrote:
Not sure what this is -- probably shouldn't be under this category.
If it is something like using OpenGL/Direct3d it should probably be
under bucket B.

These are probably both more medium sized.

The rest of this sounds pretty good.

I think it would be very useful to re-organize your list to put the
changes you list under each goal, being ruthless in placing items only
under their primary goal.  Short lists under major goals may need
attention. For example, my favorite, Developer productivity seems
slighted ;-).

Making this decision now may lock you in to risks that a few more months
may clarify. In particular, defining Moz2 as "FF3 + 1yr" OR as a certain
list of features may be more prudent than AND, software being what it is.

John.

Hey John,

Good suggestion.  I'm worried that too many tasks go under multiple
categories (.e.g mmGC...)

mmGC, Exceptions, deCOM are all squarely aimed at this (well and perf,
footprint, security, but that's point #1 :-) ).  Is there some stuff we
are missing?  Probably lots of build config stuff (e.g. make Win32 no-op
builds not take 20 minutes :-P).

Yep. Locking Scope + Time + Resources = failed project
(http://www.ambysoft.com/essays/brokenTriangle.html).

We have primary control over Time and Scope.  I'm proposing that we fix
Time (~1 year) with some slush.  Many, as you point out, of the scope
issues we won't know until we get further into them.  This is why I'm
proposing we invest in a small portfolio of big/medium/small stuff.

I guess I'd throw localization infrastructure changes (read, l20n) in
this one.

I think that I can think in these buckets allright, I miss JS2, though.
Not so much in terms of "will it happen or not" but more in the sense of
"can I make my task depend on it?". To some extent, js2 and l20n might
be similar, as it's not too bad to limit their lines-of-code impact, or
to do that incrementally, but that they come with a learning curve
throughout the project. Or, if JIT makes central, I know that a
js-implemented feature or rewrite can be really measured for
performance, but JS2 completeness would say when I'm done with coding.

Axel

I wouldn't call this "removal" but rather "fast-path DOM"... and I don't
think it's a bucket-A item. I think it could implemented as a much smaller
bucket-B item once the actionmonkey (tamarin JIT) stuff is landed.

This is tricky: we could either

* implements XBL2 and keep mozilla-XBL -- this is safe and in bucket two,
but has the problem that we're carrying around two significant
infrastructures that will interact poorly

* replace mozilla-XBL with XBL2 -- this is probably much easier to implement
correctly and quickly, but it involves rewriting our existing mozilla-XBL
bindings in XBL2. In some cases we can come up with automatic
transformations, but it will still be a pretty huge change that really
should be in bucket A above.

Note that several of these tasks have dependencies on eachother: XPCOMGC
depends on AM stage 1 and stage 2. Exception support depends on XPCOMGC.

A critical piece of this plan that I think we need to nail is directing our
community of nightly testers. It's possible of course to simply have nightly
testers on mozilla-central at all times, but I don't think this provides
maximum leverage:

* Testers may want to actively follow one of the feature branches: e.g. the
UE team will have experiments and ask for testing of those before they hit
mozilla-central... an XBL2 branch will want testing from extension authors
and interested developers before it is merged to mozilla-central.

* We don't want to orphan a nightly tester: if they have been testing a
nightly from the UE branch, we'd like to keep them up to date nightly. If
the UE branch stops being developed, we want to repatriate that user to
mozilla-central

I think we can accomplish this through the update system and major update
offers:

+-------------------------------+
| Minefield Update              |
| An update is available. You are currently using
| the Minefield stable nightly builds. We are looking for
| testers of the user experience branch. This branch contains
| experiments in the user interface design of Firefox. Would you like to
| update to the UE branch?      |
| [ Update ] [ No thanks ]      |
+-------------------------------+

Finally, I think that it may make sense to do an alpha every 4 weeks. I know
that mconnor has been planning a monthly cadence for new UE features, and
with an always-stable mozilla-central and release automation, I think we can
schedule the alphas more frequently than we have. But I think we should make
sure that we get automatic updates from alpha->alpha so we don't leave
people stranded.

--BDS

While more decentralized development is certainly favorable from the
developers' perspective, I think that we need to figure out how to make
it work for our testing community. Right now there are unified nightlies
which contain all the checkins. This will no longer be the case when
there are multiple branches, which aren't merged back to hg-central
until the work is completed. Nightlies built from hg-central will not
include large chunks of changes which need testing.

As you describe, high quality and stability of the trunk is to be
ensured by fact, that major architectural changes won't be merged until
they meet the desired quality criteria. Yet, if these branches are
tested only by a relatively small number of involved developers, then
the quality isn't going to be verified as well as it would be by
thousands of community testers. This is going to detract from the
quality gains. Even though I am convinced that the balance is still
going to be positive, we should give some thought to this issue.

How do we enable and encourage testing of the branches?

Because back-end changes aren't visible to users and not many people get
excited by the fact that their Firefox now uses garbage collection
instead of reference counting, relatively few folks would be interested
in dog fooding the branch builds. Putting out separate nightlies for
different branches is probably out of the question. It would be
impossible from the build system perspective to provide automatic
updates and multiple nightlies would be too confusing for casual
testers. Not to mention that it would impose additional burden on
developers to make sure that the branches compile and are dog-foodable
day by day.

The least that should be done then, is to make fairly regular builds
available to the public and advertise them on QMO.

Maybe these problems are inherent to branched development, maybe it's
just an unavoidable trade-off. But it would be prudent and welcomed by
the testing community if we worked out some kind of strategy.

- Adam

On 12/19/07, Benjamin Smedberg <benja...@smedbergs.us> wrote:

Removal of current mozilla-XBL sounds like a very bad idea to me.

Regards,
Martijn

Could you be more specific? Are you concerned about

1) breaking the web
2) destabilizing Firefox itself
3) harming XUL application developers who use XBL?

Of these concerns, I think 2) is most valid. We've said for a long time that
XBL is not a finished technology, and we've changed it many times already in
subtle and incompatible ways. Especially if we can provide a simple script
(XSLT) to convert simple mozXBL -> XBL2 I think that is sufficient.

--BDS

It's not clear to me why this is an either/or proposition.  Wouldn't the
easiest thing to do be to implement XBL2 and carry mozilla-XBL as long
as we need to until we're ready to do the rewrite/jettison step?

Dan

Taking a closer look, everyone would agree, that usage of a DVCS, by
itself, is not making merging even a little bit easier. Such a system
"just" provides the capability to employ the development process, that
facilitates merging or even makes it seamless.

Linux kernel is a DVCS production use example closest to Mozilla in
size, success and sophistication. Long ago they abandoned CVS in favor
of first BitKeeper, and now git. Borrowing a few tricks from them
shouldn't hurt at all ;)

This topic has recently been discussed in m.d.platform "Mozilla 2
checkin patterns" thread: http://tinyurl.com/yp4ccn

The key concept is "subsystem" trees. All development happens there. The
trees are rebased over the "stable" release branch early and often.
Rebasing is essentially the old proven patch queue approach. BTW,
Mozilla2 team is using just the same technique. And it is a matter of
taste, which solution to use for that: 'quilt', 'hg mq', 'hg transplant'
or 'git rebase'.

Automated updates of nightlies with variable changesets to test will
work! It is fairly easier to implement that an alternative, which is to
have a second branch for integration in addition to the release one.

The latter has some advantages like:
* all tester have the same executable, so less information from users is
required to identify the build that has a particular problem;
* all tester try all new features at the same time;
* it fares well with "stupid" infrastructure like ftp://, which is easy
to maintain.

But:
* with some automation, it shouldn't be difficult to determine exact
build in the former solution as well;
* given enough of testers, all branches will receive adequate attention.
In addition, it will be possible to establish testing priorities;
* once the above is true, locating the problem will be much-much easier
(changes in one module v. changes in all modules);
* last, but not least, dirty preliminary merging work will be avoidable.
Not only -mm kernel tree is unstable, but it also requires permanent
reconciliations between subsystems. And it is a huge diff against
release tree (last it was over 30 Mbs or more than 10%).

--
Sergey Yanovich

In some ways this may be more difficult than just implementing XBL2, at
least on the layout end.  That is, making all the various parts of
layout/content that have to play nice with both moz-XBL and XBL2 is
likely to be a huge, fragile, crash-prone pain.

-Boris

I'm sorry, I can't understand this paragraph. In the automatic update system
you can have multiple "update channels", so it's possible to associate an
update channel with "mozilla-central" and a different update channel with
"the UE subsystem branch".

--BDS

By variable changeset I mean that different tester may receive different
  binaries via the same "update channel". Which particular binary to
supply is to be decided on each request. There may be different
strategies like, f.e.:
* XPCOMGC - 30%;
* Tamarin JIT - 20%;
* ...
* Worker threads - 5%.

The next day, priorities may change, and tester will receive next-day's
nightlies in different proportions.

This doesn't sound like a good idea to me: if one day you're running new UI
on the UE branch and the next day it's gone, I think that's likely to drive
testers away. And some testers may want to be involved in particular projects.

--BDS

Yes splitting our testing community to focus on multiple branches at
once is a difficult problem.  We've done this before by producing 1-off
builds and asking people to download (e.g.
http://weblogs.mozillazine.org/josh/archives/2005/07/intel_mac_build....).

As you and others mention we can use automated updates with either
explicit choices or sampling amounts builds to spread out the testing
amongst multiple branches in parallel.  However, I think there is a
natural limit to how many major branches of development can happen
concurrently from a merging and testing perspective.  This is why I
think we need to limit the number of CatA and CatB tasks we take on at
once.  The hope is that some CatB and most CatC tasks can do just fine
with isolated testing and automated testing before they get merged into
the integration branch.

So I think we can do more parallel development than we have recently,
but we need to be very realistic about the costs and the limits of how
much we can do at once.

Ok. super-helpful.  I think I'll take a second pass through the details
of the buckets and update once we've consolidated feedback.

Yep.  So we can't do these in parallel and need to make sure we really
really finish the first task before we move onto the next one.  I think
the big question will be should we stop at XPCOMGC for this release or
continue on to exceptions...

Yep - spreading out or testing community is a pretty critical part of
this and should be done with care.  As I said in a later post I think we
can do more than we are now through techniques as you suggest, but we
should be careful not to run too many things in parallel.

I'm happy run at whatever cadence works best for folks.  4 weeks is just
fine.   I do think with release automation we can do a better job of
alpha->alpha updates..

On Dec 20, 3:57 am, Benjamin Smedberg <benja...@smedbergs.us> wrote:

I was thinking that too. "Implement XBL2 and keep mozilla-XBL" would
probably have so many strange interactions that it would be a bucket-A
item anyway. (I prefer the "replace" approach.)

Rob

On Dec 20, 6:46 am, Dan Mosedale <dm...@mozilla.org> wrote:

As Boris said: no.

What we might be able to do is to retain support for some subset of
XBL1 *syntax* implemented using corresponding XBL2 features, to
provide some level of ongoing XBL1 compatibility.

Rob

On Dec 19, 2:14 pm, Mike Schroepfer <sch...@mozilla.com> wrote:

Let me argue that most C-bucket and some B-bucket items should
actually be developed in hg-central or at least be pushed there on a
regular basis. Features like <video> that aren't currently used
anywhere have a very small chance of causing trunk regressions. If we
want to ship and the feature isn't ready we can disable it trivially.
Having these features in hg-central will reduce testing and merging
costs. I think it matters a great deal that we can tell people
"download this build and play all these new cool features we're
working on".

Rob

That is all true.

So the only option here is to ask users to switch "update channels", and
hope that each branch will sign up enough volunteers.

In this case, it still may be a good idea to have -mm kernel tree
equivalent, which will allow for a single nightly.

On Dec 20, 8:30 am, "Robert O'Callahan" <rocalla...@gmail.com> wrote:

Also, the synergy between features is often the spark that takes a
demo from "cool" to "jaw-dropping". (HTML and canvas3d and SVG and
<video> whoooohoo!) One of the strengths of the Web platform over
proprietary plugins is that the Web is better for mixing up different
kinds of content. (Web content can contain any other kind of content;
Flash content can only contain Flash.) It's nice to be able to show
this without having to merge half a dozen branches.

Rob

On Dec 19, 7:59 pm, Mike Schroepfer <sch...@mozilla.com> wrote:

IMHO the naming should depend on the scope. It doesn't make sense to
break compatibility in the next release unless the big changes that
demand compatibility-breakage are in that release. Therefore if those
changes don't make the next release, we should not be breaking
compatibility and we should not call it Mozilla2.

I would also argue that if that happens, we should ensure the next
release does not contain major Web-compatiblity-affecting changes, so
we could call it 1.9.1 with a straight face.

I would also argue that if it becomes apparent that "long pole" A-list
items aren't going to make the next release, we should try to
recognize that early and remove them from the blocking list, and be
ready to ship earlier than a year if that list becomes empty. I
suppose Firefox front-end scheduling may not permit this.

Rob

I suppose it depends on which developers you are targeting. From what I
understand of these, they will primarily help non-Javascript developers
who are integrating multiple mozilla C++ components. They will also help
new C++ component developers as the component-infrastructure cruft will
be less. These are important and valuable improvements. But I don't see
much on the list for C++ component maintainers or Javascript (web or
XUL) developers.  I imagine that these folks are the bulk of the Mozilla
dev community?

Anyway the feature list derives from personal interests of the
developers, so it probably does not make sense to over analyze it.

John.

On Dec 19, 11:30 am, "Robert O'Callahan" <rocalla...@gmail.com> wrote:

This is right on, and indeed for ActionMonkey we have often merged low-
risk changes toward more central repos (even into cvs.m.o) to reduce
merge headaches and get the goodness out early and often.

We should resist the DVCS temptation to fork unduly, even if only as a
calculated temporary condition.

/be