Extension driven development
Kent James
of changing the way that Thunderbird is developed and distributed, with
a bare minimum set of core code, and the main features presented as a
set of extensions, shipped with the product, that can be enabled or
disabled by users.
I don't have any illusions that this has a significant chance of being
implemented, and I'm not even sure it's a good idea myself. But I ask
you to suspend disbelief for a minute, and imagine a change to the
development culture and process.
An email client is different from a web client in many ways, but one
significant way is that there is no real need for a fat uniform core
product that developers can target (such as web developers for FireFox).
So we are free to allow wide changes in our product configuration that
would not make sense for FireFox. There is really no fundamental need
for Thunderbird to be presented as a single, fat, feature-laden client.
Instead, ship Thunderbird as a minimal base with a collection of
extensions. The extensions could be in a variety of statuses. At one
status extreme, "Core" extensions would be enabled by default, would be
fully localized, and their updates would be shipped with updates to the
core product, rather than through AMO. Many existing core features would
be converted into "Core" extensions that could be disabled if desired
(for example bayes junk processing, or gloda.) At the other extreme,
"Pilot" extensions would be shipped with the core product in a disabled
state, would be updated by AMO, and not fully localized. There would
also be "Standard" extensions that are shipped with the product, not
maintained through AMO, but would not be enabled by default. Lightning
might be one example of this. FiltaQuilla or JunQuilla could easily get
added to this category in the future, or popular extensions like
ThunderBrowse.
So why would you do such a crazy thing? For several reasons.
First, you would have a path to add features to the program that is not
as generally disruptive as has been, for example, gloda or the new
message header. By not using a new extension, an existing user would not
see changes to their workflow that they did not want or appreciate.
Also, new features need not delay the release of new versions of
Thunderbird, as "Pilot" status extensions could be updated through AMO.
Second, new complex features like gloda, even though they are developed
by the core team (well mostly asuth) are in a state of rapid flux, and
would really benefit from allowing updates more frequently than even the
accelerated release process will allow.
Third, you provide a natural path for outside developers to add
contribution to the product without having to completely submerge
themselves in the Mozilla culture, or give up complete control of their
creation.
Fourth, this really recognizes that the use of an email client is highly
personal. Basic users could be presented a basic email client. Advanced
users could easily add advanced features. (Existing AMO-based extensions
are also good for this, but the quality is not uniform, and they
frequently are not kept up to date. And the standard product is still
very fat with lots of features that are unneeded by most users.)
Fifth, this would solve the serious issue with Thunderbird of how hard
it is for the average user to install addons (because the most popular
and important addons would be shipped with the product).
There's another dimension to this, and that is the developer's
relationship with his or her extension. I know that I feel a
responsibility for my extensions that is beyond the responsibility I
feel for any core code. You can see that in the documentation that I
provide, and my reliability in responding to issues. I think that many
other extension developers are like that as well. I'm guessing that they
would be delighted to see a higher level of promotion of their work,
without the need to cede complete control that incorporation in core
might involve.
I suppose I could write a book on what this might look like, but for now
let me leave it here.
rkent
Axel Hecht
the discussion around Thunderbird development.
I saw the lack of that in comments by at least one (other) person on the
other thread, and I see it here.
Axel
Andrew Sutherland
this was cross-posted; I think my reply in the context of your blog is
somewhat different, but the content is substantially what I would have
written here directly anyways.
http://mesquilla.com/2009/11/28/extension-driven-development/
I think this has a significant chance of being implemented, or at least
I hope so, because it�s what I too am pushing for.
Gloda is an interesting example case because it actually did start out
its life as an extension. Apart from a dependency on Thunderbird locales
for localized strings, it largely retains this structure.
While I have been plugging Jetpack for quite some time as the future of
(easy) extensions for Thunderbird, there is exciting news in that
Jetpack is being rebooted to be even more appropriate for the use-cases
you discuss.
Namely, I think Jetpack2 (not sure if it will actually have the 2, but
let�s use it for the sake of argument), will allow Thunderbird to ship
with extensions built-in that are still exposed as extensions AND also
provide for the user to download more recent versions of those extensions.
This would help alleviate the need to monkey-patch 3.0�s Gloda to allow
other extensions to access/provide new Gloda features and functionality,
like GlodaQuilla�s do-not-index feature (which is fully sanctioned
monkey-patching).
Andrew
Andrew Sutherland
> I would like to lobby for localization to be a first class citizen in
> the discussion around Thunderbird development.
Indeed, localization is a fundamental need to be addressed and it needs
to be addressed now.
What is the state of the L20n effort? Is the repo at:
http://hg.mozilla.org/users/axel_mozilla.com/l20n/
the right one to be looking at? It was last updated in June; does that
mean it is in a usable state?
The canonical example in my mind of how the current situation is messed
up is the faceted search string that was originally along the lines of
'Showing X of Y messages'. We got that wrong at first despite trying to
do the right thing, and our current (hopefully sufficient) solution
involves 3 separate properties and a disturbing amount of logic.
While we can require and perform any number of contortions in the core
code, there is an obvious benefit to minimizing the number of
contortions and making it easier for extensions to be fully localizable
out of the gate rather than requiring overhauls after the fact.
Andrew
David Ascher
> What then do I mean by "extension driven development"?
> I suppose I could write a book on what this might look like, but for
> now let me leave it here.
I think this is very close to how I've been thinking about it, thanks
for writing it!
I think there are lots of hidden and not-so-hidden subtleties which we'd
need to work through, incuding:
- how do we evolve the non-dev parts of the process to fit (build,
l10n, QA, docs, nightly user testing, etc.)
- how do we manage the combinatorial complexity of testing that such
a model implies?
- how do we evolve extensions from "the outside" into the core set,
wrt to module ownership, test coverage, l10n, product direction POV,
long-term maintenance commitment, etc.
- having a system with many moving parts is harder to coordinate.
The major trick to solving some of the above I suspect is not only to
push feature out to extensions, as we've been saying, but also to
strengthen the reality of the platform core. To me this will require:
- a smaller API surface that we can actually support.
- a lot better test coverage of the platform
- a lot more churn along the way as we push
things-that-shouldn't-have-been-in-core out to extensions.
which is why I'm excited about the jetpack approach that Andrew talked
about.
I don't think any of this can or should happen quickly. But my hope is
that in the next year we will see more bugs against TB be "can we make X
available to extensions?" and more featurish things happen in add-ons.
MoMo staff will be working more with add-ons in 2010, which will I
suspect help the case for add-on authors in general.
--david
Axel Hecht
My repo is not the current state of thinking. We had an intern this
summer, Jeremy Hiatt, who's made great efforts and dug up some new
concepts, and we're trying to clean that up and incorporate other trains
of thoughts together with Gandalf.
Axel
Andrew Sutherland
> My repo is not the current state of thinking. We had an intern this
> summer, Jeremy Hiatt, who's made great efforts and dug up some new
> concepts, and we're trying to clean that up and incorporate other trains
> of thoughts together with Gandalf.
Where can I find the most up-to-date reflection of current thinking in
an implementation I can "hg clone"?
Thanks,
Andrew
Kent James
> I would like to lobby for localization to be a first class citizen in
> the discussion around Thunderbird development.
>
> I saw the lack of that in comments by at least one (other) person on the
> other thread, and I see it here.
>
I'm not sure what you mean by a "first class citizen" here. Do you mean
"nothing associated with Thunderbird should ever be provided except in
all supported languages" then I don't know how you can justify AMO's
existence.
What I hope that you mean is the we need to consider the localization
impacts as a first-class issue in envisioning plans for the Thunderbird
development process, including extension driven development. I doubt if
there is any opposition to that.
What I envision is that extensions would have a number of possible
statuses, and there would be a series of hurdles that an extension would
have to go through to advance to each higher status, starting from the
lowest status of "AMO experimental" to the highest status, "Thunderbird
Core". I doubt if there is any opposition to the concept that full
localization is NOT needed for "AMO experimental", but IS needed for
"Thunderbird Core". It's those middle statuses, that are not yet even
defined, where the issues come.
In my earlier post, I suggested a "Pilot" status, that would be higher
than "AMO Public" or even "AMO Recommended". Neither of those currently
require full localization, and I doubt if it would be practical to have
a "Pilot" status require full localization either. But there could be
some minimum localization requirements for "Pilot" status.
I'm not without localization experience personally, having started a
company in a country where educated people had to speak English, a
regional language (Russian), and a local language (Azerbaijani). I don't
think there would be objections in such an environment to an
experimental program feature being unavailable in the local language, or
even have some obscure setup screen for an experimental feature that
included some English or Russian. After all, the entire program
documentation is already only or mostly in English.
I don't really know where and how strict to draw the line, however. We
certainly don't want to promote creeping anglicization.
rkent
Joshua Cranmer
> I don't have any illusions that this has a significant chance of being
> implemented, and I'm not even sure it's a good idea myself. But I ask
> you to suspend disbelief for a minute, and imagine a change to the
> development culture and process.
I've pondered something akin to this before, but, similarly, I'm
undecided as to the merits of this approach. In any case, I think the
architecture of Thunderbird is presently unable to support such a
development approach: it is too poorly compartmentalized and the
internal APIs too poorly designed to be able to cleave into separate
core and extension components.
Certainly, that doesn't stop completely new backend logic from forming
in this fashion (think gloda), and I think that developing parallel
backend components should follow such an approach as much as feasibly
possible.
> Instead, ship Thunderbird as a minimal base with a collection of
> extensions. The extensions could be in a variety of statuses. At one
> status extreme, "Core" extensions would be enabled by default, would be
> fully localized, and their updates would be shipped with updates to the
> core product, rather than through AMO. Many existing core features would
> be converted into "Core" extensions that could be disabled if desired
> (for example bayes junk processing, or gloda.) At the other extreme,
> "Pilot" extensions would be shipped with the core product in a disabled
> state, would be updated by AMO, and not fully localized. There would
> also be "Standard" extensions that are shipped with the product, not
> maintained through AMO, but would not be enabled by default. Lightning
> might be one example of this. FiltaQuilla or JunQuilla could easily get
> added to this category in the future, or popular extensions like
> ThunderBrowse.
How would you compartmentalize Thunderbird? These are roughly what
probably could be separated, given enough mashing and fault-tolerant code:
* RSS (this was originally an extension IIRC)
* NNTP
* IMAP
* POP
* SMTP
* Movemail
* Address book
* Calendar (i.e., Lightning)
* Composition
* Core
* Junk and other filtering stuff
* Gloda
* Encrypted mail support
Right now, the NNTP/IMAP/POP/SMTP/Compose would probably be hard to
extricate from each other.
> First, you would have a path to add features to the program that is not
> as generally disruptive as has been, for example, gloda or the new
> message header. By not using a new extension, an existing user would not
> see changes to their workflow that they did not want or appreciate.
> Also, new features need not delay the release of new versions of
> Thunderbird, as "Pilot" status extensions could be updated through AMO.
I've advocated an approach of having "official" extensions in the past
(akin to DOM Inspector); the feedback I received was along the lines of
"tried it, didn't work too well."
> Fourth, this really recognizes that the use of an email client is highly
> personal. Basic users could be presented a basic email client. Advanced
> users could easily add advanced features.
To quote what I've said in the past: "An adage holds that 80% of
features are used by only 20% of users, but the problem is that the
only-used 20% is different for each user." I would expect to see
MNG-esque flamewars complaining about feature Y being or not being
considered core.
> Fifth, this would solve the serious issue with Thunderbird of how hard
> it is for the average user to install addons (because the most popular
> and important addons would be shipped with the product).
I don't know if this would be true--unless you have a really broad
cutoff, the users of the just-below-the-cutoff are probably going to be
rather vocal about failures in the installation process.
> There's another dimension to this, and that is the developer's
> relationship with his or her extension. I know that I feel a
> responsibility for my extensions that is beyond the responsibility I
> feel for any core code. You can see that in the documentation that I
> provide, and my reliability in responding to issues. I think that many
> other extension developers are like that as well. I'm guessing that they
> would be delighted to see a higher level of promotion of their work,
> without the need to cede complete control that incorporation in core
> might involve.
My personal approach tends to center more along the lines of "can
someone please take this off my hands"... I suppose that's why I'm not
an extension developer. :-)
Simon Paquet
>> I would like to lobby for localization to be a first class
>> citizen in the discussion around Thunderbird development.
>>
>> I saw the lack of that in comments by at least one (other)
>> person on the other thread, and I see it here.
>
> I'm not sure what you mean by a "first class citizen" here. Do
> you mean "nothing associated with Thunderbird should ever be
> provided except in all supported languages" then I don't know
> how you can justify AMO's existence.
I don't know exactly what Axel was hinting at, but I believe that
he was hinting at making sure that localizability (l12y) is an
integral part of such extension pilots and not a mere afterthought.
As Andrew rightly points out, making a piece of code localizable is
sometimes hard, even if you think about the l12y before or while you
do the coding. It becomes a huge mess, if you only think about it
afterward.
So my proposal would be, that not all those pilot extensions would
have to be available in all TB locales, but that it should be possible
to have all of them available in said locales.
Simon
--
Thunderbird/Calendar Localisation (L10n) Coordinator
Thunderbird l10n blog: http://thunderbird-l10n.blogspot.com
Calendar website maintainer: http://www.mozilla.org/projects/calendar
Calendar developer blog: http://weblogs.mozillazine.org/calendar
Mark Banner
> Fifth, this would solve the serious issue with Thunderbird of how hard
> it is for the average user to install addons (because the most popular
> and important addons would be shipped with the product).
With the new content tabs, we're actually almost in a position to fix
this for at least the AMO case. It needs a little bit more work, but I
believe I may be able to do an extension for TB 3 (and if not, certainly
for TB 3.1) that would allow add-on manager to integrate with content
tabs and then be able to install add-ons from there. This would actually
need some addon.mozilla.org work as well, but that's a minor point ;-)
Standard8
Axel Hecht
brainz://
There's no code and there are no final decisions made. Making the
experiments such that other people could understand them from an hg
clone wasn't the goal, and might not become the goal.
http://jeremyhiatt.wordpress.com/ has some blog posts about the
thinking, though there are trains of thought not covered there.
Axel
Ben Bucksch
> Instead, ship Thunderbird as a minimal base with a collection of
> extensions. The extensions could be in a variety of statuses. At one
> status extreme, "Core" extensions would be enabled by default, would
> be fully localized, and their updates would be shipped with updates to
> the core product, rather than through AMO. Many existing core features
> would be converted into "Core" extensions that could be disabled if
> desired (for example bayes junk processing, or gloda.) At the other
> extreme, "Pilot" extensions would be shipped with the core product in
> a disabled state, would be updated by AMO, and not fully localized.
> There would also be "Standard" extensions that are shipped with the
> product, not maintained through AMO, but would not be enabled by
> default. Lightning might be one example of this. FiltaQuilla or
> JunQuilla could easily get added to this category in the future, or
> popular extensions like ThunderBrowse.
I can see several problems with that:
* Users get confused. I typically have a hard time explaining people
what a "email application" is, given how used they are to webmail.
Having things get even more complicated, with different feature sets,
authors, versions and localizations, would be way too much for most users.
* If we ship an extension, we are responsible for it, period. And I
don't mean legally, but in the sense of reputation (impression that the
software makes), quality, usability etc.
* You won't be able to explain users that some parts are localized and
others are not.
* It gets terribly complicated with versions. Extensions are apparently
already struggling with keeping up with Thunderbird, what happens when
there are 5 different versions, in different combinations?
* When important functionality is in extensions, dependencies get a
nightmare. Ext A might depend on ext B and might use C and needs to
consider D. Add versions to that, and it won't be fun.
I guess, in one word, it would be a "patchwork".
There's nothing users can gain from *disabling* functionality. Almost
nobody wants / does that (normal users wouldn't do it out of principle
or rather fear of missing something). The only advantage I see from your
list is that users can opt to update faster. But then again it assumes
they actually understand which extension does what and looked whether
there's an update and why. I don't think the vast minority of our 12-20
million users would do that.
That said, I think gloda was a good example of what could be developed
as extension at first, and it made sense there. It was a big new module,
almost no interaction with existing code, but wildly fast development.
And asuth did it. But it should end up as part of the core product, not
extensions anymore.
For other things, which touch a lot of core code or have a shorter
development time and are intended for core, it makes less sense to
develop them as extension. For these, feature branch development makes
more sense.
> without having to completely submerge themselves in the Mozilla
> culture, or give up complete control of their creation.
The benefit of collaboration is that many ideas and concerns come
together at the beginning and form the design, and then the code is
written based on that and hopefully covers all the concerns and
viewpoints, not just one. If the extension first was written as "my
little feature", and is later shipped with Thunderbird and enabled for
everybody, you don't have that. It's then a "take it or leave
it"-situation. So, I see that as disadvantage, not advantage.
> I know that I feel a responsibility for my extensions that is beyond
> the responsibility I feel for any core code.
I'm the opposite. I consider core code to be most important, and
extensions as little side thing I maintain only when I feel like it.
Thomas Stache
>
> * It gets terribly complicated with versions. Extensions are apparently
> already struggling with keeping up with Thunderbird, what happens when
> there are 5 different versions, in different combinations?
> * When important functionality is in extensions, dependencies get a
> nightmare. Ext A might depend on ext B and might use C and needs to
> consider D. Add versions to that, and it won't be fun.
>
> I guess, in one word, it would be a "patchwork".
>
> There's nothing users can gain from *disabling* functionality. Almost
> nobody wants / does that (normal users wouldn't do it out of principle
> or rather fear of missing something). The only advantage I see from your
> list is that users can opt to update faster. But then again it assumes
> they actually understand which extension does what and looked whether
> there's an update and why. I don't think the vast minority of our 12-20
> million users would do that.
I concur in large parts with Ben in that I have a problem with imagining
Thunderbird as a "small" core with a boatload of extensions shipped with
it. Sounds like Netscape (or Evolution) consisting of (XP-)COM
components - both meanwhile reversed approaches - in new dresses.
Up until a couple of days ago I believed "extension driven development"
referred to the prototyping/rapid development phase of features, before
they would be properly integrated into the product. I was in favour of
that. But now I'm less confident about where the discussion is heading.
How do you want to market a new release 3.x with features X, Y and Z,
but those packaged in extensions? If a user (or a Cnet reviewer)
disabled a certain official MoMo extension in version 3.x-1 and later
updates, he wouldn't see the hot new stuff, and there you have a bad
review out on the Internet (and forever in search indices).
Another example: the inevitable "Improve TB performance"
recommendations. Inexperienced users follow the advice to disable the
POP, News, Gloda or Message Reader extensions/modules, and next you know
you see them in your support forums with "Thunderbird 3.x update broke
my email" questions. That support load is completely avoidable.
It's all about public perception if you want to reestablish Thunderbird
as a strong player in the messaging landscape, and I doubt the
perception is controllable if the product is different for everyone.
Andrew Sutherland
> There's no code and there are no final decisions made. Making the
> experiments such that other people could understand them from an hg
> clone wasn't the goal, and might not become the goal.
> http://jeremyhiatt.wordpress.com/ has some blog posts about the
> thinking, though there are trains of thought not covered there.
Ah, that's unfortunate. I'm actively working on Jetpack1-based
extensibility for Thunderbird right now, and it would have been nice to
have that.
What I'm working on has a compilation stage of sorts, so when a true
L20n implementation is available the representation can be transformed
as appropriate. For now I'll try and make sure it's easy and feasible
to correctly address the "Showing M of N messages" case for both the
extension author and localizers and that additional meta-data can be
propagated for eventual L20n support.
Andrew
Axel Hecht
Localizing jetpacks hasn't been addressed at all yet. I have some
thoughts on how to guide the discussion around that, mostly putting some
UE decisions upfront, but yeah.
The concept that stuck so far is that you're setting up a context with
the variables M and N, and a list of localized files, and then pull a
single named string out of it.
All the fuzziness lies behind that abstraction wall, and for the sake or
how you code things, could be ignored.
Axel