Proposal to start a new implementation of Thunderbird based on web technologies

[Ben Bucksch]

Summary:

• Our base goes away. Gecko will change dramatically in the future, and dropping many features that Firefox does not need anymore, but that Thunderbird relies on.
• Our codebase is now roughly 20 years old. It heavily and intrinsically relies on those very Gecko technologies that are now being faded out, more or less aggressively.
• JavaScript and HTML5 have evolved dramatically. Entire applications in JS are now realistic, and have been done. There are several existing JS libraries we might leverage. JavaScript is an efficient language, which allows fast development. A rewrite in JavaScript makes sense now.
• We will learn from shortcomings of existing Thunderbird, and solve them, for example a more flexible address book, and cleanly supporting virtual folders without overhead.
  
• The goal of the rewrite is to be close to the existing Thunderbird, in UI and features, as a drop-in replacement for end users, without baffling them. They should immediately recognize the replacement as the Thunderbird they love. It will install and run as normal desktop application, like Thunderbird does today. It keeps user data local and private.
  
• We can also make a new, fresh desktop UI, as alternative to the traditional one, for new users. The technology also gives us the option to run it as mobile app.
  
• While we implement the new version of Thunderbird, the old codebase based on Gecko will be maintained until the rewrite is ready to replace the old one.
• I expect this effort to take roughly 3 years: 1 year until some dogfood (usable by some developers and enthusiasts). 2 years until a basic feature set is there. 3 years until we can replace Thunderbird.
  

I will describe each point in more detail below.

Proposal

I would like to put this up for discussion. I have not discussed this with the rest of the council yet, but rather I start the discussion here, to make it public, so that everyone can follow the discussion.

Ultimately, this will be a team effort. The TB Council, even with employees, will not be able to do this alone. I hope that Thunderbird can fund a number of developers on this, but I would like this to be an open-source project in the sense that everybody can participate. Please see this also as a call for help. If you are an experienced JavaScript developer who already wrote applications and libraries in JavaScript, please step forward and write us that you want to help.

Importance

I've been working on Thunderbird since 18 years, and I care about it a lot. More importantly, I care about communicating with my friends in a way that's private and not controlled by any single entity, but decentralized and running and stored on my own computer. In other words, we need a desktop (or smartphone) email client. Not just today, but also 30 years from now, if the world didn't collapse until then.

This is important for the freedom of the individual, of communication, and of computing. There are not many contenders left that defend the freedom of communication, because most have an interest to bind users to their own services. If I want to communicate with teenagers today, or even my mother, the best way is WhatsApp or SnapChat, closed systems. In order to make a different in this world, Thunderbird needs to be a viable option, not just next year, and not only for the existing userbase, but for the whole world, all age groups and all usage patterns. The current Thunderbird codebase has served well - 25 million users are amazing - but will not be able to defend freedom for all of us. Given the current technical challenges we face, this is the perfect time for a fresh start.

Our base goes away

Firefox has discontinued the Firefox XUL extensions, killing tens of thousands of small projects this way. This is a massive cut that goes to the base of Firefox. They did that, because these extensions use XUL and XPCOM, and this is preventing Firefox developer from making more drastic changes in the internal architecture. They are gradually moving to using some Rust components, which means the XPCOM base is no longer viable. At the same time, they also want to gradually decrease reliance on XUL in favor of HTML, and remove parts of the XUL implementation that they no longer need to run Firefox. That's why they remove XUL extensions. Thunderbird heavily relies on XUL, all its UI is written in XUL. Some parts of XUL - esp. some features of the tree widget - were back then written only for Thunderbird. We can expect breakage in this area, as Mozilla has clearly stated that they will no longer care about breaking Thunderbird. In fact, I think this problem is part of the motivating force behind Mozilla asking us to be independent.

Thunderbird relies on those very Gecko technologies

Thunderbird is based entirely on XPCOM (backend module API) and XUL (user interface). If you take out XPCOM and XUL from Thunderbird, you have effectively rewritten it. This change is as dramatic as the change from Netscape 4.5 to Mozilla was, when it was rewritten with XUL and XPCOM. 90% of the code was new. We are faced with a similar problem.

Gradual change would be very costly

Of course, the changes in Gecko are gradual, but eventually, the technology will go away, so the overall work is there nonetheless.

Given that XPCOM is the very technology that creates the API between the modules, is hard to replace step by step. I think we would spent as much time integrating the new modules with the old code as we would take writing the module in the first place, which means the overall effort increases at least 2-fold by trying to do it step by step.

Having worked on Thunderbird since 18 years, I do not deem it feasible to modernize it component by component. I estimate it would be 4 times faster to start with a clean slate and start from 0.

JavaScript is today the best choice

JavaScript, if used diligently and with good design, is a very efficient language. Both in execution time, but more importantly for developers. Personally, I wrote apps in many languages, including C++, Java and JavaScript. Of those, JavaScript is by far the most productive - I am personally 4-10 times as productive as with C++.

Today, we have node.js, Electrolysis, Cordova and other platforms that are already ready-made to create desktop applications using HTML5 and JavaScript. There are already other applications that have been written this way, and there is existing knowledge and libraries on how to do this.

Solve shortcomings of Thunderbird

There are some known shortcomings of Thunderbird, as it is implemented and designed today. Some are on the feature level, for example the address book is not flexible enough to capture even 3 email addresses for one person, it cannot link 2 persons (e.g. a couple), and so on. A modern re-write would consider today's user requirements.

Deeper goes the problem of  virtual folders. Thunderbird has them implemented as a bolt-on, which essentially keeps a copy of the mails, as far as I understand, and they are slow and inflexible. A new Thunderbird implementation could take an approach of where folders are a view that is computed in real time (like views in databases). This matches what GMail does with the AllMails folder and tags as properties, and then tags also form a virtual folder. There are important implications that this has: I can have a unified inbox, one inbox per folder, and I can have the emails sorted into specific project folders on arrival, all without overhead. The same incoming email appears in all 3 folders in the same second, and is marked read at the same time. I no longer have to move emails, but they are filtered.

Even deeper are design shortcomings of Thunderbird. It establishes a rough border between user interface and backend modules for IMAP etc., but is not very consistent in enforcing the independence of the 2 layers.

Right now, large parts of the logic are written as part of the frontend code, which is one of the big reasons why Thunderbird code is so difficult to understand and modify. It also makes it practically impossible to write alternative frontend. Instead, the architecture should be flexible enough to allow this, by strictly separating logic from UI using design patterns.

Some backend modules call the frontend, to update changes. Instead, we should use a classic observer/subscriber design pattern, where the frontend subscribes to changes in the backend, and updates itself. That allows more flexibility for the frontend. All lists should be observable, which is a simple but very powerful way to decouple logic from UI. See Linq in C#, and https://wiki.mozilla.org/Jetpack/Collections , which takes it a step further. This could be the technical basis for very flexible virtual folders that update immediately and are fast. But it would be used whenever there is a list.

Be close to the existing Thunderbird

Even though we write almost all code from scratch, we will save a lot of time by having a clear goal: We want to replicate the current Thunderbird, from an end user perspective. That means, the user will find the same 3-pane window layout, the same way how folders and message lists and the thread pane operate. The theme will be similar. Existing Thunderbird users should feel right at home.

We retain the overall UI and most features and qualities like performance, even if we do not copy all little details.

Disclaimer: Given that the technological basis - particularly HTML - is completely different, there will be some things that work differently in some ways. Hopefully, many will be better. We will have some technical limitations. Some will be just different, because the underlying implementation is completely different. The goal is not to copy bug for bug, but to make existing users immediately recognize this as Thunderbird, and feel at home, even if some details are different.

We must pay attention to also keep technical qualities that many of our users rely on. An obvious one is that the new implementation must be able to quickly scroll through a list of up to 100000 messages. There is no such HTML widget that allows that, we would have to create one, but I think it's feasible, I already have ideas how to do that. We also need to preserve privacy and security qualities that Thunderbird has, or even improve on it.

Fresh UI, more platforms

In addition to replicating the current Thunderbird UI, we should also experiment with new forms of UI, in parallel. For example, we should create a UI that's suitable for the new generation of users that never used a desktop email client before. These people do not feel at home with Thunderbird today, and we should create something for them.

A lot of the new userbase is on mobile platforms and on tablets. That new UI should be "responsive" (automatically adapting to different screen sized) so that it runs well on tablets and smart phones as well. With Cordova and similar toolkits, we have a technological basis to quickly make a mobile app out it so that it installs like any other app. It would be a replacement for the system "email" app.

The goal for the new UI are 1 billion users.

Maintain old Thunderbird code base

During the time while the new rewrite is implemented, a smaller part of the staff will maintain Thunderbird, as they have done in the last 2 years. We keep up with Gecko changes, and fix smaller bugs. But we will not do major refactoring or big new features on the old codebase.

This gives users a continuously updates Thunderbird. Particularly important are security updates for security holes found in Gecko, which Thunderbird inherits. They might be exposed in HTML emails, RSS feeds or other places where Thunderbird shows HTML and renders images.

Effort

The majority of the team should concentrate on the rewrite. This is also why this decision is important to make now, because Thunderbird has now resources to hire a number of developers, and we need to decide on which efforts we spend the precious little money we have through the generous donations of our existing users.

I would estimate that we need a team of 10 full time developers, for 3 years. We need 2 persons for the framework, 3 for backend modules, 4 for frontend UI, and 1 for theming.

Year 1

In the first year, we are laying the foundation, getting the framework sorted out, building infrastructure etc..

After 1 year, I would expect a first demo that can read and write email, but with a very minimal feature set and many rough edges. A few enthusiastic first alpha users might be able to use it for their email needs, and some developers can use it for their daily needs.

Year 2

In the second year, we would concentrate on building the important features that are needed for most users.

After 2 years, we should have an email client that's appealing to most normal users. Even some power users might like it, because we have advanced features that no other client has, but they will miss some features.

Year 3

In the third year, we concentrate on feature parity to the old Thunderbird. We add any features that Thunderbird currently has and are appreciated by the existing userbase. We also add functionality that allows larger deployments, as a significant part of userbase are enterprises.

Any features that are used by significantly less than 1% of the userbase will not be implemented, in favor of other features that are desperately needed today.

After 3 years, 95% of the existing Thunderbird userbase should find the features they need in the new implementation. There will be some changes and adoptions necessary, but there should be a way to do what they need. That's what I call "feature parity".

Year 4-20

Keep improving

[Matt Harris]

On 3/25/17 3:34 AM, Ben Bucksch wrote:

Summary:

• Our base goes away. Gecko will change dramatically in the future, and dropping many features that Firefox does not need anymore, but that Thunderbird relies on.
• Our codebase is now roughly 20 years old. It heavily and intrinsically relies on those very Gecko technologies that are now being faded out, more or less aggressively.
• JavaScript and HTML5 have evolved dramatically. Entire applications in JS are now realistic, and have been done. There are several existing JS libraries we might leverage. JavaScript is an efficient language, which allows fast development. A rewrite in JavaScript makes sense now.
• We will learn from shortcomings of existing Thunderbird, and solve them, for example a more flexible address book, and cleanly supporting virtual folders without overhead.
  
• The goal of the rewrite is to be close to the existing Thunderbird, in UI and features, as a drop-in replacement for end users, without baffling them. They should immediately recognize the replacement as the Thunderbird they love. It will install and run as normal desktop application, like Thunderbird does today. It keeps user data local and private.
  
• We can also make a new, fresh desktop UI, as alternative to the traditional one, for new users. The technology also gives us the option to run it as mobile app.
  
• While we implement the new version of Thunderbird, the old codebase based on Gecko will be maintained until the rewrite is ready to replace the old one.
• I expect this effort to take roughly 3 years: 1 year until some dogfood (usable by some developers and enthusiasts). 2 years until a basic feature set is there. 3 years until we can replace Thunderbird.

I think all this relies on a powerful replacement for the XUL treeview.  Do we have one now?

Matt

[R Kent James]

I'm generally in agreement with what you have said (and I think others
are as well), with the possible exception of whether to do the
transition as a complete replacement, or as a gradual conversion of the
existing codebase.

I've been working on the gradual replacement approach for the last few
years, in C++->JS conversion. The JsAccount technology, which is now in
Thunderbird 52, allows in theory the main back-end account modules
(IMAP, POP3, SMTP, NNTP, RSS) to be replaced by JS equivalents. I know
how hard it is to trace through the muck of existing implementations to
try to figure things like, how do I get the UI to actually show the
change I just did without having to do a restart or resync? And now
Ehsan says things like "oh by the way, we're not going to allow protocol
handler newURI in the future written in JS" and now the whole scheme
collapses, and JsAccount is going to have to adapt.

But at the same time, many, many attempts at rewrites fail. The most
valuable asset that Thunderbird has is our userbase, which in spite of
lousy communications and "Thunderbird is Dead", continues to slowly
grow. ADI on March 6 was 10,258,971 which I believe is a new high,
though we have been at just over 10,000,000 now for a couple of years so
growth is very slow. It is absolutely critical that we bring our users
along with us, and I have serious doubts about whether a sudden change
to a new UI and program is something they would follow. Read "The
Phoenix Project" for a classic description of the challenges and false
hopes of the Amazing Rewrite while the core technology collapses around you.

So I have mixed feelings.

But I have plans. I am specifically looking at the Address Book rewrite
as an opportunity to explore alternate approaches. The next step is to
come up with a very simple Contact Manager UI (possibly using the New
Zealand class project as a base), and exploring how to make that work in
a variety of target environments. That would include at least 1)
embedded in existing Thunderbird as an addon, 2) as a standalone
Electron App, 3) as a traditional website, and 4) as a Progressive Web
App. There are other possibilities as well, such as 5) using Myk Melez'
qbrt Gecko-based desktop, 6) .NET, 7) React-native for mobile.

I also am aware that one of the most challenging platforms is likely to
be 1) embedded in existing Thunderbird as an addon. I know the existing
nsIAB* interfaces pretty well, and interfacing with them at a low level
(as nsIAbDirectory implementations) is extremely challenging.
Practically it is impossible, as any reasonable future API for
Thunderbird contacts would be async, while the existing API is fully
sync with a hidden async implementation used in LDAP and relying on UI
notifications. So bolting onto Thunderbird is only going to be possible
by rewriting existing Thunderbird to support an async Contacts interface.

You (Ben) are saying that rewriting Thunderbird to support a new Address
Book implementation is not worth the effort, instead we should hold
back, and wait some years before we make this available to our users. I
have serious doubts about that approach. But at the same time, this
Thunderbird adaption is not a project I would want to give to my
students, so who is going to do it?

No easy answers, but I hope to make progress in the next year, and I
hope this will be my major focus.

:rkent
_______________________________________________
tb-planning mailing list
tb-pl...@mozilla.org
https://mail.mozilla.org/listinfo/tb-planning

[Ben Bucksch]

Matt Harris wrote on 24.03.2017 18:27:
> /I think all this relies on a powerful replacement for the XUL

> treeview. Do we have one now?

> /

Indeed, this is needed.

I've once found a "fast-list" or similar on github, that goes in that
direction. But no, we don't have it now. As mentioned, I have a plan how
to do it, but I haven't realized it yet. That's one of the things we'd
need to create as part of the framework.

That problem is inherent, because the XUL <tree> widget was originally
written specifically for Thunderbird. Firefox didn't need it. And it
might also go away now, if Firefox doesn't need it anymore.

I wrote about this in my proposal.

Ben

[Ben Bucksch]

R Kent James wrote on 24.03.2017 19:41:
> I know how hard it is to trace through the muck of existing
> implementations to try to figure things like, how do I get the UI to
> actually show the change I just did without having to do a restart or
> resync? And now Ehsan says things like "oh by the way, we're not going
> to allow protocol handler newURI in the future written in JS" and now
> the whole scheme collapses, and JsAccount is going to have to adapt.

Right. How much of your time have you spent a) writing the JS code that
keeps the list of accounts, and the object representing an account, and
the associated functions and b) trying to get all this hooked up to all
the rest of Thunderbird? Can you give a very rough estimate of hours for
each part?

I can tell you that with a fresh new implementation, it would take me
less than 1 day for part a), and part b) wouldn't exist, because the UI
is written anew using HTML anyway and would use the new API straight
away. Even if my numbers are a bit off: You see why I think that a
gradual approach is at least 4 times slower than a fresh start?

And that's not even considering the fact that a fresh start can use
design patterns correctly, which Thunderbird currently doesn't.

We're not talking about a small refactoring here, but replacing all the
core technologies and languages. Essentially, converting gradually is
like converting a 1969 Corvette into a 2015 Corvette, by replacing one
part after another. It just doesn't make sense.

> But at the same time, many, many attempts at rewrites fail.

I've personally managed 2 rewrites of commercial projects, and in both
cases, we finished faster than estimated, had no major problems on
release (much less than Version 1), and went from 1 million users to 3-5
million users, within a year. Of course, that was paired with a
marketing effort, but Version 1 also got marketing.

> The most valuable asset that Thunderbird has is our userbase, which in
> spite of lousy communications and "Thunderbird is Dead", continues to
> slowly grow. ADI on March 6 was 10,258,971 which I believe is a new
> high, though we have been at just over 10,000,000 now for a couple of
> years so growth is very slow. It is absolutely critical that we bring
> our users along with us

Kent, I totally, 100% agree on that. I dedicated a whole section of my
proposal to that.

> I have serious doubts about whether a sudden change to a new UI and
> program is something they would follow.

Agreed. That's why I don't want to do that, but preserve the UI. That's
explicitly the goal, for that very reason, because I completely agree
with you.

But we don't have much choice in the matter, right? XUL (or at least
parts of it) is going to go away sooner or later, no matter what we
decide here.

> I also am aware that one of the most challenging platforms is likely
> to be 1) embedded in existing Thunderbird as an addon. I know the
> existing nsIAB* interfaces pretty well, and interfacing with them at a
> low level (as nsIAbDirectory implementations) is extremely
> challenging. Practically it is impossible, as any reasonable future
> API for Thunderbird contacts would be async, while the existing API is
> fully sync with a hidden async implementation used in LDAP and relying
> on UI notifications. So bolting onto Thunderbird is only going to be
> possible by rewriting existing Thunderbird to support an async
> Contacts interface.

Right! This is the kind of thing I mean, why a gradual rewrite would
take so much longer than a complete rewrite. Because you can also use
modern APIs.

> You (Ben) are saying that rewriting Thunderbird to support a new
> Address Book implementation is not worth the effort, instead we should
> hold back, and wait some years before we make this available to our users.

2 years for most users, to be precise.

How many years are we already talking about overhauling the address
book? 3-5 years, if my memory serves me. Indeed, my memory still serves
me: It was none less than Mike Conley, in May 2012, 5 years ago:
<https://wiki.mozilla.org/Features/Thunderbird/Modern_Address_Book>.
Where are we today? Nothing happened. Yes, you're right, there were more
attempts, afterwards, all failed. I still have that old thing here. Why?
Because it's so hard to rewrite with the given constraints of
Thunderbird. So, years later, we're nowhere. The task itself isn't that
hard, I'd guess 3 person months for a talented, experienced developer.

We need to get moving, and a fresh start will allow us to drive fast,
and get somewhere.

BTW: If some volunteer wants to then backport the new AB implementation
to old Thunderbird, that would be great. But he's in for something...

I hope this post here clarifies why I think that a fresh start is our
best option at this time.

Ben

[Wayne Mery]

Actually, "talking" about it since 2007 :) where it was a known
impediment and under consideration, but it didn't make the first and
second cut for Mozillamessaging to take on the work. 2012 was just the
first time any action was taken, much less a spec.

It was my understanding that architectural constraints were never an
issue (or at least not a major issue) - the only issue was manpower.
Part of the plan (perhaps not on the wiki) was to write a new AB backend
abstracted such that it would easily transfer to a new platform. These
were things were discussed.

[Robert Kaiser]

Ben Bucksch schrieb:

> Matt Harris wrote on 24.03.2017 18:27:
>> /I think all this relies on a powerful replacement for the XUL
>> treeview. Do we have one now?
>> /
>
> Indeed, this is needed.
>
> I've once found a "fast-list" or similar on github, that goes in that
> direction. But no, we don't have it now. As mentioned, I have a plan
> how to do it, but I haven't realized it yet. That's one of the things
> we'd need to create as part of the framework.
>
> That problem is inherent, because the XUL <tree> widget was originally
> written specifically for Thunderbird. Firefox didn't need it. And it
> might also go away now, if Firefox doesn't need it anymore.

FWIW, I think a lot of the web would profit from a widget like that - so
if we can implement something like it as e.g. a web component, I think
there may be some interest in taking this as a base for a web standard
with an in-browser widget in the future - IMHO, this is one of the
biggest pieces HTML is lacking to become a really useful UI language
nowadays.

KaiRo

[Matt Harris]

On 3/28/17 6:18 AM, Robert Kaiser wrote:

Ben Bucksch schrieb:

Matt Harris wrote on 24.03.2017 18:27:

/I think all this relies on a powerful replacement for the XUL treeview.  Do we have one now?
/

Indeed, this is needed.

I've once found a "fast-list" or similar on github, that goes in that direction. But no, we don't have it now. As mentioned, I have a plan how to do it, but I haven't realized it yet. That's one of the things we'd need to create as part of the framework.

That problem is inherent, because the XUL <tree> widget was originally written specifically for Thunderbird. Firefox didn't need it. And it might also go away now, if Firefox doesn't need it anymore.

FWIW, I think a lot of the web would profit from a widget like that - so if we can implement something like it as e.g. a web component, I think there may be some interest in taking this as a base for a web standard with an in-browser widget in the future - IMHO, this is one of the biggest pieces HTML is lacking to become a really useful UI language nowadays.

KaiRo

At the risk of being a heretic, it would be useful in a "web basedThunderbird" if all the browsers implemented the tree we wanted to use as well. (android app perhaps?)

[Ben Bucksch]

Robert Kaiser wrote on 27.03.2017 21:48:
> Ben Bucksch schrieb:
>> Matt Harris wrote on 24.03.2017 18:27:
>>> /I think all this relies on a powerful replacement for the XUL
>>> treeview. Do we have one now?
>>> /
>>
>> Indeed, this is needed.
>>
>> I've once found a "fast-list" or similar on github, that goes in that
>> direction. But no, we don't have it now. As mentioned, I have a plan
>> how to do it, but I haven't realized it yet. That's one of the things
>> we'd need to create as part of the framework.
>>
>> That problem is inherent, because the XUL <tree> widget was
>> originally written specifically for Thunderbird. Firefox didn't need
>> it. And it might also go away now, if Firefox doesn't need it anymore.
>
> FWIW, I think a lot of the web would profit from a widget like that -
> so if we can implement something like it as e.g. a web component, I
> think there may be some interest in taking this as a base for a web
> standard with an in-browser widget in the future - IMHO, this is one
> of the biggest pieces HTML is lacking to become a really useful UI
> language nowadays.

There are many HTML tree widget implementations out there, but most fail
when you start pushing 100000 entries in there. The problem is that most
create a DOM nodes for every line and cell, and once you get to a
million DOM nodes, that's a noticeable load time and costs RAM. For
Thunderbird, we need to render folders with tens of thousands and a
hundred thousand emails.

I had mentioned that I had an idea how to implement it. I had not tried
it out, so I did that now.

Here's a prototype. This is just a start. It doesn't contain tree
functionality yet, just a table. The tree functionality would be added
on top. There are also many things to be polished: Alignment, fixing
column width to not change while scrolling etc..

But the basic idea of a fast list is working. It's loading 100000
entries on load, and you can add 100000 more on a button click. You will
notice that the addition is very fast. More importantly, scrolling is
very fast. Scrolling should work both using the scroll bar and using the
mouse wheel.

Try it out: http://benbucksch.github.io/trex/fastlist-test.html

How did I do that? The basic trick is that I don't create DOM nodes for
every rows, but only for the 10 or so visible rows. The data is in a
pure data array. When the user scrolls, I do not move or destroy DOM
nodes, but merely replace their text content and leave the nodes in
place. I then listen to mouse wheel scroll events, and scroll
correspondingly. The scroll bar at the right is a dummy element, and I
set the inner height of it to the calculated height in px that the rows
would have, if all rows would be DOM nodes. Then I listen to scroll
events again, and translate them to corresponding row content changes.
From what I understand, that is the basic principle that XUL <tree>s
also work with. Just that XUL <tree>s are implemented in C++, and I
implemented it in JavaScript and HTML. (And HTML was again the most
painful part in it.)

The coolest thing is that we're no longer limited to a single line per
row, and we can have rich HTML content in each cell. XUL <tree>s can do
neither, and that's a major limitation. We've always wanted to make the
message list prettier, but we couldn't. Now we can.

As said, there's a lot more polish to be done, but this is a proof of
concept that this works.

Enjoy.

Ben

[Gervase Markham]

On 28/03/17 05:23, Ben Bucksch wrote:
> There are many HTML tree widget implementations out there, but most fail
> when you start pushing 100000 entries in there. The problem is that most
> create a DOM nodes for every line and cell, and once you get to a
> million DOM nodes, that's a noticeable load time and costs RAM. For
> Thunderbird, we need to render folders with tens of thousands and a
> hundred thousand emails.

Except you don't, because one never displays that many all at one time.
Surely one could do a lazy-loading (and lazy-unloading) implementation?
We may have to keep all the folder data in memory, but that doesn't mean
it all has to be in the form of a single DOM.

> How did I do that? The basic trick is that I don't create DOM nodes for
> every rows, but only for the 10 or so visible rows. The data is in a
> pure data array. When the user scrolls, I do not move or destroy DOM
> nodes, but merely replace their text content and leave the nodes in
> place.

Er... great minds think alike :-)

Gerv

[Paul D. Fernhout]

Hi Ben,

This is a great proposal! Some ideas below to build on it.

## Minimal Viable Product

I completely agree the complete conversion effort in total will take
years (especially to move the plugin ecosystem over). But, to maintain
significant interest in the effort, I'd suggest aspiring for having a
minimally usable system to view and send email as plain text within a
month of starting (as in, one person full-time for a month).

## Thunderbird interoperation from the start

To make such a "eat our own dogfood" application useable from the start,
consider having Node.js write to the same mailbox files, address book,
and filter specifications that regular Thunderbird uses. Then POP users
(like myself) can use the new app for what it does (plain text email),
while also always being able to fire up the new Thunderbird for what it
can't do yet (like some plugins for calendaring or encryption or doing
IMAP). We can make a lot of progress on a POP-only Thunderbird's UI
(plus RSS feeds etc.), and then at some point IMAP support could be added.

Yes, anyone who does the above is living dangerously as they put their
real email at risk and so needs to make frequent backups. In order to
make this safer, we could perhaps tweak the existing Thunderbird (as a
config option) to check a lock file (or whatever) before starting up so
the two apps will not both work at the same time on the same files and
confuse each other. Or maybe we could ideally figure out a smoother way
for both applications to both be running at the same time through
cooperative file locking or such. Or maybe to minimize the risk of
corruption we could make some less minor changes to the existing
Thunderbird codebase so it could serve up and receive messages,
addresses, and filters via a socket to a Node.js server which then
serves the web-technology UI (but that is starting to mean more
wrestling with the existing codebase which might introduce new bugs and
probably won't be as much fun).

## Mobile from the start

Also, I'd suggest adding an extra mobile aspect early so that you could,
if you want, check your Thunderbird email from any web-enabled device
like a smartphone, tablet, or Chromebook that can access your
Thunderbird backend server. That new feature could be a big win for home
or corporate users relying on POP. Obviously, security is an issue if
you let the new Thunderbird backend serve such remote apps.

## Analytic Visualizations from the start

Another thing to add to the new Thunderbird from the start is
visualization features and analytics, like showing you graphs of who you
email with over time about what. This could be done in readonly mode for
existing mboxes. A big value of this is to get people installing
this new web-technology Thunderbird for just these analytic features,
even if they do their actual emailing mainly entirely in the existing
Thunderbird at the start. In fact, I'd suggest we could even consider
start with this part (reading mbox files without writing them) as a
Thunderbird companion and then build out from there.

## Technology stack

I'd suggest using TypeScript, Mithril.js, Tachyons, and
Node.js for the base technologies -- probably with Mocha/Chai for
testing. Later we can make a more stand-alone desktop version with
Electron or such. We could use socket.io to push notifications for
changes in real or virtual folders and then use Mithril.js (a vdom
library) to quickly rerender changed data the client UI requests (or is
pushed). I could explain those choices further but it would take a while.

## A previous step forward by me

Here is my past attempt in the direction of a web-technology-based
Thunderbird Server (with some help by David Krings for design and
testing) -- but I ended up taking a full-time+ paying job doing UI work
for a SV company soon after starting that effort, so not much time for
moving it further along, sadly.
https://github.com/pdfernhout/Twirlip2

A key useful thing I developed there (improving on something first
developed for another project) was displaying HTML reasonably safely
using Mithril (crossing fingers).
https://github.com/pdfernhout/Twirlip2/blob/master/source/webapp/ts/sanitizeHTML.ts

I use that sanitizer in the only significantly working plugin there (an
RSS feed reader where you can choose your level of danger for displaying
HTML[1]):
https://github.com/pdfernhout/Twirlip2/blob/master/source/webapp/ts/plugins-for-local-libre-or-standard-services/rss/rss-plugin.ts

A demo of the RSS feed reader plugin can be found at the bottom of this
page (it's a test server and may not stay up indefinitely):
http://twirlip.com:9000/twirlip.html

## On grids and performance

On a tangent about grids since they are of concern, here is a grid in
Mithril.js that can display tens of thousands of items scrolling fluidly
in the browser using a vdom approach:
http://jsfiddle.net/7JNUy/1/

I discuss that grid scaling issue further here:
https://github.com/dojo/meta/issues/11#issuecomment-177547645

## Learning from my mistakes

One of the (many) too ambitious parts of the above effort was to use a
different storage backend. Starting using mbox in parallel with the
current Thunderbird (as above) would have been a better strategy at the
start. So would have been would have been focusing on visualizations of
those existing mailboxes.

## Future directions

Once all that works, and there is useable UI frontend, we can talk about
having plug-able backends perhaps to support advanced UI plugins (like I
brainstormed in the plugin directory names in that project).
https://github.com/pdfernhout/Twirlip2/tree/master/source/webapp/ts/plugins-for-local-libre-or-standard-services
https://github.com/pdfernhout/Twirlip2/tree/master/source/webapp/ts/plugins-for-specific-websites-or-proprietary-services

--Paul Fernhout (pdfernhout.net)
"The biggest challenge of the 21st century is the irony of technologies
of abundance in the hands of those still thinking in terms of scarcity."

[1] Ultimately the safest way to handle displaying full HTML might be to
have every email or RSS feed be displayed its own subdomain served by
the server using the item's message ID...

On 2017-03-24 1:04 PM, Ben Bucksch wrote:
> Summary:
>
> * Our base goes away. Gecko will change dramatically in the future,

> and dropping many features that Firefox does not need anymore, but

> that Thunderbird relies on. * Our codebase is now roughly 20 years

> old. It heavily and intrinsically relies on those very Gecko
> technologies that are now being faded out, more or less aggressively.

> * JavaScript and HTML5 have evolved dramatically. Entire applications

> in JS are now realistic, and have been done. There are several
> existing JS libraries we might leverage. JavaScript is an efficient
> language, which allows fast development. A rewrite in JavaScript

> makes sense now. * We will learn from shortcomings of existing

> Thunderbird, and solve them, for example a more flexible address

> book, and cleanly supporting virtual folders without overhead. * The

> goal of the rewrite is to be close to the existing Thunderbird, in UI
> and features, as a drop-in replacement for end users, without
> baffling them. They should immediately recognize the replacement as
> the Thunderbird they love. It will install and run as normal desktop
> application, like Thunderbird does today. It keeps user data local

> and private. * We can also make a new, fresh desktop UI, as

> alternative to the traditional one, for new users. The technology

> also gives us the option to run it as mobile app. * While we

> implement the new version of Thunderbird, the old codebase based on
> Gecko will be maintained until the rewrite is ready to replace the

> old one. * I expect this effort to take roughly 3 years: 1 year until

[Ben Bucksch]

Hey Paul.

Paul D. Fernhout wrote on 28.03.2017 04:46:
> This is a great proposal! Some ideas below to build on it.

Thank you for the responses :)

> ## Minimal Viable Product
>
> I completely agree the complete conversion effort in total will take
> years (especially to move the plugin ecosystem over). But, to maintain
> significant interest in the effort, I'd suggest aspiring for having a
> minimally usable system to view and send email

Completely agreed! I subscribe to the same idea.

That's why I defined 3 major milestones. I want to set expectations. It
would be highly unrealistic to have feature parity with Thunderbird in
the first release. But we can produce a product that will be usable to
some parts of the userbase much earlier.

> as plain text within a month of starting (as in, one person full-time for a month).

This, however, is a little ambitious and can be very detrimental.
There's a lot of groundwork to lay. With design and framework, it's
harmful to get started without it and hack it first and try to use a
better design later, because it's not realistically possible anymore.
You'll have to change everything again. Realistically, it will not be
done. So, whatever design you use at the start is what you will always
have. Thunderbird is a good example of this. It still, up to this day,
has the very same design and architecture that it had 18 years ago.
There are clear shortcomings, with clear disadvantages (e.g. unclear API
separation, module dependency that looks like a cat had her way with a
ball of wool, no clear extension hooks, thus hard to hack), but nobody
can solve them or has the will and manpower to solve them.
MozillaMessaging wanted to, and tried, but couldn't, despite 6-10 people
on payroll.

The framework and APIs need to be designed well from the start. Likewise
the widget set. That will massively influence the quality of the
outcome, and the joy it is to hack it.

For example, I want to create consistent and clear APIs between backend
and UI, and between different UI parts.

> ## Thunderbird interoperation from the start

I think we'll support IMAP from the start. I already have a partial,
read-only implementation of IMAP and POP3 in JavaScript. Of course it's
just a start, but it shows it's feasible.

IMAP is inherently built to support multiple clients on the same account
(preferably not at the same time, although it's possible in theory).

> Yes, anyone who does the above is living dangerously as they put their
> real email at risk and so needs to make frequent backups.

yup :)

> ## Mobile from the start

I want the new implementation to run on smart phones, yes. Using
Cordova. Obviously, we need a different UI layout, but fundamentally, it
should be possible.

That smartphone UI wouldn't be something I write from the start, but I
want to architecturally enable it from the start.

> ## Analytic Visualizations from the start

This is going directly against your MVP above :). First we need to read
and send email.

I totally want visualizations. I really hope for it. But that's
something for year 3 or 4. That's the icing on the cake. You first need
to lay the groundwork.

If you (or anybody else) have ideas for concrete visualizations, could
you please post them somewhere? I am very much looking for ideas in that
area. (I am looking at asutherland's glodastrope already.)

> ## Technology stack
>
> I'd suggest using TypeScript, Mithril.js, Tachyons, and
> Node.js for the base technologies -- probably with Mocha/Chai for
> testing. Later we can make a more stand-alone desktop version with
> Electron or such.

Thunderbird is going to be a desktop application from the start. The
most fundamental reason is what we need TCP sockets for IMAP and POP3,
and read/write local folders, using mbox, maildir or other formats.

There are several email clients already that go the web server proxy
way, e.g. Nylon N1. That's also why I don't use any of them. I want a
direct replacement for Thunderbird.

Using TypeScript is something we could consider.
We'll most likely use node.js, in some form or other - if nothing else,
then for the npm module installation.
I'll take a look at the other libraries.

If any one of you has experiences with good JS libraries, that's
probably useful. I want to be easy on them, because the more JS code we
import, the slower the app will be, but if a lib is small and provides a
huge benefit, that's helpful.

> We could use socket.io to push notifications for
> changes in real or virtual folders and then use Mithril.js (a vdom
> library) to quickly rerender changed data the client UI requests (or is
> pushed).

Mithril.js looks very similar to the mini-lib that I've been writing for
myself <http://benbucksch.github.io/trex/util.js>. It looks nice, it's
worth considering.

I want to put observable lists everywhere in the API. Virtual folders
are just one application. I've already written the library
https://github.com/benbucksch/jscollection I am not aware of anything
else that can do dynamically observable operators.

> ## A previous step forward by me
>
> Here is my past attempt in the direction of a web-technology-based
> Thunderbird Server (with some help by David Krings for design and
> testing) -- but I ended up taking a full-time+ paying job doing UI work
> for a SV company soon after starting that effort, so not much time for
> moving it further along, sadly.

Too bad, would have been nice to have you helping with this endeavor.

> https://github.com/pdfernhout/Twirlip2

> A key useful thing I developed there (improving on something first
> developed for another project) was displaying HTML reasonably safely
> using Mithril (crossing fingers).
> https://github.com/pdfernhout/Twirlip2/blob/master/source/webapp/ts/sanitizeHTML.ts

Lovely. That's the same fundamental approach that I also used in the
"HTML sanitizer" used in Thunderbird ("Message | View | As Simple HTML").

For rendering messages, I want to use a sanitizer like that, plus
additionally an <iframe> sandbox (different domain) on the browser
level. When it comes to security, I believe in multi-layered defenses.
I've seen too much ;).

> [1] Ultimately the safest way to handle displaying full HTML might be to
> have every email or RSS feed be displayed its own subdomain served by
> the server using the item's message ID...

Exactly.

> http://twirlip.com:9000/twirlip.html

Nice! to have a running demo on the web :)

> ## Learning from my mistakes
>
> One of the (many) too ambitious parts of the above effort was to use a
> different storage backend.

MVP :)

> "The biggest challenge of the 21st century is the irony of technologies
> of abundance in the hands of those still thinking in terms of scarcity."

Surely true when it comes to contributors :-) . The more inviting and
exciting we are, the more will come.


Thanks for this qualified response! Too bad you can't help us with the
implementation.

Ben

[Philipp Kewisch]

On 3/28/17 8:50 AM, Gervase Markham wrote:
>> How did I do that? The basic trick is that I don't create DOM nodes for
>> every rows, but only for the 10 or so visible rows. The data is in a
>> pure data array. When the user scrolls, I do not move or destroy DOM
>> nodes, but merely replace their text content and leave the nodes in
>> place.
> Er... great minds think alike :-)

They do indeed. I'm pretty sure roc's fast list is doing the same. See
links here: https://bugzilla.mozilla.org/show_bug.cgi?id=1183812#c3

Philipp

[Jim]

On Tue, Mar 28, 2017 at 11:08 AM, Ben Bucksch <ben.b...@beonex.com> wrote:

I think we'll support IMAP from the start. I already have a partial, read-only implementation of IMAP and POP3 in JavaScript. Of course it's just a start, but it shows it's feasible.

Have you looked at GELAM[1]? It should have a fairly-complete JS implementation of IMAP, POP3, and ActiveSync (although actually *using* ActiveSync in a product may require licensing fees). There's also glodastrophe[2], which might be a good jumping-off point for a JS-powerd email client.

- Jim

[1] https://github.com/mozilla-b2g/gaia-email-libs-and-more
[2] https://github.com/asutherland/glodastrophe

[Ben Bucksch]

Jim wrote on 31.03.2017 06:00:
> On Tue, Mar 28, 2017 at 11:08 AM, Ben Bucksch <ben.b...@beonex.com
> <mailto:ben.b...@beonex.com>> wrote:
>
> I think we'll support IMAP from the start. I already have a
> partial, read-only implementation of IMAP and POP3 in JavaScript.
> Of course it's just a start, but it shows it's feasible.
>
>
> Have you looked at GELAM[1]? It should have a fairly-complete JS
> implementation of IMAP, POP3, and ActiveSync (although actually
> *using* ActiveSync in a product may require licensing fees). There's
> also glodastrophe[2], which might be a good jumping-off point for a
> JS-powerd email client.

Yes, thank you. I'm aware of it, and I'm trying to get it to run (so far
unsuccessfully).

GELAM/gaia/glodastrophe, emailjs.org, and my first-hand experiences of
implementing IMAP and other IETF protocols in JS in very little time,
make me hopeful that this is the success path, rather than trying to
reuse code from 1995.

[Gervase Markham]

On 28/03/17 03:46, Paul D. Fernhout wrote:
> To make such a "eat our own dogfood" application useable from the start,
> consider having Node.js write to the same mailbox files, address book,
> and filter specifications that regular Thunderbird uses. Then POP users
> (like myself) can use the new app for what it does (plain text email),
> while also always being able to fire up the new Thunderbird for what it
> can't do yet (like some plugins for calendaring or encryption or doing
> IMAP). We can make a lot of progress on a POP-only Thunderbird's UI
> (plus RSS feeds etc.), and then at some point IMAP support could be added.

Why is POP and not IMAP in the MVP? I'd say if you are going to do one,
IMAP is the modern mail access protocol.

> ## Analytic Visualizations from the start
>
> Another thing to add to the new Thunderbird from the start is
> visualization features and analytics,

Do you know what the word "Minimum" means? ;-)

Gerv

[Joshua Cranmer 🐧]

On 3/24/2017 12:04 PM, Ben Bucksch wrote:

Summary:

• Our base goes away. Gecko will change dramatically in the future, and dropping many features that Firefox does not need anymore, but that Thunderbird relies on.
• Our codebase is now roughly 20 years old. It heavily and intrinsically relies on those very Gecko technologies that are now being faded out, more or less aggressively.
• JavaScript and HTML5 have evolved dramatically. Entire applications in JS are now realistic, and have been done. There are several existing JS libraries we might leverage. JavaScript is an efficient language, which allows fast development. A rewrite in JavaScript makes sense now.
• We will learn from shortcomings of existing Thunderbird, and solve them, for example a more flexible address book, and cleanly supporting virtual folders without overhead.
  
• The goal of the rewrite is to be close to the existing Thunderbird, in UI and features, as a drop-in replacement for end users, without baffling them. They should immediately recognize the replacement as the Thunderbird they love. It will install and run as normal desktop application, like Thunderbird does today. It keeps user data local and private.
  
• We can also make a new, fresh desktop UI, as alternative to the traditional one, for new users. The technology also gives us the option to run it as mobile app.
  
• While we implement the new version of Thunderbird, the old codebase based on Gecko will be maintained until the rewrite is ready to replace the old one.
• I expect this effort to take roughly 3 years: 1 year until some dogfood (usable by some developers and enthusiasts). 2 years until a basic feature set is there. 3 years until we can replace Thunderbird.
  

I will describe each point in more detail below.

We've talked about rewriting Thunderbird for, oh, a decade or more now. We've had agreement on the direction these rewrites should take for at least 3 or 4 years (rkent was really one of the last hold-outs for using JS). But we've not really made any measurable progress on rewriting Thunderbird.

The problem with rewrites is that our codebase is very full-featured, and it's hard to replicate that functionality. And I'm not talking about features like multipart/appledouble, which I don't think anyone objects to throwing in a trash can. Things like S/MIME integration, downgrading text/html to text/plain, etc. were the ones that halted my work on JSMime. And some features require deep integration into the interface to work properly (the DNS PGP/SMIME key lookup functionality or NNTP's per-group password support comes to mind, although the latter one is possibly something to be trashed).

You also have to deal with the very fundamental reality that JS, actually, isn't all that well-suited to implementing an email client. For any functionality that's not in the core ES spec, every platform has a more or less fundamentally different way of representing it. Charset conversion, crypto, TCP socket support, even how you view generic I/O primitives are all different in each platform, to say nothing of issues like FFI (which you'll want to support Kerberos or LDAP). There's also the issue of email's inherent binary/text intermingling working poorly with JS's Unicode string/binarystring/Uint8Array confusion (it's somewhat less of an issue in languages like C++ or Rust, where there's nop conversion steps to treating the bytes as binary or as strings).

Now, the final and biggest problem with rewrites is that there's no backup plan if they fail. It's quite frankly inexcusable that Thunderbird doesn't have CardDAV support yet, and there is exactly one reason why we don't: that feature has been gated on the planned AB rewrite for the past 10 years. We have a partially-functional from-scratch rewrite (mconley's Ensemble project), and we even have a CardDAV implementation for it. But when the lone developer lost the time to work on it, all of that effort went to nought. Our internationalized email support is something else that is being negatively impacted by being gated on a perpetually-soon rewrite. And with your proposal, if there's any delay in the process, if anything takes longer than planned to implement or iron out bugs, then delivering any improvement, even one completely orthogonal, needs to wait in line.

At the end of the day, though, that doesn't mean that there's many other realistic options. A complete rewrite ends up being the least bad option. The main thing I object to is your plan to skeleton-maintain the existing TB implementation and then switch over to brand-new TB all at once. Doing it like that runs the risk of having the One Big Release that everyone hates (remember Vista?), and having that after a period of basically little improvements could prove an unrecoverable PR blow. Rather, you should think about how to structure the rewrite so that you can import bits and pieces of it into the existing code base as they're ready. I mean, this is what Mozilla is doing with Firefox and Servo, so there's ample precedent to look at. They even have the problem of "there's a giant mess of interconnected APIs"--the CSS rendering bits can be thought of roughly analogous to our account API. Obvious places to start are the calendar and contacts, because those are mostly independent of the rest of the codebase (excluding search, there's like 5 calls to the address book, and mconley already documented the list somewhere). In terms of other probable components that could be moved and maintained in tandem, there's platform abstraction (including utils like charset conversion or base64), individual protocol implementations (roughly ns*Protocol), import, MIME, compose, the database, search, Junk filters, the account implementation, the front-end, and test infrastructure. Obviously, there's some dependencies in there, but that should sum up most of the pieces.

I will point out that I have partial or complete implementations of a lot of stuff. It would take me a while to catalog everything and figure out what I've made public and what I haven't, but I do know that I have:
* SASL (excluding only GSSAPI at this point, because I couldn't figure out how to test it),
* a socket abstraction (the intent is to wrap a TCP socket into a connection pool interface that supports the text/binary confusion issue, both IMAP-style read N bytes and POP-style dot quoted, and STARTTLS support)
* an NNTP implementation on top of that socket abstraction
* JSMime (my local copy is about 80% of the way to ditching nsMsgSend and friends, i.e, I have fairly robust composition support)
* Infrastructure for running Dovecot and OpenLDAP in a Docker container for testing purposes
* a very rough basis to generate a realistic-looking dataset for performance testing (as in, grabbing census data to find name distributions, looking up domain name frequencies, etc.)
* a proposal for a multithreaded, JS/WebWorker-centric DB API for Thunderbird, along with an implementation of cross-thread database notification (not trivial in WebWorkers).

A final note is that, while there do exist partial implementations of a lot stuff, it is probably better to start implementation from scratch on a lot of stuff. Things like email.js's smtpclient.js make a lot of assumptions in their API (e.g., "I'll only get UTF-8 strings") that aren't really safe if you want to scale to a 20M client base. When I brought that up with the email.js list, they did more or less agree in those deficiencies, and they were willing to let me propose an alternative implementation to handle things that we needed, like DSN support.

In short: I think you need to give a lot more thought into breaking a complete redesign idea into smaller modules that can be independently reused, and you also need to think about how to import those modules into TB in a finer-grained fashion than all-or-nothing.

-- 
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

[Tanstaafl]

On Mon Apr 03 2017 12:03:22 GMT-0400 (Eastern Standard Time), Gervase Markham <ge...@mozilla.org> wrote:

Why is POP and not IMAP in the MVP? I'd say if you are going to do one,
IMAP is the modern mail access protocol.

Actually, I suggest that IMAP is no longer the 'modern' one. While I am not seriously suggesting the new TBird shouldn't support IMAP, I suggest JMAP support be made a mandatory criteria. JMAP is designed to provide all of the advantages of IMAP, while addressing most if not all of its shortcomings:

http://jmap.io/

Both Cyrus (3+) and Dovecot (2.3+) are primed to support it, and there is a Thunderbird bug to add support for it:

https://bugzilla.mozilla.org/show_bug.cgi?id=1322991

[Tanstaafl]

On Mon Apr 03 2017 12:08:23 GMT-0400 (Eastern Standard Time), Joshua

Cranmer 🐧 <pidg...@gmail.com> wrote:
> In short: I think you need to give a lot more thought into breaking a
> complete redesign idea into smaller modules that can be independently
> reused, and you also need to think about how to import those modules
> into TB in a finer-grained fashion than all-or-nothing.

I've made the exact same suggestion and always felt it makes more sense,
but being that ianap, I couldn't/can't articulate a strong argument.

But modularizing it just makes sense. This can use the same exact
process used now for thrashing out bugs - early releas, beta, and
Release channels. Then it is just on users to make sure they keep their
profile backed up in case something trashes it.

It might actually end up being more work - even significantly more work
- but the risk is vastly reduced, even eliminated.

[Ben Bucksch]

On Mon Apr 03 2017 12:03:22 GMT-0400 (Eastern Standard Time), Gervase Markham <ge...@mozilla.org> wrote:

Why is POP and not IMAP in the MVP? I'd say if you are going to do one,
IMAP is the modern mail access protocol.

If I have a say in it, the new Thunderbird will support IMAP from the get-go. IMAP is not an issue. I already have IMAP implemented, for simple reading.

The actual problems I am facing are more technical, e.g. which JS module system do we use? <script> or require() sync or require() async or TypeScript modules or something else? How do the modules communicate?

Tanstaafl wrote on 03.04.2017 18:09:

Actually, I suggest that IMAP is no longer the 'modern' one.

I'll join you there.

I suggest JMAP support be made a mandatory criteria.

Again, MVP. You can work without JMAP, so it's not a minimal criteria, and not a mandatory one.

Whether we support JMAP or web forums or whatever you can think of is not a question at this point in time. The answer is: Eventually yes, if someone cares about it enough. It should be much much easier to implement than in current Thunderbird.

Ben

[neandr]

Following Ben's or Joshua's points about a next-TB it will raise
questions which technology stack, tools, testing, design rules etc
should be used, much more should be generalized for it.

With the 'contacts/caldav' project which has been run last year at the
New Zealand University it was obvious using a technology kit like React
gives a great boost for productivity and structuring the solution. But
already that small project -- compared what a next-TB will be -- showed
also there are many possibilities for "dialects" with the various parts.
So before starting the next-TB project it would make sense to have a
common understanding for the technology base.

Paul suggested a few points for the technology stack and more, he
pointed to an important issues: legal aspects. For React there could be
a patent problem, see here:

> https://github.com/facebook/react/blob/master/PATENTS
​And also Paul's contribution to a Drupal discussion about React usage
is worth to read:
> http://buytaert.net/comment/120741#comment-120741

React is used with the DevTools and it would be interesting to hear from
the DevTools how they live with that patent issue. Thunderbird where
ever it finds it's new home should be save from the start with new
technologies and 3rd-party contributions.

This is only one example why a serious analysis is necessary and
stringent standards are an absolute must.

[Ben Bucksch]

Joshua Cranmer 🐧 wrote on 03.04.2017 18:08:
>
> We've talked about rewriting Thunderbird for, oh, a decade or more
> now. We've had agreement on the direction these rewrites should take
> for at least 3 or 4 years (rkent was really one of the last hold-outs
> for using JS). But we've not really made any measurable progress on
> rewriting Thunderbird.

And you know why? Because nobody started. If we had started "a decade or
more" ago, we would be done since 5 years, would not be fighting Gecko
regressions today, and would be adding the features that the users today
are screaming for.

This is why I say: We have to make a bold move now. Now or never. If we
don't do it now, TB will survive another few years and then die together
with XUL and XPCOM and sleep in the same grave. And our users will be in
the winds.

I want to give them a new home. One that feels like home for them.

> The problem with rewrites is that our codebase is very full-featured,
> and it's hard to replicate that functionality.

We won't be the first email client mostly in JS, and we won't even be
the first email rewrite under the Mozilla roof. I've been there for the
4.5 -> Seamonkey rewrite, as a lowly volunteer contributor.

Inaction is our biggest risk. The task is big, but doable. It's been
done before.

Please remember many of the features Thunderbird has are features for
the pre-2000 area. Like LDAP or Kerberos. Users today are asking for
completely different things, like syncing the address book with their
Android phone. I am not saying we should cut LDAP, given that there are
still people using it, but let's not forget that Thunderbird currently
does *not* have a lot of things that users desperately need.

> Things like S/MIME integration, downgrading text/html to text/plain,
> etc. were the ones that halted my work on JSMime.

None of these are a serious problem.

S/MIME would be an extension. It's currently also implemented as
extension in Thunderbird (mailnews/extensions/smime), just shipped with
core. I don't see why we couldn't do the same in a new Thunderbird.

I wrote the plaintext <-> HTML conversion and the downgrade etc. back
then. In fact, that would be much easier with a fresh design, in JS,
than with the code written originally 1993-something in C.

Of course, it needs to be designed properly. I cannot use the same
design that libmime currently uses. That's why it's so hard to replace
e.g. libmime. And that is also why a gradual rewrite fails.

> the DNS PGP/SMIME key lookup functionality

Huch? DNS? DNS is exactly one of the biggest problems in Gecko. I
desperately needed DNS SRV support for XMPP in JS, but couldn't get it
with Gecko. I needed it so desperately, I've even implemented patches
for Gecko to implement DNS SRC and TXT and others. That was 10 years
ago, with a lot of work. They were never reviewed. :-( 10 years later,
they are still rotting away in bugzilla. All that work was wasted.

For XMPP, I also needed to override the certificate check, to check
against a different hostname. Not possible in Gecko. I made patches, it
was in fact trivial to add to PSM/NSS. The patches were rejected,
basically with rationale "We don't need that". Patch is still sitting in
bugzilla somewhere, being ignored.

So, no, we don't have many features we would *need*, and that we could
normally easily implement. Gecko has been holding us back massively.
That's concrete experience.

> Now, the final and biggest problem with rewrites is that there's no
> backup plan if they fail

We do have a fallback. Thunderbird based on Gecko would continue to be
supported on the current level.

The 2 projects would run in parallel and independent of each other, they
share the project umbrella and the goal. The teams would be separate.
The Thunderbird-based-on-Gecko team would be free to pick any JS/HTML
implementations created for the new client and backport them to the old
client. I'm pretty sure we'd finally get the new AB this way.

The old Thunderbird could still get new features through the backports,
but the team for the future would not be held back by constraints from
the pre-2000 area. It could concentrate on advancing as quickly as
possible, with the goal of 99% feature parity with Thunderbird.

The gradual rewrite is what has failed. That's not just a possibility.
We tried and failed. As you said above, we wanted to do it since a long
time, and never succeeded. Thinking that we'll suddenly succeed now,
with less resources, is merely wishful thinking. And if that fails,
Thunderbird is dead for good, because XUL is going away eventually.

Sometimes, a picture says it best. Attached.

Ben

[Ben Bucksch]

Joshua Cranmer 🐧 wrote on 03.04.2017 18:08:

> In short: I think you need to give a lot more thought into breaking a
> complete redesign idea into smaller modules that can be independently
> reused, and you also need to think about how to import those modules
> into TB in a finer-grained fashion than all-or-nothing.

FWIW, any that will be a natural fallout of any well-designed rewrite.
Of course we wouldn't write a monolithic, take-all-or-nothing webapp,
but we'd write components. And these could be used by the old
Thunderbird. I'm all for that.

The important difference is:

* the new modules need to be designed for the future, not the past. This
is why past attempts failed: Thunderbird APIs between modules are often
unfit.

* the integration into the old Thunderbird probably takes more time than
writing the component in the first place. By making the backport an
explicit, separate step, you'd see that cost clearly. More importantly,
you can separate the tasks to different teams. And you limit the risk:
If the backport fails, you still have the new implementation.

Ben

[Joshua Cranmer 🐧]

On 4/3/2017 5:00 PM, Ben Bucksch wrote:

Joshua Cranmer 🐧 wrote on 03.04.2017 18:08:

We've talked about rewriting Thunderbird for, oh, a decade or more now. We've had agreement on the direction these rewrites should take for at least 3 or 4 years (rkent was really one of the last hold-outs for using JS). But we've not really made any measurable progress on rewriting Thunderbird.

And you know why? Because nobody started. If we had started "a decade or more" ago, we would be done since 5 years, would not be fighting Gecko regressions today, and would be adding the features that the users today are screaming for.

It is a lie to say that no one started. There have been at least two projects with the clear intention of rewriting major portions of Thunderbird that produced concrete results yet have failed to come to fruition (Ensemble and JSMime), and depending on how and what you want to count, probably another 4 or 5 that didn't produce actionable results or were not clearly intended to replace existing code (e.g., Raindrop, Gloda). If you want to make progress, you need to learn from mistakes. Denying the existence of those mistakes will only lead you on the path to ruin.

Inaction is our biggest risk. The task is big, but doable. It's been done before.

Inaction is not the biggest risk. I am not proposing inaction. I am not objecting to the idea of rewriting most/all of Thunderbird. Hell, I have put more work into rewriting Thunderbird in the past decade than you or probably anyone else on this thread has. That's not the problem with your proposal. The problem is that you're suggesting... well, basically inaction (on a feature level) while you go off and work on a separate project, hopefully to force it down everyone's throats when it's done. And arguing that it will be so wonderful and timely that there's not going to be any problems forcing people to switch (I am aware that this is probably a gross caricature of what you're proposing). That is what concerns me: that you're going to hit a roadbump, and everything is going to go wrong, and there is going to be no Plan B.

The biggest risk is that one snag derails the entire project. Suppose, for example, that you have problems with unacceptable performance on message display, and that this resists all attempts to bring it to the performance level of existing Thunderbird code for years. (This is not unrealistic: this is exactly what happened to ical.js). What do you do? This is the sort of risk that concerns me deeply, and the best way to mitigate is to minimize the criticality of delivering any one individual feature. Proposing a complete rewrite and not seeking gradual uplifiting is exactly the opposite of that strategy.

I will reiterate, again, to take a look at how Mozilla is integrating Servo into Gecko. Note what they are doing (moving things piecemeal) and what they are not doing (halting development of Gecko while they implement Servo). This is a project with far more available resources, both absolutely and in proportion to its size and userbase, than Thunderbird.

We do have a fallback. Thunderbird based on Gecko would continue to be supported on the current level.

That is not a backup plan, especially not if XUL Gecko dying is one of the main motivations for embarking on this project.

The 2 projects would run in parallel and independent of each other, they share the project umbrella and the goal. The teams would be separate. The Thunderbird-based-on-Gecko team would be free to pick any JS/HTML implementations created for the new client and backport them to the old client. I'm pretty sure we'd finally get the new AB this way.

If you expect the end-goal to be "TB users get force^H^H^H^H^Hauto-upgraded to new product" and expect that to go without serious objection, you are sorely mistaken. There needs to be communication and liasons between the two teams if you want it to go smoothly. Some more points:

1. Realistically, you're proposing a skeleton maintenance team for TB, more or less what we've had for the past 5 years or so. This team will likely have to spend so much of its time keeping TB building that they are unlikely to have much time or ability to port new components to it.
2. Thunderbird has 15+M users, your new product will have maybe hundreds. Email is rife with buggy implementations and has no general conformance testsuite, so you're not going to find problems until you get actual users with their weird edge cases. Your developers should be jumping at the chance to have their work in Thunderbird.
3. I am not proposing to maintain the current APIs. Nor am I proposing to maintain the current feature set. What I am proposing is that the idea of "how can we get this into Thunderbird without getting all the unfinished stuff" should be a first-level concern. The answer might be that it's impossible--transitioning the database API is perhaps infeasible, and as a result, migrating anything that relies on the database wouldn't be possible. On the other hand, structuring your nntpclient implementation to roughly correspond to the methods on nsINntpService does seem like a good idea.
4. There is a lot of great technical knowledge stored in the TB repository (in a hard-to-read form, sure). It's easy to make assumptions based on specifications that aren't true in practice (e.g., the charset of message headers), and it's all to easy to make a design that's hostile to the changes needed to support those assumptions (this, broadly, is the failure of libmime).

The gradual rewrite is what has failed. That's not just a possibility. We tried and failed. As you said above, we wanted to do it since a long time, and never succeeded. Thinking that we'll suddenly succeed now, with less resources, is merely wishful thinking. And if that fails, Thunderbird is dead for good, because XUL is going away eventually.

The glaring failures we have are not that the rewrites failed. We could have had CardDAV 5 years ago, but we decided to have it implemented in Ensemble instead. I could have finished EAI support years ago, instead I made it contingent on JSMime. Those rewrites failed ultimately due to lack of manpower, and not because they were trying to maintain current APIs (Ensemble didn't try to, JSMime did seek to maintain compatibility in the interim but wasn't designed on it).

I keep going back to Servo, but it's a good model IMO. By all means, incubate implementations in a product outside of Thunderbird. Feel free to experiment on alternative methodologies without worrying about how exactly to squeeze it into Thunderbird. BUT when you draw up the roadmaps, when you write out the work plans, when you farm out the coding tasks, have bullet points that ask "What can we deliver to Thunderbird?" Servo is not the replacement for Firefox, it is the product in which improvements to Firefox are incubated.

[Paul D. Fernhout]

Thanks, neandr! Here is more recent stuff I collected on the
React/Facebook patent issue including opinions from a couple of legal teams:
https://github.com/Automattic/wp-calypso/issues/650#issuecomment-282932740

As indicated there (citing the next link), people are still (as of the
beginning of the year) still waiting on opinions from OSI and FSF about
the React license.
https://discuss.reactjs.org/t/legal-department-did-not-allow-use-of-react/3309

What's especially confusing about React's licensing situation is that is
has been under three license -- Apache, then a really bad version of
BSD+PATENTS, then a not quite so bad so bad version of BSD+PATENTS (but
still problematical).

--Paul Fernhout (pdfernhout.net)

"The biggest challenge of the 21st century is the irony of technologies
of abundance in the hands of those still thinking in terms of scarcity."

[Paul D. Fernhout]

Hi Joshua,

Your point on risk management makes a lot of sense. That's why I
suggested a possibility of having a new Thunderbird that could run
alongside the existing one and share the same files while offering a
different UI. That is the simplest risk-management strategy I can think
of for one developer with web technology experience to take.

To address this risk in another way (one requiring coordinating multiple
developers with current Thunderbird codebase knowledge), consider an
option I mentioned but mostly discounted as as not fun. We could modify
Thunderbird-as-it-is to work as a backend to serve content to a web
browser. Essentially, we add a simple embedded web server to Thunderbird
written in C/C++ which supports an API (maybe all JSON, maybe even
JMAP?) to fetch and store email-related content, where the existing
backend still handles retrieving messages from servers and writing local
files. The existing full Thunderbird UI would continue to work in
parallel with this web server and be maintained in parallel. The goal
then would be to get the entire Firefox-based UI replaced first. After
we have a workable UI, then we could throw away the embedded Firefox UI
while still keeping the backend. After that, then we could talk about
redoing the Thunderbird backend if desired (maybe in Node.js, Java,
Python, Rust, C/C++ again, or whatever). And all along, there would just
be one system.

Anyway, this other approach would be a lot more work and coordination,
and would require making riskier changes to the existing Thunderbird
codebase (including because the existing UI code might not play nice
with some other UI given shared state, so that might take extra work to
resolve). As an individual, I would never consider doing such a thing
myself. But it is possibly the most incremental thing we could do as a
group.

Also, maybe someone more conversant than I with the range of existing
Firefox/Mozilla technologies might be able to point to embedded server
code we could leverage? Maybe related to past work for FirefoxOS?
https://hacks.mozilla.org/2015/02/embedding-an-http-web-server-in-firefox-os/

Obviously, at some point the effort involved in making such changes
incrementally might exceed the from effort needed to rewrite from
scratch. So, its a tradeoff about risk mitigation at potentially a big
extra labor cost. I don't know enough about the Thunderbird codebase to
decide about that tradeoff -- other than that from what little I've seen
and learned the existing Thunderbird-specific codebase seems hard to
understand and maintain (even the backend) because of the entanglement
with Firefox code and a complex Firefox-related build process. But,
perhaps that impression is incorrect? Or even if it is correct, given
Thunderbird-as-is will likely be maintained for another year or two,
maybe the extra work of adding in a server part won't be too much?

--Paul Fernhout (pdfernhout.net)
"The biggest challenge of the 21st century is the irony of technologies
of abundance in the hands of those still thinking in terms of scarcity."

On 2017-04-03 8:58 PM, Joshua Cranmer 🐧 wrote:
> The problem is that you're suggesting... well, basically inaction (on a
> feature level) while you go off and work on a separate project,
> hopefully to force it down everyone's throats when it's done. And
> arguing that it will be so wonderful and timely that there's not going
> to be any problems forcing people to switch (I am aware that this is

> probably a gross caricature of what you're proposing). *That* is what

> concerns me: that you're going to hit a roadbump, and everything is
> going to go wrong, and there is going to be no Plan B.
>
> The biggest risk is that one snag derails the entire project. Suppose,
> for example, that you have problems with unacceptable performance on
> message display, and that this resists all attempts to bring it to the
> performance level of existing Thunderbird code for years. (This is not
> unrealistic: this is exactly what happened to ical.js). What do you do?
> This is the sort of risk that concerns me deeply, and the best way to
> mitigate is to minimize the criticality of delivering any one individual
> feature. Proposing a complete rewrite and not seeking gradual uplifiting
> is exactly the opposite of that strategy.

_______________________________________________

[Paul D. Fernhout]

Hi Gerv,

On POP vs. IMAP/JMAP, to me, POP with delete (maybe after a week
retention in case of disk crashes) is about keeping your email private
and local -- especially with previous US laws that considered email on
your ISP more than six months old as abandoned and so subject to request
without a warrant. IMAP is more about letting your ISP (or your own
local server like dovecot) keep your mail for you -- maybe so you can
access it in multiple places. A Thunderbird server could potentially
support an IMAP-like function to serve email to other devices on a
network, so supporting IMAP to a server to collect mail is less
important in a Thunderbird server than POP for that use case. I'm not
saying IMAP or JMAP should not be supported eventually -- just that it's
less important at the start in my view. Granted, others may use
Thunderbird very differently than I do right now.

On analytics, my thinking was that if we think about minimum *valuable*
product to end users, a new version of Thunderbird that does less but
uses web technology is not valuable to end users. Why should they bother
to install it? But, if this new version has some extra feature the
current Thunderbird does not have (like personal mail analytics or
making your mail available to your mobile or tablet devices over your
LAN or something else) then it may be worth it to some Thunderbird users
to use it (alongside Thunderbird for now). I feel analytics in
particular will be a huge publicity win for relatively little effort.

--Paul Fernhout (pdfernhout.net)
"The biggest challenge of the 21st century is the irony of technologies
of abundance in the hands of those still thinking in terms of scarcity."

[Paul D. Fernhout]

Thanks so much for pointing out JMAP! JMAP looks like a great option for
moving forward towards any web-technology-based UI for Thunderbird. JMAP
seems especially well-suited for mobile devices which hopefully implies
good support for the desktop need of scrolling through thousands of
messages in a folder.

The biggest short-term benefit from using JMAP may be if we consider
flipping around the client/server idea of Thunderbird "supporting" JMAP.
Instead of Thunderbird first being a JMAP client to a remote mailstore,
Thunderbird could first be a JMAP server to a web-based UI client.

With that initial approach, the UI connection to the Thunderbird core
would benefit from all the great JMAP design as an email-domain-focused
communications protocol. Otherwise, we would be essentially be
reinventing JMAP when we think about how a web-based Thunderbird UI
would talk to a Thunderbird server.

I have rough ideas on a more general API based on other work (like I
experimented with in Pointrel/Twirlip versions) -- but JMAP is a much
lower-risk strategy to start with and could fit well with the
email-centric Thunderbird culture.

And, if we need more features (say, to make Thunderbird into a more
general federated knowledge-sharing platform -- which is my ultimate
hope for Thunderbird's future), we could perhaps suggest improvements to
JMAP.

And JMAP as-is might be good enough anyway to build other things on top
of -- in the same way IMAP has been used for other more general purposes
in Kolab. See:
https://en.wikipedia.org/wiki/Kolab
"The special idea behind Kolab is the use of IMAP as an underlying
protocol not only for email, but also for contact and calendar entries.
These entries are simply saved in special IMAP-folders utilising the
Kolab XML-format and the IMAP-server takes care of the storage and
access rights."

Here is another post I made has more details on why adding a JSON/JMAP
server to Thunderbird could make sense to manage risk by supporting an
incremental migration to web technologies:
https://mail.mozilla.org/pipermail/tb-planning/2017-April/005357.html

--Paul Fernhout (pdfernhout.net)
"The biggest challenge of the 21st century is the irony of technologies
of abundance in the hands of those still thinking in terms of scarcity."

[Ben Bucksch]

Joshua Cranmer 🐧 wrote on 04.04.2017 02:58:
> On 4/3/2017 5:00 PM, Ben Bucksch wrote:
>> Joshua Cranmer 🐧 wrote on 03.04.2017 18:08:
>>>
>>> We've talked about rewriting Thunderbird for, oh, a decade or more
>>> now. We've had agreement on the direction these rewrites should take
>>> for at least 3 or 4 years (rkent was really one of the last
>>> hold-outs for using JS). But we've not really made any measurable
>>> progress on rewriting Thunderbird.
>>
>> And you know why? Because nobody started. If we had started "a decade
>> or more" ago, we would be done since 5 years, would not be fighting
>> Gecko regressions today, and would be adding the features that the
>> users today are screaming for.
>
> It is a lie to say that no one started.

There are many email clients written in JavaScript, and many of them
work wonderfully. So, it's definitely technically feasible. However,
they are not a replacement for Thunderbird, because they are based on
custom servers, and they do not even try to have the features that
Thunderbird has. That's not because it's impossible - they have other
difficult features, like PGP or telephone integration -, but because it
was not their goal. They just had a different user group in mind.

I meant a full email client, to replace Thunderbird. The Thunderbird
project has never seriously tried that.

I specifically excluded attempts to gradually rewrite, by replacing
certain components of the old Thunderbird. That was my whole point. We
have tried that, indeed, and it hasn't gotten us anywhere. If it had
worked, we would be finished by now, with a renewed Thunderbird. After
"a decade", we're very far from done. We simply don't have another 5 years.

> There have been at least two projects with the clear intention of
> rewriting major portions of Thunderbird that produced concrete results
> yet have failed to come to fruition (Ensemble and JSMime)

I know your work around JSMime, and I appreciate and respect your work.

> what concerns me: that you're going to hit a roadbump, and everything
> is going to go wrong, and there is going to be no Plan B.

There is a plan B. I am not proposing to stop Thunderbird based on
Gecko. It will continue until the replacement is ready. Everybody has
the option to switch at the moment they feel the replacement is better
for them than the old Thunderbird. That will be a different moment for
different people.

At the end, we will make a specific effort to migrate the features that
the remaining users need.

Also, Thunderbird-on-Gecko can backport the new developments. I also
described that. That's another risk mitigation.
That means there *are* 2 plans. There is a Plan B.

If you gradually rewrite, *then* you don't have a plan B. The
integration of new components into old codebase is hard. If the rewrite
fails, or does not finish in time (same thing), you are dead.

> force [the replacement] down everyone's throats

I don't think that's a fair argument. You also force your rewritten
components "down the throat of everyone" (to use your own language),
even in the old client. What, if I don't like the replacement part or
how it works? Same thing.

> for example, that you have problems with unacceptable performance on
> message display, and that this resists all attempts to bring it to the
> performance level of existing Thunderbird code for years.

Please don't pretend that Thunderbird is perfect. Here, my Thunderbird
keeps freezing for 3-5 seconds every 15-30 seconds. I have one of the
fastest desktop machiens you can buy. It's like that since years, but
it's getting worse. Do you any idea how disruptive this is? I've filed
bugs about this, posted all information and symptoms I have there,
without any action. This makes me want to trash Thunderbird. And that's
saying something, because I've put a lot of my time into it.

> What do you do?

Fix it. There are enough email clients built in JS to prove that it's
possible.

> This is the sort of risk that concerns me deeply, and the best way to
> mitigate is to minimize the criticality of delivering any one
> individual feature.

You're welcome to backport individual features of the new implementation
to the old Thunderbird.

> Proposing a complete rewrite and not seeking gradual uplifiting is
> exactly the opposite of that strategy.

No, it's not. It's just 2 efforts running in parallel: Creating the new
app, with a clean code design, and backporting the new components to old
Thunderbird. And that can be done in separate teams. You do effectively
still do a gradual rewrite of the old Thunderbird.

You just a) share the load and b) don't compromise the new design with
old APIs and c) if the integration with old Thunderbird does not work
out, you still have a working result: the new version.

I am so insistent, because I'm convinced that the gradual rewrite will
not finish in time. It will be late by years, if it ever finishes at
all. If project history is any indication, I think it will outright
fail. So, what I propose is in fact risk mitigation.

(The other reason is that the Thunderbird code design is 20 years old
and just not fit for today, and a gradual rewrite would keep the overall
design.)

All our attempts of something new have always failed, because we held on
too much on old code and old thinking. The task itself is doable. It's
just extremely hard within the constraints of the old thing.

> 3. I am not proposing to maintain the current APIs.

How did that work out for JsMime? See what I mean?

Ben

[Magnus Melin]

On 4/4/17 1:00 AM, Ben Bucksch wrote:
> The gradual rewrite is what has failed. That's not just a possibility.
> We tried and failed. As you said above, we wanted to do it since a
> long time, and never succeeded. Thinking that we'll suddenly succeed
> now, with less resources, is merely wishful thinking. And if that

> fails, Thunderbird is dead for good, because XUL is going away
> eventually.

Unfortunately AFAIK there was never any serious resources from MoMo/MoCo
put into rewriting anything to web technologies.

One of the valuable things they did put resources into was creating and
setting up the testing infrastructure. With your proposed start from
scratch model all you'll get no test coverage at all, and that's really
bad. Things just aren't that easy to get right in the details.

-Magnus

[Philipp Kewisch]

On 4/4/17 10:38 PM, Magnus Melin wrote:
> On 4/4/17 1:00 AM, Ben Bucksch wrote:
>> The gradual rewrite is what has failed. That's not just a
>> possibility. We tried and failed. As you said above, we wanted to do
>> it since a long time, and never succeeded. Thinking that we'll
>> suddenly succeed now, with less resources, is merely wishful
>> thinking. And if that fails, Thunderbird is dead for good, because
>> XUL is going away eventually.
>
> Unfortunately AFAIK there was never any serious resources from
> MoMo/MoCo put into rewriting anything to web technologies.
>
> One of the valuable things they did put resources into was creating
> and setting up the testing infrastructure. With your proposed start
> from scratch model all you'll get no test coverage at all, and that's
> really bad. Things just aren't that easy to get right in the details.

Yes, we won't be able to re-use our current tests a lot, but when
rewriting from scratch, part of the work is ensuring a testing framework
is set up, so I'd consider this specifically a non-issue.

Philipp

[Joshua Cranmer 🐧]

On 4/4/2017 3:05 PM, Ben Bucksch wrote:

If you gradually rewrite, *then* you don't have a plan B. The integration of new components into old codebase is hard. If the rewrite fails, or does not finish in time (same thing), you are dead.

No. The plan B is that you *don't* hold new feature development hostage to rewrites. If the rewrite succeeds, great we can use it. But if it doesn't succeed, well, at least users get new features in some fashion anyways.

This is the sort of risk that concerns me deeply, and the best way to mitigate is to minimize the criticality of delivering any one individual feature.

You're welcome to backport individual features of the new implementation to the old Thunderbird.

With what manpower? Thunderbird has a critical problem: it's short of manpower, and it's been short of it for half a decade. Everything else is just a multiplier on the burden that this manpower shortage represents. We stand at a point now where we can rectify that situation and, to my eyes, you are proposing that what we should do is continue to deprive Thunderbird of resources. And proposing that it doesn't matter since Thunderbird can do what it wants anyways, but all initiative should come from Thunderbird, and Thunderbird shouldn't inconvenience your new project--which is exactly the kind of demeaning attitude that Mozilla has displayed over the past several years.

3. I am not proposing to maintain the current APIs.

How did that work out for JsMime? See what I mean?

Quite well, in fact. The problem with JSMime is that I lacked the time to bring it to fruition, and no one else was willing to pick it up.

[Ben Bucksch]

Joshua Cranmer 🐧 wrote on 4/4/17 11:54 PM:

> No. The plan B is that you *don't* hold new feature development
> hostage to rewrites.

I'm not holding anyone hostage. Why this martial terminology?

> If the rewrite succeeds, great we can use it. But if it doesn't
> succeed, well, at least users get new features in some fashion anyways.

That's exactly what I said.

> With what manpower? Thunderbird has a critical problem: it's short of
> manpower, and it's been short of it for half a decade.

And a gradual rewrite is at least 5x more work than a rewrite with a
clean slate, because of thee extra burden of the old architecture,
design, integration etc.. That in combination with the size of the
effort and the limited resources is why I know it will fail.

OTOH, we have data on how long it takes to implement a new email client
in JS, because it's been done before multiple times. Several teams did
it with 3-5 developers in about 2 years. We might even be able to reuse
existing JS components. We add on top the features that Thunderbird
needs - that's just extra work, nothing conceptually different -, and
we're done. The hard part is the architecture, and that's proven to
work, by others.

> Everything else is just a multiplier on the burden that this manpower
> shortage represents.

How did you develop JSMime? You first developed it as a standalone
component, then integrated it in Thunderbird, didn't you? What I propose
here would be doing the same thing. How is that a multiplier?

(skipping all the personal attacks on me)

[Ben Bucksch]

Magnus Melin wrote on 4/4/17 10:38 PM:

> setting up the testing infrastructure. With your proposed start from
> scratch model all you'll get no test coverage at all

If the testsuite is any good, it can be reused for the new implementation.

Ben

[R Kent James]

On 3/24/2017 10:04 AM, Ben Bucksch wrote:
> The goal of the rewrite is to be close to the existing Thunderbird, in
> UI and features, as a drop-in replacement for end users, without
> baffling them. They should immediately recognize the replacement as
> the Thunderbird they love. It will install and run as normal desktop
> application, like Thunderbird does today. It keeps user data local and
> private.

It seems like you are specifically ruling out the role of other
platforms, including a pure website, and mobile. I don't think it
accurately reflects the current set of expectations and actual work
practices of our users.

Given the extreme challenge in funding even the desktop version of this,
it might not be practical to consider the other use cases initially in a
fully functional release, but from a conceptual perspective the design
needs to plan for the eventual mobile and website versions. If this is
not explicitly planned, and probably partially prototyped, then the end
result will be a desktop-only product that will never meet user
expectations of access using multiple platforms.

For the Contacts experiment we will be running, an explicit goal is to
investigate whether it is reasonable to use the same basic core
components to provide a seamless user experience over multiple
platforms, including mobile, website, and desktop. That is expectation
of users today, and certainly represents my typical use.

:rkent

[Magnus Melin]

On 5.4.2017 03:09, Ben Bucksch wrote:
> How did you develop JSMime? You first developed it as a standalone
> component, then integrated it in Thunderbird, didn't you? What I propose
> here would be doing the same thing. How is that a multiplier?

Implementing different features as components with modern design is all good.
What's not so good is that you seem to say actually integrating those
components into Thunderbird for real usage is a secondary priority. It needs
to be *the* priority, because that's the only way you could get enough real
world feedback.

And you keep telling developing "a new email client in JS" in a short time can
be done, and have been done before. Well, there's a reason people still prefer
Thunderbird. You can develop something, but what that something is... I mean,
my kid plays soccer, but from that there's a loooong way to the Champions League.

-Magnus

[Ben Bucksch]

R Kent James wrote on 4/5/17 7:35 AM:

> On 3/24/2017 10:04 AM, Ben Bucksch wrote:
>> The goal of the rewrite is to be close to the existing Thunderbird,
>> in UI and features, as a drop-in replacement for end users, without
>> baffling them. They should immediately recognize the replacement as
>> the Thunderbird they love. It will install and run as normal desktop
>> application, like Thunderbird does today. It keeps user data local
>> and private.

> Given the extreme challenge in funding even the desktop version of
> this, it might not be practical to consider the other use cases
> initially in a fully functional release, but from a conceptual
> perspective the design needs to plan for the eventual mobile and
> website versions.

I completely agree, and I explicitly state that in my proposal.

I wrote, directly after: "The technology also gives us the option to run
it as mobile app." and
"
In addition to replicating the current Thunderbird UI, we should also
experiment with new forms of UI, in parallel. For example, we should
create a UI that's suitable for the new generation of users that never
used a desktop email client before. These people do not feel at home
with Thunderbird today, and we should create something for them.

A lot of the new userbase is on mobile platforms and on tablets. That
new UI should be "responsive" (automatically adapting to different
screen sized) so that it runs well on tablets and smart phones as well.
With Cordova and similar toolkits, we have a technological basis to
quickly make a mobile app out it so that it installs like any other app.
It would be a replacement for the system "email" app.

The goal for the new UI are 1 billion users.
"


Please read more than the summary. ;-)

I agree with you. Android is where most users are today. Personally, I
would really like to see a mobile client based on that new
implementation. I will probably develop one on the side. If the code
design is right, it would not be a lot of work.

However, to focus the development effort, we should concentrate the main
development stream on the most important goal: a replacement for current
Thunderbird, and a desktop client that the masses can not only use, but
enjoy using.

Ben

[Mark Banner]

On 24/03/2017 17:04, Ben Bucksch wrote:

Summary:

• While we implement the new version of Thunderbird, the old codebase based on Gecko will be maintained until the rewrite is ready to replace the old one.

I've been reading your original proposal as "Start a new fresh email client on a new code base from scratch, eventually swap users over to that new code base". You now seem to be saying "Start a new fresh client, but integrate parts of it back to Thunderbird". Is that correct? If not can you please clarify.


Starting a client from scratch does have a huge amount of risk, but that is not to say we shouldn't. What I would suggest is to start in a different manner, with some experimentation (similar to what Servo did originally).

So, define an experiment which is limited in scope and goals, has limited (but enough) resources, but is definitely an experiment. For that experiment:

• Define the scope/goals clearly, e.g.
• Obtain knowledge of the most viable code base?
• Is it likely to be able to support the performance we require? (e.g. multi-threaded/process etc).
• Will it be flexible enough that we can support all the existing Thunderbird features & more?
• Can the existing tests be re-used/easily ported?
• How are L10n, Add-ons, profiles, etc... supported?
• How long might it take to reimplement all of Thunderbird?
  
• What is the minimum MVP we need to be able to answer the scopes/goals?
• As this is an experiment, Thunderbird still continues developing at roughly its current pace.
• The experiment is reviewed regularly and at some stage all the goals are answered or are impractical to be answer (e.g. would take too long).
• There should then be enough known about the expected future code base that a decision could be made for how compatible is the technology - should we migrate the existing code base/continue fresh/continue fresh but with porting parts of the existing code base.

This could be seen as "let's delay the decision", which in part it is, but I think it would give a much better basis on which to make a decision than what seems to be "let's stop work, start something new, currently undefined and hopefully switch to it later".

The 'trick' will be in correctly forming the scope to get enough top-level questions answered, but not going into too much detail and never getting answers.

The advantage is that this will help inform a clearer path and eliminate some of the risk up-front. There will still be lots of risk, but hopefully it'll be a better size than the current proposal.

Mark.

[Ben Bucksch]

Magnus Melin wrote on 4/5/17 9:14 AM:

On 5.4.2017 03:09, Ben Bucksch wrote:

How did you develop JSMime? You first developed it as a standalone component, then integrated it in Thunderbird, didn't you? What I propose here would be doing the same thing. How is that a multiplier?

Implementing different features as components with modern design is all good. What's not so good is that you seem to say actually integrating those components into Thunderbird for real usage is a secondary priority. It needs to be *the* priority, because that's the only way you could get enough real world feedback.

I want to build a new desktop email client, based on web tech with a modern code design, which imitates Thunderbird and eventually has feature parity.

I want to have real world usage!
Year 1: I think it will take about 1 year until dogfood, meaning the first most courageous people can try to use it.
Year 2: After another year, I expect most people to be happy with it, in terms of features. At that time, many users should migrate voluntarily, because the new product is already better for them than the old Thunderbird, even if it doesn't have all TB features, but it has other cool features that TB can't do.
Year 3: It will take a third year to achieve reasonable feature parity with Thunderbird.

Rome wasn't built in a day. But it was built. And new big cities were built after Rome, too.

So, the new components will form the new client. That is the priority.

If you believe in gradual rewrite of Thunderbird, then you can take the new components and integrate them into the old Thunderbird. This would effectively be more or less the same as you do when you gradually rewrite Thunderbird without a new client: You first write the new component, then integrate it into the live product. It's not significantly more work than if you do only the rewrite.

This approach has the following benefits:

• You're decoupling both tasks into separate teams. One team writes the new components, with a clean, modern API, and another team integrates it into Thunderbird. That allows each team to focus on their tasks. They are very different tasks with different skill sets.
  
• If the rewrite fails to replace all of Thunderbird in time before Gecko becomes unmaintainable, we at least have the new client to fall back on.
• The new components can have a clean code design in their APIs and implementation. The new app is a coherent whole free of legacy ideas. Concrete examples: XPCOM, communication between modules using URLs, decoupling frontend, logic and network/storage tiers.

And you keep telling developing "a new email client in JS" in a short time can be done, and have been done before. Well, there's a reason people still prefer Thunderbird. You can develop something, but what that something is...

I would probably be using Nylon N1 today, if it wasn't tied with a server component. I don't need that many features, mostly UI speed, tree views, filters, and multiple identities.

Don't kill the past. But build the future at the same time.

[Ben Bucksch]

Mark Banner wrote on 4/5/17 10:42 AM:

> On 24/03/2017 17:04, Ben Bucksch wrote:
>>
>> Summary:
>>

>> * While we implement the new version of Thunderbird, the old

>> codebase based on Gecko will be maintained until the rewrite is
>> ready to replace the old one.
>>
> I've been reading your original proposal as "Start a new fresh email
> client on a new code base from scratch, eventually swap users over to
> that new code base".

Yes, that is my idea.

> You now seem to be saying "Start a new fresh client, but integrate
> parts of it back to Thunderbird". Is that correct? If not can you
> please clarify.

This thought is a concession, because most people here seem to strongly
believe in a gradual rewrite. I've adapted my proposal based on that
feedback. I'm saying that a gradual rewrite is not necessarily in
conflict with my idea, but can work in tandem, both efforts can
complement each other.

But to be frank: Personally, I'm pretty sure that the gradual rewrite
will not complete until Gecko becomes unmaintainable in practice, and
will therefore fail. It's just too much work to marry the old codebase
with new components gradually, while keeping other old components in
place. But if other people feel that this is the right way, then fine.
They can do that. And they can use the components that the effort for
the new client produces.

I think the truth is probably in the middle somewhere. I think the
address book will be possible to backport, and will significantly help
the userbase, until the new client is usable. There will probably be
other low cost, high gain cases like that. Such backports will be very
useful. But I don't think it's possible to replace all of old
Thunderbird step by step, before Gecko sky crashes on us.

> So, define an experiment which is limited in scope and goals, has
> limited (but enough) resources, but is definitely an experiment.

Great idea. I am doing just that at the moment.

> For that experiment:
>
> * Define the scope/goals clearly, e.g.
> o Obtain knowledge of the most viable code base?
> o Is it likely to be able to support the performance we require?
> (e.g. multi-threaded/process etc).
>

For example, we've identified the speed of the thread pane (message
list) as major problem in HTML. That's why I wrote the fastlist test
<http://benbucksch.github.io/trex/fastlist-test.html>. This was the main
missing widget in HTML, and it's proven to work, in concept. In fact, it
can do more than XUL trees could. The rest - hierarchical tree view,
selection, rich content rows, clean API etc. - is just leg work now.

> o How are L10n, Add-ons, profiles, etc... supported?
> o How long might it take to reimplement all of Thunderbird?
> o What is the minimum MVP we need to be able to answer the
> scopes/goals?
>

Right. Good questions!

I would like to prove feasibility. It would be nice, if we could define
some realistic goals, which would - if achieved or at least shown
possible - give those who have doubts about the project the comfort that
this is a promising route. I have some goals in mind. But I fear that
won't help.

It needs to be those who have doubts that define them. But, please be
realistic. Reasonable milestones. Not all of Rome, in 2 weeks.

> This could be seen as "let's delay the decision", which in part it is,
> but I think it would give a much better basis on which to make a
> decision than what seems to be "let's stop work, start something new,
> currently undefined and hopefully switch to it later".

FWIW, I don't suggest to stop work. My original proposal was maybe
formulated a little drastic. I am proposing that old Thunderbird
continues as it has been in the last years. In no way "stop work" on it.

> The 'trick' will be in correctly forming the scope to get enough
> top-level questions answered, but not going into too much detail and
> never getting answers.

+1

[Matt Harris]

On 4/5/17 6:17 PM, Ben Bucksch wrote:

Magnus Melin wrote on 4/5/17 9:14 AM:

On 5.4.2017 03:09, Ben Bucksch wrote:

How did you develop JSMime? You first developed it as a standalone component, then integrated it in Thunderbird, didn't you? What I propose here would be doing the same thing. How is that a multiplier?

Implementing different features as components with modern design is all good. What's not so good is that you seem to say actually integrating those components into Thunderbird for real usage is a secondary priority. It needs to be *the* priority, because that's the only way you could get enough real world feedback.

I want to build a new desktop email client, based on web tech with a modern code design, which imitates Thunderbird and eventually has feature parity.

We have a mature product and a lot of mature users.  You do not seriously believe that launching a "new" product that is not at feature parity will fly with the users do you?.  You will probably satisfy some users,  but a significant number of them would be happy if you just updated outlook express to run properly on Windows 10.  They are very unlikely to be those putting their money or time where their mouth is.  They like free as in beer and will just move on to the next free as in beer product with more features.

I would suggest anyone on this list thinking a modular approach is a good idea should try using the Lithium version of SUMO.  That is a perfect example of a modular building process that simply does not work well.  Lithium is well funded and has lots of staff,  Mozilla have been throwing people at it and it simply is not flying.  I see it, and read this thread and can not help drawing comparisons.  New, shiny, JavaScript and not at feature parity.

Matt

[Gervase Markham]

On 05/04/17 10:52, Matt Harris wrote:
> We have a mature product and a lot of mature users. You do not
> seriously believe that launching a "new" product that is not at feature
> parity will fly with the users do you?.

<cough> Firefox?

But I do think a new product which does not have feature parity would
need a new name.

Gerv

[Disaster Master]

On Tue Apr 04 2017 00:02:21 GMT-0400 (Eastern Standard Time), Paul D. Fernhout <pdfer...@kurtz-fernhout.com> wrote:

Your point on risk management makes a lot of sense. That's why I
suggested a possibility of having a new Thunderbird that could run
alongside the existing one and share the same files while offering a
different UI.

First, such a possibility already exists - if you are using IMAP.

But connecting some brand new client, in beta, alpha, or even pre alpha stages of development? This presents a very large risk to your precious data, one that I would never be willing to accept, and if anyone did, the PR backlash could be devastating when their X years worth of precious emails were nuked by some bad code.

[Joshua Cranmer 🐧]

There's been a lot of discussion about this proposal, and I think some
people have modified their opinions or clarified them in ways that
aren't necessarily collected in one place. I think we can come to a
better agreement if we go back and start our positions more clearly.

For my part, I have one non-negotiable requirement: we should be able to
ship CardDAV support to our users by 2018. Beyond that, I think there
should be means to implement new features in TB in a more rapid basis
"if prudent" (and I'm leaving it vague on what exact criteria that entails).

To that end, here are some suggestions to the proposal:
1. Delete the requirement that feature development in Thunderbird is
unfunded. It's okay to leave it vague which features would be
implemented sooner--the only two that really come to my mind are CardDAV
and EAI. Beyond that, I don't think that there are many features which
are urgent to implement (e.g., JMAP would be nice, but no one's really
clamoring for it).
2. Make it clear that being able to transfer features to Thunderbird is
a priority. i.e., you'd rather a 100% complete address book and 50%
complete IMAP than 80% of both. This also means that whatever platform
support library you have would include support for
xpcshell/xpconnect/whatever else is being used by Gecko as a base layer.

I realize this is something you're likely to disagree with, but I
suspect the disagreement is more over perception than fundamental
semantics. I believe that it would be fairly easy to replace the address
book, import, mime, and compose code; that replacing the search and
protocol implementations are probably doable (with SMTP being easiest
and IMAP hardest); that replacing the database is challenging, but
possible if you look to only implement a well-defined new DB API that
coexists with the old one but doesn't eliminate it; and that replacing
the account implementations and associated frontend bits (e.g.,
nsMsgDBView) is probably so difficult it's not worth attempting.
Accordingly, to me, there seems to me to be little reason not to throw
some people at porting over the first few bits rapidly.

3. This may be controversial, but I suspect it's less controversial than
it seems. Paid support, if necessary, would be available to do the
porting of bits to Thunderbird. I'm sure we disagree on where the line
should be drawn, but I'm not proposing that everything would be
considered a viable candidate (i.e., I'm fine with you objecting to
having to port nsMsgDBView-replacement), and I suspect you're open to
letting mostly standalone stuff get done (e.g., the address book). So
I'm happy if we can both agree that a line should be drawn, it should be
drawn somewhere in the middle, and we can agree to defer the discussion
of where it should be drawn until an actual relevant case comes up. The
question of who is paid to do so is also something that I'm fine with a
fuzzy answer.

Beyond that, I think the crux of the issue is in the manner of a pitch.
Most users aren't going to notice backend changes at all unless it
impacts them, but front-end changes tend to engender a lot of
complaints. I've mentioned several times the Servo model, and if that's
essentially what you're proposing, I don't have problems with it. That
model (to me, at least) is that you use an entirely different product to
develop all of the backend pieces which you can then reintegrate into
Thunderbird as they become feasible replacements--and in terms of pitch,
you don't describe it as being the replacement for Thunderbird. It's
also clear that, unlike Firefox, the UI of Thunderbird cannot be easily
decoupled from its backend, and as to how to deal with that... I have no
opinion; I'm a backend guy through-and-through.

I would also suggest an effective mandate that developers on the new
project should be sensitive to disruption to Thunderbird UI (if the
intent is to replace the UI of Thunderbird with new UI). As well, they
should also have explicit mandates to have better testing infrastructure
and documentation than Thunderbird has traditionally maintained,
including performance tests (as a rough baseline, the question should be
"how well do we perform on a 1M message folder?"--and yes, I'm aware
that Thunderbird doesn't look good on that regard right now). I also
think that the testing infrastructure should be shared with TB as much
as practicable--e.g., sharing the same datasets for performance tests.

How does that sound?

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

_______________________________________________

[neandr]

Yes, Joshua's opening makes the point:

On 07.04.2017 23:30, Joshua Cranmer  wrote:

There's been a lot of discussion about this proposal, and I think some people have modified their opinions or clarified them in ways that aren't necessarily collected in one place. I think we can come to a better agreement if we go back and start our positions more clearly.

It would be great to see TB contributors come together to define a starting point for the "TB-next". No question it will be hard to
bring all the conflicting interests and knowledge together and get a balanced solution. But we need to do so! Now!

Joshua tried to define a starting point which sounds doable ...

For my part, I have one non-negotiable requirement: we should be able to ship CardDAV support to our users by 2018. Beyond that, I think there should be means to implement new features in TB in a more rapid basis "if prudent" (and I'm leaving it vague on what exact criteria that entails).

... and it has the appeal, the opportunity to probe a lot of the necessary technologies. But not only probe but also to start defining standards for "TB-next". (See also my previous post to this proposal.)

A project like the CardDAV/contacts would be relatively separated from the big TB project but enables us to understand how the "new" technologies has to be handled to meet requirements for
-- user interface / supporting different platforms (desktop, mobile, tables) with responsive designs
-- database
-- handling of large data sets
-- test suites (using existing, defining new as appropriated)
for sure not complete!?

We see projects and contributions to support such a project, just to name a few:
-- Kent is working with a special group of people and I understood his goal is to contribute with building box for the TB-next
-- Ben's tested with tables coded with html/css to demonstrate fast display / performance is possible with html/css
-- mconley's project for a new address book
-- it sounds a long year contributor and friend with TB can help to define the right approach -- THANKS to Mark
-- Cardbook also very close to the current mature TB could contribute at least with expertise
-- the New Zealand 'contacts' project show a first proof of concept

Not to repeat Joshua's arguments here, but it's important to start! Now!

How does that sound?

For me his arguments sounds more than GOOD, I'm looking forward to see more of this and a team is formed for a start!

Günter

[Paul D. Fernhout]

On 2017-04-08 1:23 PM, neandr wrote:
> We see projects and contributions to support such a project, just to
> name a few:
> -- Kent is working with a special group of people and I understood his
> goal is to contribute with building box for the TB-next
> -- Ben's tested with tables coded with html/css to demonstrate fast
> display / performance is possible with html/css
> -- mconley's project for a new address book
> -- it sounds a long year contributor and friend with TB can help to
> define the right approach -- THANKS to Mark
> -- Cardbook also very close to the current mature TB could contribute at
> least with expertise
> -- the New Zealand 'contacts' project show a first proof of concept
>
> Not to repeat Joshua's arguments here, but it's important to start! Now!

Personally, I feel making a new email client from scratch would be a lot
of fun -- especially if it went in new directions with new features for
knowledge exchange. But, to try to be practical, below I try to make the
case for considering joining forces with another existing
communications-oriented community that is already going in similar
directions rather than starting from scratch.

--Paul Fernhout

=== What Thunderbird still has going for it

Most people posting recently seem to agree that the Thunderbird codebase
is rapidly becoming obsolete given Mozilla's recent choices -- even as
that codebase may still provide a great reference for building better
email clients. So, something needs to change.

Given that, what Thunderbird has going for it most of all right now is
the community, the trademark and related good will, and the collective
knowledge about email within the community.

So, a broader question is, what does it make sense for the Thunderbird
community to do with its current resources of time, attention, and
knowledge to support current and future Thunderbird users who share
whatever values lead people to choose Thunderbird?

Starting from scratch to make a new Thunderbird is certainly possible.
But, imagine it was completely impossible to do a new version from
scratch for whatever reason and all Thunderbird users had to migrate
their email out of Thunderbird-as-is by the end of 2017. Imagine if the
Thunderbird Council set an End-of-Life date on Thunderbird support, like
a prize-fighter deciding to retire as a champion instead of fighting
until a final defeat. What would the community consider doing for
alternative options in that case?

=== Some alternatives

I liked the idea Mark Banner mentioned of experimenting to gather more
information if we were to start from scratch. Such experiments could
also include determining whether it was feasible to leverage all that
community email knowledge to make other email systems better with a lot
less work than starting from scratch (including perhaps tools to make
migrating Thunderbird plugins and email archives easier).

If we are seriously considering "burning the disk packs" as Alan Kay
calls a full rewrite in new conceptual directions based on previous
learning, there are other codebases that could be leveraged for future
work.

For example, Ben already mentioned N1 as something he might consider if
it were somewhat different than it is, writing: "I would probably be

using Nylon N1 today, if it wasn't tied with a server component. I don't
need that many features, mostly UI speed, tree views, filters, and

multiple identities." Is it really easier to redo Thunderbird from
scratch than to change N1 either from within the community or via a fork?

So, another possibility is for this group to pick an existing FLOSS mail
system (Nylas N1, Mailpile, KMail, Mutt, Claws Mail, Zimbra, or whatever
especially it used web technologies) and make that into a new blessed
"Thunderbird" either as a fork or by joining that community.

Another broader possibility is joining forces with non-email systems
like Matrix.org or MatterMost (or perhaps even NEPOMUK or some other
Semantic Desktop project) and then integrating email into those projects
in some fantastic way. That might entail rethinking email and how it
interacts with many other communication, archiving, and indexing services.

Could the same amount of effort going into redoing Thunderbird from
scratch instead be able to make, say, Mailpile into something even better?

Or could that same effort perhaps create a robust email gateway for
Matrix.org as well as a related UI?
https://en.wikipedia.org/wiki/Matrix_(communication_protocol)
"Development of Matrix is led by Matrix.org, a non-for-profit initiative
based in the United Kingdom, which hopes to make it an open standard for
decentralised, persistent and interoperable communications over the
Internet. Matrix targets use cases like Voice over IP, Internet of
Things and instant messaging, including group communication, along with
a longer-term goal to be a generic messaging and data synchronization
system for the web. The protocol supports security and replication,
maintaining full conversation history, with no single points of control
or failure. Existing communication services can integrate with the
Matrix ecosystem."

=== Social difficulties in considering alternatives

Communities and community members are not fungible, of course. Socially,
long-term Thunderbird users might feel uncomfortably strange becoming,
say, KMail advocates. And the dynamics of power sharing by the current
leadership if moving to some other community with an existing leadership
may be problematical.

But, if Thunderbird does not soon have a compelling story about what is
going to happen with a radically new version, it seems to me that some
sort of splintering from Thunderbird and joining other communities is
quite likely.

So, why not at least consider that option now as a fallback to manage
the risk of some new from scratch effort not coming together quickly?
Granted, a good reason "why not" is that such a hypothetical discussion
might become a self-fulfilling prophecy. So, exploring such options is
not without social risk to the community. And of course, wading through
someone else's email client code may seem less fun than starting from
scratch.

=== Making objections to alternatives into technical requirements

If the option of joining forces with other communities is completely
unthinkable for *technical* reasons, then outlining why it is
unthinkable might at least help in coming up with requirements for a new
Thunderbird. For example, perhaps we might want to use JMAP as a great
new standard to talk between a local email store and various web-tech
clients and we all think it would be hard to integrate such an approach
into existing codebase?

--Paul Fernhout (pdfernhout.net)
"The biggest challenge of the 21st century is the irony of technologies
of abundance in the hands of those still thinking in terms of scarcity."

[op...@optosolar.com]

As much as a new start may make sense from a programming point of view, from a user's perspective, I just want to point out that this will kill nearly all addons/extensions.

I am confident that members of this list may update their extensions, but most other extensions will be lost, especially if not only XUL is gone, but if the API totally changes.

Also, we might be facing 3 years of no new features for TB, but also 3 years of no new extensions for TB. Can there be any mechanism that extensions might be reused after rewriting? Or having need of minimal changes only?

my 2 cts, Klaus

[Joshua Cranmer 🐧]

On 4/9/2017 3:01 PM, op...@optosolar.com wrote:
> As much as a new start may make sense from a programming point of view, from a user's perspective, I just want to point out that this will kill nearly all addons/extensions.
>
> I am confident that members of this list may update their extensions, but most other extensions will be lost, especially if not only XUL is gone, but if the API totally changes.
>
> Also, we might be facing 3 years of no new features for TB, but also 3 years of no new extensions for TB. Can there be any mechanism that extensions might be reused after rewriting? Or having need of minimal changes only?

Most add-ons today rely on infrastructure that Mozilla has expressed
interest in killing, namely XPCOM, XUL, and XPConnect. The sad truth is
that Thunderbird has no real resources to maintain these technologies
itself: XUL is deeply baked into the display engine, and XPConnect
relies on a very fickle JS engine API and has serious design flaws baked
into it (XPCOM is essentially a giant hashtable, with some mild C/C++
reflection and a few assorted helper ADTs--it basically requires so
little maintenance that large portions of it effectively haven't been
touched for a decade). It's not really in our hands to decide whether
these extensions can be reused, although we do have some measure of
control over how painful it would be transition them to a new system.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

_______________________________________________

[Ben Bucksch]

Hello Joshua,

first off: Thank you so much for the nice, level-headed response! This is the kind of discussion I was hoping for. This is productive and great. Thank you!

Joshua Cranmer wrote on 4/5/17 7:31 PM:

every estimation for feature implementation I've seen from Mozilla has been wildly optimistic

I'm not Mozilla. :-)
I've done rewrites and challenging tasks before, and typically kept my estimates.

I would rather convince others on a rational level, or if I'm missing some aspects or ideas, improve my proposal to accommodate valid needs that I've overlooked.

Let's talk about the proposal then. For my part, I have one non-negotiable requirement: we should be able to ship CardDAV support to our users by 2018. Beyond that, I think there should be means to implement new features in TB in a more rapid basis "if prudent"

Now, we're talking! This is getting somewhere. That's exactly the concrete level at which we should be talking.

Yes, I totally agree that the address book can't stay as-is until 2020, even in the old Thunderbird. Just 2 email addresses per person are just ridiculous. Sync is critically important these says, and CardDAV is part of that, yes. And I think everybody agrees that the old AB implementation is ripe to be trashed. So, this would be an excellent component to backport from the new Thunderbird to the old. I totally agree.

There are probably other such low-hanging fruits.

1. Delete the requirement that feature development in Thunderbird is unfunded. It's okay to leave it vague which features would be implemented sooner--the only two that really come to my mind are CardDAV and EAI. Beyond that, I don't think that there are many features which are urgent to implement (e.g., JMAP would be nice, but no one's really clamoring for it).

I fully subscribe to this.

We are actually in violent agreement. What you wrote is exactly what I think as well.

(I've formulated it too extreme in my original proposal. I've tried to be short - already hardly anybody read my entire proposal, even though I tried to be short.)

Later:

I've mentioned several times the Servo model, and if that's essentially what you're proposing, I don't have problems with it. That model (to me, at least) is that you use an entirely different product to develop all of the backend pieces which you can then reintegrate into Thunderbird as they become feasible replacements

Yes. And no. It's a balance of how much you backport and where you focus.

I think you described it best under point 1. You phrased the balance exactly as I have it in mind.

2. Make it clear that being able to transfer features to Thunderbird is a priority. i.e., you'd rather a 100% complete address book and 50% complete IMAP than 80% of both.

I'm not sure what you mean. But I don't think that when we rewrite, we should first complete 100% of the Thunderbird IMAP features before we start with address book, or vise versa. Rather, I subscribe to the MVP (Minimum viable product) idea. I want to first have a functional, but minimal and simple email client using only web techs (plus socket and file I/O of course, but still pure JS). That includes all of IMAP, POP3, AB etc., but not at the level of TB yet, but just the basics to read and send mail. Then, we build it out to be a good (not simple) email client, so that 80% of the users are happy with it. Then, finally, we build all the remaining TB features that the remaining 19% need. That's the idea of MVP.

(And yes, I left the last 1% of users. There will always be holdouts that insist on the most arcane features. There's no way to make them happy, no matter what we do, and trying to will consume so much energy that we will ignore the needs of the remaining 80%.)

This also means that whatever platform support library you have would include support for xpcshell/xpconnect/whatever else is being used by Gecko as a base layer.

I don't understand. XPCOM is slated to be killed in Gecko in the long term, it's one of the technologies we need to get away from. Thunderbird on Android is not going to use XPCOM, nor a compat layer emulating XPCOM, but a pure JS module system.

If things are backported from the new implementation into old Thunderbird, TB can use it on the pure JS level. Or build a small XPCOM wrapper in JS, if absolutely necessary.

I believe that it would be fairly easy to replace the address book, import, mime, and compose code

heh! I agree that these are components that would be easiest in a relative sense. But not easy in an absolute sense. "Fairly easy" is quite a statement. :) You of all people should know how hard it is to replace libmime, within existing Thunderbird.

replacing the account implementations and associated frontend bits (e.g., nsMsgDBView) is probably so difficult it's not worth attempting. Accordingly, to me, there seems to me to be little reason not to throw some people at porting over the first few bits rapidly.

Right!

I think you're actually saying here that you agree with my approach. Because the conclusion is that a gradual rewrite of Thunderbird, with the result of eventually everything being replaced with JS components without XPCOM and XUL, is "probably so difficult it's not worth attempting". That's precisely what I keep saying! :)

3. Paid support, if necessary, would be available to do the porting of bits to Thunderbird.

What do you mean? If you meant that Thunderbird or Mozilla could hire people specifically to backport parts of the new implementation to old Thunderbird, then yes, I think that would be a great way to distribute the work load among several developers.

One way to visualize the problem is the components on the x axis and the different tasks for each component on the y axis. Instead of slicing work in x axis, you can slice in the y axis. I think that produces much better and more homogeneous results, because you use the individual talents better. (Of course, you can slice in both directions, too.)

That means: One team builds the new client, another team backports the new components to old Thunderbird, and a third team fixes Gecko breakage in old Thunderbird. All of these need different skills.

Even within the new client team, we could slice work. In another project, I need 10% for the implementation of the client, 10% for the server, 40% for tests, and 40% for reviews and landing. If you add integration into old Thunderbird on top, numbers look several times worse. If we could reduce that overhead (make sure reviews and CI are efficient) and slice work horizontally (e.g. let another dev build the tests, yet another backport etc.), that would help a lot. Like a factory line. A good test suite is independent of the implementation, anyway, and can be written even before the component.

Beyond that, I think the crux of the issue is in the manner of a pitch. Most users aren't going to notice backend changes at all unless it impacts them, but front-end changes tend to engender a lot of complaints.

Agreed! :-)

Luckily, with CSS, we can style the new HTML-based implementation very much like the old Thunderbird, so that people will immediately recognize it as TB.

I've once reimplemented a C# desktop app as webapp. Not only was it 100 times faster to load, and 50 times faster to implement, but I could make it look almost like the desktop app, using CSS. It was immediately recognizable. You can do amazing things with JS and CSS.

So, I don't worry about that part.

--and in terms of pitch, you don't describe it as being the replacement for Thunderbird.

How would you describe it?


Ben

[Ben Bucksch]

Paul D. Fernhout wrote on 09.04.2017 04:44:

So, another possibility is for this group to pick an existing FLOSS mail
system (Nylas N1, Mailpile, KMail, Mutt, Claws Mail, Zimbra, or whatever
especially it used web technologies) and make that into a new blessed
"Thunderbird" either as a fork or by joining that community.

We should definitely look at existing projects and existing state of the art.

However, I think we all agree we want the Thunderbird replacement to be a desktop client (plus other platforms like mobile), and based on web technologies, most importantly written in JavaScript. So, all webapps are not helpful. Solutions that try to combine a server and client are fragile and hard to develop in, because you need to create the client/server protocol for everything.

We are also very clear that we want to take our users with us, and therefore want to keep a UI that is reasonably similar to current Thunderbird, at least as transitional option.

Let me quickly go through the list:

Not applicable:

• Mailpile
• Looks nice (from the website) https://www.mailpile.is/
  
• It's a webapp written in Python. It runs in a browser.
• https://github.com/mailpile/Mailpile/tree/master/mailpile
  
• All Python => Not applicable.
• Mutt
• You gotta be kidding
• This is a text-based console client
  
• Claws
• Written in C with GTK => Not applicable
• KMail
• Written in C++ with Qt => Not applicable
• Zimbra
• Webapp => not applicable
  
• Mozilla used it, then stopped, in favor of Gmail.

Interesting are:

• Nylon N1
• Desktop app written in JS, but married with a server components.
• UI applicable
• Need to look at this
• Gaia
• Mobile app written purely in JS (with some Firefox OS APIs)
• Desktop prototype as asutherlands's glodastrophe
• Need to look at this
• Uses emailjs.org libraries
• emailjs.org
• Implementations of MIME, IMAP, SMTP
• Definitely need to look at this, potentially use it.

So, none of the existing projects are directly applicable, but some have libraries or code that we might be able to use. Even if not, we might be able to get ideas or interesting solutions there.

Ben

[Joshua Cranmer 🐧]

On 4/10/2017 8:42 PM, Ben Bucksch wrote:
> I'm not sure what you mean. But I don't think that when we rewrite, we
> should first complete 100% of the Thunderbird IMAP features before we
> start with address book, or vise versa. Rather, I subscribe to the MVP
> (Minimum viable product) idea. I want to first have a functional, but
> minimal and simple email client using only web techs (plus socket and
> file I/O of course, but still pure JS). That includes all of IMAP,
> POP3, AB etc., but not at the level of TB yet, but just the basics to
> read and send mail. Then, we build it out to be a good (not simple)
> email client, so that 80% of the users are happy with it. Then,
> finally, we build all the remaining TB features that the remaining 19%
> need. That's the idea of MVP.

I agree that basic implementations of "everything" comes first. Beyond
that, well, it's best to explain with a hypothetical. Suppose we decided
that LDAP and QRESYNC support were both necessary final features but not
earlier mandatory features, and suppose that LDAP were the last feature
not found in the new address book. I'm saying that we should prioritize
finishing the LDAP implementation (via FFI to OpenLDAP, say) over
implementing QRESYNC. Outside of that specific kind of situation, there
isn't much difference in various approaches, and it's hard to say that
sort of situation will even come up in the first place. So I'm not going
to push very hard on this.

>
>> This also means that whatever platform support library you have would
>> include support for xpcshell/xpconnect/whatever else is being used by
>> Gecko as a base layer.
>
> I don't understand. XPCOM is slated to be killed in Gecko in the long
> term, it's one of the technologies we need to get away from.
> Thunderbird on Android is not going to use XPCOM, nor a compat layer
> emulating XPCOM, but a pure JS module system.
>
> If things are backported from the new implementation into old
> Thunderbird, TB can use it on the pure JS level. Or build a small
> XPCOM wrapper in JS, if absolutely necessary.

The problem with JS is that it doesn't have much in the way of the
standard library. A TCP socket, for example, is handled very differently
in node.js, in Chromium extensions, in XPCOM, in the B2G bindings, etc.
At some point, you need a library that presents a common API for all the
platforms to its different users, much the same way that (say) Rust's
libstd bridges over i686-linux-gnu, i686-windows-msvc, etc. My point is
that a) the JS environment of Thunderbird (which I'm referring to as
xpcshell/xpconnect/whatever) is included as one of the platforms in this
list and b) it should show up as such in, e.g., a tbpl-like matrix view.
It is extremely hazy-feely on how this backend works in Thunderbird. (In
particular, how to handle ES6 modules versus require.js, etc. is not
something I can recommend a solution to at this moment).

The original proposal gave the impression that Thunderbird development
was to be unfunded, and certainly much of my wording was meant to focus
on overturning that viewpoint. It matters much less to me who is being
paid to do what as long as we agree that those acts deserve paid
developers. Which means we're in total agreement here.

>> --and in terms of pitch, you don't describe it as being the
>> replacement for Thunderbird.
>
> How would you describe it?

If I had a good example, I would have given it. I'm concerned that being
upfront with replacing Thunderbird will prime people to bring out the
pitchforks, but lying obviously helps nobody. So starting out with
descriptors like "next-generation" or "experimental" or "incubator" that
are somewhat evasive but not really dishonest probably work better.

[R Kent James]

On 4/10/2017 7:47 PM, Ben Bucksch wrote:

• Nylon N1

• Desktop app written in JS, but married with a server components.
• UI applicable
• Need to look at this

I had a chat with the CEO of Nylas a year ago about possibly working together. My impression at the time was that they were primarily interested in the mail client as a promotion of the web services, rather than as a widely available client, and were not really interested in working with us. Of course, I could be wrong in my impressions. Note they have been quite active in the Electron community reporting on various issues with Electron in their application. Their code is very close to what we have been doing, and some might ask "Does the world need 2 Electron/React open source email clients?" Lead developer Ben Gotow left in 2016, now principal engineer at https://www.foundry376.com/

GPL code :(  Primarily the work of 2 people. I wonder if we would ever consider switching to GPL?

:rkent

[R Kent James]

On 4/10/2017 7:57 PM, Joshua Cranmer 🐧 wrote:
> It matters much less to me who is being paid to do what as long as we
> agree that those acts deserve paid developers.

You can look at Nylas N1, with its exciting mix of hot technologies
(Node on Electron, React) which tend to attract volunteers, and see that
the vast majority of commits are from paid developers
(https://github.com/nylas/nylas-mail/graphs/contributors top 4 are Nylas
staff). So I think that we can agree that the hope that there are armies
of qualified, volunteer programmers just waiting to work on Thunderbird
or related projects is so 20th Century (and unlikely). While we have
some amazing volunteers that have kept the project alive, yet email is
serious business and it is not reasonable to believe we will have a
serious product done mostly by volunteers (and I say that as someone
actively recruiting about 10 volunteer students to work on aspects of this).

The problem is, I have had a very, very hard time getting anyone to
seriously discuss the question of how to fund this. I have been
disappointed that my suggestions to the Council to devote some resources
to financial development are not received well. Elsewhere, I have
suggested a coop model but got very little traction with that. Yes we
have donations, really more than we expected, but keeping the existing
product alive for 2-3 years is going to take most of those resources.

We cannot compete technologically if we do not have a business model
that can compete with models such as the Postbox paid user, or the Nylas
paid server services. What is the plan here?

:rkent

[Óvári]

Hello :rkent,

1. Yes, move Thunderbird to GPL :)

2. Also move the project to The Document Foundation:

http://www.documentfoundation.org/

On 11/4/17 1:09 pm, R Kent James wrote:
> GPL code :( Primarily the work of 2 people. I wonder if we would ever
> consider switching to GPL?

[Gervase Markham]

On 11/04/17 04:09, R Kent James wrote:
> GPL code :( Primarily the work of 2 people. I wonder if we would ever
> consider switching to GPL?

From a purely legal (so not wisdom or community) perspective, MPLed code
can be combined with GPLed code into a Larger Work, so as long as you
start with a GPLed codebase, there is no problem, and not even a
requirement for a license "switch".

Gerv

[Ben Bucksch]

R Kent James wrote on 11.04.2017 05:09:

> Primarily the work of 2 people.

That's interesting to know. This speaks about the feasibility of the
project, the duration, and the number of man years required. They didn't
work for more than 2 years on this, so this means roughly 5 man years
maximum. And they got quite an impression result.

Of course, that's not at the feature level of where Thunderbird-Next
needs to be. But OTOH, they ad to master many of the difficult
architectural challenges that come with a project like this. This is a
feat it itself.

So, I think the 30 man years we've been estimating are fairly realistic.

> GPL code :(

That of course seriously hinders code changes.

> I wonder if we would ever consider switching to GPL?

FWIW: I would consider that. GPL2 or GPL3 or LGPL is fine with me,
personally.

And the Mozilla project has opened up the way for this, by the
tri-licensing under MPL1.1/GPL2/LGPL2, and adapting the MPL2 to the
GPL2, so that MPL2 code can be combined with GPL2 code.

I'm not saying we should take this path, but I'd be open for it.

Ben

[Ben Bucksch]

R Kent James wrote on 11.04.2017 05:26:
> While we have some amazing volunteers that have kept the project
> alive, yet email is serious business and it is not reasonable to
> believe we will have a serious product done mostly by volunteers (and
> I say that as someone actively recruiting about 10 volunteer students
> to work on aspects of this).

I agree here. I think volunteers will give excellent support, and some
components might even be written by volunteers. Many of the people
participating in the thread here have contributed entire modules.

That said, an effort like this needs to be concentrated, focused, and
most importantly directed with clear leadership. Herding cats takes
serious resources. So, I agree that the bulk work will have to be done
by paid developers. And the kind of devs we need here are the most
experienced ones.

But I do look forward to some individuals volunteering for certain
parts, and implementing them entirely. We have a huge feature set to
cover, and if some people contribute filter UI, or MDN, or PGP support,
or Kerberos, or things like that, this would be great. I am hoping that
maybe 20% of the effort will be done by volunteers.

And that's not to minimize their influence. Volunteers are a critically
important corrective for things that are overlooked or misguided.
Everybody makes mistakes in judgement, and that outside corrective is
important for making the result mature. That's in part where the power
of open-source comes from. So, these 20% are important.

> The problem is, I have had a very, very hard time getting anyone to
> seriously discuss the question of how to fund this.

Kent, I'd like to discuss this. However, we need to make one step after
the other. We first all need to agree on a overall direction. Only when
we're united behind a goal, can we make sound decisions afterwards, and
choose the right path. Financing is then the second step.

Just one example, we could make a donor campaign specifically for
Thunderbird-Next. Even people who are not donating right now. Once
people have a specific goal, they are much more happy to fund things.
That "shiny object" reflect.

> We cannot compete technologically if we do not have a business model
> that can compete with models such as the Postbox paid user, or the
> Nylas paid server services. What is the plan here?

Kent, we don't have a classic "business model". We're an open-source
project.

I think it's great we're getting so many donations. This is encouraging
that this might actually work only with goodwill, instead of a hard
sale. That said, we can and should do more to encourage donations, and
be more visible.

And most of all, show more action. See the "Outreach" thread. I am
immediately getting a front of people telling us "To something first,
that we can be excited about". Our users are expecting action.

I'm surprised so many people are donating despite the situation. Imagine
what happens when we have a concrete, exciting, useful project to fund!

Ben

[Disaster Master]

On Mon Apr 10 2017 23:09:37 GMT-0400 (Eastern Standard Time), R Kent James <ke...@caspia.com> wrote:

GPL code :(  Primarily the work of 2 people. I wonder if we would ever consider switching to GPL?

Interesting question - how would such a change affect Addons? Would that depend on exactly what the Addon does, or would it affect all Addons (they would all have to be GPL licensed)?

[Ben Bucksch]

Joshua Cranmer 🐧 wrote on 11.04.2017 04:57:
> On 4/10/2017 8:42 PM, Ben Bucksch wrote:
>> I'm not sure what you mean. But I don't think that when we rewrite,
>> we should first complete 100% of the Thunderbird IMAP features before
>> we start with address book, or vise versa. Rather, I subscribe to the
>> MVP (Minimum viable product) idea. I want to first have a functional,
>> but minimal and simple email client using only web techs (plus socket
>> and file I/O of course, but still pure JS). That includes all of
>> IMAP, POP3, AB etc., but not at the level of TB yet, but just the
>> basics to read and send mail. Then, we build it out to be a good (not
>> simple) email client, so that 80% of the users are happy with it.
>> Then, finally, we build all the remaining TB features that the
>> remaining 19% need. That's the idea of MVP.
>
> I agree that basic implementations of "everything" comes first. Beyond
> that, well, it's best to explain with a hypothetical. Suppose we
> decided that LDAP and QRESYNC support were both necessary final
> features but not earlier mandatory features, and suppose that LDAP
> were the last feature not found in the new address book. I'm saying
> that we should prioritize finishing the LDAP implementation (via FFI
> to OpenLDAP, say) over implementing QRESYNC. Outside of that specific
> kind of situation, there isn't much difference in various approaches,
> and it's hard to say that sort of situation will even come up in the
> first place. So I'm not going to push very hard on this.

So, you're saying we should prioritize existing features (LDAP) over new
features (QRESYNC) that Thunderbird-Gecko does not have? Yes, I
generally agree on that. More specifically, the fact That Thunderbird
currently has that feature makes that feature rise considerably in the
feature list. I wouldn't make this an absolute, though. I think that
some way to sync the address book with mobile phones is more important
than LDAP, these days.

Things like prioritization are decisions that are best made numerically.
You count number of users who use the feature in current Thunderbird, or
you let people vote what they want and why, or things like that.
Essentially, you make sure you really know what people need, and why.
And then, on top of that, existing features get certain bonus points
over new features, to make sure we cover the old Thunderbird feature set.

>>
>>> This also means that whatever platform support library you have
>>> would include support for xpcshell/xpconnect/whatever else is being
>>> used by Gecko as a base layer.

> The problem with JS is that it doesn't have much in the way of the

> standard library. A TCP socket, for example, is handled very
> differently in node.js, in Chromium extensions, in XPCOM, in the B2G
> bindings, etc. At some point, you need a library that presents a
> common API for all the platforms to its different users

Ah, I see what you mean. Yes, I agree. For example, for my IMAP and POP3
implementations in JS (written by me for a XUL extension), I first wrote
a small Socket.js that abstracts the XPCOM API. It's not very big, and
it's simpler than the XPCOM API. When I port things to node.js, I have
to re-implement this Socket.js. But the IMAP and POP3 implementations
have no dependency on XPCOM and should hopefully then work after
porting. I'm expecting a day of work for this. In fact, that's the one
of the things I wanted to do this week or next week, if I can find the
time, so we'll see.

Luckily, JS covers a lot already. I've identified the following parts
that are not well standardized:
* Module system (I've spent the day yesterday to convert JSM to
npm-style require())
* Interprocess communication
* File system access (in node.js: require("fs"))
* TCP Sockets

FS and Sockets are rather easy, because relatively easy to abstract.
Module system is harder, but require() is reasonable.

The biggest problem in practice is the communication between UI and
backend (IPC). That can be vastly different, and can break the neck of
the project. Concretely, this is why I abandoned my port from XUL
extension to Jetpack. Luckily, that appears to be easy in Electron, with
the shared |global| object. I have yet to really test it in practice,
but it appears to work really nicely.

> the JS environment of Thunderbird is included as one of the platforms
> in this list

Yes, XULRunner vs. Electron vs. Cordova is the "platform" for TB-Next,
indeed.
I am hoping to keep it reasonably platform-abstracted, without too much
of a compat layer. Nothing like NSPR again, please! Luckily, JS gives us
most of what we need (basic types, etc.)

I will use the latest and greatest JS version, though. I really want to
use "=>" functions, and |class| if possible.

Too bad modules are not there yet.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import

> (In particular, how to handle ES6 modules versus require.js, etc. is
> not something I can recommend a solution to at this moment).

hihi! ditto! I was just asking myself the very same question this week.
Comforting to hear that you're facing the same dilemma. I've picked
require() for now, and it seems to work reasonably well (once I figured
out how to initialize "app-module-path").

> The original proposal gave the impression that Thunderbird development
> was to be unfunded, and certainly much of my wording was meant to
> focus on overturning that viewpoint.

Sorry, I worded it too drastically. I mean we should maintain status quo
for Thunderbird based on Gecko.

The danger to Thunderbird-on-Gecko is not this project, but increasing
breakage caused by Gecko, which will soon have unsustainable levels.
That's why I propose this project as solution.

> Which means we're in total agreement here.

:-)

> So starting out with descriptors like "next-generation" or

> "experimental" or "incubator" ... probably work better.

OK. I'll go with "next-generation".

Greetings to Jean Luc Picard! :)

Ben

[Joshua Cranmer 🐧]

On 4/10/2017 10:09 PM, R Kent James wrote:
> GPL code :( Primarily the work of 2 people. I wonder if we would ever
> consider switching to GPL?

Personally, I'm strongly averse to the GPL, particularly for code that I
want to make available to other people, for two reasons. First, as you
implied, GPL code limits the ability of other worthy, but non-GPL
projects, to use it (even if open-source). Second, the GPL actually
comes with some surprisingly onerous compliance requirements with
implications for how you build and deploy software, and I don't think
the philosophical goals of the GPL merit all that legal compulsion.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

_______________________________________________

[Disaster Master]

On Mon Apr 10 2017 22:47:28 GMT-0400 (Eastern Standard Time), Ben Bucksch <ben.b...@beonex.com> wrote:

We are also very clear that we want to take our users with us, and therefore want to keep a UI that is reasonably similar to current Thunderbird, at least as transitional option.

One point for clarity.

As with Firefox, it isn't just the base/default UI, it is a) the ease with which it can be customized by the end user by simply dragging things around and enabling/disabling toolbars, and b) Addons, that provide even more/deeper UI AND functional changes.

My Thunderbird UI is barely even recognizable compared to the default out-of-the-box UI, and would be even less recognizable if two current/open bugs were implemented.

[Magnus Melin]

In general, GPL core => GPL add-on.

 -Magnus

[Magnus Melin]

On 4/11/17 6:09 AM, R Kent James wrote:
> I wonder if we would ever consider switching to GPL?

Good question, the MPL really hasn't served Thunderbird very well, with
Postbox being the prime example, resulting in zero win for the base
project. This would not be possible with GPL.

You could also consider a dual licensing of GPL + commercial licensing
for proprietary software. (Like MySQL, Stanford CoreNLP etc.)

-Magnus

[Nomis101 🐝]

Am 11.04.17 um 20:53 schrieb Magnus Melin:

For people not so familiar with licensing, what are the differences
between MPL and GPL and how would this affect people who are building
Thunderbird from source?

[neandr]

Just in case you missed details about MPL vs GPL -->

> https://en.wikipedia.org/wiki/Mozilla_Public_License

[Óvári]

On 12/4/17 4:53 am, Magnus Melin wrote:

On 4/11/17 6:09 AM, R Kent James wrote:

I wonder if we would ever consider switching to GPL?

Good question, the MPL really hasn't served Thunderbird very well, with Postbox being the prime example, resulting in zero win for the base project. This would not be possible with GPL.

You could also consider a dual licensing of GPL + commercial licensing for proprietary software. (Like MySQL, Stanford CoreNLP etc.)

 -Magnus

Why did R Kent James demand CardBook be licensed under MPL then? Zero win for CardBook...

LibreOffice seems to be working, why not copy their licensing model? Or even join The Document Foundation (TDF)?

[Magnus Melin]

On 12.4.2017 02:06, Óvári wrote:
> Why did R Kent James demand CardBook

> <https://addons.mozilla.org/thunderbird/addon/cardbook/> be licensed under MPL

> then? Zero win for CardBook...

Depends on the perspective and your goals for your code. Being license
compatible to core code makes it possible to include it in core. In parts or
as a whole. If inclusion of some sort is the goal, the license needs to be
compatible.

> LibreOffice <https://www.libreoffice.org/> seems to be working, why not copy
> their licensing model?

They are actually using the MPLv2, like Thunderbird.

As I see it the problem with MPL is that you don't have to provide your whole
derivative work as open source. Only the actual changes, which makes porting a
feature extremely impractical as the changes can rely on other code which you
don't have to provide.

[Gervase Markham]

On 11/04/17 19:53, Magnus Melin wrote:
> You could also consider a dual licensing of GPL + commercial licensing
> for proprietary software. (Like MySQL, Stanford CoreNLP etc.)

Now _that_ would require permission from all copyright holders, which
would be a mammoth task. As well as the fact that this business model is
not well liked in some corners of the open source community.

Gerv

[Matt Harris]

On 4/12/17 5:08 PM, Magnus Melin wrote:

On 12.4.2017 02:06, Óvári wrote:

Why did R Kent James demand CardBook
<https://addons.mozilla.org/thunderbird/addon/cardbook/> be licensed under MPL
then? Zero win for CardBook...

Depends on the perspective and your goals for your code. Being license compatible to core code makes it possible to include it in core. In parts or as a whole. If inclusion of some sort is the goal, the license needs to be compatible.

LibreOffice <https://www.libreoffice.org/> seems to be working, why not copy
their licensing model?

They are actually using the MPLv2, like Thunderbird.

As I see it the problem with MPL is that you don't have to provide your whole derivative work as open source. Only the actual changes, which makes porting a feature extremely impractical as the changes can rely on other code which you don't have to provide.

I would prefer a license that requires any derivative to be open source.  That is a derivative mail client.  I have no issue with say a JS library being sucked into another project,  with only an attribution.
Matt

[Gervase Markham]

On 12/04/17 11:04, Matt Harris wrote:
> I would prefer a license that requires any derivative to be open
> source. That is a derivative mail client. I have no issue with say a
> JS library being sucked into another project, with only an attribution.

Can I suggest, without pointing fingers at anyone in particular, that
this excellent discussion of Thunderbird's future direction not rathole
on licensing questions? :-)

Gerv's First Law of New Projects states that any software project whose
first output is their own logo is doomed to failure. It might need a
corollary to incorporate projects which begin with an extended
discussion of licensing ;-)

Gerv

[Gervase Markham]

On 12/04/17 13:00, Jim Jagielski wrote:
> Well, if future direction depends on the leverage or consumption of 3rd
> party FOSS, then licensing compatibility is a major issue that should be
> resolved.

When a team has been assembled and prime candidates for the specific
pieces of 3rd party FOSS have been identified and there's a plan for how
they might be integrated, then let's have the discussion about
specifics. Discussions of the form of "I prefer GPL" and "I don't like
MPL" are not going to be productive.

[Jim Jagielski]

Well, if future direction depends on the leverage or consumption of 3rd party FOSS, then licensing compatibility is a major issue that should be resolved.

[The Wanderer]

Sorry for responding so relatively late to the original mail; I've been
distracted by other things, and only just read through the thread today.

Most of the discussion thus far seems to have been about the proposal to
reimplement in broad terms, and that's probably as it should be.
However, I'd like to respond to a couple of the more detailed
possibilities brought up in the original proposal mail.

For what it's worth, I approve of the idea of rewriting from scratch
with the aim of UI equivalence and feature parity (within reasonable
limits, although reasonable people may disagree where those limits are),
but I also have serious reservations (which I think I read Joshua
Cranmer as sharing) about whether there is even any plausible hope of
having sufficient manpower available to staff the various "teams" Ben
has suggested to effective levels.

(For the record, I'm 99% an interested user, although I've made a couple
of comparatively minor contributions to Thunderbird in the past - in the
form of code patches, not money - and am not inherently averse to making
more in the future.)

On 2017-03-24 at 13:04, Ben Bucksch wrote:

> Deeper goes the problem of virtual folders. Thunderbird has them
> implemented as a bolt-on, which essentially keeps a copy of the
> mails, as far as I understand, and they are slow and inflexible. A
> new Thunderbird implementation could take an approach of where
> folders are a view that is computed in real time (like views in
> databases). This matches what GMail does with the AllMails folder and
> tags as properties, and then tags also form a virtual folder. There
> are important implications that this has: I can have a unified inbox,
> one inbox per folder, and I can have the emails sorted into specific
> project folders on arrival, all without overhead. The same incoming
> email appears in all 3 folders in the same second, and is marked read
> at the same time. I no longer have to move emails, but they are
> filtered.

The mention of GMail-like views leaves me cautious. Is this related to
what GMail does with message deduplication (only store one copy of the
message, and display it in multiple locations), or is it something
different and more folder-centric rather than message-centric?

If the latter, this is probably fine, but the former could be
problematic. So far as I can tell / am aware, Gmail appears to do its
message-level deduplication by treating messages with the same
Message-ID as being identical and discarding any duplicates, even though
in some cases (e.g. sending an E-mail to a mailing list and then
receiving a list-modified copy back to yourself) the second message with
the same Message-ID may in fact include important differences.

I consider that a distinct misfeature, and would not be at all pleased
to see it replicated in Thunderbird, or in a successor. (I also
occasionally _want_ to mark one copy of a message read while leaving
another marked as unread, e.g. when I've been CCed directly on a message
which was also sent to a mailing list to which I subscribe.)

Assuming that sort of issue (the question of what criteria to use to
determine message uniqueness) can be addressed, however, I approve of
this concept.

> 0.8. Be close to the existing Thunderbird
>
> Even though we write almost all code from scratch, we will save a lot
> of time by having a clear goal: We want to replicate the current
> Thunderbird, from an end user perspective. That means, the user will
> find the same 3-pane window layout, the same way how folders and
> message lists and the thread pane operate. The theme will be similar.
> Existing Thunderbird users should feel right at home.
>
> We retain the overall UI and most features and qualities like
> performance, even if we do not copy all little details.

What about those of us who prefer a Thunderbird UI more like that which
came *before* "current Thunderbird", and apply various hacks (add-ons,
userChrome.css, et cetera) to restore aspects of that UI? That is, those
who may have felt "right at home" in a previous Thunderbird version, and
felt that a later version took that home away from them.

Will this rewrite take those people into account, and either provide
options for such UI variations, or make sure that it remains feasible to
hack the UI locally (by whatever means) to achieve that result?

(Examples fall into at least two categories: modifications which are
possible in current Thunderbird, and ones which are not. Examples of the
former which I change for my own Thunderbird include the To/CC portion
of the compose UI, and zebra-striping of the message list / thread pane;
examples of the latter which I would change if I could include the use
of tabs vs. the use of separate windows.)

--
The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw

[Ben Bucksch]

The Wanderer wrote on 12.04.2017 16:04:
> For what it's worth, I approve of the idea of rewriting from scratch
> with the aim of UI equivalence and feature parity (within reasonable
> limits, although reasonable people may disagree where those limits are),
> but I also have serious reservations

nod

> On 2017-03-24 at 13:04, Ben Bucksch wrote:
>
>> Deeper goes the problem of virtual folders.

> The mention of GMail-like views leaves me cautious. Is this related to
> what GMail does with message deduplication (only store one copy of the
> message, and display it in multiple locations), or is it something
> different and more folder-centric rather than message-centric?
>
> If the latter, this is probably fine, but the former could be
> problematic. So far as I can tell / am aware, Gmail appears to do its
> message-level deduplication by treating messages with the same
> Message-ID as being identical and discarding any duplicates, even though
> in some cases (e.g. sending an E-mail to a mailing list and then
> receiving a list-modified copy back to yourself) the second message with
> the same Message-ID may in fact include important differences.

Yes, I am not talking about message de-duplication, but operating on the
folder level. I want to have virtual folders that e.g. show me all new
mails in all folders (or a reasonable subset that I subscribed to and
can open). Or see all mails that are flagged "TODO" or similar.

Once we combine several folders, there will certainly some kind of
de-duplication, and most likely based on the message-ID. But that will
be an optional additional feature. You will certainly still have the
ability to see only a single IMAP folder or local mbox, and all messages
in it, like you do today in Thunderbird. Virtual folders would be an
additional feature on top. But I want them to be usable and fast, not
like in today's Thunderbird.

> Assuming that sort of issue (the question of what criteria to use to
> determine message uniqueness) can be addressed, however, I approve of
> this concept.

>
>> 0.8. Be close to the existing Thunderbird
>>
>> Even though we write almost all code from scratch, we will save a lot
>> of time by having a clear goal: We want to replicate the current
>> Thunderbird, from an end user perspective. That means, the user will
>> find the same 3-pane window layout, the same way how folders and
>> message lists and the thread pane operate. The theme will be similar.
>> Existing Thunderbird users should feel right at home.
>>
>> We retain the overall UI and most features and qualities like
>> performance, even if we do not copy all little details.
> What about those of us who prefer a Thunderbird UI more like that which
> came *before* "current Thunderbird", and apply various hacks (add-ons,
> userChrome.css, et cetera) to restore aspects of that UI? That is, those
> who may have felt "right at home" in a previous Thunderbird version, and
> felt that a later version took that home away from them.

Which version did you like, and why?

I always liked userChrome.css, and we can preserve that feature, given
that the style will still be based on CSS.

> Will this rewrite take those people into account, and either provide
> options for such UI variations

Concretely, I plan eventually 3 different UIs:
* Imitate current Thunderbird
* Make a new, fresh UI for those who have not used email clients
* Mobile UI for smartphones and tablets.

That will be down the road. We should not get distracted. But the
technical possibility should exist.

That possibility currently does not exist in Thunderbird. There's too
much logic mixed in the UI to make it feasible to make a different UI
without outright forking.

> (Examples fall into at least two categories: modifications which are
> possible in current Thunderbird, and ones which are not. Examples of the
> former which I change for my own Thunderbird include the To/CC portion
> of the compose UI, and zebra-striping of the message list / thread pane;

FWIW, zebra-striping of the message list / thread pane should be a trivial change in userChrome.css. And it will probably be part of one of the new UIs anyway.

[Gervase Markham]

On 12/04/17 15:04, The Wanderer wrote:
> Will this rewrite take those people into account, and either provide
> options for such UI variations, or make sure that it remains feasible to
> hack the UI locally (by whatever means) to achieve that result?

Oh, goodness, no. One massive advantage of producing a new product is
that you don't have to carry over all the UI configuration baggage of
the old one! This should certainly not be a goal; if it happens to
become possible in some way, fine, the people who want it can make the
effort to make it happen.

Gerv

[The Wanderer]

On 2017-04-12 at 10:29, Ben Bucksch wrote:

> The Wanderer wrote on 12.04.2017 16:04:

>> On 2017-03-24 at 13:04, Ben Bucksch wrote:
>>
>>> Deeper goes the problem of virtual folders.
>>
>> The mention of GMail-like views leaves me cautious. Is this related
>> to what GMail does with message deduplication (only store one copy
>> of the message, and display it in multiple locations), or is it
>> something different and more folder-centric rather than
>> message-centric?
>>
>> If the latter, this is probably fine, but the former could be
>> problematic. So far as I can tell / am aware, Gmail appears to do
>> its message-level deduplication by treating messages with the same
>> Message-ID as being identical and discarding any duplicates, even
>> though in some cases (e.g. sending an E-mail to a mailing list and
>> then receiving a list-modified copy back to yourself) the second
>> message with the same Message-ID may in fact include important
>> differences.
>
> Yes, I am not talking about message de-duplication, but operating on
> the folder level. I want to have virtual folders that e.g. show me
> all new mails in all folders (or a reasonable subset that I
> subscribed to and can open). Or see all mails that are flagged "TODO"
> or similar.

That seems entirely reasonable.

> Once we combine several folders, there will certainly some kind of
> de-duplication, and most likely based on the message-ID. But that
> will be an optional additional feature. You will certainly still have
> the ability to see only a single IMAP folder or local mbox, and all
> messages in it, like you do today in Thunderbird. Virtual folders
> would be an additional feature on top. But I want them to be usable
> and fast, not like in today's Thunderbird.

Depending on how this is implemented in practice, it's still something I
could be mildly concerned about, but that does sound as if it could be
reasonably good overall.

>>> 0.8. Be close to the existing Thunderbird
>>>
>>> Even though we write almost all code from scratch, we will save a
>>> lot of time by having a clear goal: We want to replicate the
>>> current Thunderbird, from an end user perspective. That means,
>>> the user will find the same 3-pane window layout, the same way
>>> how folders and message lists and the thread pane operate. The
>>> theme will be similar. Existing Thunderbird users should feel
>>> right at home.
>>>
>>> We retain the overall UI and most features and qualities like
>>> performance, even if we do not copy all little details.
>> What about those of us who prefer a Thunderbird UI more like that
>> which came *before* "current Thunderbird", and apply various hacks
>> (add-ons, userChrome.css, et cetera) to restore aspects of that UI?
>> That is, those who may have felt "right at home" in a previous
>> Thunderbird version, and felt that a later version took that home
>> away from them.
>
> Which version did you like, and why?

Essentially, Thunderbird 2, with the exception of the fact that thread
sorting was by (approximately) "order in which the thread-root message
was moved into this folder" rather than "date of the thread-root message".

"Why" is more complicated, and would probably require examples and
side-by-side comparisons. I could easily enough list off the changes I
dislike and what I do to (try to) revert them (I have a notes file for
whenever I need to spin up a new TB install or profile for myself), and
start explaining the whys in each case, but that seems like it would be
a tangent to the thread.

> I always liked userChrome.css, and we can preserve that feature,
> given that the style will still be based on CSS.

That says a lot about how much ability to hack the UI locally there will
be, just as a baseline.

>> Will this rewrite take those people into account, and either
>> provide options for such UI variations
>
> Concretely, I plan eventually 3 different UIs:
> * Imitate current Thunderbird
> * Make a new, fresh UI for those who have not used email clients
> * Mobile UI for smartphones and tablets.
>
> That will be down the road. We should not get distracted. But the
> technical possibility should exist.
>
> That possibility currently does not exist in Thunderbird. There's too
> much logic mixed in the UI to make it feasible to make a different
> UI without outright forking.

Assuming I understand what you're talking about (which I think I do),
that sounds reasonable.

>> (Examples fall into at least two categories: modifications which
>> are possible in current Thunderbird, and ones which are not.
>> Examples of the former which I change for my own Thunderbird
>> include the To/CC portion of the compose UI, and zebra-striping of
>> the message list / thread pane;
>
> FWIW, zebra-striping of the message list / thread pane should be a
> trivial change in userChrome.css. And it will probably be part of one
> of the new UIs anyway.

At the moment, it is the following snippet:

#threadTree > treechildren::-moz-tree-row(even) {
background-color: -moz-oddtreerow;
}

The "odd" vs. "even" mismatch is because the alternative needs several
additional CSS overrides (with !important), overriding one another in
succession, to produce the desired result in all cases; I'm not clear
whether Thunderbird 2 shaded odd rows or even ones, but the result is
functionally the same regardless, since I'm rarely at the "zero" end of
the list in any case.

This was only an example, however. (The compose-UI facet can also be
reverted by userChrome.css hacks, but it takes _much_ more extensive
code, and I just use something I found in a forum which someone else
wrote.)

[Axel Grude]

Subject:Re: Proposal to start a new implementation of Thunderbird based on web technologies
From:Gervase Markham <ge...@mozilla.org>
To:The Wanderer; Tb-planning
Sent: Wednesday, 12/04/2017 16:16:49 16:16 GMT ST +0100 [Week 15]

On 12/04/17 15:04, The Wanderer wrote:
> Will this rewrite take those people into account, and either provide
> options for such UI variations, or make sure that it remains feasible to
> hack the UI locally (by whatever means) to achieve that result?

Oh, goodness, no. One massive advantage of producing a new product is
that you don't have to carry over all the UI configuration baggage of
the old one! This should certainly not be a goal; if it happens to
become possible in some way, fine, the people who want it can make the
effort to make it happen.

Gerv

I don't know about UI hacking, but some entry point for a (completely fresh) Addons infrastructure sure would be nice. I wouldn't mind completely rewriting my addons if there was a chance of entering new markets.

Certainly overlays were a neat thing, and userChrome.css was one of the biggest advantages Firefox / Thunderbird had over programs of other origins.

Axel

[Tanstaafl]

On 4/12/2017, 10:04:01 AM, The Wanderer <wand...@fastmail.fm> wrote:
> Sorry for responding so relatively late to the original mail; I've
> been distracted by other things, and only just read through the
> thread today.

I'm glad you did, because I missed Ben's mention of virtual folders (so
will now go back and re-read his entire message in full to see what else
I missed).

> On 2017-03-24 at 13:04, Ben Bucksch wrote:
>> Deeper goes the problem of virtual folders. Thunderbird has them
>> implemented as a bolt-on, which essentially keeps a copy of the
>> mails, as far as I understand, and they are slow and inflexible. A
>> new Thunderbird implementation could take an approach of where
>> folders are a view that is computed in real time (like views in
>> databases). This matches what GMail does with the AllMails folder
>> and tags as properties, and then tags also form a virtual folder.
>> There are important implications that this has: I can have a
>> unified inbox, one inbox per folder, and I can have the emails
>> sorted into specific project folders on arrival, all without
>> overhead. The same incoming email appears in all 3 folders in the
>> same second, and is marked read at the same time. I no longer have
>> to move emails, but they are filtered.

> The mention of GMail-like views leaves me cautious. Is this related
> to what GMail does with message deduplication (only store one copy of
> the message, and display it in multiple locations), or is it
> something different and more folder-centric rather than
> message-centric?

I love the idea of 'virtual folders'... BUT...

I played with them in Thunderbird in the past, and while the idea was
awesome and at first I was very excited to start using them, I soon
realized the fact that they were local only made them a non starter for me.

This feature needs to be just as if not more focused on server
side/based virtual folders as it is Local. Dovecot calls these 'virtual
mailboxes'.

I would absolutely love a way to create/manage Dovecot virtual mailboxes
from Thunderbird. This way, the 'virtual folders' would be available to
all email clients (mainly my phone and tablet), and to other Thunderbird
instances (without having to recreate and manage them on each one
manually), and even better, would automatically have the advantage of
dovecots server side indexes for blindingly fast searches, even full
text searches (if you have the enabled), without GLODA totally disabled.
As nice as GLODA is, for someone who lives in IMAP and does NOT keep
their messages stored locally, GLODA is useless (apologies to the GLODA
developers - as I said, it is nice, but only for POP users - or IMAP
users who only use Thunderbird and/or keep everything on their local
PC('s). I don't, and never will.

In addition to dovecot, TB could also support GMail's out of the box,
then add support for additional servers as they add support for it.

> If the latter, this is probably fine, but the former could be
> problematic. So far as I can tell / am aware, Gmail appears to do its
> message-level deduplication by treating messages with the same
> Message-ID as being identical and discarding any duplicates, even
> though in some cases (e.g. sending an E-mail to a mailing list and
> then receiving a list-modified copy back to yourself) the second
> message with the same Message-ID may in fact include important
> differences. I consider that a distinct misfeature,

I fully agree, but I don't think that such a 'feature' automatically
follows a server-side virtual folder implementation, and hopefully the
TB dev(s) who end up implementing this could at least be convinced to
provide a way to disable it if they decided they liked it.

I do, however, in general, like the idea of de-duplication, and dovecot
does have an option for 'single instance storage', but when enabled, it
doesn't de-dupe list messages (meaning, it doesn't do what gmail does).

So, I don't think this is something that TB should be bothering with, at
least for IMAP accounts. It might make sense for POP/Local mail only
users, but in my opinion, those are getting fewer and fewer as time
marches on - IMAP (and its enhancements, like maybe JMAP) are the way of
the future.

>> 0.8. Be close to the existing Thunderbird
>>
>> Even though we write almost all code from scratch, we will save a lot
>> of time by having a clear goal: We want to replicate the current
>> Thunderbird, from an end user perspective. That means, the user will
>> find the same 3-pane window layout, the same way how folders and
>> message lists and the thread pane operate. The theme will be similar.
>> Existing Thunderbird users should feel right at home.

Yes, and those of us who are not coders can than participate in other
ways, such as collecting any/all UI bug/enhancement requests so that
whoever is 'recreating' the UI, can also focus on making improvements as
they do so - for just one example, improving the Vertical View (ala
Outlook), which would make Outlook users much happier if/when using
Thunderbird. I have a decent list of these...

>> We retain the overall UI and most features and qualities like
>> performance, even if we do not copy all little details.

> What about those of us who prefer a Thunderbird UI more like that
> which came *before* "current Thunderbird", and apply various hacks
> (add-ons, userChrome.css, et cetera) to restore aspects of that UI?
> That is, those who may have felt "right at home" in a previous
> Thunderbird version, and felt that a later version took that home
> away from them.

I think the main goal here should just be that we also want to duplicate
and enhance the ability to customize the UI, so that, even more than
now, we can modify the UI to the way we like it.

That said, it is still very customizable. To date there is only one UI
change I am unable to recreate from the older version 2 that I would
still like to be able to change - put all buttons on one toolbar, ie the
menu bar (yes, this can be done), show the menu bar above the tab bar
(can this still be done?), not show the tab bar when only one tab is
open (this can not be done when the window is not maximized and), and
shove the menu bar up into the Window Titlebar). This is currently
impossible to do.

Allow both mail.tabs.drawInTitlebar and mail.tabs.autoHide to be True
when window is maximized
https://bugzilla.mozilla.org/show_bug.cgi?id=1273199

Allow grabbing blank space on the Menu Bar to move the window
https://bugzilla.mozilla.org/show_bug.cgi?id=1336959

[The Wanderer]

On 2017-04-12 at 11:10, Tanstaafl wrote:

> On 4/12/2017, 10:04:01 AM, The Wanderer <wand...@fastmail.fm>
> wrote:

>> On 2017-03-24 at 13:04, Ben Bucksch wrote:

>>> 0.8. Be close to the existing Thunderbird
>>>
>>> Even though we write almost all code from scratch, we will save a
>>> lot of time by having a clear goal: We want to replicate the
>>> current Thunderbird, from an end user perspective. That means,
>>> the user will find the same 3-pane window layout, the same way
>>> how folders and message lists and the thread pane operate. The
>>> theme will be similar. Existing Thunderbird users should feel
>>> right at home.

>>> We retain the overall UI and most features and qualities like
>>> performance, even if we do not copy all little details.
>
>> What about those of us who prefer a Thunderbird UI more like that
>> which came *before* "current Thunderbird", and apply various hacks
>> (add-ons, userChrome.css, et cetera) to restore aspects of that
>> UI? That is, those who may have felt "right at home" in a previous
>> Thunderbird version, and felt that a later version took that home
>> away from them.
>
> I think the main goal here should just be that we also want to
> duplicate and enhance the ability to customize the UI, so that, even
> more than now, we can modify the UI to the way we like it.

I would mostly agree, but Gervase's reply leads me to conclude that
others may not.

> That said, it is still very customizable. To date there is only one
> UI change I am unable to recreate from the older version 2 that I
> would still like to be able to change - put all buttons on one
> toolbar, ie the menu bar (yes, this can be done), show the menu bar
> above the tab bar (can this still be done?), not show the tab bar
> when only one tab is open (this can not be done when the window is
> not maximized and), and shove the menu bar up into the Window
> Titlebar). This is currently impossible to do.
>
> Allow both mail.tabs.drawInTitlebar and mail.tabs.autoHide to be
> True when window is maximized
> https://bugzilla.mozilla.org/show_bug.cgi?id=1273199
>
> Allow grabbing blank space on the Menu Bar to move the window
> https://bugzilla.mozilla.org/show_bug.cgi?id=1336959

So far as I recall, there are only two main things I still want from the
Thunderbird 2 UI which I've been unable to regain:

* Tab-free access to all features, including e.g. the Add-ons Manager.
(I don't want a tab bar in my Thunderbird, and anything which would open
in a separate tab I want to be in a separate window instead.)

* Quick-filter functionality in a text-box widget, rather than a
separate toolbar. (I've actually stopped using quick-filter
functionality entirely since that change came in, since the appearance
and disappearance of that additional toolbar is just so unpleasant for me.)

Neither is likely to ever come in "current Thunderbird", and it's far
from clear that either would come in a complete rewrite either, but I
think the odds are better in the latter case.

(Then again, I'm still using Thunderbird 31, because I haven't been able
to take the time to do upgrade testing in a VM so I don't risk my live
data on something that might have negative UI changes and no way to
revert them. So it's entirely possible that there have been other UI
changes I haven't yet taken into account.)

[Philipp Kewisch]

Hi All,

I'm sorry if I missed some of the discussion, but I have the feeling we
are getting into a lot of technical details that are better discussed at
a later stage. I think we should come back to the high level approach
and ask ourselves what exactly we want to achieve and why we are doing this.

I think it would also be helpful to summarize the discussions and all
the pro/contra arguments. Maybe someone can to through all the email and
make a list. Ben maybe?

Philipp

[Gervase Markham]

On 12/04/17 18:44, Jim Jagielski wrote:
> But certainly the selection of a license semi-defines the community that
> rallies around a s/w project... As well as how said community wishes to
> distribute the s/w project itself.

But, particularly when building on existing codebases and libraries,
licenses are not selected in a vacuum.

Here's my point in a nutshell. We could:

a) have a big argument about licenses, come to a conclusion which
results in some people going off in a huff, and then try and find code
to achieve our goal compatible with those license choices; or

b) find the codebase(s) to start from which we think will give us the
best chance of success, and then look at their licensing, and see if
they can be combined and if the result upsets anyone. If so many people
don't like the result that you think your chances of success are now
small, repeat the process with different choices.

I suggest b) is much better than a).

I am not a "don't talk about licenses" person. I spend a lot of my
professional life talking about licenses. I'm Mozilla's license guy. But
that means I know a license rathole when I see one, and I know when the
most productive time for that discussion is.

[Ben Bucksch]

Gervase Markham wrote on 13.04.2017 14:41:
> On 12/04/17 18:44, Jim Jagielski wrote:
>> But certainly the selection of a license semi-defines the community that
>> rallies around a s/w project... As well as how said community wishes to
>> distribute the s/w project itself.
> But, particularly when building on existing codebases and libraries,
> licenses are not selected in a vacuum.
>
> Here's my point in a nutshell. We could:
>
> a) have a big argument about licenses, come to a conclusion which
> results in some people going off in a huff, and then try and find code
> to achieve our goal compatible with those license choices; or
>
> b) find the codebase(s) to start from which we think will give us the
> best chance of success, and then look at their licensing, and see if
> they can be combined and if the result upsets anyone. If so many people
> don't like the result that you think your chances of success are now
> small, repeat the process with different choices.

To be fair, b) is exactly how we arrived here. We talked about Nylas N1,
Kent briefly considered it as base, and then saw that it's GPL. That's
how this discussion came to be.

But you're right. This is not the time to have this discussion. It's
rathole and distraction, indeed.

Your point about being specific about what codebase we need is
particularly well-suited right here. I just double-checked the license
of Nylas N1, and it's in fact GPL v3. Which, if I remember correctly, is
not compatible with GPL v2. So, that's again an entirely different problem.

OTOH, emailjs.org libraries (which asutherland called 'excellent', IIRC)
are under MIT license, so the question might not even pose itself. MIT
is compatible to pretty much everything else, and we already have MIT
code in the Mozilla tree. And such libraries are probably the part that
we need the most.

So, we don't actually have any license problem.

[Phillip Hallam-Baker]

​If you are committed to really open source then MIT or BSD are by far the best options. There is a reason they are compatible with pretty much everything - they make no demands. 

GPL does make demands which is why quite a few semi commercial projects I have worked on used it. In one case we bought out a crypto library a couple of people had put together and released it under GPL to stop our competitors using it in their product. I think we would have done better to have just asked them to chip in a fair contribution and make it MIT License.

One of the big concerns I had with GPL3 was that it is a very long and very complicated contract written with the objective of achieving a particular set of ideological goals. And it is very much rooted in the techno-political debates of the 1990s, not today.

https://www.gnu.org/licenses/gpl-3.0.html

The part that affects me, security is that RMS had been going off on an anti-DRM jihad which is a problem because at a technical level it is really difficult to distinguish DRM measures and use of hardware protections for public keypairs. Using a private key that is bound to the machine and cannot be exported from it is a really useful and powerful technique. Does GPL3 permit that? Well I have no idea and neither does anyone else as you have to wade through a very long contract that uses a lot of language and a lot of principles that have never been tested in court.

Trying to force other people to follow your idea of freedom really isn't freedom, its something else.

[Jim Jagielski]

But certainly the selection of a license semi-defines the community that rallies around a s/w project... As well as how said community wishes to distribute the s/w project itself.

[Gervase Markham]

On 13/04/17 14:36, Phillip Hallam-Baker wrote:
> ​If you are committed to really open source then MIT or BSD are by far
> the best options.

<snip>

With respect, Phil, this sort of advocacy is precisely the sort of thing
I am suggesting is not helpful at this juncture.

It would be marginally less unhelpful if you were, by contributing to
the discussion, agreeing to commit coding resources under your control
to the Thunderbird++ project. Are you?

Gerv

[Ben Bucksch]

Ben Bucksch wrote on 24.03.2017 18:04:

Summary:

• Our base goes away. Gecko will change dramatically in the future, and dropping many features that Firefox does not need anymore, but that Thunderbird relies on.
• Our codebase is now roughly 20 years old. It heavily and intrinsically relies on those very Gecko technologies that are now being faded out, more or less aggressively.
• JavaScript and HTML5 have evolved dramatically. Entire applications in JS are now realistic, and have been done. There are several existing JS libraries we might leverage. JavaScript is an efficient language, which allows fast development. A rewrite in JavaScript makes sense now.
• We will learn from shortcomings of existing Thunderbird, and solve them, for example a more flexible address book, and cleanly supporting virtual folders without overhead.
  
• The goal of the rewrite is to be close to the existing Thunderbird, in UI and features, as a drop-in replacement for end users, without baffling them. They should immediately recognize the replacement as the Thunderbird they love. It will install and run as normal desktop application, like Thunderbird does today. It keeps user data local and private.
  
• We can also make a new, fresh desktop UI, as alternative to the traditional one, for new users. The technology also gives us the option to run it as mobile app.
  
• While we implement the new version of Thunderbird, the old codebase based on Gecko will be maintained until the rewrite is ready to replace the old one.
• I expect this effort to take roughly 3 years: 1 year until some dogfood (usable by some developers and enthusiasts). 2 years until a basic feature set is there. 3 years until we can replace Thunderbird.
  

I will describe each point in more detail below.

Based on the discussion, I'd like to amend or correct my proposal in the following points:

• Thunderbird based on Gecko would continue to be maintained on the current level of effort. I did write that above, but many apparently misunderstood me.
  
• If some of the newly written components can be backported to old Thunderbird, with a reasonable amount of effort, we can do that. The address book would be a prime example for that. However, that backporting is a separate effort and would not be done by the same team that writes the new app, but other developers, so that one team can concentrate on the new Thunderbird-Next-Generation and another team on the old Thunderbird.
• There are some existing libraries that we might be able to use for TB:NG, e.g. emailjs.org . There are also a lot of modules on npm.

Ben

[neandr]

ad TB.next Addressbook:
It's important to note the ICAL.js lib [*] written by Philipp Kewisch /
Fallen supports vCard/RFC6350. This has been proven with the NewZealand
university project.
Also interesting that project makes use of REACT and indexedDB, so it
could be a good candidate for the next generation.

[*] [iCAL.js](https://github.com/mozilla-comm/ical.js)
[iCAL.js API Documentation](http://mozilla-comm.github.io/ical.js/api/)


On 14.04.2017 23:32, Ben Bucksch wrote:
> ...

> The address book would be a prime example for that.

> ...

> There are some existing libraries that we might be able to use for
> TB:NG, e.g. emailjs.org . There are also a lot of modules on npm.

[Robert Kaiser]

Magnus Melin schrieb:

> On 4/11/17 6:09 AM, R Kent James wrote:
>> I wonder if we would ever consider switching to GPL?
>
> Good question, the MPL really hasn't served Thunderbird very well,
> with Postbox being the prime example, resulting in zero win for the
> base project. This would not be possible with GPL.

Wrong. From all I heard, this is a common misconception. A fork of a
GPLed project does not have to contribute code back to any upstream.
They only need to make the code available to their customers under GPL
as well.

KaiRo

[Phillip Hallam-Baker]

On Fri, Apr 14, 2017 at 2:53 AM, Gervase Markham <ge...@mozilla.org> wrote:

On 13/04/17 14:36, Phillip Hallam-Baker wrote:
> ​If you are committed to really open source then MIT or BSD are by far
> the best options.
<snip>

With respect, Phil, this sort of advocacy is precisely the sort of thing
I am suggesting is not helpful at this juncture.

It would be marginally less unhelpful if you were, by contributing to
the discussion, agreeing to commit coding resources under your control
to the Thunderbird++ project. Are you?

Gerv

​With respect Gerv, my long experience is that many people pushing for GPL3 suddenly stop doing so after they are required to read it and explain what it means.​

Since my code is under MIT License, my resources are available regardless. Though I would have to know which language to target first if I was going to write the back end of the code synthesizer. I have done C#, C, FORTRAN and Occam. Never targeted Javascript though.

[Tanstaafl]

On Wed Apr 12 2017 11:50:11 GMT-0400 (Eastern Standard Time), The

Wanderer <wand...@fastmail.fm> wrote:
> So far as I recall, there are only two main things I still want from the
> Thunderbird 2 UI which I've been unable to regain:
>
> * Tab-free access to all features, including e.g. the Add-ons Manager.
> (I don't want a tab bar in my Thunderbird, and anything which would open
> in a separate tab I want to be in a separate window instead.)

I also prefer windows to tabs in Thunderbird (love tabs in Firefox
though), and would push for this capability as an option in any rewrite.
Basically, everything that by default opens in a new tab should be able
to be set via a preference to open in a window - and for me, better
would be a global option to disable tabs completely. I love tabs in
Firefox, but have no use for them in my mail client.

> * Quick-filter functionality in a text-box widget, rather than a
> separate toolbar. (I've actually stopped using quick-filter
> functionality entirely since that change came in, since the appearance
> and disappearance of that additional toolbar is just so unpleasant for me.)

While I have grudgingly gotten accustomed to the loss of vertical space
caused by the Quick Filter bar, I too much prefer the old textbox widget
you could add/move to any toolbar and would switch to that in a
heartbeat given the opportunity.

There was an excellent attempt in bug 570815
(https://bugzilla.mozilla.org/show_bug.cgi?id=570815) in late 2010
(after the existing search widget was removed) to bring it back as an Addon:

Unified Search
https://addons.mozilla.org/en-US/thunderbird/addon/unified-search-187593/

I used this for years, but the quirks just got to be too much trouble to
deal with, and I eventually grudingly uninstalled it. I really wish that
Iago (the one who made the Addon) had continued working on it until it
was perfected. It had one serious problem (it would somehow get out of
sync with the Quickfilter toolbar settings and filtering went wonky),
and some less serious UI issues that seemed like would be easily
resolvable, but he eventually just stopped responding, not sure why.

> Neither is likely to ever come in "current Thunderbird",

See above Unified Addon. It is entirely doable, and some (much?) of the
work has already been done in the form of the Addon. Some of the
problems with it could be remedied by not trying to incorporate the
'Global' searching capability, just the regular filtering of current
folder, which is one of the last things Iago was considering when he
disappeared (you can follow what happened mostly in the bug).

> and it's far from clear that either would come in a complete rewrite
> either, but I think the odds are better in the latter case.

Yes, it would definitely be high on my list of 'improvements ' to
incorporate during the rewrite, but as long as the Addon framework
(WebExtensions?) still allowed for powerful Addons, it would still be
possible to do it that way.

Note: it was explained on one of the other lists (tb-planning? dev?)
that it is entirely possible for Thunderbird to add WebExtension APIs
independently of what the Firefox devs add for Firefox.

> (Then again, I'm still using Thunderbird 31, because I haven't been able
> to take the time to do upgrade testing in a VM so I don't risk my live
> data on something that might have negative UI changes and no way to
> revert them. So it's entirely possible that there have been other UI
> changes I haven't yet taken into account.)

There is no risk whatsoever - as long as you simply duplicate/backup
your profile folder. This is trivial, so that is no excuse not to
upgrade. Also, if you're running a VM, you can also take snapshots for
this purpose (although that is way overkill for just upgrading TB). It
is very doubtful there are UI changes that would be deal breakers, but
it may take a little work to fix anything you don't like. I heavily
customize my UI, and it has never been a real problem for me since the
carnage during the transition from 2 to 3+.

[The Wanderer]

(This is drifting offtopic, if not already there. Also, I apologize for
length.)

On 2017-04-13 at 11:21, Tanstaafl wrote:

> On Wed Apr 12 2017 11:50:11 GMT-0400 (Eastern Standard Time), The
> Wanderer <wand...@fastmail.fm> wrote:
>
>> So far as I recall, there are only two main things I still want
>> from the Thunderbird 2 UI which I've been unable to regain:

<snip>

I've thought of a third one: the message header pane. AFAIK it's still
not possible to set things up so you get the same information in the
same space as was the case with TB2, and although the tweaks from
http://kb.mozillazine.org/Thunderbird_3.1_-_New_Features_and_Changes#Gaining_Vertical_Space
and ones based on but not identical to the ones from
http://kb.mozillazine.org/Make_Thunderbird_3.1_look_like_Thunderbird_2
come close enough that I can live with the result, they're not perfect.

>> Neither is likely to ever come in "current Thunderbird",
>
> See above Unified Addon. It is entirely doable, and some (much?) of
> the work has already been done in the form of the Addon. Some of the
> problems with it could be remedied by not trying to incorporate the
> 'Global' searching capability, just the regular filtering of current
> folder, which is one of the last things Iago was considering when he
> disappeared (you can follow what happened mostly in the bug).

It's good to know that I'm not the only one wanting these features, in
both cases.

(I should perhaps clarify slightly: I don't insist that the quick-filter
widget be a text box in the old style. I would have no problem if it
behaved exactly like the collection of widgets that appear in the
toolbar, albeit with less horizontal space between them than the toolbar
provides, and if those widgets could only be moved as a unit - as long
as that unit could be moved around like any other widget, rather than
being fixed in a separate toolbar.)

>> and it's far from clear that either would come in a complete
>> rewrite either, but I think the odds are better in the latter
>> case.
>
> Yes, it would definitely be high on my list of 'improvements ' to
> incorporate during the rewrite, but as long as the Addon framework
> (WebExtensions?) still allowed for powerful Addons, it would still
> be possible to do it that way.

I've had it confirmed, on the dev-addons mailing list, that it is
explicitly intended that WebExtensions not provide any way to
significantly modify the existing UI (rather than do things like add
toolbars and so forth), apparently for maintainability and
separation-of-layers and keep-working-in-the-next-release reasons...

> Note: it was explained on one of the other lists (tb-planning? dev?)
> that it is entirely possible for Thunderbird to add WebExtension
> APIs independently of what the Firefox devs add for Firefox.

...and the underlying structural reasons why that decision was made are
such that, much though I dislike it, I'm not sure it would be possible
(much less practical) to make a different decision on the Thunderbird
side with its significantly less available manpower.

>> (Then again, I'm still using Thunderbird 31, because I haven't been
>> able to take the time to do upgrade testing in a VM so I don't risk
>> my live data on something that might have negative UI changes and
>> no way to revert them. So it's entirely possible that there have
>> been other UI changes I haven't yet taken into account.)
>
> There is no risk whatsoever - as long as you simply duplicate/backup
> your profile folder. This is trivial, so that is no excuse not to
> upgrade.

I do that as a matter of course for upgrades, but it's not a panacea.

The trouble is that I have "connect automatically" mail accounts, and
filter rules set up which move new mail from the inboxes of those
accounts into Local Folders, as soon as the new mail is seen.

If I launch the new version, and it sees new mail and moves it, and then
I discover UI problems and revert to the old version and old profile,
then I will have lost those new mails with - as far as I can see - no
practical way to recover them.

In order to avoid that, as far as I can see I'd have to launch the new
version with the network disconnected and do all the UI testing before
reconnecting it (and hope I hadn't missed anything, and that there are
no regressions around receiving new mail or the filters I use), and at
that point it's almost less of a hassle just to do the VM test in the
first place.

If anyone knows of a practical Third Way around this problem, I'd be
glad to know about it!

> Also, if you're running a VM, you can also take snapshots for this
> purpose (although that is way overkill for just upgrading TB).

I don't run in a VM normally - I just spin one up for upgrade testing.
(And doing so is rather more of a hassle than it used to be, for
unrelated reasons, which is why I've taken so long to get to it for
post-TB31 releases.)

> It is very doubtful there are UI changes that would be deal breakers,
> but it may take a little work to fix anything you don't like. I
> heavily customize my UI, and it has never been a real problem for me
> since the carnage during the transition from 2 to 3+.

For me, the radical changes to the To/CC section of the compose UI would
have been a deal-breaker for the upgrade to TB31 - except that I didn't
do this type of testing for that upgrade, since I'd been working with
relatively-late pre-release builds as part of getting a couple of
restore-TB2-behavior-with-an-option patches in and hadn't seen the
changes there.

Discovering them after the upgrade, when I no longer had a viable way to
downgrade, was a nasty shock. I had to scramble to find the complex -
and probably fragile - userChrome CSS stanzas which someone on a forum
put together to revert those changes.

Having had that experience (immediately after having worked to fix the
remaining issues which had blocked me from upgrading from TB2), I no
longer trust that negative UI changes will not come in with a new
Thunderbird release, and I feel it necessary to apply the same
pre-upgrade testing practices as I use with Firefox. (I also don't trust
that the userChrome tweaks I'm applying - which I don't even fully
understand myself, in this case - won't break with the upgrade, since
there was talk about making further changes in this area specifically to
address people who thought the result was bad.)

[Paul D. Fernhout]

On 2017-04-04 7:25 AM, Disaster Master wrote:
> On Tue Apr 04 2017 00:02:21 GMT-0400 (Eastern Standard Time), Paul D.
> Fernhout <pdfer...@kurtz-fernhout.com> wrote:
>> Your point on risk management makes a lot of sense. That's why I
>> suggested a possibility of having a new Thunderbird that could run
>> alongside the existing one and share the same files while offering a
>> different UI.
>
> First, such a possibility already exists - if you are using IMAP.

Yeah, but I use POP. :-)

> But connecting some brand new client, in beta, alpha, or even pre alpha
> stages of development? This presents a very large risk to your precious
> data, one that I would never be willing to accept, and if anyone did,
> the PR backlash could be devastating when their X years worth of
> precious emails were nuked by some bad code.

You're absolutely right of course.

But, one of the toughest things in a battle is to get soldiers to
abandon their dug-in foxholes to retreat in the face of advancing enemy
fire. The soldiers indeed are safer in the foxholes with all the random
shells flying around -- until the battle line advances to their foxhole
and they are overrun. And the closer the battle line comes, the more
dangerous it is for soldiers to leave their foxholes.

The battle line of entropy for Firefox has been advancing for the past
couple of years as the Firefox browser codebase that Thunderbird depends
on is deprecated and then abandoned. And there is also Mozilla
abandoning financial support of Thunderbird -- it's one area of solid
user growth -- to redirect money to various quixotic activities that
have little to do with either maintaining a popular web browser or
supporting distributed secure communications and web identity management
via email.

At some point, Thunderbird-as-is may fail in a big way given it will be
in practice unmaintainable and so at risk of security issues like a
maliciously-crafted email exploit or perhaps some bad interaction with
an updated OS. Thunderbird also has a known failure mode of 4GB
mailboxes getting corrupted that many long-time Thunderbird users may be
reaching.

So, we're going to have to take some risks sooner or later -- one way or
another. Unfortunately, as time grows smaller and smaller until that day
of failure, the risks grow bigger and bigger because there is less time
to experiment and test. It feels safer to keep using Thunderbird-as-is
and it is safer -- until something bad happens from lack of maintenance.

I've been using Thunderbird almost since it began, migrating from
Mozilla Application Suite. It would be sad to have to abandon Thunderbird.

But I know based on looking at the Thunderbird code and thinking about
the risks that it seems very unlikely Thunderbird-as-it-is will be
maintainable either technically or socially much longer. Dealing with
complex legacy C/C++ code that embeds an abandoned browser in a
convoluted architecture is going to be a high-stress no-fun job. It will
have a high learning curve. Even if some team does support the codebase
for pay, it is going to be hard to do it well for the long haul because
of the complexity of relying on complex embedded browser mixed with the
unforgiving nature of C/C++.

So, we need to do something. It is prudent either to start making new
alternatives or to migrate to existing well-tested proven alternatives.

The biggest thing Thunderbird has going for it is a community of loyal
users. So the best solutions would leverage that somehow -- either by:
* an incremental move such as changing Thunderbird to serve JMAP to web
clients,
* creating a parallel companion to Thunderbird with the risk you mention
(but without the risk of changing much about Thunderbird itself),
* a complete from-scratch-build that gets the Council's blessing
assuming regular Thunderbird users take notice -- or perhaps even
* a collective decision to move to another existing platform (like, say,
Nylas N1 that Ben mentioned) and put energy into enhancing it to have
all of Thunderbird's features, or,
* making something just so compellingly better than Thunderbird that we
all want to switch to it (like a social semantic desktop that just
happens to also handle email well as well as chat, whiteboards,
federated wikis, and so on -- as I tried with Twirlip).

Most likely, Thunderbird users being a large and independent-minded
group (otherwise we'd all be using gmail's web client instead), all
these paths will be tried if/when something big breaks (or perhaps before).

Anyway, with just $1 per Thunderbird user, we could fund the core of
something amazing.

I might forward my job application to the Mozilla Foundation from 2011
to work on Thunderbird to perhaps spark more conversation on bigger
picture alternatives like a semantic desktop.

--Paul Fernhout (pdfernhout.net)
"The biggest challenge of the 21st century is the irony of technologies
of abundance in the hands of those still thinking in terms of scarcity."

[Magnus Melin]

On 4/15/17 2:37 AM, Robert Kaiser wrote:
> Magnus Melin schrieb:
>> On 4/11/17 6:09 AM, R Kent James wrote:
>>> I wonder if we would ever consider switching to GPL?
>>
>> Good question, the MPL really hasn't served Thunderbird very well,
>> with Postbox being the prime example, resulting in zero win for the
>> base project. This would not be possible with GPL.
>
> Wrong. From all I heard, this is a common misconception. A fork of a
> GPLed project does not have to contribute code back to any upstream.
> They only need to make the code available to their customers under GPL
> as well.

I did not mean to imply they have to upstream every change, they don't
have to do anything except release the code. But, with the *complete
code* available, the upstream project *could* fairly easily include
changes/features they like. With MPL a fork can easily make it very
difficult for anyone to do that.

-Magnus

[Ben Bucksch]

Phillip Hallam-Baker wrote on 14.04.2017 15:35:
> many people pushing for GPL3 suddenly stop doing so after they are
> required to read it and explain what it means.​

Nobody pushed for GPL3.

> Since my code is under MIT License, my resources are available
> regardless. Though I would have to know which language to target first
> if I was going to write the back end of the code synthesizer. I have
> done C#, C, FORTRAN and Occam. Never targeted Javascript though.

The next TB is going to be written in JavaScript, that much is for sure.

Ben

[Robert Kaiser]

Magnus Melin schrieb:

> On 4/15/17 2:37 AM, Robert Kaiser wrote:
>> Magnus Melin schrieb:
>>> On 4/11/17 6:09 AM, R Kent James wrote:
>>>> I wonder if we would ever consider switching to GPL?
>>>
>>> Good question, the MPL really hasn't served Thunderbird very well,
>>> with Postbox being the prime example, resulting in zero win for the
>>> base project. This would not be possible with GPL.
>>
>> Wrong. From all I heard, this is a common misconception. A fork of a
>> GPLed project does not have to contribute code back to any upstream.
>> They only need to make the code available to their customers under
>> GPL as well.
>
> I did not mean to imply they have to upstream every change, they don't
> have to do anything except release the code. But, with the *complete
> code* available, the upstream project *could* fairly easily include
> changes/features they like. With MPL a fork can easily make it very
> difficult for anyone to do that.

They do not need to publicly make the code available (or make it
available to upstream), they only have to make it available to anyone
they give the binaries to, i.e. their customers. Of course, that needs
to happen under GPL again, so any customer can make that code public or
contribute it back, but the company changing the code doesn't have to do
that themselves.

And IIRC, that is very much the same for MPL, just that it only affects
the specific files that were under MPL in the first place and not any
files they added to the code (unless they contain pieces of code copied
or moved from MPLed files).

The GPL and the MPL are pretty much equivalent on the level of the files
containing covered code, the differences are that GPL also "virally"
applies to all added files of code and even code linked into the final
executable - and that the GPL is much harder to read and potentially
surprising while the MPL2 is easy to read and understand.

KaiRo

[Ben Bucksch]

Paul D. Fernhout wrote on 06.04.2017 05:51:
> But, one of the toughest things in a battle is to get soldiers to
> abandon their dug-in foxholes to retreat in the face of advancing enemy
> fire. The soldiers indeed are safer in the foxholes with all the random
> shells flying around -- until the battle line advances to their foxhole
> and they are overrun. And the closer the battle line comes, the more
> dangerous it is for soldiers to leave their foxholes.

Paul, this is an excellent analogy for the situation Thunderbird finds
itself in.

So far, we've been keeping up with Gecko with volunteer work only. Now
we have Jörg K hired to keep up to date with Gecko, and nonetheless the
situation is already so dire that he can't manage. He is facing new
breakage every day, and not enough time to fix it all properly.

And the hardest blow is still to come, after Gecko 57, when Firefox XUL
extensions are killed. The Firefox teams are already scratching to make
breaking changes. The worst is yet to come.

So, I think that analogy is very much true. And unfortunately, the
reaction of the Thunderbird community is the same as in that analogy,
trying to hold on to status quo.

Right now, the biggest risk for Thunderbird is to not move fast enough.
Getting going is not longer sufficient. We need to run now, race.

[Disaster Master]

On 4/16/2017, 10:36:59 AM, Robert Kaiser <ka...@kairo.at> wrote:

They do not need to publicly make the code available (or make it 
available to upstream), they only have to make it available to anyone 
they give the binaries to, i.e. their customers.

But if the binaries are publicly available - say, in the form of trial versions - wouldn't they still have to provide public access to the source?

I think you meant that they wouldn't have to only if the restrict access to ANY of their binaries to paying customers.

[Disaster Master]

On 4/15/2017, 9:53:25 AM, The Wanderer <wand...@fastmail.fm> wrote:

(This is drifting offtopic, if not already there. Also, I apologize for
length.)

On 2017-04-13 at 11:21, Tanstaafl wrote:

On Wed Apr 12 2017 11:50:11 GMT-0400 (Eastern Standard Time), The
Wanderer <wand...@fastmail.fm> wrote:

So far as I recall, there are only two main things I still want
from the Thunderbird 2 UI which I've been unable to regain:

<snip>

I've thought of a third one: the message header pane. AFAIK it's still
not possible to set things up so you get the same information in the
same space as was the case with TB2, and although the tweaks from
http://kb.mozillazine.org/Thunderbird_3.1_-_New_Features_and_Changes#Gaining_Vertical_Space
and ones based on but not identical to the ones from
http://kb.mozillazine.org/Make_Thunderbird_3.1_look_like_Thunderbird_2
come close enough that I can live with the result, they're not perfect.

I'm curious what you were missing that CompactHeader doesn't give you for the MEssage Pane HEader?

(I should perhaps clarify slightly: I don't insist that the quick-filter
widget be a text box in the old style. I would have no problem if it
behaved exactly like the collection of widgets that appear in the
toolbar, albeit with less horizontal space between them than the toolbar
provides, and if those widgets could only be moved as a unit - as long
as that unit could be moved around like any other widget, rather than
being fixed in a separate toolbar.)

Well, if you read the comments in the bug and/or try the Unifed Search Addon, I think you'd like what you saw. It still works (I think), but as I said, it isn't perfect.

Note: it was explained on one of the other lists (tb-planning? dev?)
that it is entirely possible for Thunderbird to add WebExtension
APIs independently of what the Firefox devs add for Firefox.

...and the underlying structural reasons why that decision was made are
such that, much though I dislike it, I'm not sure it would be possible
(much less practical) to make a different decision on the Thunderbird
side with its significantly less available manpower.

Yes, while it would apparently be trivial for TB to manage extensions of the Webextensions API to provide hooks as deep as we wanted/needed, independent of Mozilla/Firefox, the matter of finding devs capable of writing/extending said APIs is obviously not trivial.

[Disaster Master]

On 4/16/2017, 11:05:08 AM, Ben Bucksch <ben.b...@beonex.com> wrote:

The next TB is going to be written in JavaScript, that much is for sure.

Is it? When was this decision formally adopted?

I'm not saying I'm for or against it, I'm just asking, I don't recall seeing any serious discussion about this, including the all important debating of alternatives.

[Tito]

>
> Right now, the biggest risk for Thunderbird is to not move fast enough.
> Getting going is not longer sufficient. We need to run now, race.

Ben, unfortunately your are right i.e. there is not much time left and
TB needs to hurry but as Kent spoke about architecture, design patterns
and Philip about the goal of that future client. I really believe that
this is crucial before moving forward with any implementation.

> The next TB is going to be written in JavaScript, that much is for sure.

As much as i would like to agree with that since in this case there such
complicated technologies as XPCOM will not be needed, I would like to
put my 2 cents:

There is a very good reason why binary websocket together with
WebAssembly are being so actively pushed by all major vendors on the
market. I am not saying JavaScript is bad i am just saying that my
target will be to make a future such project much more robust by any big
changes. For example Mozilla decided to drop xul, xpcom etc ... and the
whole projects collapses. I completely agree that JavaScipt must be
present in the such project, but I also believe that some of the core
needs to be written in some other strongly typed languages since i
believe 2 things will happen in the future:

1) WebAssembly with websockets will take over because of one simple
reason this will kill all of the addons such as Privacy Badger, No
Script JS, Adblock plus etc ... this will result in some devaluation of
the JavaScript. Large vendors are just waiting for the right moment for
that.

2) large game development business will use exactly the 2 technologies
above and not java script (and by the way this is already happening!!!)

3) Unfortunately WebAssembly is difficult to debug as well impossible to
steal the code i.e. proprietary. Companies love that!

Now considering all this above I am not saying JavaScript will disappear
but it will definitely decrease from the networking perspective. I get
it that we are really productive with JavaScript and that we can save
time on development but also I believe that the above arguments needs to
be considered before going forward. I would like you to consider Java
here as well because of 6 reasons:

a) We need to try to bring that future client not only to the end users
but also to the to the enterprise. Enterprise today live in Java

b) By bringing it to the enterprise we have good change of getting a
companies financing the project.

c) "Write once, run anywhere" - Platform independent

d) Phones are using Java - future code will run on the phones as well,
not changes what so ever, this will be really the killer feature.

e) No one can drop JDK i.e. there is OpenJdk , IBM jdk etc. etc.

f) There it is plethora of libraries written on Java.

g) It is the most popular language today according to Tiobe 2017.



regards,

Tito
(very old dino)