A preliminary review of smtpclient.js for Thunderbird

[Joshua Cranmer 🐧]

Hello,

First an introduction: I am one of the primary developers of
Thunderbird, and I am also the person behind jsmime (cf.
<https://github.com/mozilla-comm/jsmime>, although I have a lot of
updates to that code present only in my fork since I'm still working on
them).

Recently, I have started looking into moving larger portions of our
codebase into JavaScript for easier maintenance. It makes a good deal
sense to share and reuse open-source JS libraries for core protocol
aspects (e.g., IMAP, POP, SMTP, NNTP, MIME). As SMTP is the simplest of
these protocols and the most isolated in our codebase, as well as
something I'm intending to modify anyways in the near future to
introduce EAI support, I've started by taking a look at smtpclient.js to
see how much work it would take to be possible to use in Thunderbird.

Since I've not communicated too much in the past with the primary
maintainers of these projects, I don't know what the willingness would
be to, for example, adopt newer ES6 features (such as generators and
promises, see later) or make changes to the API. I'm aware that the
SpiderMonkey engine and Gecko DOM APIs are far more aggressive in
picking up new features than Node.js, so I personally have a strong bias
towards using these new features as they are available and I'm aware of
the impact that causes for attempting to use libraries as web client
pages or node.js.

What follows is a list of major issues I see in the present codebase
based on the current implementation, roughly ordered in decreasing
severity. I've not tried running it in detail or doing any porting work
to see what works and what breaks.

1. smtpclient.js appears to assume that messages will always be valid
UTF-8 strings when encoded. Unfortunately, this is most definitely not
the case. My current designs for JSMime involve potentially creating
non-UTF-8 8-bit parts for attachments if they can be legally encoded as
such. I've also been repeatedly informed by the Japanese localization
team that supporting ISO-2022-JP as the charset body is a non-negotiable
requirement [1], so I get to have to support encoding strings to
non-UTF-8 charsets as much as I hate doing so. :-(

2. There is very little support for different authentication mechanisms.
Thunderbird uses effectively a tiered auth system: we deduce from prefs
the set of acceptable authentication methods, compare with that the list
of mechanisms that the server supports and tries them in turn. The
source code can be seen at
<https://dxr.mozilla.org/comm-central/source/mailnews/compose/src/nsSmtpProtocol.cpp#873>
(warning: this isn't a permalink, so it may cease to be valid in a few
months), if you're willing to dig into strange C++ code.

Supporting CRAM-MD5 is necessary to pass tests, and I suspect that
dropping support for NTLM or GSSAPI is untenable. At the same time, I
also recognize that asking for a JS implementation of NTLM or GSSAPI is
completely unworkable. Given the architecture of how SASL works (it's
effectively a middleman between authentication protocols and other
protocols), and its utility as well for the other protocols, it's
plausible that we could permit an interactive SASL session
authentication mechanism to handle this mechanism. I'll also note that,
judging from bug activity, it's unlikely that we would need to support
the data security layer.

2a. Another minor difference in authentication is that Thunderbird
lazily processes the authentication when it's done at the protocol,
instead of providing the parameters when necessary. Only for NNTP is
there a benefit to doing it Thunderbird's way, but I don't think it's
worth adversely impacting the API for current clients for the sake of
symmetry for a protocol most clients don't care about. Hence why this is
a separate sub-point.

3. smtpclient.js does not support the delivery status notification
extension.

4. The debug logging infrastructure seems underwhelming. Our current
implementation saves to a log all of the traffic that is sent and
received, merely blanking out the authentication steps (see
<https://dxr.mozilla.org/comm-central/source/mailnews/compose/src/nsSmtpProtocol.cpp#1569>).
I am aware that some potential users may be concerned about that level
of debugging knowledge, but having had to resort to following these logs
before, I believe there is benefit in these steps that requires logging
this much data.

5. There appears to be no provisioning for reporting EHLO capabilities.
These are useful to, for example, know which authentication methods may
be attempted, or whether or not the server is EAI-compliant, or if DSN
is supported. I would suggest utility of 8BITMIME and BINARYMIME, but
just-send-8 has been the prevailing view among MTAs, and I'm still wary
of using the Content-Transfer-Encoding: binary encoding.

6. There seems to be a strong footgun effect if you attempt to use EAI
or IDN addresses on a non-EAI server: it looks like the client is purely
responsible for applying UTR46 (or IDNA2003/2008) processing to IDN
hostnames in EHLO or constituent pieces of email addresses.

The following points are issues of design that are purely my own opinions:
7. Having used Promises a lot, I've come to view event-based
notification systems as increasingly cumbersome and problematic to use
[2]. For example, take the simplicity of this API:

client.send({from: "some...@example.com", to: ["ot...@example.com"]},
stream).then(
() => console.log("Message sent!"),
(e) => console.log("Message failed: " + err)
);

8. If you combine promises and generators, you can build a framework
that very easily supports the line-oriented, semi-UTF-8 nature of SMTP,
POP, and NNTP. For example, a message generator might look like:

function* sendMessage(conn, to, from, msg) {
let [code, text] = yield conn.runCommand("MAIL FROM:<" + from + ">");
if (code < 250 || code > 259)
throw new Error("Your SMTP server won't let you send");
for (let rcpt of from) {
[code, text] = yield conn.runCommand("RCPT TO:<" + rcpt + ">");
if (code != 250)
throw new Error("Not doing error handling in this example");
}
let [code, text] = yield conn.runCommand("DATA");
if (code != 354)
throw new Error("Still not doing this");
yield conn.sendMultilineStream(msg);
}

[Side note: this example is adapted from an experimental NNTP client I
wrote, since I figure people here might understand SMTP better than
NNTP. The client used a small connection pool, which influenced how I
did some of the API.]

This kind of implementation strategy I personally find to be much easier
to read, but it also absolutely requires generators and nearly
absolutely requires promises. You get the clarity of imperative-style
code (including proper exception handling) but still get the
asynchronous I/O processing that is desired.

I hope that I am not coming off as someone who is saying "your
implementation sucks; let me design a new one for you" (although I do
feel strongly about the utility of Promises, so some amount of
evangelism there was intentional). I think it would be a shame not to
settle on one high-quality source of libraries for email utilities, and
the current implementation appears to be a good start. It's just that
Thunderbird demands more features in its SMTP code than the current
library is supplying.

Thoughts/questions/comments/concerns/flames/diatribes/rants?

[1] It gets worse: the jp team would rather mangle text bodies than
silently fallback from ISO-2022-JP to UTF-8. As I've said in prior
communications about jp email, "The logic that the JP locale wants is
incompatible with good logic for the rest of the world."

[2] The newer drafts of TCPSocket specify a Streams-based API for
working with the stream instead of the event-based Mozilla variant. The
first thing I did when playing with TCPSocket was create a
quick-and-dirty wrapper to give me something akin to the stream-based
API, because I preferred that version so much. Unfortunately, it does
not look like Mozilla (or anyone else?) is looking to implement the
streams-based version anytime soon.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist


--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

[Joshua Cranmer 🐧]

[Andris Reinman]

Hi Joshua,

Thanks for evaluating smtpclient.js! Shortly about the project – email.js components were created for the Whiteout.io e-mail client which is a PGP capable HTML5 e-mail client. Most of our users use it for Gmail, so we try to be compatible with Gmail first and support others if we have time and resources (we’re not Gmail-only but Gmail usually comes first). The components are also meant to be working together, even though you do not have to. So smtpclient.js expects input from mailbuild.js etc. Thus there are some shortcuts taken that might not be acceptable for Thunderbird which should support anything the world has ever seen about the e-mail.

1. smtpclient.js appears to assume that messages will always be valid UTF-8 strings when encoded. Unfortunately, this is most definitely not the case. My current designs for JSMime involve potentially creating non-UTF-8 8-bit parts for attachments if they can be legally encoded as such. I've also been repeatedly informed by the Japanese localization team that supporting ISO-2022-JP as the charset body is a non-negotiable requirement [1], so I get to have to support encoding strings to non-UTF-8 charsets as much as I hate doing so. :-(

Actually we do not assume utf-8 but ascii as this is expected to be the output from the mailbuild component. We know about the JP issue but our market share in Japan is 0, so no-one has complained so far. Besides it is not possible in the Whiteout app to change output encoding. All text output is always ascii or utf-8 in quoted-printable if needed (even though I have wanted to change this behavior to use base64 for non-latin alphabets – quoted-printable does not make much sense for chinese). Attachments are base64.

2. There is very little support for different authentication mechanisms.

Email.js components are developed using complaint-driven-development model, so if no-one is complaining we try to not add excess baggage – not that we have anything against it, we just have to plan our limited resources wisely. CRAM-MD5 should be pretty easy to add but I have never done anything with the network based authentications, so I have no idea if and how these could be implemented.

3. smtpclient.js does not support the delivery status notification extension.

Haven’t seen any need for this as Gmail has limited support for it. Shouldn’t be too difficult to add though

4. The debug logging infrastructure seems underwhelming. Our current implementation saves to a log all of the traffic that is sent and received, merely blanking out the authentication steps (see <https://dxr.mozilla.org/comm-central/source/mailnews/compose/src/nsSmtpProtocol.cpp#1569>). I am aware that some potential users may be concerned about that level of debugging knowledge, but having had to resort to following these logs before, I believe there is benefit in these steps that requires logging this much data.

Agree, we’ll probably change this behavior anyway across email.js, since it is really hard to debug errors if available data is fairly limited.

5. There appears to be no provisioning for reporting EHLO capabilities. These are useful to, for example, know which authentication methods may be attempted, or whether or not the server is EAI-compliant, or if DSN is supported. I would suggest utility of 8BITMIME and BINARYMIME, but just-send-8 has been the prevailing view among MTAs, and I'm still wary of using the Content-Transfer-Encoding: binary encoding.

Currently the following items are detected from EHLO response: AUTH PLAIN, AUTH LOGIN, AUTH XOAUTH2, SIZE (even though this value is not actually used for anything) and STARTTLS. These values are only used internally by the module but not reported outside, even though it might make sense to make this information available.

6. There seems to be a strong footgun effect if you attempt to use EAI or IDN addresses on a non-EAI server: it looks like the client is purely responsible for applying UTR46 (or IDNA2003/2008) processing to IDN hostnames in EHLO or constituent pieces of email addresses.

Yes, this is handled in the client, or actually by the mailbuild library – you feed all from/to/cc/bcc headers to mailbuild and it returns the envelope for SMTP https://github.com/whiteout-io/mailbuild#getenvelope

It would make sense to add a second check in the smtpclient.js as well though.

The following points are issues of design that are purely my own opinions:
7. Having used Promises a lot, I've come to view event-based notification systems as increasingly cumbersome and problematic to use [2]

This kind of implementation strategy I personally find to be much easier to read, but it also absolutely requires generators and nearly absolutely requires promises. You get the clarity of imperative-style code (including proper exception handling) but still get the asynchronous I/O processing that is desired.

Actually we would love to use promises and generators this way and actually it is not Node that is the culprit (for server side testing we could use iojs instead which has a lot better ES6 support) but Phantomjs that we use for client-side testing. Phantomjs doesn’t seem to reflect reality at all by now as it deviates so much from Chrome and Firefox but it’s also the best tool so far. Other option would be using a ES6to5 transpiler https://github.com/6to5/6to5 but we haven’t tried it.

I hope that I am not coming off as someone who is saying "your implementation sucks; let me design a new one for you" (although I do feel strongly about the utility of Promises, so some amount of evangelism there was intentional). I think it would be a shame not to settle on one high-quality source of libraries for email utilities, and the current implementation appears to be a good start. It's just that Thunderbird demands more features in its SMTP code than the current library is supplying.

Not at all, I totally agree with the issues. If we had unlimited time and unlimited budget, we’d probably start supporting GSSAPI and all the other stuff one day. Right now I really doubt if we can do such things. In the less-longer term than eternity we’d probably like to switch over to use Promises, Generators and the upcoming Streaming API for all the components. So far we have done something with Promises (mostly for BrowserBox) but nothing with Generators and the Streams API isn’t even ready yet.

Best regards,

Andris Reinman

Whiteout Networks GmbH c/o Werk1

Grafinger Str. 6

D-81671 München

Geschäftsführer: Oliver Gajek

RG München HRB 204479

[Joshua Cranmer]

On 1/23/2015 9:39 AM, Andris Reinman wrote:

Hi Joshua,

Thanks for evaluating smtpclient.js! Shortly about the project – email.js components were created for the Whiteout.io e-mail client which is a PGP capable HTML5 e-mail client. Most of our users use it for Gmail, so we try to be compatible with Gmail first and support others if we have time and resources (we’re not Gmail-only but Gmail usually comes first). The components are also meant to be working together, even though you do not have to. So smtpclient.js expects input from mailbuild.js etc. Thus there are some shortcuts taken that might not be acceptable for Thunderbird which should support anything the world has ever seen about the e-mail.

Not everything, more like 99.9% :-)

Actually we do not assume utf-8 but ascii as this is expected to be the output from the mailbuild component. We know about the JP issue but our market share in Japan is 0, so no-one has complained so far. Besides it is not possible in the Whiteout app to change output encoding. All text output is always ascii or utf-8 in quoted-printable if needed (even though I have wanted to change this behavior to use base64 for non-latin alphabets – quoted-printable does not make much sense for chinese). Attachments are base64.

... I'm surprised you don't use typed array output for mailbuild? Not supporting non-UTF-8 is also understandable (JSMime itself doesn't support text/* conversion using non-UTF-8; the ISO-2022-JP conversion gunk happens in TB pre-JSMime. Which does mean I have to duplicate features like magic EOL conversion, but oh well). I'm also surprised you're hard-cording QP/base64 decisions--it's not too bad to do the metrics to decide format: <https://github.com/jcranmer/jsmime/blob/mime-emit/mimeemitter.js#L24> (to be fair, you do need the entire body to do the stats--but I would expect you to get the body as one giant string anyways instead of a stream).

Email.js components are developed using complaint-driven-development model, so if no-one is complaining we try to not add excess baggage – not that we have anything against it, we just have to plan our limited resources wisely. CRAM-MD5 should be pretty easy to add but I have never done anything with the network based authentications, so I have no idea if and how these could be implemented.

I found out last night that webcrypto doesn't support MD5. :-(

CRAM-MD5, and the newer SCRAM replacements are basically a challenge-response protocol: you take random nonces and you use cryptographic hashes together with HMAC. Effectively, you mostly need a decent crypto library and a crypto-secure random number generator.

NTLM and GSSAPI, on the other hand, are protocols that you implement by calling the relevant OS API. Particularly for GSSAPI, which typically means Kerberos, and where you really do want to use the local keytab instead of running the protocol yourself.

Not at all, I totally agree with the issues. If we had unlimited time and unlimited budget, we’d probably start supporting GSSAPI and all the other stuff one day. Right now I really doubt if we can do such things. In the less-longer term than eternity we’d probably like to switch over to use Promises, Generators and the upcoming Streaming API for all the components. So far we have done something with Promises (mostly for BrowserBox) but nothing with Generators and the Streams API isn’t even ready yet.

So it sounds like you wouldn't mind if I started prototyping a variant of smtpclient.js using the Promise/generator approach with the intent of eventually submitting a pull request?

-- 
Joshua Cranmer
News submodule owner
DXR coauthor

[Andris Reinman]

... I'm surprised you don't use typed array output for mailbuild? Not supporting non-UTF-8 is also understandable (JSMime itself doesn't support text/* conversion using non-UTF-8; the ISO-2022-JP conversion gunk happens in TB pre-JSMime. Which does mean I have to duplicate features like magic EOL conversion, but oh well). I'm also surprised you're hard-cording QP/base64 decisions

Mailbuild behavior is mostly “inherited” from my other smtp project Nodemailer that generates and streams ascii only data to smtp. QP is not hard coded, the requirement is detected on compile step. If the message is ascii only with short lines, then no transfer encoding is applied. If it is ascii but with long lines, then format=flowed is applied and if it contains non-ascii characters, then QP is applied. Attachments use base64 by default, even for ascii content. The goal was to keep it simple for the module user – I guess it is more important in the server side like the case with Nodemailer, where an automated app just wants to send out an email and not worry about the details but I can’t see why this approach wouldn't work in the client side as well.

I found out last night that webcrypto doesn't support MD5. :-(

MD5 is relatively simple to implement in javascript and the hashed values are really small, so even if you use the worst implementation then you are still faster than tcp. Including a full MD5 implementation in smtpclient.js itself surely doesn’t make much sense. In the Whiteout app we can use other included crypto modules like openpgp or forge but obviously in the context of smtpclient we can’t always assume that these modules are available. Besides I do not like Cram-md5/sha1 anyway since this requires the server to store clear passwords or do some kind of shenanigans. But if the server supports it and MD5 is somehow available then it wouldn’t hurt to support Cram-md5.

So it sounds like you wouldn't mind if I started prototyping a variant of smtpclient.js using the Promise/generator approach with the intent of eventually submitting a pull request?

Not at all! As far as we are concerned, you can go crazy with this module.

Best regads,

Andris

--
You received this message because you are subscribed to the Google Groups "Email.js Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emailjs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.