Email SASL status

[Joshua Cranmer]

Hi all,

I've started working on a pure-JS SASL library for use in the email
protocols. The code's not public yet, because, well, I'm a neat freak
when it comes to clean history, and the code's not up to my standards
for releasing publicly. :-)

Presently, the library supports PLAIN, LOGIN, XOAUTH2, CRAM-MD5, and
SCRAM-* (specifically using SHA-1, SHA-256, SHA-384, and SHA-512)
mechanisms. I definitely want to get the EXTERNAL and ANONYMOUS support
in before first public release. I have not thoroughly tested that code
works in the case where the server sends garbage data, although this is
only really a problem for SCRAM and NTLM which require detailed parsing
of server challenges.

The code itself is lightweight on requiring ES6 support. It needs:
* String.prototype.normalize (although String.prototype.normalize =
function () { return this; } would likely be sufficient for normal use,
just not passing tests).
* Math.trunc.... I kind of used this without realizing it's an ES6
feature :-)
* ES6 promises
* Generator support--the SASL modules are implemented as generators over
the server input. It was a very clean way of doing it until I switched
the SCRAM implementation to using WebCrypto and realized I couldn't mix
generator and async-function-via-generator syntax.

In addition to ES6 features, emailsasl requires the following web
standards, polyfilled for node:
* TextEncoder, specifically only UTF-8.
* WebCrypto (but see below)

Support for CRAM-MD5, naturally, requires MD5 and HMAC-MD5 support. I
filed a bug (<https://bugzilla.mozilla.org/show_bug.cgi?id=1130876>) on
it for Gecko, although I expect that it will likely be marked
WONTFIX--although hearing support from B2G folks might influence minds
in a way that a TB developer can't. Chrome, I suspect, is unlikely to
implement it, as their main point-of-contact on the WebCrypto mailing
lists has generally resisted the idea of supporting "legacy" (i.e.,
anything that actually exists today) protocols.

Speaking of support for MD5, I more recently looked into NTLM. I had
been under the impression that it was a Single sign-on-y protocol like
XOAUTH2 or GSSAPI, but it's actually more akin to CRAM-MD5--although, if
you use the SSPI functions on Windows, it looks like you can reuse
credential cache or something to avoid needing to give the application a
password (needs more manual testing). Thunderbird/Firefox presently
maintain support for cross-platform NTLM support by manually
implementing the protocol. There's actually a bewildering variety of
different responses--LM, NTLM, NTLMv2, LMv2, and NTLM2. I think it's
only necessary to support NTLMv2 (and maybe LMv2), which is nice because
the older variants require DES keys. They still do require MD4 hash of
the passwords--and MD4 has no chance of getting added to WebCrypto, I
think. Given all of this, supporting NTLM "eventually" is sort of a
goal, but one I'm willing to defer to post-public-release.


The reason why I'm bothering to send this email without a copy of the
code to inspect is that I want to gather some feedback on the design.
I'm assuming that no one cares to implement the SASL security wrapping
(which is really only possible with GSSAPI, NTLM, or SCRAM and largely
pointless in the face of TLS--if you really want channel binding, you
could achieve it with client certificate authentication instead)--it is
invasive into the protocol, since it's a wrapping layer (and
protocol-specific if it's above or below TLS! ... and IMAP doesn't
specify which is the case [1]!). I try to design my APIs to be
idiot-proof, although the more I've read on SASL, the harder that is:
the rules for sending an initial response in SMTP are actually quite
complicated.

For your feedback, here is the API from the client's perspective:
var auth = new sasl.Authenticator("imap", "example.com", ["PLAIN",
"LOGIN", "XOAUTH2"], opts.auth);

To try to authenticate:
var details = auth.tryNextAuth();
if (details == null)
throw new Error("No acceptable authentication mechanisms!");
var authMechanism = details[0]; // e.g., PLAIN
var isClientFirst = details[1]; // true or false
// When isClientFirst is true, you can send a SASL-IR. I'd demonstrate
it here,
// but getting all of the edge cases right is difficult without a lot of
testing.

To send a step:
var serverChallenge = /* */; // In case of SASL-IR, this is ""
auth.authStep(serverChallenge).then(
function (str) {
// Example for IMAP continuation line.
send("+ " + str + "\r\n");
}, function (err) {
// Internal error in authentication protocol, abort
send("*\r\n"); // I didn't know this was a thing until today
onAuthError(err); // Go back and try to authenticate with the next
mechanism...
});

The inherent looping nature of many steps (LOGIN, NTLM both require 2;
SCRAM 3; XOAUTH2 may require 2; don't know about GSSAPI), and the
possibility to fallback to other authentication mechanisms makes it hard
to demonstrate how to code this without resorting to an async/await
style function.

The SASL options parameter presently has up to three values--user, pass,
and oauthbearer, all of which ought to be fairly self-explanatory in
their meaning. I modeled them off of smtpclient's options.auth parameter
already, although I renamed xoauth2 to oauthbearer to future-proof the
meaning when the OAUTHBEARER command passes to RFC status [2].

Asides from wanting feedback on the API, the other issue I have is
trying to find a way to automatically test execution in a browser
context. Phantom.js is not going to work, since this is intimately
dependent on crypto support. I would try hacking my way up an
xpcshell-based build for Thunderbird's use, but that also doesn't have
webcrypto (unless I tried running all the tests in a web worker...).
It's also unkind to people who don't have xpcshell test runners handy.
I'm curious if anyone else on this list has a better testing environment
I could leverage.

Thoughts/comments/questions/concerns/flames?

[1] Actually, since I wrote that line, I did find a more definitive
answer... in the RFC for IMAP COMPRESS.
[2] The current OAUTHBEARER draft has the error JSON optionally return
somewhat useful information for attempting to configure oauth without
prior information, e.g., identifying dynamic registration endpoints. I
don't have a way to feedback this yet, since Google's use of XOAUTH2 is
underdocumented in this regard, but it would be nice to support it if I
can reliably assume it will exist.

--
Joshua Cranmer
News submodule owner
DXR coauthor

[Andris Reinman]

Thats great to know that someone is actually willing to step into the U+1F4A9 of SASL :D

In Whiteout we mostly care about Chrome support and all the listed ES6 features including String.normalize are supported by Chrome. Currently we run unit tests in node but we can switch to iojs that also should support most of the stuff (besides TextEncoder and WebCrypto that can be polyfilled).

> Support for CRAM-MD5, naturally, requires MD5 and HMAC-MD5 support

> ….

> They still do require MD4 hash of the passwords--and MD4 has no chance of getting added to WebCrypto, I think

Its kind of shocking that MD4 is actually still used somewhere. In generally I don’t see an issue if we use javascript implementations of MD5 and MD4 instead of WebCrypto as these aren’t the most CPU intensive operations and the values that need to be hashed are short. There are tons of md5 implementations already in npm registry.

> send("*\r\n"); // I didn't know this was a thing until today

I had no idea of the * response either. There even seems to be a separate ABNF grammar rule for it in rfc4954 called "cancel-response”. I tried it out with AUTH LOGIN in some SMTP servers (should return 501) and while Dovecot gave me "501 5.7.0 Authentication aborted” for * and "535 5.7.8 Error: authentication failed” for a non-valid response, then Gmail gave "501 5.5.2 Cannot Decode response” both for * and non-valid response. Hotmail/Outlook also seems to know about * by returning “501 5.0.0 Authentication aborted” but Yahoo returns "500 5.5.1 Invalid command” so it doesn’t seem something that is widely understood or known.

> For your feedback, here is the API from the client's perspective:

I think the proposed API is really good, I like the fallback from preferred mechanism to a less preferred one.

> Phantom.js is not going to work, since this is intimately dependent on crypto support.

Crypto can be polyfilled but using Generators or even “let” would blow it up since you can’t polyfill syntax. In the long run we need to consider options for dumping phantomjs entirely. For now the only viable option would seem some kind of combination of mocha + selenium + headless chrome or firefox. As browser engines are developing quite rapidly it makes no sense trying to polyfill everything (in some cases it is not even possible). I, for one, am looking forward to use the Streams API once it is available but I highly doubt that Phantomjs is ever going to catch up with it.

Best regards,
Andris

> --
> You received this message because you are subscribed to the Google Groups "Email.js Mailing List" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emailjs+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


--
Whiteout Networks GmbH c/o Werk1
Grafinger Str. 6
D-81671 München
Geschäftsführer: Oliver Gajek
RG München HRB 204479

[Joshua Cranmer]

On 2/10/2015 3:09 AM, Andris Reinman wrote:
> Thats great to know that someone is actually willing to step into the U+1F4A9 of SASL :D
>
> In Whiteout we mostly care about Chrome support and all the listed ES6 features including String.normalize are supported by Chrome. Currently we run unit tests in node but we can switch to iojs that also should support most of the stuff (besides TextEncoder and WebCrypto that can be polyfilled).

I'm using npm test = "node --harmony_generators
./node_modules/.bin/grunt test" for the moment, which suffices to run tests.

> Its kind of shocking that MD4 is actually still used somewhere. In
> generally I don’t see an issue if we use javascript implementations of
> MD5 and MD4 instead of WebCrypto as these aren’t the most CPU
> intensive operations and the values that need to be hashed are short.
> There are tons of md5 implementations already in npm registry

Well, using non-crypto-hardened implementations might have a challenging
time passing sec-review, since a poorly-written variant of cryptographic
methods opens up the possibility to leakage via side-channel attacks.
And given that SASL is one of those places where crypto concerns is
actually important, relying on third-party or custom JS implementations
of MD4 and MD5 make me queasy. I'll have to schedule some time to talk
with the security folks at Mozilla, though.

>> send("*\r\n"); // I didn't know this was a thing until today
> I had no idea of the * response either. There even seems to be a separate ABNF grammar rule for it in rfc4954 called "cancel-response”. I tried it out with AUTH LOGIN in some SMTP servers (should return 501) and while Dovecot gave me "501 5.7.0 Authentication aborted” for * and "535 5.7.8 Error: authentication failed” for a non-valid response, then Gmail gave "501 5.5.2 Cannot Decode response” both for * and non-valid response. Hotmail/Outlook also seems to know about * by returning “501 5.0.0 Authentication aborted” but Yahoo returns "500 5.5.1 Invalid command” so it doesn’t seem something that is widely understood or known.

Reading RFC 4954 carefully, the proper response should probably be a
501. It is definitely worth pointing out that clients should ignore the
returned "fail" status code, since it was "caused" by the authentication
abort. Which goes to show--writing client or server protocol
implementations are far, far, far harder then they look.

> Crypto can be polyfilled but using Generators or even “let” would blow
> it up since you can’t polyfill syntax. In the long run we need to
> consider options for dumping phantomjs entirely. For now the only
> viable option would seem some kind of combination of mocha + selenium
> + headless chrome or firefox. As browser engines are developing quite
> rapidly it makes no sense trying to polyfill everything (in some cases
> it is not even possible). I, for one, am looking forward to use the
> Streams API once it is available but I highly doubt that Phantomjs is
> ever going to catch up with it.

I've heard about selenium when trying to find more details, although it
would be nice to find someone that took a mocha webdriver and automated
the selenium portion of the hookup. The closest I found was one suite
that was using browserify, which kind of defeats the purpose of trying
to test with a web browser...