gmail and the infamous "530 5.7.0 Must issue a STARTTLS command first"

[Paul Bibbings]

There is a lot of discussion from many sources on the net about setting
up gmail smtp in gnus and the "530 5.7.0 Must issue a STARTTLS command
first" error. What I haven't been able to find is a solution that works
(in Emacs 22.3.1 with gnus v5.11). Has anyone been able to find a
workaround for this that uses either starttls or gnutls-cli?

I've done some debugging and the problem is, as the error messages
suggest, that gnus is trying to send an AUTH command /before/ it has
issued a STARTTLS. The error can be reproduced at the command line with
something like:

/usr/bin $gnutls-cli --crlf -s -p 587 smtp.gmail.com
Resolving 'smtp.gmail.com'...
Connecting to '74.125.77.109:587'...

- Simple Client Mode:

220 mx.google.com ESMTP 14sm14646818ewy.15
ehlo test
250-mx.google.com at your service, [86.15.147.8]
250-SIZE 35651584
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250 PIPELINING
auth plain AHBhdWwuYmliYmluZ3NAZ21haWwuY29tAHdoaXRlc294
530 5.7.0 Must issue a STARTTLS command first. 14sm14646818ewy.15

I've traced the beginning of the problem down to the code around line
719 in mail/smtpmail.el:

(if (and do-starttls
(smtpmail-find-credentials smtpmail-starttls-credentials host port)
(member 'starttls supported-extensions) // line 719
(numberp (process-id process)))
(progn
(smtpmail-send-command process (format "STARTTLS"))
(if (or (null (car (setq response-code (smtpmail-read-response process))))
(not (integerp (car response-code)))
(>= (car response-code) 400))
(throw 'done nil))
(starttls-negotiate process)
(setq do-starttls nil))
(setq do-ehlo nil))))

Here, (member !startls supported-extensions) is failing, not because
gmail doesn't require a STARTTLS here, but because, apparently, one of
the other extensions "isn't supported." Consequently, the sending of
STARTTLS is skipped and the whole thing fails. I haven't gone into that
much detail to remember which it is that is failing, and I have to admit
that I don't know Lisp at all. Does anyone here know enough to be able
to dig a little deeper, and perhaps offer a solution?

Perhaps better still, does anyone have an alternative or add-on
implementation that uses openssl directly, rather than through starttls?
openssl has a very different command line invocation to both starttls
and gnutls-cli, but if it could be hooked in in their place, it handles
the STARTLS directly, so the above code wouldn't fail for not being able
to issue it in this case. Call it a challenge, but I would like to get
gmail smtp working with one of these (starttls or gnutls-cli), even
though I am aware and have tested many other alternatives - mostly
without success, or at least with issues of their own.

Regards

Paul Bibbings

[Ted Zlatanov]

On Thu, 07 Jan 2010 01:59:43 +0000 Paul Bibbings <paul.b...@tesco.net> wrote:

PB> (if (and do-starttls
PB> (smtpmail-find-credentials smtpmail-starttls-credentials host port)
PB> (member 'starttls supported-extensions) // line 719
PB> (numberp (process-id process)))
PB> (progn
PB> (smtpmail-send-command process (format "STARTTLS"))
PB> (if (or (null (car (setq response-code (smtpmail-read-response process))))
PB> (not (integerp (car response-code)))
PB> (>= (car response-code) 400))
PB> (throw 'done nil))
PB> (starttls-negotiate process)
PB> (setq do-starttls nil))
PB> (setq do-ehlo nil))))

PB> Here, (member !startls supported-extensions) is failing, not because
PB> gmail doesn't require a STARTTLS here, but because, apparently, one of
PB> the other extensions "isn't supported." Consequently, the sending of
PB> STARTTLS is skipped and the whole thing fails. I haven't gone into that
PB> much detail to remember which it is that is failing, and I have to admit
PB> that I don't know Lisp at all. Does anyone here know enough to be able
PB> to dig a little deeper, and perhaps offer a solution?

How about a variable smtpmail-do-starttls-always which can be set to a
list of server names? It could be inserted in the list above. I don't
know if it's a good idea to change the logic of the supported
extensions, which is the other route to solve this.

PB> Perhaps better still, does anyone have an alternative or add-on
PB> implementation that uses openssl directly, rather than through starttls?
PB> openssl has a very different command line invocation to both starttls
PB> and gnutls-cli, but if it could be hooked in in their place, it handles
PB> the STARTLS directly, so the above code wouldn't fail for not being able
PB> to issue it in this case. Call it a challenge, but I would like to get
PB> gmail smtp working with one of these (starttls or gnutls-cli), even
PB> though I am aware and have tested many other alternatives - mostly
PB> without success, or at least with issues of their own.

I'm pretty sure you can wrap your SMTP connection in a shell script that
sets it up the way you want, then just does STDIN/STDOUT as far as Gnus
is concerned. This seems to be a problem on Windows, though (many
bug reports and few people have reported that it all just works).

The best solution is to provide TLS support inside Emacs as a core
function to transform a network connection but I don't know if that's
ever going to happen. Maybe someone else here does; if not I can ask on
emacs-devel. I'm cc-ing Simon who probably has some idea.

Ted