bug#14380: 24.3; `network-stream-open-tls' fails in some imap servers on w32

[João Távora]

Here's a emacs -Q recipe:

$ emacs -Q
M-x eval-expression RET (setq gnus-select-method '(nnimap "imaps.mycompany.com")) RET
M-x gnus

An error pops up:

nnimap (imaps.mycompany.com) open error: ''gnutls-cli' is not recognized as an internal or external command,

In the *Message* buffer, we can see at at least one tls command succeeded

Opening connection to imaps.mycompany.com via tls...
gnutls.c: [1] (Emacs) GnuTLS library not found
Opening TLS connection to `imaps.mycompany.com'...
Opening TLS connection with `gnutls-cli --insecure -p 993 imaps.mycompany.com'...failed
Opening TLS connection with `gnutls-cli --insecure -p 993 imaps.mycompany.com --protocols ssl3'...failed
Opening TLS connection with `openssl s_client -connect imaps.mycompany.com:993 -no_ssl2 -ign_eof'...done
Opening TLS connection to `imaps.mycompany.com'...done
nnimap (imaps.mycompany.com) open error: ''gnutls-cli' is not recognized as an internal or external command,
'. Continue? (y or n) n

This works fine, on the same server with linux.

My analysis and possible fix is as follows: The problem is that
`network-stream-open-tls' returns invalid information in its final form:

(list stream
(network-stream-get-response stream start eoc)
(network-stream-command stream capability-command eoc)
'tls)

This is turn is because code immediately preceding it

;; If we're using tls.el, we have to delete the output from
;; openssl/gnutls-cli.
(when (and (null use-builtin-gnutls)
eoc)
(network-stream-get-response stream start eoc)
(goto-char (point-min))
(when (re-search-forward eoc nil t)
(goto-char (match-beginning 0))
(delete-region (point-min) (line-beginning-position))))

is not really doing what it says it's doing: it's not deleting enough
input, and the search for capabilities actually returns the IMAP
greeting.

This in turn is related to the CRLF lineendings that the function
`open-tls-stream' running under w32 only (in linux it doesn't happen)
inserts in the process buffer before the end of the tls handshake. The
code in `network-stream-open-tls' for getting rid of useless starting
output deletes up to the first CRLF (and apparently it should do so just
once, since other CRLF's are legitimate). But if fails and catches a
very early CRLF which is still part of the now useless TLS
handshake. The search performed by

(network-stream-command stream capability-command eoc)

doesn't work. That said, a reasonable way to fix this might be adding a
line

(delete-region (point-min) (point))

as the first form in the body of the `when'. This is because
`open-tls-stream' apparently guarantees that point will be left where
the tls handshake ended, as per its code snippet and comment:

(if start-of-data
;; move point to start of client data
(goto-char start-of-data))

Alternalively, make `open-tls-stream' do that itself in the end. I'm
doing just that with.

(defadvice open-tls-stream (after dont-forget-to-erase-buffer-up-to-point activate)
(delete-region (point-min) (point)))

Or maybe improve searches by `network-stream-get-response' or be smarter
when deleting output.

Here's the contents of the nnimap process buffer. Notice the CRLFs.


'gnutls-cli' is not recognized as an internal or external command,
operable program or batch file.

Process *nnimap* exited abnormally with code 1
'gnutls-cli' is not recognized as an internal or external command,
operable program or batch file.

Process *nnimap* exited abnormally with code 1
depth=0 C = PT, L = Lisboa, O = SISCOG.COM, OU = IMAP server, CN = *.siscog.com, emailAddress = sup...@siscog.pt
verify error:num=18:self signed certificate
verify return:1
depth=0 C = PT, L = Lisboa, O = SISCOG.COM, OU = IMAP server, CN = *.siscog.com, emailAddress = sup...@siscog.pt
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP server/CN=*.siscog.com/emailAddress=sup...@siscog.pt
i:/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP server/CN=*.siscog.com/emailAddress=sup...@siscog.pt
---
Server certificate
-----BEGIN CERTIFICATE-----
[elided]
-----END CERTIFICATE-----
subject=/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP server/CN=*.siscog.com/emailAddress=sup...@siscog.pt
issuer=/C=PT/L=Lisboa/O=SISCOG.COM/OU=IMAP server/CN=*.siscog.com/emailAddress=sup...@siscog.pt
---
No client certificate CA names sent
---
SSL handshake has read 1240 bytes and written 518 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: 9334E31E9FD9A90C9D301227A2799EB8210AAC9FFD583B6597251744CB6EBFB3
Session-ID-ctx:
Master-Key: 7A5A40FEFC3A3493C9AFC4B80BCB044FAC48C94E0B4DE6E35035866F90EC78FFDE18D1B40E93B125F4F3F73468A9D5F0
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Compression: 1 (zlib compression)
Start Time: 1368189548
Timeout : 300 (sec)
Verify return code: 18 (self signed certificate)
---
* OK Dovecot ready.

In GNU Emacs 24.3.1 (i386-mingw-nt5.1.2600)
of 2013-03-17 on MARVIN
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
`configure --with-gcc (4.7) --cflags
-ID:/devel/emacs/libs/libXpm-3.5.8/include
-ID:/devel/emacs/libs/libXpm-3.5.8/src
-ID:/devel/emacs/libs/libpng-dev_1.4.3-1/include
-ID:/devel/emacs/libs/zlib-dev_1.2.5-2/include
-ID:/devel/emacs/libs/giflib-4.1.4-1/include
-ID:/devel/emacs/libs/jpeg-6b-4/include
-ID:/devel/emacs/libs/tiff-3.8.2-1/include
-ID:/devel/emacs/libs/gnutls-3.0.9/include
-ID:/devel/emacs/libs/libiconv-1.13.1-1-dev/include
-ID:/devel/emacs/libs/libxml2-2.7.8/include/libxml2'

Important settings:
value of $LANG: C.UTF-8
locale-coding-system: cp1252
default enable-multibyte-characters: t

[Eli Zaretskii]

> From: joaot...@gmail.com (João Távora)
> Date: Fri, 10 May 2013 13:49:20 +0100

>
>
> (list stream
> (network-stream-get-response stream start eoc)
> (network-stream-command stream capability-command eoc)
> 'tls)
>
> This is turn is because code immediately preceding it
>
> ;; If we're using tls.el, we have to delete the output from
> ;; openssl/gnutls-cli.
> (when (and (null use-builtin-gnutls)
> eoc)
> (network-stream-get-response stream start eoc)
> (goto-char (point-min))
> (when (re-search-forward eoc nil t)
> (goto-char (match-beginning 0))
> (delete-region (point-min) (line-beginning-position))))
>
> is not really doing what it says it's doing: it's not deleting enough
> input, and the search for capabilities actually returns the IMAP
> greeting.

Why don't you use the built-in gnutls? This is the recommended way of
using GnuTLS on Windows.

Or did I misunderstand your analysis?

[João Távora]

More or less, you didn't misunderstand my analysis, but I didn't use
built-in tls because its not the default (I did see some
references to it, and the function seems to be fboundp (whereas in linux
its not) but the code path didn't lead that way for some reason).

With emacs -Q, setting gnus-select-method that way and then doing M-x gnus
fails.

If the builtin tls fixes this and its the "recommended way" it should
be the default no?

Maybe it *is* the default but emacs -Q possibly configured itself to use
some external gnutls and configure itself after it. I'm using cygwin, so
that might be the case. In that case, this might be seen as a cygwin bug,
in which case I apologize for the noise.

btw: how to I configure emacs to use builtin gnutls?

J

--
João Távora

[Eli Zaretskii]

> From: João Távora <joaot...@gmail.com>
> Date: Fri, 10 May 2013 17:00:32 +0100
> Cc: 14...@debbugs.gnu.org

>
> If the builtin tls fixes this and its the "recommended way" it should
> be the default no?

It _is_ the default, AFAIK, but you need to install the GnuTLS
libraries (DLLs) for it to work. Try installing this one:

http://sourceforge.net/projects/ezwinports/files/gnutls-3.0.9-w32-bin.zip/download

> btw: how to I configure emacs to use builtin gnutls?

I think you just need to install the above (and restart Emacs, if it's
already running).

[Eli Zaretskii]

> From: João Távora <joaot...@gmail.com>
> Date: Fri, 10 May 2013 21:44:12 +0100
> Cc: 14...@debbugs.gnu.org
>
> OK, I see.
>
> Honestly I thought builtin tls was some kind of tls library in elisp.
> Needing external libraries doesn't make it very builtin :-), but
> that's just my opinion.

It's builtin in the sense that it does not need external programs.

> In any case, my description shows how it "almost worked" with whatever
> tls client it found (it was probably cygwin's, but I didn't check).
> You might want to avoid these bug reports by fixing it the way I
> suggested or some other way. Or not. All is OK from my part.

The native Windows build of Emacs doesn't work very well with an
external tls client, because their interaction is through signals,
which are not supported by Windows.

[Ted Zlatanov]

On Fri, 10 May 2013 21:44:12 +0100 João Távora <joaot...@gmail.com> wrote:

JT> Honestly I thought builtin tls was some kind of tls library in elisp.
JT> Needing external libraries doesn't make it very builtin :-), but
JT> that's just my opinion.

Implementing TLS in ELisp is possible, sure, but it would be extremely
slow and there's considerable risk in reimplementing that protocol. We
chose to use GnuTLS instead, which is a fast reliable C implementation
and is available for W32 and all our other platforms.

JT> In any case, my description shows how it "almost worked" with whatever
JT> tls client it found (it was probably cygwin's, but I didn't check).
JT> You might want to avoid these bug reports by fixing it the way I
JT> suggested or some other way. Or not. All is OK from my part.

I've seen dozens of bugs related to "almost working" external TLS
binaries on all platforms. It was one of the reasons I did the initial
GnuTLS integration with Emacs. My vote is to require GnuTLS with Emacs
and to only support it, but there are some questions there, mainly for
W32 and Mac OS X: do we auto-update GnuTLS? What happens when the
GnuTLS we install conflicts with another system install? And so on...

Ted

[Ted Zlatanov]

On Fri, 24 May 2013 23:27:07 +0300 Eli Zaretskii <el...@gnu.org> wrote:

>> From: Ted Zlatanov <t...@lifelogs.com>
>> Cc: 14...@debbugs.gnu.org, joaot...@gmail.com
>> Date: Fri, 24 May 2013 15:48:20 -0400
>>
EZ> What risk? what responsibility?
>>
>> The risk is that their version of GnuTLS is out of date.

EZ> That happens with dozens of packages on each user's machine. There's
EZ> nothing in GnuTLS that makes it unique in this regard.

Yes, of course. I don't know the other packages we require to enable
extra features on W32, sorry. I think GnuTLS is somewhat unique in this
regard by being the only way to do secure communications with the
outside world, but it's worth considering putting the other packages
under the same mechanism as GnuTLS for installations and updates.

EZ> Moreover, the latest and greatest GnuTLS sometimes simply won't build
EZ> on some systems. Like with the latest release, for example. How is
EZ> it a good idea to upgrade to a version that is broken? And if the
EZ> latest version is not always the one to upgrade to, then who will make
EZ> the research required to tell users to which version to upgrade?
EZ> You?

Yes, possibly.

EZ> I did that research for the single version whose Windows port I made
EZ> available. I built it, fixed the build problems, tested it, fixed the
EZ> problems revealed by that, and after doing all that I could in good
EZ> faith tell people they can use that version without too much fear.
EZ> Why is it safer for users to upgrade to a newer version than to stay
EZ> with the one I tested? Shouldn't whoever wants to tell them to
EZ> upgrade invest a similar effort in that newer version? If she
EZ> doesn't, she is actually shifting the responsibility to the users
EZ> anyway!

You're right. It's a lot of work. I appreciate it very much. I hope
to be able to find the resources to make these reviews happen.

>> Installing and keeping GnuTLS up to date should not be the
>> responsibility of the user.

EZ> Says you. But since there's no one else to pick up the gauntlet,
EZ> that's where this responsibility will need to rest. If J.R. Hacker
EZ> needs GnuTLS today, he has no one else but himself to rely on. All
EZ> we, the Emacs developers, do is just talk.

I like to ask before I make changes, hence my request for votes in
emacs-devel. Sorry if it seems like empty talk to you.

>> As far as I know GnuTLS status is back to "kosher."

EZ> Not sure based on what you say this.

Monitoring the GnuTLS mailing lists. I don't mean there are no issues,
only that the FSF has not made a statement about changing its preference
for GnuTLS.

>> I see it as a responsibility we're avoiding. But if we had these
>> regular builds, how would the user know about a critical update he
>> really must install?
>>
>> See here http://bugs.python.org/issue17425 for an example of how the
>> Python community dealt with an security issue in the OpenSSL libraries
>> they ship for Windows. I guess we have to answer the question of
>> whether that's a standard we as Emacs developers should aspire to, or
>> not.

EZ> I'm sorry, but you are expecting from the Emacs development something
EZ> it can never provide in its present shape and form. Tracking security
EZ> issues to this degree in even a single package is a very time
EZ> consuming job. Unless we have several volunteers on board taking
EZ> responsibility for the various packages which Emacs supports, what you
EZ> seem to want is nothing more than a pipe dream. I don't see any such
EZ> volunteers; in fact, I don't even see a single one. If we had such an
EZ> individual, my year-old port would have been replaced by newer ones
EZ> already. (Of course, the Windows build in GnuTLS is regularly broken,
EZ> so it's not really easy, either.) Until that changes, all this talk
EZ> is just a huge waste of energy.

EZ> If you think this kind of effort is possible, how about if you present
EZ> a complete realistic plan for having a secure Emacs, name individuals
EZ> who would test the releases of those packages for security issues, and
EZ> make sure any problems that are detected are promptly fixed on all
EZ> platforms we support, etc.? Otherwise, let's just stop these endless
EZ> discussions and admit that we don't have the resources to live up to
EZ> it.

I'm trying to get the work started by first and foremost deciding if
Emacs as a project wants to do this at all. This is a decision for the
maintainers and you've voted against it on emacs-devel, so let's see
what the vote count is and what the maintainers say.

If the maintainers are OK with this direction, I will start working on
automating the builds (which will need your help initially, if you're
willing, to replicate your build process), asking for volunteers, and
packaging the libraries. I don't have a 5-year plan but hope to get far
enough to make it something sustainable.

Ted

[Eli Zaretskii]

> From: Ted Zlatanov <t...@lifelogs.com>
> Date: Fri, 24 May 2013 18:20:55 -0400

>
> >> The risk is that their version of GnuTLS is out of date.
>
> EZ> That happens with dozens of packages on each user's machine. There's
> EZ> nothing in GnuTLS that makes it unique in this regard.
>
> Yes, of course. I don't know the other packages we require to enable
> extra features on W32, sorry.

Those are the image libraries and libxml2. And, of course, any
prerequisite packages needed to build those: zlib, libintl, libiconv,
and the libraries, like libnettle, that GnuTLS depends on.

> I think GnuTLS is somewhat unique in this regard by being the only
> way to do secure communications with the outside world, but it's
> worth considering putting the other packages under the same
> mechanism as GnuTLS for installations and updates.

Security is not just about secure communications. I'm sure you know
that.

> >> Installing and keeping GnuTLS up to date should not be the
> >> responsibility of the user.
>
> EZ> Says you. But since there's no one else to pick up the gauntlet,
> EZ> that's where this responsibility will need to rest. If J.R. Hacker
> EZ> needs GnuTLS today, he has no one else but himself to rely on. All
> EZ> we, the Emacs developers, do is just talk.
>
> I like to ask before I make changes, hence my request for votes in
> emacs-devel. Sorry if it seems like empty talk to you.

It is empty because it doesn't lead to any action. We've been talking
about this for months already.

> >> As far as I know GnuTLS status is back to "kosher."
>
> EZ> Not sure based on what you say this.
>
> Monitoring the GnuTLS mailing lists.

The actual discussions were private, not on any public list.

> I'm trying to get the work started by first and foremost deciding if
> Emacs as a project wants to do this at all. This is a decision for the
> maintainers and you've voted against it on emacs-devel, so let's see
> what the vote count is and what the maintainers say.

I think it's well beyond our capacities and resources, yes.