[Shib-Dev] OpenSSL 0.9.8j and Tomcat

[Scott Cantor]

Since life's been boring lately, my favorite open source projects have
decided to stir things up. Took me a while to track this down, but I believe
there's an incompatibility between OpenSSL 0.9.8j clients (e.g. libcurl,
i.e. the SP) and Tomcat TLS/SSL servers (most likely anything using the Sun
Java SSL implementation).

The trigger for this is the disabling of SSLv2 or the explicit selection of
SSLv3 or TLSv1 in the client, and it results in various handshaking errors
of different types depending on the option used.

Curl as of version 7.18.1 disables SSLv2 by default if you don't choose a
specific protocol, so as a result, no matter what the SP code does, one of
those triggers is in effect with any recent version.

The actual bug appears to show up only with OpenSSL 0.9.8j. The last SP
release on Windows used either g or h, and nothing else is using j yet for
the most part, since it's very recent. With a cygwin build of 0.9.8i, I
don't get the error.

I've reproduced the behavior with both my own build and a Cygwin version of
0.9.8j. Haven't tried a Unix version yet. I don't need curl to show it,
openssl s_client alone reveals the error. My Tomcat's the latest 6.0, but I
think any version's probably affected.

This leaves me in a little bit of a quandary for the next release, and the
usual response when you raise these sorts of issues is deafening silence,
but I guess I'll try. Independent confirmation of this would be helpful,
particularly with different Tomcat/Java versions.

-- Scott

[Tom Scavo]

On Wed, Feb 4, 2009 at 7:08 PM, Scott Cantor <cant...@osu.edu> wrote:
>
> This leaves me in a little bit of a quandary for the next release, and the
> usual response when you raise these sorts of issues is deafening silence,
> but I guess I'll try. Independent confirmation of this would be helpful,
> particularly with different Tomcat/Java versions.

Can you provide a stacktrace for comparison? We've seen some
unexplained SSL/TLS errors recently, and since we no longer ship a
particular version of openssl with our product, your description
sounds like a possible explanation of the problem we're seeing.

Thanks,
Tom

[Kaspar Brand]

> Since life's been boring lately, my favorite open source projects have
> decided to stir things up. Took me a while to track this down, but I believe
> there's an incompatibility between OpenSSL 0.9.8j clients (e.g. libcurl,
> i.e. the SP) and Tomcat TLS/SSL servers (most likely anything using the Sun
> Java SSL implementation).

Looking at the changelog, one candidate for causing handshake troubles
could be

> Changes between 0.9.8i and 0.9.8j [07 Jan 2009]
[...]
> *) Enable TLS extensions by default.
> [Ben Laurie]

If you add "disable-tlsext" to the OpenSSL config command and recompile,
does the error disappear? With the s_client commannd (from a "stock"
0.9.8j version), you could also try "-no_ticket" and see if that helps
(will disable one of the TLS extensions, which might be the culprit here).

Kaspar

[André Cruz]

On Feb 5, 2009, at 1:08 , Scott Cantor wrote:

> Since life's been boring lately, my favorite open source projects have
> decided to stir things up. Took me a while to track this down, but I
> believe
> there's an incompatibility between OpenSSL 0.9.8j clients (e.g.
> libcurl,
> i.e. the SP) and Tomcat TLS/SSL servers (most likely anything using
> the Sun
> Java SSL implementation).

I don't have any system with 0.9.8j yet. I'll have to find time to
install it and try. Which version of the JDK are you using?

> I've reproduced the behavior with both my own build and a Cygwin
> version of
> 0.9.8j. Haven't tried a Unix version yet. I don't need curl to show
> it,
> openssl s_client alone reveals the error. My Tomcat's the latest
> 6.0, but I
> think any version's probably affected.

For the record, what happens?

Also, you mention using tomcat 6.0. I was under the impression that
the IDP 1.3 required jdk 5 and tomcat <= 5.5. Does it work well?

André

[Scott Cantor]

Kaspar Brand wrote on 2009-02-05:
> If you add "disable-tlsext" to the OpenSSL config command and recompile,
> does the error disappear?

That'll take me some time to try, but...

> With the s_client commannd (from a "stock"
> 0.9.8j version), you could also try "-no_ticket" and see if that helps
> (will disable one of the TLS extensions, which might be the culprit here).

Yep. That appears to fix it (just from s_client of course).

I don't know if there's any way to get libcurl to match that option's
behavior. I suspect the extension disabling will work around it as you
suggest, if/when rebuilding is an option.

Sounds like a probable bug in the Java server code though.

Thanks,
-- Scott

[Scott Cantor]

André Cruz wrote on 2009-02-05:
> I don't have any system with 0.9.8j yet. I'll have to find time to
> install it and try. Which version of the JDK are you using?

Latest 1.6 patch.

> For the record, what happens?

C> openssl.exe s_client -connect hostname:443 -no_ssl2
CONNECTED(00000003)
2272:error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert
unexpected message:s23_clnt.c:596:

C> openssl.exe s_client -connect hostname:443 -tls1
CONNECTED(00000003)
5624:error:140943F2:SSL routines:SSL3_READ_BYTES:sslv3 alert unexpected
message:s3_pkt.c:1060:SSL alert number 105624:error:1409E0E5:SSL
routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:530:

C> openssl.exe s_client -connect hostname:443 -ssl3
CONNECTED(00000003)
4316:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version
number:s3_pkt.c:284:

Omitting those options, or adding no_ticket allows it to handshake
successfully.

> Also, you mention using tomcat 6.0. I was under the impression that
> the IDP 1.3 required jdk 5 and tomcat <= 5.5. Does it work well?

I can't imagine any up to date advice of that nature. I doubt this is Tomcat
influenced, though it could be JDK influenced. Kind of doubt that too
though.

-- Scott

[Kaspar Brand]

>> With the s_client commannd (from a "stock"
>> 0.9.8j version), you could also try "-no_ticket" and see if that helps
>> (will disable one of the TLS extensions, which might be the culprit here).
>
> Yep. That appears to fix it (just from s_client of course).
>
> I don't know if there's any way to get libcurl to match that option's
> behavior.

Upgrade to HEAD, yes :-) See the (recent) commit for r1.215:

http://cool.haxx.se/cvs.cgi/curl/lib/ssluse.c

They now disable the ticket extension by default (through OpenSSL's
SSL_OP_NO_TICKET option).

Kaspar

[André Cruz]

On Feb 5, 2009, at 14:06 , Scott Cantor wrote:

> I can't imagine any up to date advice of that nature. I doubt this
> is Tomcat
> influenced, though it could be JDK influenced. Kind of doubt that too
> though.

JDK 1.5, Tomcat 5.5 - OK
JDK 1.6, Tomcat 5.5 - NOK
JDK 1.5, Tomcat 6 - OK
JDK 1.6, Tomcat 6 - NOK

I guess it's the JDK.

André

[André Cruz]

On Feb 5, 2009, at 14:06 , Scott Cantor wrote:

>> Also, you mention using tomcat 6.0. I was under the impression that
>> the IDP 1.3 required jdk 5 and tomcat <= 5.5. Does it work well?
>
> I can't imagine any up to date advice of that nature.

On Mon, 07 Apr 2008 17:10:26 +0100 Chad sent a message to shibboleth-
users list replying to a mail from me in which he specifically said
that IDP 1.3 would not work correctly with jdk 1.6 while the 2.0 IDP
did in fact work with 1.5, 1.6 and 1.7. That was from where I got that
information.

Is this outdated? Are people using the IDP 1.3 in production with JDK
6 and tomcat 6?

I made some tests and didn't have any problems but since I don't have
any problems with jdk 1.5 as well...

André

[Scott Cantor]

Kaspar Brand wrote on 2009-02-05:

> Upgrade to HEAD, yes :-) See the (recent) commit for r1.215:
>
> http://cool.haxx.se/cvs.cgi/curl/lib/ssluse.c
>
> They now disable the ticket extension by default (through OpenSSL's
> SSL_OP_NO_TICKET option).

Ok, that might change the situation a bit. This is why I figured I'd
ask...thanks for the pointer.

-- Scott

[Scott Cantor]

André Cruz wrote on 2009-02-05:

> Is this outdated? Are people using the IDP 1.3 in production with JDK
> 6 and tomcat 6?

I'm definitely using JDK 1.6. I'm not using Tomcat 6 at the moment with 1.3,
no.

-- Scott

[Scott Cantor]

Kaspar Brand wrote on 2009-02-05:

> They now disable the ticket extension by default (through OpenSSL's
> SSL_OP_NO_TICKET option).

Ok, that gave me the work-around I needed. I've manually overridden this
option in all cases, regardless of what libcurl does, so the versions of the
libraries shouldn't matter with the next version of the SP.

According to the threads I found, the bug also affects some non-Java
servers, including some older openssl versions (thus, Apache/mod_ssl).

-- Scott

[André Cruz]

FYI

On Feb 5, 2009, at 14:37 , André Cruz wrote:

> JDK 1.5, Tomcat 5.5 - OK
> JDK 1.6, Tomcat 5.5 - NOK
> JDK 1.5, Tomcat 6 - OK
> JDK 1.6, Tomcat 6 - NOK

I had tested with jdk 1.6.0u6. The latest 1.6.0u12 works fine.

André

[Kaspar Brand]

> I had tested with jdk 1.6.0u6. The latest 1.6.0u12 works fine.

Ok, then it's this bug here (from
http://java.sun.com/javase/6/webnotes/6u12.html):

> 6728126 jsse runtime Parsing Extensions in Client Hello message is done in a wrong way

The bug is not public, but OpenJDK is, fortunately enough ;-)

http://hg.openjdk.java.net/jdk7/tl/jdk/rev/16efbe49c725

(Previously JSSE was stumbling over an empty TLS ticket extension - but
the latter is perfectly legal according to RFC 4507/5077, when
negotiating a new TLS session.)

Kaspar