Handshake Exception with Firefox and Jetty Servlet Container

[skleinei]

We do have a strange problem with Firefox 2 and Jetty 6.1 (a Servlet
Container) using HTTPS to communicate: At some stage Firefox (FF)
sends a ClientHelloV2, to which Jetty responds with a ServerHelloV3.
This leads to a handshake failure for this handshake and for any
subsequent attempt.

To avoid the handshake failures there are 2 options:
1.) restart FF
2.) change the servername or port, by using ssltap [1] as a proxy.
[FF] --> [ssltap] --> [Jetty]
That way it is possible to change the port without restarting
FF and Jetty, and to debug the SSL traffic.

Both options make FF to start a SSL session, by sending a
ClientHelloV3. Then everything works until we get to the same stage
described above.

Question ======

Why is Firefox a ClientHelloV2, although SSL v2 is disabled in Firefox
2?
Is there any work-around in Firefox, e.g. by making Firefox to start a
new SSL session everytime it accesses a server?
I am pretty stuck on solving our problem. Can you give me any advice,
how to further investigate this problem?

Logs =====

From what I can see in the logs of ssltap, Firefox is sending some
data, which might be something like a partial handshake (?).
--> [
alloclen = 63 bytes
(63 bytes of 63)
[Wed Jan 09 12:18:41 2008] [ssl2] ClientHelloV2 {
version = {0x03, 0x00}
cipher-specs-length = 36 (0x24)
sid-length = 0 (0x00)
challenge-length = 16 (0x10)
cipher-suites = {
(0x000039) TLS/DHE-RSA/AES256-CBC/SHA
(0x000038) TLS/DHE-DSS/AES256-CBC/SHA
(0x000035) TLS/RSA/AES256-CBC/SHA
(0x000033) TLS/DHE-RSA/AES128-CBC/SHA
(0x000032) TLS/DHE-DSS/AES128-CBC/SHA
(0x000004) SSL3/RSA/RC4-128/MD5
(0x000005) SSL3/RSA/RC4-128/SHA
(0x00002f) TLS/RSA/AES128-CBC/SHA
(0x000016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
(0x000013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
(0x00feff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
(0x00000a) SSL3/RSA/3DES192EDE-CBC/SHA
}
session-id = { }
challenge = { 0xa954 0x2122 0x3e82 0xb993 0xd72f 0xea54 0x779f
0x958c }
}
]

The server responds with:
<-- [
(1161 bytes of 1156)
SSLRecord { [Wed Jan 09 12:18:41 2008]
0: 16 03 00 04 84 |....ä
type = 22 (handshake)
version = { 3,0 }
length = 1156 (0x484)
handshake {
0: 02 00 00 46 |...F
type = 2 (server_hello)
length = 70 (0x000046)
ServerHello {
server_version = {3, 0}
random = {...}
0: 47 84 ad 91 a1 f6 cb e5 f8 e2 f0 46 60 4b dd 48 | Gä¡æí.╦.°.­F`K.H
10: 13 a8 93 96 d2 4f 2b d8 2d fe 49 2f 22 e5 29 5e | .¿ôûÊO+Ï-.I/".)^
session ID = {
length = 32
contents = {..}
0: 47 84 ad 91 84 b7 ef 62 92 fb 03 d7 8a 41 ae 82 | Gä¡æäÀ´bƹ.Î.A«.
10: d5 57 a3 e0 24 cc b5 2e b2 c7 29 3d 3a 37 a6 11 | ıWúÓ$╠Á.▓Ã)=:7ª.
}
cipher_suite = (0x0032) TLS/DHE-DSS/AES128-CBC/SHA
}
0: 0b 00 02 f6 |....
type = 11 (certificate)
length = 758 (0x0002f6)
CertificateChain {
chainlength = 755 (0x02f3)
Certificate {
size = 752 (0x02f0)
data = { saved in file 'cert.001' }
}
}
0: 0c 00 01 38 |...8
type = 12 (server_key_exchange)
length = 312 (0x000138)
0: 0e 00 00 00 |....
type = 14 (server_hello_done)
length = 0 (0x000000)
}
}
]

Then FF responds with a handshake failure:
--> [
(7 bytes of 2)
SSLRecord { [Wed Jan 09 12:18:41 2008]
0: 15 03 00 00 02 |.....
type = 21 (alert)
version = { 3,0 }
length = 2 (0x2)
fatal: handshake failure
0: 02 28 |.(
}
]

Thanks,
-Stefan

[1] http://www.mozilla.org/projects/security/pki/nss/tools/ssltap.html

[skleinei]

I did some more analysis:

When I replace the keystore by another keystore provided by the Jetty
distribution, Firefox will not complain about the handshake
exception. Nevertheless there are handshake exceptions, and the SSL
level switches from TLS to SSLv3.

Now I guess the question is: What is the difference between the two
keystores?
Answer: As far as I can see, the Keystore provided in the Jetty distro
has expired in 2001, while my keystore is still valid (although also
self-generated).

Could it be, that Firefox some is less strict regarding handshake
failures, after you have told it to accept an expired certificate?

[skleinei]

It turned out, that difference was the certificate algorithm. - DSA
seems to cause the problem, with RSA there is no problem. This result
can also be verified with Tomcat, so it is no Jetty-specific error.

I wonder, whether this may be related to these bugs:
https://bugzilla.mozilla.org/show_bug.cgi?id=398680
https://bugzilla.mozilla.org/show_bug.cgi?id=273861

Comments, anyone?

[Nelson Bolyard]

Robert Relyea wrote, On 2008-01-16 17:01:
>> Question ======
>>
>> Why is Firefox a ClientHelloV2, although SSL v2 is disabled in Firefox
>> 2?
>>

> This is a big question. Firefox2 has turned on a number of SSL3/TLS
> extensions which require and SSL3 hello. I suspect that for some reason
> you don't really have SSL2 turned off (and old profile?). We know that
> we are sending SSL3 hello's because we can connect to sites that require
> them.
>
> bob

>> Logs =====
>>
>> From what I can see in the logs of ssltap, Firefox is sending some
>> data, which might be something like a partial handshake (?).
>> --> [
>> alloclen = 63 bytes
>> (63 bytes of 63)
>> [Wed Jan 09 12:18:41 2008] [ssl2] ClientHelloV2 {
>> version = {0x03, 0x00}

Bob, An SSLv2 format client hello advertising SSL 3.0 is what we send
out to "TLS intolerant" servers.

Stefan, you've begun exploratory surgery on the patient, but haven't yet
recorded the patient's temperature and blood pressure (IOW, the basics).

What error does FF report when trying to connect to your server?

[skleinei]

On Jan 17, 3:57 am, Nelson Bolyard <NOnelsonS...@NObolyardSPAM.com>
wrote:

Hi Nelson,

I agree. Here are the basics:

First of all, I am using version 2.0.0.11. The following parameters
might be of interest:security.enable_ssl2=false,
security.enable_ssl3=true, security.enable_tls=true
The error I am getting after a few clicks or reloads is "Could not
establish an encrypted connection because certificate presented by
localhost has an invalid signature."

As I mentioned this happens with DSA certificates only. RSA seems not
to cause a problem. Also, I've only been able to test this on Java-
based servers (Jetty, Tomcat).

Originally we thought, that this was a Jetty bug. So, some of the
history can be found here:
http://www.nabble.com/jetty-ssl-problems-to13726768.html#a13816984
http://jira.codehaus.org/browse/JETTY-484

Please let me know, if there is additional information I can provide.

Thanks,
-Stefan

[Nelson Bolyard]

skleinei wrote, On 2008-01-17 09:44:
> [...] Here are the basics:

>
> First of all, I am using version 2.0.0.11. The following parameters
> might be of interest:security.enable_ssl2=false,
> security.enable_ssl3=true, security.enable_tls=true
> The error I am getting after a few clicks or reloads

After a few reloads?
Are you saying that it works for a while and then fails?
Are you able to connect to this site at all when it is using that
particular certificate?

> is "Could not
> establish an encrypted connection because certificate presented by
> localhost has an invalid signature."

OK, so there you have the root of the problem, signatures that cannot be
verified and therefore are declared invalid. The problem is either
with the signature in one of the certificates in the server's cert
chain, or with the signature in the server key exchange message.
It would be necessary to examine the entire server cert chain to
determine which of those is the case.

> As I mentioned this happens with DSA certificates only. RSA seems not
> to cause a problem.

I'd guess that your answer to my questions above will be that you are
not able to communicate with the https server at all while it is
configured to use the DSA certificate. Assuming that guess is right,
then the problem is likely that no certificate in the DSA certificate
chain contains the PQG parameters for the DSA public key.

There also also other possibilities. Complete diagnosis cannot be
made without the answers to the questions above and the complete
server certificate chain.

> Please let me know, if there is additional information I can provide.

Did you get this DSA certificate from a professionally run CA?
or did you make the cert yourself?

If you made the DSA cert yourself, then the problem is likely that the
certificate (key) is incomplete or incorrectly made. Try some other
approach, one that works for you. Explaining all the intricacies
of DSA certs is beyond the charter of this newsgroup. Sorry.

OTOH, if you can reproduce this with a DSA cert from a real CA, then
I'm willing to pursue this further.

/Nelson

[skleinei]

Hi Nelson,

thanks for the Info. We are indeed using a self-signed certificate.
I need to get our clients some decisions made, whether we pursue this
issue or not.

Thanks anyway for your help!

-Stefan