SSL connection fails on the server with SSL_ERROR_HANDSHAKE_FAILURE_ALERT
Honzab
We are building a firefox extension and we are using our own
implementation of SSL server and client implemented closely to the ssl
sample (security\nss\cmd\SSLsample). We are not using PSM socket
provider implementation.
All certificates (CAs and client/server) are RSA/SHA1 signed. We are
using a root CA + several intermediate CAs to sign user's certificates.
The user's certificates are used for SSL server and SSL client
authentication too.
All CAs are added to the software token and marked as trasted CA. The
user certificate is also added to the software token and marked as SSL
client and server cert, email signer, email recipient, SSL CA (users
sign other certs using its private key) and also is visible as Status
Responder Certificate.
The server is configured to request client authentication.
We are using SSL socket prototypes to setup client and server sockets.
Just a few cipher suits are selected to be used by clients and the same
set (+ few more) accepts the server socket.
This all is working very well with Firefox 1.5 (NSS 3.10). Now we have
moved to Firefox 2.0 and NSS 3.11 and experience the following problem:
Two peers tries to connect each other. On the server I read from the
socket with error SSL_ERROR_HANDSHAKE_FAILURE_ALERT. It is described on
http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html as
"SSL peer was unable to negotiate an acceptable set of security
parameters." I traced the code on the client and the negotiation is
probably broken in ssl3con.c line 4498 (the cipher suite was not
found).
What is strange, that the cipher suite sent from the server is c014 -
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA. This suite is disabled on the
server.
The client socket gets to state with error SSL_ERROR_NO_CYPHER_OVERLAP
and send to the server our SSL_ERROR_HANDSHAKE_FAILURE_ALERT alert. The
connection is then broken.
Other curious think is, that connection to and also from (!) Firefox
1.5 is established correctly.
Julien Pierre
> What is strange, that the cipher suite sent from the server is c014 -
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA. This suite is disabled on the
> server.
>
> The client socket gets to state with error SSL_ERROR_NO_CYPHER_OVERLAP
> and send to the server our SSL_ERROR_HANDSHAKE_FAILURE_ALERT alert. The
> connection is then broken.
NSS only supports RSA ECDHE cipher suites on the client side at this
time, so this is expected. If you are using NSS on the server side, you
need to enable alternate cipher suites - and of course you need to
enable them on the client side as well.
Honzab
Julien Pierre napsal:
>
> NSS only supports RSA ECDHE cipher suites on the client side at this
> time, so this is expected. If you are using NSS on the server side, you
> need to enable alternate cipher suites - and of course you need to
> enable them on the client side as well.
Thanks for advise, unfortunatelly this invokes another problem. I
enabled for client and sever another 4 suites:
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Server closes the connection with error number -12227
(SSL_ERROR_WRONG_CERTIFICATE) described as "Client authentication
failed: private key in key database does not correspond to public key
in certificate database." I am 100% sure the private key (RSA) is
corresponding to the public key provided in the cert. I am not sure in
this point, but it seems that NSS needs the certificates by signed by
ECC. Maybe I am wrong.
And yet another question: why do you restrict usage to just the ECC
cryptography? Means this to stop using classic DH and RSA?
Julien Pierre
Honzab wrote:
> Julien Pierre napsal:
>
>>NSS only supports RSA ECDHE cipher suites on the client side at this
>>time, so this is expected. If you are using NSS on the server side, you
>>need to enable alternate cipher suites - and of course you need to
>>enable them on the client side as well.
>
>
> Thanks for advise, unfortunatelly this invokes another problem. I
> enabled for client and sever another 4 suites:
>
> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
These cipher suites all require a certificate with an EC public key.
I believe for the first 2, the certificate must be signed by ECDSA, for
the last 2, by RSA.
> And yet another question: why do you restrict usage to just the ECC
> cryptography? Means this to stop using classic DH and RSA?
I'm sorry, I made a mistake earlier. All the EC cipher suites are
supported on both sides.
Only the DHE/RSA cipher suites are supported on the client-side only.
The list of client-side only cipher suites is :
SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
SSL_DHE_DSS_WITH_DES_CBC_SHA
SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_DHE_RSA_WITH_DES_CBC_SHA
SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
All other cipher suites are supported for both client and server sides.
Honzab
setting for ECC suites were inside of #ifdef NSS_ENABLE_ECC which was
not defined. We define this symbol now and disable all ECC suites for
all prototypes of socket we use (client and server too).
I used Wireshark to watch the traffic and found a very strange behavior
(the reason of the connection failure):
- ClientHello packet contains (among others) suite 0xC014
(TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA).
- ServerHello packet contains this suite as negotiated to be used for
the ssl session
- Client answers with fatal alert: Handshake Failure (40)
I did not investigate the reason deeply, but it might be potentialy a
bug in NSS 3.11 (?). Code in mozilla\security\nss\lib\ssl\ssl3con.c
line 4488 doesn't consider the suite as suitable for the session and
breaks the negotiation with fatal alert. This is strange, because the
client socket sent this suite in the list of suits as available for the
session.
Julien Pierre
Does this error also occur with NSS 3.11.3 ? Many ECC related bugs were
fixed after the original NSS 3.11 . If you aren't using the latest,
please upgrade.
If the problem still occurs, you might need to file a bug. You'll need
to provide all the information to reproduce the problem, ideally your
server cert and private key in a PKCS#12 file if you can, and
instructions on how to reproduce with the NSS tools tstclnt and selfserv.
Nelson B
>> I used Wireshark to watch the traffic and found a very strange behavior
>> (the reason of the connection failure):
Please use ssltap instead. Save all the output and any cert.NNN files that
it creates.
>> - ClientHello packet contains (among others) suite 0xC014
>> (TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA).
>> - ServerHello packet contains this suite as negotiated to be used for
>> the ssl session
>> - Client answers with fatal alert: Handshake Failure (40)
>>
>> I did not investigate the reason deeply, but it might be potentialy a
>> bug in NSS 3.11 (?). Code in mozilla\security\nss\lib\ssl\ssl3con.c
>> line 4488
Are you referring to the call to ssl3_config_match_init() in function
ssl3_HandleServerHello() ?
>> doesn't consider the suite as suitable for the session and
>> breaks the negotiation with fatal alert. This is strange, because the
>> client socket sent this suite in the list of suits as available for the
>> session.
Indeed. It makes me quite curious to see the ssltap output.
I really want to see the client hello message, and the server's full
response to it. Please use ssltap from NSS 3.11.3.
Invoke ssltap like this:
ssltap -sxlp <listen port> <server DNS name>:<server port>
e.g. if running on the system with the SSL server, this can be
ssltap -sxlp 444 localhost:443
then connect to port 444 on the server host, and ssltap on that host will
then connect to its localhost port 443.
Or if running ssltap on the SSL client system,
ssltap -sxlp 443 remotehost:443
then connect the SSL client to localhost:443 and ssltap will connect to
remotehost:443
Or run ssltap on a third machine (neither client nor server) and run it
as
ssltap -sxlp 443 serverhost:443
then connect to thirdmachine:443 and it will connect to serverhost:443
Julien Pierre wrote:
>
> Does this error also occur with NSS 3.11.3 ? Many ECC related bugs were
> fixed after the original NSS 3.11 . If you aren't using the latest,
> please upgrade.
Yes, the version of NSS in some of the FF 2.0 Betas had some issues.
If you're running one of those versions, do try it with the latest NSS,
or an older one.
> If the problem still occurs, you might need to file a bug. You'll need
> to provide all the information to reproduce the problem, ideally your
> server cert and private key in a PKCS#12 file if you can, and
> instructions on how to reproduce with the NSS tools tstclnt and selfserv.
We won't need your private key. I'd prefer the ssltap output, including
the output to stdout and stderr, and any cert.NNN files that ssltap creates.
--
Nelson B