NSS 3.12.5.0: Error '-8152' (SEC_ERROR_INVALID_KEY) when connecting to ssl-enabled servers

[Bernhard Thalmayr]

Hi experts, an OpenAM community member is using OpenAM policy agent to
connect to an ssl-secured server.

The policy agent uses NSPR 4.8.2, NSS 3.12.5.0 optimized build for Linux
(RHEL) 64bit.

If the agent tries to open a connection to a specific, ssl-enabled
OpenAM server, error '-8152' is raised.

What might be the root-cause for this error?

Could I get some additional output from an optimized build or do I
really need a 'DEBUG' build to leverage NSS environment variables
(https://developer.mozilla.org/en/NSS_reference/NSS_environment_variables)?

Interestingly the same agent can connect to other ssl-enabled servers.

Unfortunately the community member will / can not provide a network
trace showing the handshake messages.

TIA,
Bernhard

--
Painstaking Minds
IT-Consulting Bernhard Thalmayr
Herxheimer Str. 5, 83620 Vagen (Munich area), Germany
Tel: +49 (0)8062 7769174
Mobile: +49 (0)176 55060699

bernhard...@painstakingminds.com - Solution Architect

This e-mail may contain confidential and/or privileged information.If
you are not the intended recipient (or have received this email in
error) please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

[Nelson B Bolyard]

On 2012/05/08 04:53 PDT, Bernhard Thalmayr wrote:
>
> Hi experts, an OpenAM community member is using OpenAM policy agent to
> connect to an ssl-secured server.
>
> The policy agent uses NSPR 4.8.2, NSS 3.12.5.0 optimized build for Linux
> (RHEL) 64bit.
>
> If the agent tries to open a connection to a specific, ssl-enabled
> OpenAM server, error '-8152' is raised.
>
> What might be the root-cause for this error?
>
> Could I get some additional output from an optimized build or do I
> really need a 'DEBUG' build to leverage NSS environment variables
> (https://developer.mozilla.org/en/NSS_reference/NSS_environment_variables)?
>
> Interestingly the same agent can connect to other ssl-enabled servers.
>
> Unfortunately the community member will / can not provide a network
> trace showing the handshake messages.
>
> TIA,
> Bernhard

Bernhard,
I think the most likely explanations are these:

1) Server certificate has a public key that is too small, too large, has a
too small public exponent (if RSA), an unknown key type, or a key for an
Elliptic Curve that is not supported by NSS.

2) Some other certificate in the server's cert chain has one of the above
problems.

3) The server is attempting to use "Server Key Exchange" for forward
secrecy, and the key it is offering for that purpose has one of the problems
mentioned above.

4) The server is selecting a cipher suite that is incompatible with the type
of key in its public key certificate.

Ii suggest you use tcpdump or ssltap to get a trace of your own.

Regards,
/Nelson

[Wan-Teh Chang]

On Tue, May 8, 2012 at 7:33 PM, Nelson B Bolyard <nel...@bolyard.me> wrote:
>
> Bernhard,
> I think the most likely explanations are these:
>
> 1) Server certificate has a public key that is too small, too large, has a
> too small public exponent (if RSA), an unknown key type, or a key for an
> Elliptic Curve that is not supported by NSS.
>
> 2) Some other certificate in the server's cert chain has one of the above
> problems.
>
> 3) The server is attempting to use "Server Key Exchange" for forward
> secrecy, and the key it is offering for that purpose has one of the problems
> mentioned above.
>
> 4) The server is selecting a cipher suite that is incompatible with the type
> of key in its public key certificate.

Nelson is right.

I looked into a check we added recently for 3). It was added in NSS 3.12.7:
https://bugzilla.mozilla.org/show_bug.cgi?id=554354

Since you're using NSS 3.12.5.0, that makes 3) less likely, but still possible.

> Ii suggest you use tcpdump or ssltap to get a trace of your own.

Yes. To track this down, we need the server's certificate chain and the
"Server Key Exchange" handshake message, if it is used.

Wan-Teh

[Bernhard Thalmayr]

Hi Wan-Teh, Nelson, could it be that this error is also raised by the
client if the client can not 'participate' in ssl client-auth?

Unfortunately I only got a text-output of 'ssldump', not sure if this is
would be helpful.

The end of the handshake shows ...

1a0: f3 6e fc 04 ab 79 e1 13 | .n...y..
0: 0d 00 2b 36 | ..+6
type = 13 (certificate_request)
length = 11062 (0x002b36)
CertificateRequest {
certificate types[3] = { 01 02 40 }
certificate_authorities[11056] = {

<<<<<....List Truncated....>>>>>

}
}
0: 0e 00 00 00 | ....
type = 14 (server_hello_done)
length = 0 (0x000000)
}
}
]
--> [
(7 bytes of 2)
SSLRecord { [Mon May 14 13:25:27 2012]
0: 15 03 00 00 02 | .....
type = 21 (alert)
version = { 3,0 }
length = 2 (0x2)
fatal: bad_certificate
0: 02 2a | .*
}


Interestingly enough the community member told be that this error does
not happen with

NSS: 3.12.8
NSPR: 4.8.6


TIA,
Bernhard

Am 5/9/12 7:43 PM, schrieb Wan-Teh Chang:

[Nelson B Bolyard]

On 2012/05/21 05:21 PDT, Bernhard Thalmayr wrote:
>
> Hi Wan-Teh, Nelson, could it be that this error is also raised by the
> client if the client can not 'participate' in ssl client-auth?
>
> Unfortunately I only got a text-output of 'ssldump', not sure if this is
> would be helpful.

[snip]

The client is telling the server that the server's certificate is bad.

> Interestingly enough the community member told be that this error does
> not happen with
>
> NSS: 3.12.8
> NSPR: 4.8.6

NSS is less forgiving of certain key size errors than it was before.

[Wan-Teh Chang]

On Mon, May 21, 2012 at 5:21 AM, Bernhard Thalmayr
<bernhard...@painstakingminds.com> wrote:
> Hi Wan-Teh, Nelson, could it be that this error is also raised by the client
> if the client can not 'participate' in ssl client-auth?

Yes, this is possible.

> Unfortunately I only got a text-output of 'ssldump', not sure if this is
> would be helpful.
>

> The end of the handshake shows ...
>
> 1a0: f3 6e fc 04  ab 79 e1 13                            | .n...y..
>   0: 0d 00 2b 36                                         | ..+6
>      type = 13 (certificate_request)
>      length = 11062 (0x002b36)
>         CertificateRequest {
>            certificate types[3] = { 01 02 40 }
>            certificate_authorities[11056] = {
>
>                <<<<<....List Truncated....>>>>>
>
>            }
>         }
>   0: 0e 00 00 00                                         | ....
>      type = 14 (server_hello_done)
>      length = 0 (0x000000)
>   }
> }
> ]

This shows a client certificate was requested.

> --> [
> (7 bytes of 2)
> SSLRecord { [Mon May 14 13:25:27 2012]
>   0: 15 03 00 00  02                                     | .....
>   type    = 21 (alert)
>   version = { 3,0 }
>   length  = 2 (0x2)
>   fatal: bad_certificate
>   0: 02 2a                                               | .*
> }

The -> arrow is from client to server. As Nelson said, most likely
the public key in the server's certificate is bad.

Wan-Teh