I'm trying to create a FIPS 140-2 compliant SSL connection using the Sun
JSSE (SunPKCS11) and NSS.  I set up the CA DB and Server DB as per the
instructions on this page:

http://www.mozilla.org/projects/security/pki/nss/ref/ssl/gtstd.html

In my code, I'm able to create a keystore that can pull my certificate (for
127.0.0.1) from the NSS db and display it's information. However, when I try
to connect a client socket and a server socket, I receive this exception:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:136)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:165 7)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:93 2)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocke tImpl.java:1096)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.jav a:1123)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.jav a:1107)
...

javax.net.ssl.SSLHandshakeException: bad handshake record MAC
    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1591)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1549)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:84 7)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocke tImpl.java:1096)
    at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.jav a:744)
    at
com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
    at java.io.InputStreamReader.read(InputStreamReader.java:167)
    at java.io.BufferedReader.fill(BufferedReader.java:136)
    at java.io.BufferedReader.readLine(BufferedReader.java:299)
    at java.io.LineNumberReader.readLine(LineNumberReader.java:182)
...

I'm not sure what "Bad Record MAC" means, or how to go about fixing it. I
thought that perhaps it was a hint that my certificate was corrupt, but
running

certutil -V -d server_db -u V -n 127.0.0.1

shows that the certificate is valid. Does anyone have any suggestions on
where to start looking? Is this a sign that I have NSS or certificates
configured incorrectly, or should I be talking with the Sun gurus about my
implementation?

Roy Donaldson wrote, On 2008-12-19 12:27 PST:

I suppose you mean a FIPS compliant TLS connection.  TLS is SSL version 3.1
(or newer).  FIPS 140 compliance requires the use of TLS.

Crypto questions are best asked in mozilla.dev.tech.crypto rather than
in mozilla.dev.security.  I have redirected followups to m.d.t.crypto.

It usually means that, at the end of the handshake, the client and the
server ended up with different keys, and the attempt to decrypt an encrypted
record received from the remote peer party produced a decrypted result that
is unintelligible.

No, there are different errors that occur when a cert is bad.
It might mean that your server is using the wrong private key,
not the one that corresponds to the public key in the cert.

Your message may get help from others in the mozilla.dev.tech.crypto
newsgroup (which is also the dev-tech-crypto AT lists.mozilla.org mailing
list).  Sun Java gurus who best know the internals of the class
com.sun.net.ssl.internal.ssl.SSLSocketImpl may be able to provide more help.

Are you sure that your client and server are actually using NSS's PKCS#11
module?  The exception stacks shown above provide no evidence that proves
or disproves that they do.  Still, a common problem is that the software
is not using the provider that the developer imagines it is using.