NSS - TLS Session Hash and Extended Master Secret doesn't work in FIPS mode with Java 1.8.0_161

[James Covington]

Not sure if this post belongs here or in mozilla.dev.tech.crypto, have posted to both:

I am using nss 3.23.0.0 with nspr 4.12.0.0 in FIPS mode to provide the cryptography for my Red Hat JBoss EAP 6.4.14 application server hosted on Windows 2008 R2 server. After updating my JDK version from 1.8.0_152 to 1.8.0_162, I received this exception during TLS handshake in my server.log:

RSA master secret generation error:
java.security.InvalidAlgorithmParameterException: Key format must be RAW
at com.sun.crypto.provider.TlsMasterSecretGenerator.engineInit(TlsMasterSecretGenerator.java:67)
at javax.crypto.KeyGenerator.init(KeyGenerator.java:454)
at javax.crypto.KeyGenerator.init(KeyGenerator.java:430)
at sun.security.ssl.Handshaker.calculateMasterSecret(Handshaker.java:1261)
at sun.security.ssl.Handshaker.calculateKeys(Handshaker.java:1183)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:301)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2306)
at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:201)
at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.java:1114)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
at java.lang.Thread.run(Thread.java:748)

In reviewing Oracle's release notes, I found that one of the new features effective in build 161 is the addition of TLS Session Hash and Extended Master Secret extension support in the jsse provider. By following the guidance in the release note, and adding -Djdk.tls.UseExtendedMasterSecret=false to my JAVA_OPTS variable, the exception is no longer raised.

I have read that NSS supports Extended Master Secret extension effective with version 3.21, so I'm not sure where to direct a bug report. Perhaps this extension is not supported in FIPS mode?

[James Covington]

Doing more reading on Extended Master Secret extension support in NSS, I understand that support is "off" by default. But I can't see in any documentation how to "turn it on". Any help out there?

[Martin Thomson]

SSL_OptionSet or SSL_OptionSetDefault with
SSL_ENABLE_EXTENDED_MASTER_SECRET and PR_TRUE. I'm not sure how that
manifests on the Java side of the house though.

On Tue, Feb 27, 2018 at 11:59 PM, James Covington
<james.h....@gmail.com> wrote:
> Doing more reading on Extended Master Secret extension support in NSS, I understand that support is "off" by default. But I can't see in any documentation how to "turn it on". Any help out there?
>
> On Monday, February 26, 2018 at 8:57:16 AM UTC-5, James Covington wrote:

> _______________________________________________
> dev-security mailing list
> dev-se...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security

[Martin Thomson]

Oh, I just noticed. From your stack trace, you aren't hitting NSS
libssl, that looks like the Java implementation of TLS.

On Wed, Feb 28, 2018 at 1:53 PM, Martin Thomson <m...@mozilla.com> wrote:
> SSL_OptionSet or SSL_OptionSetDefault with
> SSL_ENABLE_EXTENDED_MASTER_SECRET and PR_TRUE. I'm not sure how that
> manifests on the Java side of the house though.
>
> On Tue, Feb 27, 2018 at 11:59 PM, James Covington
> <james.h....@gmail.com> wrote:
>> Doing more reading on Extended Master Secret extension support in NSS, I understand that support is "off" by default. But I can't see in any documentation how to "turn it on". Any help out there?
>>
>> On Monday, February 26, 2018 at 8:57:16 AM UTC-5, James Covington wrote:

[James Covington]

Thanks for the reply, just to clarify the configuration, I'm using NSS as described in the JDK 8 PKCS#11 Reference Guide

https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html#NSS