Certificate unknown during connection

[Daniel Martínez Navarro]

Hi,

We're struggling with the setup of the new APNs Provider API to be able to send VoIP push notifications to our client app. Our current lab setup is a Centos 6.4 VM with a just a main class instantiating a single client (we wanted a simple PoC before industrializing the solution) but we're kind of stuck here. Compared to what it took us to make the same thing using Google's GCM this seems to be a complete PITA :)

The approach we're following id to use a statically linked netty-tcnative jar (we use the boringssl for linux x86-64 one) and it looks like the native library is properly loaded and ALPN support is found:

2016-05-09 20:55:22,009|INFO|main|com.relayrides.pushy.apns.ApnsClient|OpenSSL (via netty-tcnative) is available and supports ALPN; will use OpenSSL.

2016-05-09 20:55:22,088|DEBUG|main|io.netty.handler.ssl.OpenSslContext|Default cipher suite (OpenSSL): [ECDHE-RSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-SHA, ECDHE-RSA-AES256-SHA, AES128-GCM-SHA256, AES128-SHA, AES256-SHA, DES-CBC3-SHA]

But then, when we try to connect, the following exception is raised:

2016-05-09 20:55:24,689|WARN|nioEventLoopGroup-2-1|io.netty.handler.ssl.ApplicationProtocolNegotiationHandler|[id: 0x4bb2bd51, L:/10.0.2.15:50374 - R:api.development.push.apple.com/17.172.238.203:443] TLS handshake failed:

javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN

        at io.netty.handler.ssl.OpenSslEngine.shutdownWithError(OpenSslEngine.java:575)

        at io.netty.handler.ssl.OpenSslEngine.sslReadErrorResult(OpenSslEngine.java:778)

        at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:733)

        at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:810)

        at io.netty.handler.ssl.OpenSslEngine.unwrap(OpenSslEngine.java:853)

        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1098)

        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:970)

        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:904)

        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:387)

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:245)

        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)

        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:154)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:354)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:145)

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:1078)

        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)

        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:527)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:484)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:398)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:370)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)

        at java.lang.Thread.run(Thread.java:745)

The classpath for the sample test we try to run is the following:

lib/pushy-0.7.1.jar

lib/netty-all-4.1.0.CR7.jar

lib/netty-tcnative-1.1.33.Fork15.jar

lib/netty-tcnative-boringssl-static-1.1.33.Fork15-linux-x86_64.jar

lib/slf4j-api-1.7.21.jar

lib/slf4j-log4j12-1.7.21.jar

lib/log4j-1.2.17.jar

lib/gson-2.3.1.jar

lib/alpn-api-1.1.2.jar

I initially thought the problem had to do with a wrongly generated p12 file, but then I converted it to pem format and tried APNs Provider API using cURL 7.48 + OpenSSL 1.0.2f and we found no problems with that.

Do you know what could be the issue here?

Thanks in advance,

Daniel

[j...@turo.com]

It's possible that the problem is actually a failure on the client's part to trust the server. You may need to add the Entrust CA root certificate to your trust store; see https://developer.apple.com/library/ios/technotes/tn2265/_index.html#//apple_ref/doc/uid/DTS40010376-CH1-TNTAG31 for more information.

-Jon

[Daniel Martínez Navarro]

Hi,

Finally we managed to fix this problem by "sanitizing" the PKCS12 file exported from the Keychain, as we read someone else was doing to fix some other issue (not having exactly the same symtomps anyway):

openssl pkcs12 -in apns_certs_from_keychain.p12 -out apns_cert.pem -clcerts -nokeys

openssl pkcs12 -in apns_certs_from_keychain.p12 -out apns_key.pem -nocerts -nodes

openssl pkcs12 -export -inkey apns_key.pem -in apns_cert.pem -out apns_all.p12

This way we ended up with an apns_all.p12 file having the same certificates and keys as the initial one but this time working ok in our PoC :)

I hope this helps anybody else having the same issue

Thanks for your responses,

Daniel

[j...@turo.com]

That's really strange; thanks for digging in and sharing your solution!

Out of curiosity, what was in your original p12 file? Were there duplicate keys or "stray" certificates? It might be worth adding a check for common failure modes to Pushy.

Cheers!

-Jon

[Daniel Martínez Navarro]

Hi,

Well, apart from minor differences like the working .p12 not having friendlyName attributes for all certificates/keys the main difference is that in the original .p12 there were 4 keys as opposed to one in the sanitized one:

Original P12 structure

$ openssl pkcs12 -in apns_certs_from_keychain.p12 -info -noout

Enter Import Password:

MAC Iteration 1

MAC verified OK

PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048

Certificate bag

Certificate bag

Certificate bag

Certificate bag

PKCS7 Data

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Working p12 structure

$ openssl pkcs12 -in apns_all.p12 -noout -info

Enter Import Password:

MAC Iteration 2048

MAC verified OK

PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048

Certificate bag

Certificate bag

Certificate bag

Certificate bag

PKCS7 Data

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

Daniel

[j...@turo.com]

Huh… and were all four keys the same key? I'm surprised that Pushy didn't complain about having multiple keys. We may need to update some tests here.

Thanks very much for following up!

-Jon

[j...@turo.com]

…and to follow up, I've opened a bug for the "we probably should have complained about the multiple keys" issue at https://github.com/relayrides/pushy/issues/292.

Cheers!

-Jon

[Daniel Martínez Navarro]

No, they were different keys. The first one is the one that got copied to the working p12

[j...@turo.com]

Sorry to take so long to get around to this one, but I'm totally baffled as to what's going on here. I added a test for detecting multiple private keys, but it passes when testing a keystore deliberately crafted to contain multiple keys. Out of curiosity, I export a P12 that contains multiple keys from Keychain Access, and discovered that the test fails for that file. The issue seems to be that Java and OpenSSL (and presumably Keychain Access) disagree about the contents of the keystore.

Here's what OpenSSL sees:

$ openssl pkcs12 -in ~/Desktop/science.p12 -info -noout
MAC Iteration 1

MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Certificate

bag
PKCS7 Data

Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048

…and here's what keytool (and presumably everything else in Java) sees:

$ keytool -list -storetype PKCS12 -keystore ~/Desktop/science.p12

Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 1 entry
 
http/2 push test, May 26, 2016, PrivateKeyEntry,  
Certificate fingerprint (SHA1): [redacted]

So it seems like this might actually be a bug in Java. I'll dig a little bit and see if I can find other people with this issue and try to report it upstream if necessary.

Cheers!

-Jon

[j...@turo.com]

Ohhhhhhhhhh… I bet I know what's going on here.

> apart from minor differences like the working .p12 not having friendlyName attributes for all certificates/keys

I bet that difference is significant. Keystores in Java seem to be pretty heavily reliant upon "aliases" to identify the different entries. From Java's perspective, I bet the lack of friendlyName attributes makes everything look like it has the same alias, which might be messing with entry counting/enumeration. I'm not sure who to blame for that ;)

-Jon

[Mohamed Hafez]

I added Daniel's fix to the wiki at https://github.com/relayrides/pushy/wiki/Certificates since its the only thing that worked for me, hope that's ok with everyone!

[Jon Chambers]

Excellent. Thank you!

-Jon

On Fri, Jun 17, 2016 at 12:10 AM, Mohamed Hafez <mohamed....@gmail.com> wrote:

I added Daniel's fix to the wiki at https://github.com/relayrides/pushy/wiki/Certificates since its the only thing that worked for me, hope that's ok with everyone!

--
Pushy is an open-source Java library for sending APNs (iOS and OS X) push notifications. Pushy is brought to you by the engineers at RelayRides.
---
You received this message because you are subscribed to the Google Groups "pushy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pushy-apns+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vyachesl...@team.wrike.com]

Hi,

Just met similar issue. However your solution wasn't helping me. It turns out this failure caused by outdated keys. This, combined with the fact that keytool sees only one key leads me to conclusion that initial problem caused by selecting an outdated key in a keychain which contains both old and new keys. If in your particular scenario we can see multiple keys then we could use the only key which isn't outdated at this moment and so bypass this problem.