grpc java mutual auth BAD certificate

[cr2...@gmail.com]

Hi
Seeing this error though it seems the all the certificates do check out ok.

grpc java version 1.10.0

javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE

Server side auth only works just fine.
This is is how the code looks for setting up the SSL connection:   Anything here that's missing ?

// Client code >>>
try {

                        SslProvider sslprovider = sslp.equals("openSSL") ? SslProvider.OPENSSL : SslProvider.JDK;
                        NegotiationType ntype = nt.equals("TLS") ? NegotiationType.TLS : NegotiationType.PLAINTEXT;

                        InputStream myInputStream = new ByteArrayInputStream(pemBytes); // <<<<<< pemBytes here is the CA ROOT cert.  This is all we use for server side auth only.
                        SslContext sslContext = GrpcSslContexts.forClient().trustManager(myInputStream)
                                .sslProvider(sslprovider).keyManager(clientKey, clientCert).build();   /// <<<<<<<<<<<<< This is how we set the client key (java) and the client x509 cert.
                        this.channelBuilder = NettyChannelBuilder.forAddress(addr, port).sslContext(sslContext)
                                .negotiationType(ntype);
                        if (cn != null) {
                            channelBuilder.overrideAuthority(cn);
                        }
                        addNettyBuilderProps(channelBuilder, properties);
                    } catch (SSLException sslex) {
                        throw new RuntimeException(sslex);
                    }


Generated on the fly so nothing lost in exposing this :)


-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIIhesX/tiUKgsNIs8eIR4+BZl3Bdx0VFEAv2DYLLt/4RoAoGCCqGSM49
AwEHoUQDQgAE62w7CxmQUPksTD/Sisw9fOjqlvfPkx/IH/nmMYwfF9BemhPNvzN6
BzGEHH8EFxCeMyPe0F3HoR9alzmjVQnsIw==
-----END EC PRIVATE KEY-----




-----BEGIN CERTIFICATE-----
MIICOTCCAd+gAwIBAgIUJ2ru30tMFlLgwI9yh5/oGG55gEYwCgYIKoZIzj0EAwIw
czELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNh
biBGcmFuY2lzY28xGTAXBgNVBAoTEG9yZzIuZXhhbXBsZS5jb20xHDAaBgNVBAMT
E2NhLm9yZzIuZXhhbXBsZS5jb20wHhcNMTgwMjI4MjIzNzAwWhcNMTkwMjI4MjI0
MjAwWjAhMQ8wDQYDVQQLEwZjbGllbnQxDjAMBgNVBAMTBWFkbWluMFkwEwYHKoZI
zj0CAQYIKoZIzj0DAQcDQgAE62w7CxmQUPksTD/Sisw9fOjqlvfPkx/IH/nmMYwf
F9BemhPNvzN6BzGEHH8EFxCeMyPe0F3HoR9alzmjVQnsI6OBojCBnzAOBgNVHQ8B
Af8EBAMCA6gwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB
/wQCMAAwHQYDVR0OBBYEFHrH0c2hLOfzHFcK0MMQOIBEvPz8MCsGA1UdIwQkMCKA
IHsrdlPEUS1s6VNeOBQGNfU5vYoTC+VKyU9+Ext1oPI+MBQGA1UdEQQNMAuCCWxv
Y2FsaG9zdDAKBggqhkjOPQQDAgNIADBFAiEA2ylucwIh6aanmPcntEN2+sRyp6nM
+Z08yH2u3NVTW1QCIHolG76Hqlu49K1WV5Rz4o84uyYZ8sI5pgXMNW4+1aX/
-----END CERTIFICATE-----



2018-02-28 22:44:13,006 grpc-default-executor-1 ERROR OrdererClient:157 - Received error on channel foo, orderer orderer.example.com, url grpcs://localhost:7050, UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:419)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41)
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41)
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:391)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:475)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:557)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:478)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:590)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(Thread.java:795)
Caused by: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:876)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1124)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1080)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1146)
    at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1189)
    at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1247)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
    ... 1 more
201

[Carl Mastrangelo]

Can you try with some of the default certs provided?   We use https://github.com/grpc/grpc-java/tree/master/testing/src/main/resources/certs  for our tests.

[cr2...@gmail.com]

Not tried that yet but I removed the keyManger setting and got the very same error. So I'm really now thinking it's nothing actually wrong with the client  keys/certs being set there.

[cr2...@gmail.com]

Does this just wrapper the local native oppenssl implementation ?

On Thursday, March 1, 2018 at 2:16:14 PM UTC-5, Carl Mastrangelo wrote:

[Carl Mastrangelo]

Yes, basically.  If you can use the openssl tool (like s_client) to use the certs, then we would know this is a problem on the java side.

[cr2...@gmail.com]

I captured our generated   cert and the private key in files .. tlsCertPem tlsKeyPem
Then did a `openssl s_client -connect localhost:7050 -cert tlsCertPem -key tlsKeyPem -debug`
I got what was a telnet like session .. typed a few lines hit enter : and ..

```

orderer.example.com       | 2018-03-02 00:24:45.019 UTC [grpc] Printf -> DEBU 0ba transport: http2Server.HandleStreams received bogus greeting from client: "hitherieh\njfsfjskfjs\njfk"

So I think this rules out invalid client certs / keys.

[cr2...@gmail.com]

This has been reproduced on Fedora 27 with IBM Jdk / Ubuntu OpenJdk  and Mac not sure of what JDK.

On Wednesday, February 28, 2018 at 5:47:41 PM UTC-5, cr2...@gmail.com wrote:

[cr2...@gmail.com]

It's possible to give multiple client certificates.  Is there some selection criteria when connecting to a specific server that determines which if any client certs are acceptable ?
Will it simpie try all ?  (not done much with mutual auth :)

 
On Wednesday, February 28, 2018 at 5:47:41 PM UTC-5, cr2...@gmail.com wrote:

[cr2...@gmail.com]

Hi I see the certificate in the builder. Have not found a reference to private key .. should I?
Some guidance where to look ? >>>
https://pasteboard.co/Ha2YMA2.jpg

On Wednesday, February 28, 2018 at 5:47:41 PM UTC-5, cr2...@gmail.com wrote:

[Carl Mastrangelo]

You don't happen to have the server requesting a client side cert do you?

Otherwise, I don't have much else to guess.  Personally I would fire up gdb and step through the authentication, but that may be a lot of effort.

[cr2...@gmail.com]

I think we made some progress.  We've got a case where it worked.  There were several issue at play making it difficult to nail down and leading to false theories..
It's a bit preliminary but one issue with a particular JDK not seeming to work at all, we think.  The other is the client certificates I think have to match server organization.
Which they did'nt and a bit of lacking of understanding on my part. However the error SSLV3_ALERT_BAD_CERTIFICATE is not very helpful .. like how about NO_MATCH_CLIENT_CERTIFICATE?

On Wednesday, February 28, 2018 at 5:47:41 PM UTC-5, cr2...@gmail.com wrote:

[Carl Mastrangelo]

One other thing you can try is using Jetty ALPN (See the SECURITY.md file in the grpc-java repo).  That would tell you if it is Netty's integration with OpenSSL that is broken.