gRPC Netty exception when using proxyless xDS based example with TLS

203 views
Skip to first unread message

Oleg Cohen

unread,
Feb 2, 2023, 10:22:02 PM2/2/23
to grpc.io

Greetings,

I have gotten the XDS example working with Java Control Plane. When I added TLS-based creds, the example works correctly, but I observe the following execption on the gRPC xDS-based client:

java.util.NoSuchElementException: SecurityProtocolNegotiators$ClientSdsHandler#0

Below is more info on the contex where the issue occured.

In the code in SecurityProtocolNegotiators class the exception occurs in this code segment:

@Override

public void updateSslContext(SslContext sslContext) {

logger.log(

Level.FINEST,

"ClientSdsHandler.updateSslContext authority={0}, ctx.name={1}",

new Object[]{grpcHandler.getAuthority(), ctx.name()});

ChannelHandler handler =

InternalProtocolNegotiators.tls(sslContext).newHandler(grpcHandler);

// Delegate rest of handshake to TLS handler

ctx.pipeline().addAfter(ctx.name(), null, handler);

fireProtocolNegotiationEvent(ctx);

ctx.pipeline().remove(bufferReads);

}

@Override

public void onException(Throwable throwable) {

ctx.fireExceptionCaught(throwable);

}

}

Would appreciate any info on what causes the exception and whether it needs to be handled/ignored!

Thanks,
Oleg

A20:08:52.692 [fileWatcher-0] DEBUG io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl - Supported protocols (OpenSSL): [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2, TLSv1.3]

20:08:52.692 [fileWatcher-0] DEBUG io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl - Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256]

[2023-02-02 20:08:52] [FINE ] Selecting OPENSSL

20:08:52.830 [fileWatcher-0] DEBUG io.grpc.netty.shaded.io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.grpc.netty.shaded.io.netty.util.ResourceLeakDetector@7cf88c53

[2023-02-02 20:08:52] [FINEST ] ClientSdsHandler.updateSslContext authority=be-srv, ctx.name=SecurityProtocolNegotiators$ClientSdsHandler#0

[2023-02-02 20:08:52] [FINEST ] ClientSdsHandler.updateSslContext authority=be-srv, ctx.name=SecurityProtocolNegotiators$ClientSdsHandler#0

[2023-02-02 20:08:52] [FINEST ] ClientSdsHandler.updateSslContext authority=be-srv, ctx.name=SecurityProtocolNegotiators$ClientSdsHandler#0

[2023-02-02 20:08:52] [FINEST ] [NettyClientTransport<23>: (be.cluster.local/127.0.0.1:50051)] WaitUntilActive started

[2023-02-02 20:08:52] [FINER ] [NettyClientTransport<23>: (be.cluster.local/127.0.0.1:50051)] ClientSds completed

20:08:52.838 [grpc-nio-worker-ELG-1-7] WARN io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

java.util.NoSuchElementException: SecurityProtocolNegotiators$ClientSdsHandler#0

   at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.getContextOrDie(DefaultChannelPipeline.java:1073)

   at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:302)

   at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:290)

   at io.grpc.xds.internal.security.SecurityProtocolNegotiators$ClientSdsHandler$2.updateSslContext(SecurityProtocolNegotiators.java:220)

   at io.grpc.xds.internal.security.SslContextProviderSupplier$1.updateSslContext(SslContextProviderSupplier.java:70)

   at io.grpc.xds.internal.security.SslContextProvider$1.run(SslContextProvider.java:123)

   at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)

   at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)

   at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)

   at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)

   at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

   at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

   at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

   at java.base/java.lang.Thread.run(Thread.java:833)

20:08:52.838 [grpc-nio-worker-ELG-1-8] WARN io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

java.util.NoSuchElementException: SecurityProtocolNegotiators$ClientSdsHandler#0

   at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.getContextOrDie(DefaultChannelPipeline.java:1073)

   at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:302)

   at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:290)

   at io.grpc.xds.internal.security.SecurityProtocolNegotiators$ClientSdsHandler$2.updateSslContext(SecurityProtocolNegotiators.java:220)

   at io.grpc.xds.internal.security.SslContextProviderSupplier$1.updateSslContext(SslContextProviderSupplier.java:70)

   at io.grpc.xds.internal.security.SslContextProvider$1.run(SslContextProvider.java:123)

   at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)

   at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)

   at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)

   at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)

   at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

   at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

   at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

   at java.base/java.lang.Thread.run(Thread.java:833)

[2023-02-02 20:08:52] [FINER ] [NettyClientTransport<23>: (be.cluster.local/127.0.0.1:50051)] WaitUntilActive completed

[2023-02-02 20:08:52] [FINEST ] [NettyClientTransport<23>: (be.cluster.local/127.0.0.1:50051)] ClientTls started

20:08:52.842 [grpc-nio-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.grpc.netty.shaded.io.netty.util.ResourceLeakDetector@758eb546

[2023-02-02 20:08:52] [FINE ] X509Certificate: Alg:SHA256withRSA, Serial:18c0ee8b17a8e338, ... Key type:RSA, Length:2048, Cert Id:381443688, Valid from:9/13/22, 5:00 PM, Valid until:9/13/25, 4:59 PM

[2023-02-02 20:08:52] [FINE ] ValidationChain: 2066922785, 381443688

20:08:52.873 [grpc-nio-worker-ELG-1-6] DEBUG io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler - [id: 0xed37e2a9, L:/127.0.0.1:56566 - R:be.cluster.local/127.0.0.1:50051] HANDSHAKEN: protocol:TLSv1.3 cipher suite:TLS_AES_128_GCM_SHA256

[2023-02-02 20:08:52] [FINER ] TLS negotiation succeeded.

SSLEngine Details: [

OpenSSL, Version: 0x1010107f (BoringSSL), ALPN supported: true

TLS Protocol: TLSv1.3

Application Protocol: h2

Need Client Auth: false

Want Client Auth: false

Supported protocols=[SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2, TLSv1.3]

Enabled protocols=[SSLv2Hello, TLSv1.2, TLSv1.3]

Supported ciphers=[ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES256-GCM-SHA384, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-CHACHA20-POLY1305, ECDHE-RSA-CHACHA20-POLY1305, ECDHE-PSK-CHACHA20-POLY1305, ECDHE-ECDSA-AES128-SHA, ECDHE-RSA-AES128-SHA, ECDHE-PSK-AES128-CBC-SHA, ECDHE-ECDSA-AES256-SHA, ECDHE-RSA-AES256-SHA, ECDHE-PSK-AES256-CBC-SHA, AES128-GCM-SHA256, AES256-GCM-SHA384, AES128-SHA, PSK-AES128-CBC-SHA, AES256-SHA, PSK-AES256-CBC-SHA, DES-CBC3-SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, AEAD-AES128-GCM-SHA256, AEAD-AES256-GCM-SHA384, AEAD-CHACHA20-POLY1305-SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, SSL_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, SSL_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, SSL_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256, SSL_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA, SSL_ECDHE_PSK_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA, SSL_ECDHE_PSK_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, SSL_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, SSL_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_AES_128_CBC_SHA, TLS_PSK_WITH_AES_128_CBC_SHA, SSL_PSK_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_AES_256_CBC_SHA, TLS_PSK_WITH_AES_256_CBC_SHA, SSL_PSK_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, null]

Enabled ciphers=[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256]

]

[2023-02-02 20:08:52] [FINER ] [NettyClientTransport<23>: (be.cluster.local/127.0.0.1:50051)] ClientTls completed

Message has been deleted
Message has been deleted

Sanjay Pujare

unread,
Mar 13, 2023, 11:36:01 PM3/13/23
to grpc.io
- would you be able to share your reproducible test-case?

- could you share more of the log? I would like to see why there are multiple calls to updateSslContext

I suspect this might be something specific about the Java control plane or how it is used but I am not sure.

sanjay...@google.com

unread,
Mar 20, 2023, 11:47:03 AM3/20/23
to grpc.io
Pinging again for a reproducible test-case...
Reply all
Reply to author
Forward
Message has been deleted
0 new messages