Disable application protocol negotiation by usage of nginx ssl termination

[jozef...@gmail.com]

Hi, 

I want to use nginx for ssl termination and load balancing. I found, there is one way how to do it, to use stream proxy configuration.

But by SSL GRPC request occurs exception "Failed ALPN negotiation" (bellow), where are checking supported ALPN/NPN protocols io.grpc.netty.GrpcSslContexts#NEXT_PROTOCOL_VERSIONS. It happens after successful SSL handshake and nginx returns by ALPN negotiation NULL value, reason why "Failed ALPN negotiation" is thrown. 

Therefore I've tried to disable ALPN negotiation by create of client GRPC SSL context using applicationProtocolConfig(ApplicationProtocolConfig.DISABLED).

In this case is thrown exception "ALPN must be enabled and list HTTP/2 as a supported protocol." through io.grpc.netty.GrpcSslContexts#ensureAlpnAndH2Enabled.

I there some way how to disable ALPN negotation by GRPC client SSL context?

Thanks.

Jozef R.

Failed ALPN negotiation exception:

io.grpc.StatusRuntimeException: UNKNOWN
 at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:227)
 at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:208)
 at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141)
 at io.grpc.examples.helloworld.GreeterGrpc$GreeterBlockingStub.sayHello(GreeterGrpc.java:218)
 at com.test.client.HelloWorldClient.greet(HelloWorldClient.java:80)
 at com.test.client.HelloWorldClient.cycleGreet(HelloWorldClient.java:117)
 at com.test.client.HelloWorldClient.main(HelloWorldClient.java:107)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Failed ALPN negotiation: Unable to find compatible protocol.
 at io.grpc.netty.ProtocolNegotiators$BufferUntilTlsNegotiatedHandler.userEventTriggered(ProtocolNegotiators.java:608)
 at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:330)
 at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:316)
 at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:308)
 at io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)
 at io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:336)
 at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:330)
 at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:316)
 at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:308)
 at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1329)
 at io.netty.handler.ssl.SslHandler.setHandshakeSuccessIfStillHandshaking(SslHandler.java:1314)
 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1188)
 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1039)
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
 ... 1 common frames omitted

ALPN must be enabled and list HTTP/2 as a supported protocol exception:

java.lang.IllegalArgumentException: ALPN must be enabled and list HTTP/2 as a supported protocol.
 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
 at io.grpc.netty.GrpcSslContexts.ensureAlpnAndH2Enabled(GrpcSslContexts.java:194)
 at io.grpc.netty.NettyChannelBuilder.sslContext(NettyChannelBuilder.java:193)
 at com.test.client.HelloWorldClient.createWithSsl(HelloWorldClient.java:209)
 at com.test.client.HelloWorldClient.createPublicWithSsl(HelloWorldClient.java:186)
 at com.test.client.HelloWorldClient.createChannel(HelloWorldClient.java:163)
 at com.test.client.HelloWorldClient.cycleGreet(HelloWorldClient.java:115)
 at com.test.client.HelloWorldClient.main(HelloWorldClient.java:108)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
 at java.lang.Thread.run(Thread.java:745)

[Carl Mastrangelo]

Hmm, Not sure that is supported.  The RFC says that the ALPN string needs to be present to avoid cross protocol attacks (RFC 7540 Section 10.2).  I *think* NPN identifiers still work, so you could use that if nginx supports sending.  That said, NPN support will be going away and it isn't a good idea to depend on it.

I suppose getting Nginx to send the "h2" alpn string is infeasible?

[Jozef R.]

Hi Carl, thanks for answer. I was googling / reading nginx docs and I didn't find way how getting Nginx to send the "h2" ALPN string.

I agree that NPN isn't a good idea to depend on it.

I found out that by nginx are two ways of SSL termination. The first is using SSL termination over HTTPS, here seems everything working properly also with support of HTTP/2, but problem is that nginx supports afterwards proxy pass only for HTTP/1.1 and for my GRPC bakends I need proxy pass of HTTP/2. The second is using SSL termination for TCP upstreams and this seems to be for me right way, but this SSL termination send by APLN negotiation NULL (nothing). Therefore I'm trying to disable ALPN negotiation on GRPC client or find out some other ways how to communication with nginx using SSL termination on it.

Dňa utorok, 6. júna 2017 20:01:43 UTC+2 Carl Mastrangelo napísal(-a):

[Mehrdad Afshari]

This is tangential to the discussion (and I don't have recent experience with Nginx and HTTP/2), but it seems like at least till a year ago or so, Nginx didn't support HTTP/2 trailer proxying that is used by gRPC. Has that situation improved on that dimension? If not, that might remain an issue even if the ALPN problem is solved... I also hear the proxy du jour to consider might be Envoy (https://github.com/lyft/envoy).