gRPC Java: What are caveats with using large message size (100 MiB)

[Anjaneya Srujan Narkedamalli]

Are there any caveats with using large message sizes of around 100 MiB in gRPC?

[Carl Mastrangelo]

It should be able to handle it.   You will need to modify the max message size on the Channel / Server to accommodate it, as the default is 4MB.  

If you are sending a lot at the same time, you may want to tune your heap size as well as direct memory (-Xmx, -Xms, and -XX:MaxDirectMemorySize).     gRPC prefers to use direct memory, which is normally unused by most applications.    It should be a fairly high value, but probably not more than young gen.

[Anjaneya Srujan Narkedamalli]

Thank you. 

On a similar note, I noticed few exceptions (indicated below) on server due to streams not being closed properly (I guess) when either server response or client request message size exceeds default limit. 

These exceptions are confusing as I initially thought there was some other problem apart from message exceeding max message size. I am using grpc-java 1.7.

Error on server when Message from Client Exceeds max message size:

Jan 24, 2018 1:41:37 PM io.grpc.netty.NettyServerHandler onStreamError

WARNING: Stream Error

io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 3

at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)

at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:535)

at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:187)

at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)

at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)

at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)

at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)

at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)

at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)

at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:383)

at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:443)

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

at java.lang.Thread.run(Thread.java:745)

Error on Server when message response from server exceeds max message size:

Jan 24, 2018 1:48:38 PM io.grpc.netty.NettyServerHandler onStreamError

WARNING: Stream Error

io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place

at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:149)

at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:499)

at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:480)

at io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105)

at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:349)

at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:985)

at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:941)

at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:497)

at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153)

at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:221)

at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:422)

at io.grpc.netty.NettyServerHandler.channelInactive(NettyServerHandler.java:478)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)

at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917)

at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

at java.lang.Thread.run(Thread.java:745)

[Carl Mastrangelo]

Hmm, those seem like the wrong exceptions to throw.   If you can reliably cause it, we can probably fix it.  I know for sure the top stack trace is wrong.  

Would you mind filing an issue?  https://github.com/grpc/grpc-java/issues

[Anjaneya Srujan Narkedamalli]

I am able to reproduce these errors consistently. Sure, I will file a issue. Just to be clear these are only server side exceptions. The client does show accurate exceptions.

Also in the case where server receives oversized message it does throw a correct exception in addition to previously mentioned exceptions. I will add all these details on github issue.

[Anjaneya Srujan Narkedamalli]

Filed this issue: https://github.com/grpc/grpc-java/issues/3996