Avoid gRPC message exceeds maximum size error

[Noam Isachar]

I'm using gRPC to send messages from Java to a Python service. Recently I started getting larger messages that sometimes exceed the maximum message size for gRPC. Increasing that size is not possible for me due to lack of resources and timeouts, so I had to implement a chunking mechanism in order to split the large messages.

In Java, I check the message size using getSerializedSize() and if it exceeds the maximum size, I do the chunking. However, I noticed that the value I'm getting from getSerializedSize() is different and much smaller than the one shown in the error "gRPC message exceeds maximum size" which means that I still send messages that exhaust the resources.

For example, how is it possible that getSerializedSize() returns 75274 and the error shows a size of 179651? Both are supposed to show the size in bytes. I also tried getting the size using the length of the byte array, but it is identical to getSerializedSize().

How can I know the actual size gRPC will get before sending the message so I can apply the chunking when I should?

[Larry Safran]

Hi Noam,

  If you set CallOptions.withMaxOutboundMessageSize, then your write will get a Status.RESOURCE_EXHAUSTED based upon the actual size being sent which will be present in the description.  You could catch that and then break your message apart when needed.

I'm not clear how your chunking is helping compared to the chunking that is done at the network level.  The example size you gave is much smaller than the 4MB default; have you tried using a larger max size?

Thanks,

  Larry

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/602108bd-15fd-421a-96ad-4f7c52bf4a2en%40googlegroups.com.

[Noam Isachar]

Thanks for the reply. My chunking helps because the large message contains a lot of timeseries for forecasting, and if they exceed they maximum size I can split them and send them in several messages instead of in a single one. 

Increasing the max size is not possible because my service crashes with larger messages, so the only way here is to send messages lower than the current maximum size. The problem is that when I check the message size before sending it I get a size that should be good, but then I get a Status.RESOURCE_EXHAUSTED with a different size than the one I checked with before.

[Noam Isachar]

Hi, is it possible that because of one of these properties I will experience the issues?
```
grpc.client.enable-keep-alive=true
grpc.client.keep-alive-without-calls=true
grpc.client.negotiation-type=plaintext
```