recovering from server sending too large a message in grpc-java

[Steven Parkes]

This is related to https://github.com/grpc/grpc-java/issues/2563. The tl;dr of https://github.com/grpc/grpc-java/issues/2563 is "don't do that" which we agree with in principle but don't have an efficient way to implement.

We've had cases where our gRPC server has sent messages greater than a client was prepared to handle. (FWIW, server is java, client is go and the call is streaming in both directions.)

The clients enforce a maximum receipt size but we don't have a way to cap the size sent by the server: the message is dependent on user code in a moderately complex way. Trying to send this message is an error case and we're prepared to deal with it, but we haven't found a graceful way to detect it.

What happens now is that the client fails the streaming call when it receives the too-large payload and we have to reestablish the call. This is pretty disruptive for us.

Given that there doesn't appear to be any method on the java side for us to cap the outbound message size, our only alternative appears to be manually serialize the protos before sending them and check the size. This is feels pretty expensive. What we'd prefer is to get an error on the sender side, at the point of send, when trying to send a message that exceeds the size. That way serialization is only done once and we can error-handle immediately, before anything goes out on the wire.

Thoughts?

[Steven Parkes]

Typo: don't have to serialize but do have to go through the size calculations. Isn't as bad but ...

[Eric Anderson]

On Thu, Sep 19, 2019 at 1:18 PM Steven Parkes <smpa...@smparkes.net> wrote:

This is related to https://github.com/grpc/grpc-java/issues/2563. The tl;dr of https://github.com/grpc/grpc-java/issues/2563 is "don't do that" which we agree with in principle but don't have an efficient way to implement.

There's three options: increase the limit on client-side, address it on server-side, or live with the failures. Commonly I think it is addressed on server-side with a heuristic. Most of the time this error is hit is because of a large bytes or lots of repeated fields, and those are easier to resolve than it sounds like in your case. For repeated fields, for example, you can just have "no more than 1000 entries", or something similarly appropriate for your particular service.

The clients enforce a maximum receipt size but we don't have a way to cap the size sent by the server: the message is dependent on user code in a moderately complex way. Trying to send this message is an error case and we're prepared to deal with it, but we haven't found a graceful way to detect it.

Detecting the error should be easy. I'm assuming you mean detect the problem before the failure.

Given that there doesn't appear to be any method on the java side for us to cap the outbound message size, our only alternative appears to be manually serialize the protos before sending them and check the size. This is feels pretty expensive. What we'd prefer is to get an error on the sender side, at the point of send, when trying to send a message that exceeds the size. That way serialization is only done once and we can error-handle immediately, before anything goes out on the wire.

As mentioned in your reply, you can ask the proto to do its size calculations. Those are cached in Java so it shouldn't impact performance much.

Any cap on outbound message size would also fail the RPC, so that would be a dead-end. The server doesn't actually know the client's limit. gRPC could potentially send the limit as metadata, but that would cause a small performance loss on all RPCs, even when this isn't a problem.