grpc streaming exceeds message limit of 4MB

[guav...@gmail.com]

I'm using gRPC streaming between a golang client and a golang server. I'm seeing quite a few streaming error such as the following on the client side:

stream error:rpc error: code = ResourceExhausted desc = grpc: received message larger than max (14993559 vs. 4194304)

I understand that gRPC does have a message limit size of 4MB on the receiving end. My usage is that client and the server establishes a stream and the server will feed the client with stream of objects during throughout the lifetime of client. So the server side streaming is defined as:

func (s *server) FilterResult(req *filter.ResultStreamRequest, stream filter.FilterService_FilterResultServer) error {

    // stream the data forever

    for {

        // wait for more data

        result := waitData()

        // stream the result, 

        stream.Send(result)

    }

    return nil

}

The size of each result object is no more than 4k in length. But quite often I would get the above error on the client side. It seems to me that the server buffered too much results before it sends to the client. 

Thanks for any help.

[Yuxuan Li]

Hi Joe,

Buffering happens in lower level (transport layer) and shouldn't be able to modify the message size.

There are two places where the error you mentioned could happen, and both in rpc_util.go. The first place (in func recvMsg()) is before we apply decoding, where we compare the encoded message length to the limit. The second place (in func recv()) is after we decode the message, where we compare the decoded (decompressed if compression was enabled) message length to the limit. It could be possible that your encoded message is within 4MB (you said no more than 4k in length), but the decoded message is much larger and beyond the limit.

What you can probably do is:

on client side, verify where does the error get generated, before the decoding or after the decoding.

on server side, print out the message size before and after encoding, and see if it's larger than 4MB.

Please let me know if it doesn't help and we can investigate more. And in that case, it would be great if you can provide us with a reproducible example.Thanks!

Yuxuan

[Joe Lin]

Thanks. It was a user error. I do have large message > 4M. Setting MaxCallSendMsgSize does the trick for me.