What is the maximum message size

Oregonduckman

unread,

Dec 13, 2018, 1:14:06 PM12/13/18

to grpc.io

I have read several posts about the maximum size gRPC will transport and posts range from 4M to 2G. I am interested to know what others are doing to transport large buffers on the order of 200M using gRPC.

Robert Engels

unread,

Dec 13, 2018, 1:41:58 PM12/13/18

to Oregonduckman, grpc.io

Pretty trivial to use the streaming protocol and send it in chunks to avoid large buffering.

On Dec 13, 2018, at 12:14 PM, Oregonduckman <joe.p...@decisionsciencescorp.com> wrote:

I have read several posts about the maximum size gRPC will transport and posts range from 4M to 2G. I am interested to know what others are doing to transport large buffers on the order of 200M using gRPC.

This email and its contents are confidential. If you are not the intended recipient, please do not disclose or use the information within this email or its attachments. If you have received this email in error, please report the error to the sender by return email and delete this communication from your records.
--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0cd893b7-fb28-4128-ab3e-e72b5f5d2232%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Carl Mastrangelo

unread,

Dec 17, 2018, 1:39:53 PM12/17/18

to grpc.io

The largest I've seen is about 40M, and it was due to extremely complex proto files, rather than large byte arrays. The large proto had to be updated transactionally, so it couldn't be split into smaller parts.

Oregonduckman

unread,

Dec 17, 2018, 3:57:27 PM12/17/18

to grpc.io

Thanks for the feedback. Was this an off the shelf version of gRPC or were any changes made to accommodate such large buffers.

zun...@humu.com

unread,

Dec 17, 2018, 5:05:15 PM12/17/18

to grpc.io

The 40M that Carl mentions makes my eyes water. You're going to have to tweak your network *so* hard to make that work.

In a case like that, I would strongly recommend doing what Robert Engels suggested -- use the streaming protocol and send it in chunks. If you need transactional atomicity, you just send a "OK, that's the end of the batch, commit this now" message as part of the stream, and essentially have your server accumulate the data until it's ready to go. This is going to be *way* more robust and reliable, and not require you to do anything too unspeakable to networks or the like.

(I don't remember if the gRPC API exposes raw messages, but if it does and you are for some reason receiving a 40M protobuf that you want to chunk up, the wire format of a serialized protobuf lends itself very naturally to splitting, either by replacing SerializeToString() with your own logic, or by chopping up the output. OTOH, if you're receiving a single 40M protobuf in a single lump and that isn't because one field is 40MB of raw bytes, you might want to rethink your data structures on that end too, to make managing it easier. I've had to work with cases where these things scaled much bigger -- a distributed storage system where multi-T files were not uncommon things to deal with -- and setting a chunk size of ~256kB was a very important thing for all sorts of reasons, not just for making gRPC happy)