Woes with the gRPC 1.0.0 4MB limit

[prot...@gmail.com]

I have a large message type with relatively complicated fields (nested repeated bytes etc.). The recent change of 4MB frame limit broke a lot of things and is making life rather difficult for us. We just happen to have a lot of messages in the 4-30MB ballpark. Note there's no way to increase the 4MB limit on the python server side (github.com/grpc/grpc/issues/7927).

Our current thinking is to take existing messages that could be > 4MB:

message Foo {

bytes file1 = 1; // chunkable

bytes file2 = 2; // chunkable

repeated bytes listOfFiles = 3; // chunkable

int64 someField = 4;

}

and add a union-like serialized bytes tag that is empty save for gRPC use:

message Foo {

bytes file1 = 1; // chunkable

bytes file2 = 2; // chunkable

repeated bytes listOfFiles = 3; // chunkable

int64 someField = 4;

        bytes serialized = 9999;

}

So we'd do something like:

# server side

f = someMessageFoo(file1=someData, file2=someData2);

g = Foo() # emptyFoo

g.serialized = f.SerializeToString() # all the other fields of g is empty.

server.Send(g) # use the streaming API calls with a chunk size of 4MB

Then on the client side we'd deserialize this similarly via original_f = ParseFromString(new_g.serialized) to get back our message.

Granted we'd need to stuff everything into ram along the way, but it's not a big issue for us. Granted there's a fair bit of boilerplate to do this (and that we'd need to add the rather unfortunate serialized field to all our messages). 

Is there a better way?

[Nathaniel Manista]

On Wed, Sep 21, 2016 at 1:35 PM, <prot...@gmail.com> wrote:

I have a large message type with relatively complicated fields (nested repeated bytes etc.). The recent change of 4MB frame limit broke a lot of things and is making life rather difficult for us. We just happen to have a lot of messages in the 4-30MB ballpark. Note there's no way to increase the 4MB limit on the python server side (github.com/grpc/grpc/issues/7927).

While that issue does refer to maximum message size it is about something completely different: that it should be illegal to specify a too-low maximum message size (like 2 bytes).

Our current thinking is to take existing messages that could be > 4MB:

message Foo {

bytes file1 = 1; // chunkable

bytes file2 = 2; // chunkable

repeated bytes listOfFiles = 3; // chunkable

int64 someField = 4;

}

and add a union-like serialized bytes tag that is empty save for gRPC use:

message Foo {

bytes file1 = 1; // chunkable

bytes file2 = 2; // chunkable

repeated bytes listOfFiles = 3; // chunkable

int64 someField = 4;

        bytes serialized = 9999;

}

So we'd do something like:

# server side

f = someMessageFoo(file1=someData, file2=someData2);

g = Foo() # emptyFoo

g.serialized = f.SerializeToString() # all the other fields of g is empty.

server.Send(g) # use the streaming API calls with a chunk size of 4MB

Then on the client side we'd deserialize this similarly via original_f = ParseFromString(new_g.serialized) to get back our message.

Granted we'd need to stuff everything into ram along the way, but it's not a big issue for us. Granted there's a fair bit of boilerplate to do this (and that we'd need to add the rather unfortunate serialized field to all our messages). 

Is there a better way?

As of a few days ago it is possible to set the maximum message size on a gRPC Python server, but our guidance is still to use messages of a few tens of kilobytes.

-Nathaniel