Inquiry regarding chunking in gRPC

[Saurav Pawar]

Hello, hope everything is well with you.

I am currently using grpc for communication of LLMs (large language models) having 1 to 7 billion parameters. I know that there is a 2 GB serialization limit and that is why I have used chunking. So basically in my chunking, let's say I have a model having 100 layers, where each layer is less than 2 GB. I basically send `batch_size` amount of layers in a single go. For example, let's say I can send 5 layers in a single go so I will need 20 rounds to communicate the whole 100 layers. Also, this chunking I am doing is using python and not grpc.

But right now I have a model which has a single layer itself which has a size greater than 2 GB so in that case I am not sure how to proceed. 

Can anyone please give me some info on how can I leverage grpc chunking for this issue?

Kind regards,

Saurav

[Saurav Pawar]

Just a follow up.

[Kannan Jayaprakasam]

The 2 GB serialization limit comes protobuf itself so there is nothing we can do about it. You need to write logic to break your layers at possibly arbitrary places not just at layer boundaries, so that each chunk is <= 2GB and regather them on the receiver side with your service code logic.

[Larry Safran]

You could define your message as a byte array instead of the type you really want to send.  Then use ProtoUtils.marshaller(Type.getDefaultInstance()).stream(proto) to turn the proto you are sending into a byte stream which is easy to chunk.  Then on the other side, use ProtoUtils.marshaller(Type.getDefaultInstance()).parse(inputStream) to reassemble it.

Larry

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/grpc-io/1e1e2a34-5525-416f-a064-822801548cb8n%40googlegroups.com.