gRPC for large data transfer

[Philip]

I have the large size of request data to be submitted to remote server
API. (something like machine learning's feature data).
Will I use thrift or gRPC (protobuf behind it) to implement it for
better performance/security etc?

Thank you.

[Dean Hiller]

Having worked at Twitter(heavy thrift!!!) I would say it doesn't really matter as long as you deliver it in pieces typically.  I would not deliver one HUGE object (HUGE is relative) and instead might call N times to deliver the whole thing.  This has huge advantages in slamming it in many cases and when done right, servers can remain stateless.  My 2 cents is that 80% of your perf issue will be in 20% of the code and there is a good chance, it's not the protocol unless you have been optimizing your server for a while.  (Donald Knuth - Premature optimization is the root of all evil)

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/07228c47-81c2-57bb-f196-7fb4588e68ea%40list.199903.xyz.

[Srini Polavarapu]

FWIW, gRPC can support msg size up to 4GB (minus a few bytes) but you may be limited to 2GB or less due to protobuf library limitations. For example, python protobuf plugin default is 64MB which can be increased. See https://github.com/grpc/grpc/issues/19221.