dealing with variable network latency and streams of big request messages

[Piotr Morgwai Kotarbinski]

Hello All,
I currently have a bi-di gRPC service dealing with request streams that often contain quite big messages (several MBs). Resources needed to process request messages are limited, so to avoid buffering of such big messages, the service uses manual flow-control and requests only as many messages as it can process at the given time without a need to wait for resources (if there's enough resources, messages from a single stream may be processed concurrently in separate threads).

Currently all clients of this service are in the same cluster, so network delays are negligible: request messages arrive almost instantly when requested, so the fraction of time when resources are idle is thus also negligible. However soon there will be client calls from different regions and later also calls from mobile clients. Am I assuming correctly, that the current flow-control strategy will result in much more wasted idle times of the processing resources? The service is implemented in Java in case it matters.

As I understand, if both the delay and the average processing time are stable, the service could be requesting additional `networkDelay / averageMessageProcessingTime` messages each time. This condition however may not always be the case, especially for mobile clients. Do I understand correctly, that in order to optimize resource utilization, I would need to implement something similar to dynamic TCP window size negotiation (but for messages, not bytes) or is there maybe some built-in, ready to use mechanism already in place to deal with this?
In case I need to deal with it myself, are there any APIs in the Java library that may be helpful? For example for measuring message network delivery delay of messages, number of request messages currently buffered by the server side that are awaiting processing, etc.

Thanks!

[sanjay...@google.com]

> Am I assuming correctly, that the current flow-control strategy will result in much more wasted idle times of the processing resources?

Yes

>  I would need to implement something similar to dynamic TCP window size negotiation (but for messages, not bytes) or is there maybe some built-in, ready to use mechanism already in place to deal with this?

I am not aware of any.

> are there any APIs in the Java library that may be helpful? For example for measuring message network delivery delay of messages, number of request messages currently buffered by the server side that are awaiting processing, etc.

Take a look at channelz and see if the server-data works for you https://grpc.io/blog/a-short-introduction-to-channelz/ . BTW, channelz can be accessed internally in your Java application (and not just through the web pages grpc-zpages mentioned in the article).