Flow Control Timeouts

[Rod Prada]

Hi,

I am hoping to get some guidance about how to deal with flow control back-ups similar to the ones depicted in the Downstream connection backs-up and backpressure overview figure.

Based on our observations when one downstream client triggers the flow control mechanism (e.g. the client neglects to consume gRPC streams depleting the flow control window) other clients' streams on the the same shared upstream connection (i.e. connection pooling) also experience a pause in the flow of data. This pause can be indefinite if there are no timeouts configured and the bad client keeps the unconsumed streams opened.

Does Envoy provide flow control timeouts that would terminate any streams that triggered it. If not, what other settings / mechanisms could we use to prevent a bad client from affecting a shared upstream connection like this. We are aware that there are per-stream timeout settings, but unfortunately we have event-based gRPC services that expect streaming RPCs to be idle (i.e. no events) for long periods of time. 

Thanks in advance,

Rod

[Rod Prada]

The stream_idle_timeout setting doc mentions it also serves for terminate streams that don't open enough flow control window in a timely manner. However, in our case we have to set this to zero because, as I mentioned, some of our event streaming RPCs could validly not have any data to send for long periods of time. Would it be possible to implement a separate timeout setting solely for the flow control case instead?

Another solution that I haven't explored yet is using extensible filters to plug into the flow control mechanism of each stream to implement the timeout, so it'd be great if experienced folks here can comment of whether this is feasible and some guidance on how to achieve this.

Rod