Server restart during client side streaming

[Paulin Todev]

Hi all,

During client side streaming, I'd like to know how the client could detect that the server process has been abruptly restarted (e.g. due to some critical failure), and how the client could recover from this.

Apparently for server side streaming we can use IsCancelled() and AsyncNotifyWhenDone() to detect such issues. But is there any similar mechanism on the client side?

This is what I have tested so far:

1. I started a synchronous client side streaming RPC. The channel is in a GRPC_CHANNEL_READY state
2. Killed the server process. After the server dies, the channel is briefly GRPC_CHANNEL_IDLE. Then the channel started alternating between GRPC_CHANNEL_CONNECTING and GRPC_CHANNEL_TRANSIENT_FAILURE
3. Then the server is started again. Тhe channel then goes in a GRPC_CHANNEL_READY state. However, the server no longer receives any of the streamed messages. I suppose at this stage the client should start a new RPC and abandon the old one? But how would the client know that it has to do this? Is there a reliable way for the client to know that it has to re-establish the connection?

I attempted to detect the issue using the retval of WaitForConnected() and Write() but both of these functions returned “true” while the server was down.

Also, I tried setting these configuration variables on both the server and client, but it didn't seem to help:

args.SetInt(GRPC_ARG_KEEPALIVE_TIME_MS, 10000);

args.SetInt(GRPC_ARG_KEEPALIVE_TIMEOUT_MS, 5000);

args.SetInt(GRPC_ARG_KEEPALIVE_PERMIT_WITHOUT_CALLS, 1);

args.SetInt(GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA, 0);

Regards,

Paulin

[yas...@google.com]

I can think of a few ways to achieve this (with keepalives configured ofcourse). One way to detect this would be to perform a `Read()` on the stream. If the channel dies, the stream would die too and the read would fail.

[borong]

is it solved?