Unexpected disconnects

[Aleksandr Yeganov]

Hey folks! I have an async C++ GRPC service that uses a completion queue with two threads - one running the queue and the second that consumes the tags from the queue. I have been doing reliability testing and found that in about 6% cases the server ServerCompletionQueue.Next(tag, bool) returns false for the read tag while the client is still actively sending data. The server then closes the connection and goes on its merry way, while the client is not aware of the connection closure and eventually times out (non-grpc triggered timeout).

I tried setting the GRPC_ARG_KEEPALIVE_TIME_MS to 1000 ms and GRPC_ARG_KEEPALIVE_TIMEOUT_MS to 5000ms but it had no effect. I am honestly a little at a loss what else to check for. What would you recommend to look at or check to find the culprit?

[dfa...@google.com]

Is there a maximum connection age set on the server (GRPC_ARG_MAX_CONNECTION_AGE_MS)? If so, make sure you have an adequate grace period set (GRPC_ARG_MAX_CONNECTION_AGE_GRACE_MS).

Otherwise, enable logging on both the server and the client to determine why the connection is being closed (which side initiated, etc).