What to do with a failing TCP client connection?

[grpc_client]

Language: C++

gRPC version: 1.2.5

Role: gRPC client

A quick question here. I'm observing a situation where the client attempts to connect to a server, repeatedly printing out the following error (SOME_IP:PORT edited out)

[gRPC] watch_completion_error: {"created":"@1537198702.954601066","description":"Connect Failed","file":"src/core/ext/client_channel/subchannel.c","file_line":700,"grpc_status":14,"referenced_errors":[{"created":"@1537198702.954515638","description":"Failed to connect to remote host: FD Shutdown","file":"src/core/lib/iomgr/ev_epoll_linux.c","file_line":1208,"os_error":"Timeout occurred","referenced_errors":[{"created":"@1537198702.954497141","description":"connect() timed out","file":"src/core/lib/iomgr/tcp_client_posix.c","file_line":148}],"target_address":"ipv4:SOME_IP:PORT"}]}

This is all fine during some init time when the server might still be unreachable. However, this continues even when the server does become reachable and a new grpc connection succeeds, but the initial one keeps retrying. I can also see that the same FD is being used in all connect attempts.

To summarize:

(a) there is some networking condition that prevents a successful initial tcp connection

(b) when the condition in (a) is resolved, the initial connection still cannot recover and continues to fail (from the logs I can see that it uses the same FD every time for the socket)

(c) a new, 'fresh' connection is successful when (a) is a resolved, but the initial connection is still failing

My question is: is there a way to 'force' gRCP not to reuse the same socket for subsequent 'connect' attempts?

Any advice on how to mitigate the scenario above will be highly appreciated. My current thinking is to destroy and create a new channel if it fails repeatedly, but I was hoping for something more elegant.

Thanks for being helpful as always!

[Muxi Yan]

A new connection will be created if (and only if) the channel arguments are different; otherwise the same connection will be reused.

[grpc_client]

Thanks for the insight!