gRPC not dropping disconnected channel

[manat...@gmail.com]

Steps to reproduce

1. Start server

2. Send a client RPC to server

3. Restart server

4. Using the same client, send another RPC. The call will fail

5. Send another RPC, this call will success

Also I found that if the server is leave stopped for a long time before starting up again, the call in step 5 will return "channel is in state TRANSIENT_FAILURE" as well.

Example code: https://github.com/whs/grpc-repro

(Install from requirements.txt then run main.py)

Expected result

All call should success.

Tested with Python grpcio==1.19.0 server/client and with go-grpc server. I tried setting grpc.max_connection_age_grace_ms, grpc.max_connection_age_ms, grpc.max_connection_idle_ms, grpc.keepalive_time_ms, grpc.keepalive_permit_without_calls but they doesn't seems to help.

From my own investigation, I believe that gRPC channels doesn't drop closed channels and tries to send calls in closed channel. There is also TCP RST sent.

[Sidhartha Thota]

This is known issue afaik. May be your issue is related to this one: https://github.com/grpc/grpc/issues/9767

If the connection is down the session will not be re-useful but channel is always useful. Here session is individual RPC/Streaming RPC call. So, when session is in progress and connection is down then session cannot be restored. Channel is different and maintains connection with other end (here server) and retries to connect periodically so it takes time to reconnect to to server.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/e8465f2b-96f1-4ad1-816f-ee6521595179%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Sidhartha Thota.

[Lidi Zheng]

gRPC Python server can be closed by "server.stop()".

Underlying, the gRPC Python server has a background polling thread. The deallocation of the server object is not directly terminating the polling thread since there is no way to interrupt thread in Python. So, during the server deallocation, it will set a destruction flag, and if the polling thread sees the flag it will terminate itself. It's possible that there are some lag before the server completely shuts down. To eliminate this issue, we recommend you close the server explicitly by "server.stop()" which will block until the server is completely shut down.

Lidi Zheng

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALguN-Gg0iiDSEUrO-VEOuGu5Mhemc9bZVqxqi_BcY7-pc%3DQWg%40mail.gmail.com.

[Manatsawin Hanmongkolchai]

Thank you for your answer Lidi and Sidhartha,

I've updated the code in GitHub repository to gracefully stop server.
Unfortunately, the same problem still occurs.
Previously I also have tested the same issue using Python client and a
Go server (stopping by manually ctrl+c the server) which also has the
same problem.

In proposal A6 section "Transparent Retries" mentions that in case of
RPC failure which RPC never leaves the client, it will be retried
automatically.
I've confirmed using Wireshark that the RPC never got sent out in the
wire, and there is only 2 outgoing calls (no retry).
I'm not sure whether the proposal is implemented or I have to set a
settings somewhere?

[Lidi Zheng]

AFAIK, the transparent retries is not fully implemented.

The feature you want probably is "wait_for_ready". In case of TRANSIENT_FAILURE (server not available temporarily), it will automatically wait for the channel become READY again without failing. Read more about wait for ready.

Lidi Zheng