GRPC fails to connect to from Python client to C# echo server once a day

[Aleksey Telyshev]

Hello,

I have simple GRPC echo service on Win server 2012 and python script on Ubuntu Linux that get fired from cron every 1 min

Once in a day I get an exception 

, in run res = stub.Echo(req, _TIMEOUT_SECONDS) File "/usr/local/lib/python2.7/dist-packages/grpc/framework/crust/implementations.py", 

line 75, in __call__ protocol_options, metadata, request) File "/usr/local/lib/python2.7/dist-packages/grpc/framework/crust/_calls.py", 

line 109, in blocking_unary_unary return next(rendezvous) File "/usr/local/lib/python2.7/dist-packages/grpc/framework/crust/_control.py", 

line 415, in next raise self._termination.abortion_error NetworkError: NetworkError(code=StatusCode.UNAVAILABLE, details="") 

Next call works fine. What could be a reason for such behavior ? Should I just ignore it and implement reconnect logic ?

Thanks

Aleksey

[Nathaniel Manista]

On Mon, Oct 3, 2016 at 7:36 AM, Aleksey Telyshev <aleksey....@gmail.com> wrote:

I have simple GRPC echo service on Win server 2012 and python script on Ubuntu Linux that get fired from cron every 1 min

Once in a day

Be more specific about "once in a day"? Does it happen exactly every twenty-four hours, give or take a few seconds, or do you observe the issue once a day on average?

I get an exception 

, in run res = stub.Echo(req, _TIMEOUT_SECONDS) File "/usr/local/lib/python2.7/dist-packages/grpc/framework/crust/implementations.py", 

line 75, in __call__ protocol_options, metadata, request) File "/usr/local/lib/python2.7/dist-packages/grpc/framework/crust/_calls.py", 

line 109, in blocking_unary_unary return next(rendezvous) File "/usr/local/lib/python2.7/dist-packages/grpc/framework/crust/_control.py", 

line 415, in next raise self._termination.abortion_error NetworkError: NetworkError(code=StatusCode.UNAVAILABLE, details="")

This stack trace suggests that you are working with a version of gRPC Python <0.15 - if that's the case, what is the reason for it? If you are able to do so please upgrade to 1.0.0-or-later and look into whether the issue continues to recur?

Next call works fine. What could be a reason for such behavior ?

There are scores of possible reasons. Just out of curiosity, are you connecting over a secure or insecure channel?

Should I just ignore it and implement reconnect logic ?

Probably, for now - although do you mean "reconnect" as something different than "retry"?. We intend in the future to implement automatic retries within gRPC itself, but implementing retry logic yourself would be the way to have it available for your use today.

-N

[Aleksey Telyshev]

1. No, it is happening randomly but inside 12-32 hours interval. Something like "issue once a day on average"

2. I will discuss upgrade with the team.

3. I do connect over the local network, dont use any security.
4. Yes, I mean just retry. But we have small battle with my colleague, if it is network interface problem or GRPC implementation 

He wants to investigate possible network interface errors instead just implement retry. I personally dont see any problem with network, especially LAN and afraid it is something related with python implementation.

Regards

Aleksey

[Nathaniel Manista]

On Mon, Oct 3, 2016 at 11:33 AM, Aleksey Telyshev <aleksey....@gmail.com> wrote:

4. Yes, I mean just retry. But we have small battle with my colleague, if it is network interface problem or GRPC implementation 

He wants to investigate possible network interface errors instead just implement retry. I personally dont see any problem with network, especially LAN and afraid it is something related with python implementation.

We implemented some very dramatic stability improvements between 0.14 and 0.15. If you are able to reproduce the trouble that you are observing using gRPC Python 0.15 or later, we are very interested to hear about it in as much detail as possible. If the problem only appears in 0.14 and earlier, we apologize for the inconvenience.

-Nathaniel

[Aleksey Telyshev]

Sounds like a plan. Will update you once I upgrade gRPC and run it for couple days