gRPC C++ Interface Switching

[Steve Ayers]

Hello,

We are using the gRPC C++ 1.24.2 library and we are running into an issue when switching routes/interfaces for comms.

For example, if our device is plugged in via fiber, the gRPC channel works fine.  However, if we unplug, our device moves to LTE but the gRPC channel doesn't seem to know this has happened and does not successfully reconnect.  The channel state still shows READY but all calls begin to time out and permanently block.  In addition, we are unable to cancel them as I don't see anything that works this way in the API.

Does gRPC permanently bind to some virtual interface and then never switch to say eth0 or something when the interface providing valid comms changes?

[mx...@google.com]

Hi!

One question I have on this: before you unplug the fiber, are both fiber and LTE interfaces connected and reachable to the server?

Could you collect some logs for us? Run the client with environment variables `GRPC_TRACE=all,-timer,-timer_check` and `GRPC_VERBOSITY=DEBUG` and share us the log captured during the problem.

If you could help us check whether you can reproduce this problem without unplugging things (e.g. turn down network interface instead) that would be very helpful to us. Right now we do not have an environment that allows us to reproduce this issue by unplugging cables or something like that.

Muxi

[Steve Ayers]

Thanks for responding.

At the time we unplug the fiber, both interfaces would appear as connected, yeah.  We have sort of a unique situation/architecture right now though.  We route traffic at the OS layer through the fiber interface (which are directly controlled by DHCP right now).  Once the DHCP lease on fiber expires, we know to start routing traffic via the LTE interface.  The problem is our gRPC channel doesn't seem to realize that any of this happened.  It's almost like the rug is pulled out from underneath it and it can't reconnect.  It continues to show a READY state this entire time, yet all calls just block indefinitely.

I'm wondering if it actually *is* trying to reconnect via the new interface and is just caught up in the late stages of exponential backoff, but I'm skeptical of that since the channel shows READY the entire time.

I'll collect some logs but wanted to get some information to you in case any of this is useful first.

[Muxi Yan]

I see what you mean. Thanks Steve.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/a281622f-5971-4ba8-8fcc-b0dc36fb5c2cn%40googlegroups.com.

[Steve Ayers]

Is there a less noisy GRPC_TRACE/VERBOSITY combination we could add to provide some logs?  We transmit data at decently high rates and it is generating a ton of logs that would be difficult to send plus we need to make sure we are not exposing any internal data.

[Muxi Yan]

A list of environment variables is available in this doc. If possible, I would suggest you try excluding items that you would not like to share with "-xxxx" in GRPC_TRACE as that gives us most information. For example "secure_endpoint" and "TCP" could be excluded; we don't need to see your data on the wire. 

If for any reason exclusion does not work well, I would suggest using "api,channel,client_channel_call,connectivity_state,handshaker,subchannel,subchannel_pool" for GRPC_TRACE as a first step. We'll go from there and see if we need other types of log.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/134fedee-252b-4191-b60f-55a4725d763dn%40googlegroups.com.