gRPC netty NoRouteToHostException

[Bhuvi Viji]

My application calls a gRPC service(application B) using blocking stub to fulfill the request. End to End flow works fine without any issues.

If there are any deployments in the gRPC service (application B) which my application consumes we get into NoRouteToHostException. We are trying the request only after the successful deployment of Service B. I understand the channel is interrupted and it needs time to recreate the connections and re-establish. Is there any recommendation for get ridding of this kind of exception during the upstream servers deployments/new pod creation?

Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: *******.com/Ipaddr:443

[Eric Anderson]

You get that exception because all backends failed to be connected to. I expect this is a server deployment-approach problem. Does the service do a rolling restart?

The only thing the client can do to avoid the error is use stub.withWaitForReady(). But that implies you care more about reliability than latency. I'd consider using it for this purpose to be a hack.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/398ac163-8a7f-470b-8edb-df79014cd423n%40googlegroups.com.

[Bhuvi Viji]

Thanks Eric. Will check this option

[Bhuvi Viji]

Hi - This is issue is re-occurring even when there are no deployments on the upstream servers.

My gRPC service is a unary service , Batch application calls my gRPC service using blocking stub.

My gRPC service intern calls two different gRPC services to retrieve the data (upstream gRPC service 1) and do some calculation using the data retrieved((upstream gRPC service 2)

Since it runs in batch mode, earlier My gRPC service was not able to limit the request that it gets, and it went to unhealthy status so as per the suggestion #10164 from this ticket, Have implemented netflix concurrency limiter at my gRPC service so it allows the request based on the system capacity and for the rest of the other request it throws Server limit reached error.

Upstream services that my GRPC calls has also has limitation so it can handle a subset of request what i get mygRPC service can handle. To apply backoff on the client calls from my application have used same netflix client concurrency limiter. Even after having proper concurrency limiting , I'm getting the NoRouteToHostException. I understand this is something due to overloading the service , is that understanding right or what else are the possibility of getting into this exception?

Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: applicationALB/...:443
Caused by: java.net.NoRouteToHostException: No route to host
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)

[Bhuvi Viji]

Before I get this exception almost 4Million request processed without any issues and it's suddenly throwing error later after sometime it works fine.

You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/umB8apUGJ3I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/ac3b290d-3745-41af-909b-b6494ceda94cn%40googlegroups.com.

[Eric Anderson]

It is unlikely this is related to overloading the backends, unless some monitoring system thinks the backends are broken and reboots/replaces them. This is a network error, where packets can't be sent from your client to the server. For example, if someone unplugged a network cable, this is the sort of error you'd get. It could also mean internet/WAN connectivity has been lost.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAKJZ7ySRrKUAO-D3HnMC%2BzHLoi5FzbEh%2B%3D4kPqRNW-O3C279fQ%40mail.gmail.com.