gRPC Client thread hangs on blocking APIs

[tuk]

I am using gRPC 1.55.1 and observing an issue similar to the one discussed in the below GitHub issue

• https://github.com/grpc/grpc-java/issues/9069
  

I have set the KeepAlive on both the client and server-side as suggested in the above issue 

I am creating the stub like below

var channel = ManagedChannelBuilder.forAddress(network.getIp(), network.getPort())
        .keepAliveTime(130, TimeUnit.SECONDS)
        .maxInboundMessageSize(maxInboundMessageSize)
        .maxInboundMetadataSize(maxInboundMetadataSize)
        .enableRetry()
        .build();

var stub = HelloServiceGrpc.newBlockingStub(channel).withDeadline(Deadline.after(115, TimeUnit.SECONDS));
stub.sayHello();
stub.sayHello();

In server side also keepAliveTime is set as suggested in the above github issue.

Grpc.newServerBuilderForPort(port, InsecureServerCredentials.create())
        .addService(new GreeterImpl())
        .keepAliveTime(130, TimeUnit.SECONDS)
        .build()
        .start();

In my case, client calls server 1 then server 1 acts as a client to server 2.

I am observing that when the deadline is exceeded in server2 the same deadline error is coming to server 1 like below which is also coming to client. It is working as expected.

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)


But on some rare occasions I am observing the client thread hanging like the below and not getting the deadline exceeded error

at jdk.internal.misc.Unsafe.park(java...@17.0.9/Native Method)
- parking to wait for  <0x0000000767a53a00> (a io.grpc.stub.ClientCalls$ThreadlessExecutor)
at java.util.concurrent.locks.LockSupport.park(java...@17.0.9/LockSupport.java:211)
at io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:748)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:157)

I waited for about 2 hours & it did not recover. The only way to recover from this is to restart the client application. It happened 3-4 times in the last couple of months for us.

Can someone let me know

1. What I am doing wrong or is there any known issue in the grpc-java 1.55.1 that I am using?
2. Is there any timeout config I can set on the gRPC client side so that the client threads do not hang indefinitely?

[Yuri Golobokov]

Maybe you are hitting this bug: https://github.com/grpc/grpc-java/issues/10838

It is fixed and backported to 1.59 and later versions.

[Larry Safran]

There was an issue with a test in Google's internal repository, so this fix was temporarily backed out and is now targeted for 1.63.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/d2700afd-762e-49ee-a15b-7f7901a6d10cn%40googlegroups.com.

[tuk]

Thanks for replying.

Just confirming for now the workaround is to disable the retry as suggested in the linked GitHub issue.

[Larry Safran]

yes, disabling the retry eliminates the problem.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/80d5231c-d509-4c20-824b-34d71ee97f18n%40googlegroups.com.