Java GRPC async - io.grpc.StatusRuntimeException: CANCELLED

[maren.s...@gmail.com]

Hi all,

I've been experimenting with the following scenario.

Service A gets a synchronous GRPC call (handled by thread-A from the default grpc thread pool). Within that call, it calls Service B via an asynchronous call (stub). It provides a StreamObserver and expects a streamed response from Service B. The ManagedChannel to Service B has it's own executor (though the problem persists with default executor as well). 

Once, the thread-A calls Service B asynchronously, it returns and finishes.

Observation:

In the moment, when thread-A
finishes, the request to Service B gets cancelled. (Status: CANCELLED, see Stacktrace below).

io.grpc.StatusRuntimeException: CANCELLED
 at io.grpc.Status.asRuntimeException(Status.java:526) ~[grpc-core-1.5.0.jar:1.5.0]
 at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:380) [grpc-stub-1.5.0.jar:1.5.0]
 at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:419) [grpc-core-1.5.0.jar:1.5.0]
 at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:60) [grpc-core-1.5.0.jar:1.5.0]
 at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:493) [grpc-core-1.5.0.jar:1.5.0]
 at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$500(ClientCallImpl.java:422) [grpc-core-1.5.0.jar:1.5.0]
 at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:525) [grpc-core-1.5.0.jar:1.5.0]
 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.5.0.jar:1.5.0]
 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102) [grpc-core-1.5.0.jar:1.5.0]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_40]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_40]
 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]

If I debug and put a breakpoint before thread-A returns (breakpoint that only stops thread-A), the call to Service B is successful, the streamed responses get handled correctly by threads of the executor from B's channel etc. As soon as I let thread-A finish, I run again into the exception above. 

Can you think of an explanation for this?

Why can't the calling thread of an async thread not be finished? Shouldn't it be enough to pass the StreamObserver to the async call?

Looking forward to your help and happy to provide further details!

[maren.s...@gmail.com]

Maybe a bit more information on what happens, once I get the exception:

My StreamObserver.onError gets called, and the error gets handled correctly (in a thread from the channel's executor). 

[Carl Mastrangelo]

I'm not sure where that RST Stream is coming from but I sent out  https://github.com/grpc/grpc-java/pull/3642   to try and find the source of the error.

[Eric Anderson]

This sounds like auto-cancellation. When a RPC completes on the server, the server's io.grpc.Context is cancelled. That propagates to any outgoing RPCs. This is the same wiring that will cancel those RPCs when the initial RPC is cancelled and provides deadline propagation.

I think you want to run the RPC to Server B within a forked context. For example: Context.current().fork().run(yourStuff);

Carl: it looks like Contexts is the source of this description-less Status. I'll send out a PR.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/dc5e0c86-3722-4858-8a2a-1bf2fb250b70%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[maren.s...@gmail.com]

Thanks! 

That makes sense, I was not aware of that. Forking the context solves the issue. 

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[Eric Anderson]

On Thu, Nov 2, 2017 at 1:15 AM, <maren.s...@gmail.com> wrote:

That makes sense, I was not aware of that. Forking the context solves the issue.

Excellent!

Yeah, it is a useful feature, but it is there even if you aren't aware of it. Hopefully https://github.com/grpc/grpc-java/pull/365 will help others encountering the problem a little bit more.

We really need an example for it though, I think. I've added it to the "we need an example of this" dumping ground.