[Question] BlockingResponseStream.hasNext blocks forever

[ran...@addepar.com]

I run into an issue that `BlockingResponseStream.hasNext` occasionally blocks forever while sending response for a streaming RPC. 

GRPC server is still able to serve requests after client hangs. No exception is found in server log, and server does call "responseObserver.onCompleted()" after each GRPC.

It only happens to me when streaming millions of messages.

I haven't found any reliable way to reproduce this issue, but following code and comment in `ClientCalls.java` caught my attention:

try {
  // Will block here indefinitely waiting for content. RPC timeouts defend against permanent
  // hangs here as the call will become closed.
  last = waitForNext();
} catch (InterruptedException ie) {
  Thread.currentThread().interrupt();
  throw Status.CANCELLED.withCause(ie).asRuntimeException();
}

I am not vary familiar with GRPC codebase. Could someone tell me when is this call going to block and which thread usually unblocks it? What could cause the call being blocked forever?

It seems GRPC does not timeout by default. Should I specify RPC timeout? 

What's the best practice for streaming millions of messages?

[Carl Mastrangelo]

If the connection is alive and the server doesn't send anything, the call will just wait.  What does your server code look like?

[ran...@addepar.com]

I don't think that is the problem in my case.

My server is basically like

for all data

  stream.onNext(data)

stream.onComplete()

I don't see any chance to cause the problem you described.

[Eric Anderson]

On Tue, Jul 19, 2016 at 12:20 PM, ran.bi via grpc.io <grp...@googlegroups.com> wrote:

I don't think that is the problem in my case.

My server is basically like

for all data

  stream.onNext(data)

stream.onComplete()

I don't see any chance to cause the problem you described.

Hmm... How much data are we talking here (total)? What version of grpc-java are you using?

[ran...@addepar.com]

1.2 millions protos, and the total serialized data is a few GB I think.

The client thread is blocked here

    - waiting on <0x7552c6db> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7552c6db> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:499)
    at io.grpc.stub.ClientCalls$BlockingResponseStream.waitForNext(ClientCalls.java:421)
    at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:434)

[ran...@addepar.com]

Anyone has any idea about the potential cause?

[ran...@addepar.com]

I just noticed that the problem only happens when I make multiple large server->client streaming RPC simultaneously. 

I haven't been able to reproduce the issue in a single thread.

Could it be a race condition on the client side?

[Eric Anderson]

It looks like isCancelled() is a bit useless right now, since it's value changes too late. Thanks for bringing this to our attention. Sorry it took so long for me to look at it.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/ec7d7406-18da-424e-afb8-4fd776a62377%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eric Anderson]

Oops. Sorry. Wrong thread. That response is actually for this issue instead.

For your actual issue, I'm still wondering what version of grpc-java you're using. Also, are you using any proxy between the client and server?

As a shot-in-the-dark, you may try updating to 3.0.0-pre1, but I'm not aware of anything specifically that I would expect to fix your issue, unless you are using a proxy that adds in padding (rare).