Java gRPC: multiple bidirectional streams, onNext blocking problems

[Matt Mitchell]

Hello. I'm debugging bidirectional streaming application. A server thread obtains a client StreamObserver, sends a message and waits (countDownLatch), while a call back receives and processes client messages until it receives a "done" message. When "done", countdown() is called on the latch and the StreamObserver is made available to other server threads.

When a client receives a call, it blocks while also emitting to the serverStream, ending with the "done" message. That is when the onNext(serverMessage) call becomes unblocked. This blocking causes all other threads to go into a WAITING state. Even streams created from different channels/thread-pools have this problem. I'm guessing this is expected behavior, having to do with the event execution? How can this be avoided?

Thanks,

- Matt

[Carl Mastrangelo]

I'm having a hard time understanding your example, can you provide a sample snippet of your sever side StreamObserver?

[Matt Mitchell]

Hi Carl,

I meant to get back to you sooner, but while building up an example I figured out what the problem was. All of the gRPC onNext threads were performing the application work, and blocking until done. Some of this work involves reading from file systems, network calls etc..

During that work, the gRPC thread is then blocked, and if the blocking lasts for more than just a second or so, all other gRPC/onNext threads halt as well.

I do have other questions on the best way for application threads to "claim" a StreamObserver though (there is a "pool" of them). Most examples I've seen use synchronized, and maybe that'll be fine, but I could see something like a ring buffer / circular queue working well for this too?

- Matt 

[Carl Mastrangelo]

StreamObserver is not threadsafe, and they need external synchronization to be shared.   

Also, gRPC threads are provided by your application when you pass an Executor to the ManagedChannel or the Server.   You can control that blocking.

[matt.m...@lucidworks.com]

Thanks Carl. I have tried using various executors on the client channel and stub, but I still see that if an onNext thread blocks, all other client streams go into a waiting state. By all other client threads, I mean that when the client JVM starts up, it sends multiple bidi calls and those streams sit around, waiting to respond to server message, which usually results in the clients sending lots of message back. The only way I've been able to get the client app threads working really well (all threads busy) is to immediately push messages from the client's onNext into a queue + return asap. From there, a client app thread pool pulls from the queue and eventually begins sending messages back.

- Matt

[Carl Mastrangelo]

What executor are you using?  If you are using a fixed threadpoolexecutor, having multiple hanging calls could starve the other ones.   Do you notice the same blocking using ForkJoinPool or cached threadpoolexecutor?