Handling Multiple Outstanding Asynchronous Requests Responsively in Client

[chh...@mesosphere.io]

Hi all,

I'm from the Apache Mesos community and I'm designing a module to support the Container Storage Interface which is based on gRPC. Our application needs a client to handle multiple outstanding asynchronous gRPC calls. My initial design is to create a special thread looping on `CompletionQueue::Next()`:

CompletionQueue cq;

thread looper {
  void* tag;
  bool ok = false;
  while (cq.Next(&tag, &ok)){
    if (ok) {
      ... // Ask a worker thread to handle the response.
    }
  }
}

so other threads could send asynchronous calls in a non-blocking way:

template <class S, class Q, class R> void call(shared_ptr<Channel> channel, const Q& request) {
  shared_ptr<R> response(new R);
  shared_ptr<ClientContext> context(new ClientContext);
  shared_ptr<Status> status(new Status);

  auto stub = S::NewStub(channel);
  shared_ptr<ClientAsyncResponseReader<R>> reader =
    std::move(stub->AsyncSomeCall(context.get, request, &cq);

  auto callback = new function<void()>([response, context, status, reader] {
    ... // Callback to handle the response.
  });

  reader->Finish(response.get(), status.get(), callback);
}

The above function template could be called from different threads, with different types of requests and responses, and the order of the asynchronous calls is not important. The goal of the above design is to have a responsive client that can handle an asynchronous response immediately when it is ready, as long as there's an available worker thread.

However, I noticed that in gRPC 1.3.2, the server does not receive asynchronous calls after `reader->Finish(...)` is called in the client process; the server only receive call after `cq.Next(...)` is called. Is it true that `cq.Next(...)` needs to happen after `reader->Finish(...)` in order to send out an asynchronous call?

If that's the case, then it seems to me that there's no way to implement the above design without introducing any delay. I can make the looper thread loop on `cq.AsyncNext(...)` with a certain delay, but then an asynchronous call committed right after a `cq.AsyncNext(...)` call won't be sent out to the server until the current `cq.AsyncNext(...)` times out. Is there a workaround to achieve my original goal stated above?

Thanks,

Chun-Hung

[chh...@mesosphere.io]

I did some more experiments related to this, and it seems the conclusion I got was not quite right.

`cq.Next(...)` needs not to happen after `reader->Finish(...)` to send out the asynchronous call.

However, either `cq.Next(...)` or `cq.AsyncNext(...)` needs to be called,

otherwise the sever won't receive any request even if `reader->Finish(...)` has been called at the client.

(I somehow disabled the looper thread in my experiments, and that's why I got the previous incorrect conclusion.

[ncte...@google.com]

This seems like expected behavior to me. If is work is queued up, then some version of Next must be called in order for it to take place.