How to achieve server side throttling in GRPC?

[weib...@gmail.com]

Hi all,

I'm implementing a backend service using GRPC 0.9.0. Because the rps of our service may fluctuate, I'd like to add a throttling strategy to the server side. Any suggestion on doing this? Can I achieve this by ServerInterceptor or other mechanism provided by GRPC framework itself? Or I'm only able to implement this in my service implementation layer?

Thanks a lot. 

[Louis Ryan]

Which language is your backend in?

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/be8b87da-585d-4aaa-9ed8-ae51071c6b93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[weib...@gmail.com]

I'm using Java. My expectation is that when server side is on high load, it sends fail message to the client immediately so that clients don't need to waiting until timeout.

[Louis Ryan]

OK. ServerInterceptor is the appropriate way to do this. 

Out of curiosity how do you intend to measure load (max requests in flight, memory pressure, ...)

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/7996ed62-a1d5-4405-9990-a67e62f5baa5%40googlegroups.com.

[weib...@gmail.com]

I will simply decide whether to turn on throttling by measuring concurrent requests. Because the most time costing part of my service is to query DB. If DB's getting slow, many request will be blocked so that I have to drop further requests to prevent DB crashing. 

And could please explain more specific that how could I use ServerInterceptor to do this? IMHO, I can not just ignore the request, in case that clients will wait until request timeout, or causing client side Future object leak. I think it's better to send a error message(onError) to client immediately. How can I do this in ServerInterceptor?

[Eric Anderson]

On Wed, Oct 14, 2015 at 9:18 PM, <weib...@gmail.com> wrote:

And could please explain more specific that how could I use ServerInterceptor to do this? IMHO, I can not just ignore the request, in case that clients will wait until request timeout, or causing client side Future object leak.

Yes, you wouldn't want to ignore it.

I think it's better to send a error message(onError) to client immediately. How can I do this in ServerInterceptor?

Interceptors have full control over the call; they can do basically anything the application can do.

You can simply call serverCall.close without forwarding to the next handler.

ServerCall.Listener interceptCall(MethodDescriptor method,

    ServerCall call, Metadata headers,

    ServerCallHandler next) {

  if (tooManyConcurrentRequests) {

    call.close(Status.UNAVAILABLE, new Metadata());

    return;

  }

  // Note that really, you would likely want this atomic with

  // the check above.

  incrementRequestCounter();

  return new SimpleForwardingServerCallListener(

      next.startCall(method, call, headers)) {

    @Override

    public void onCancel() {

      try {

        // The application hasn't called call.close(), so it

        // may still be executing a DB query. Even after

        // onCancel returns, a query may be running, but doing

        // better requires application involvement.

        super.onCancel();

      } finally {

        decrementRequestCounter();

      }

    }

    @Override

    public void onComplete() {

      decrementRequestCounter();

      super.onComplete();
    }

  };

}

[weib...@gmail.com]

It looks the right way to solve my problem.

But interceptCall() method should return something. When the call is closed due to too many concurrent requests, what should the code return?

[Eric Anderson]

Oh, right.

In that case, you can just return a Listener that does nothing; the call will already have been closed, so you don't really care about any of the events.

return new ServerCall.Listener() {};

(Hmm... Maybe ServerCall.Listener should no longer be abstract, since all of its methods are non-abstract. That can easily be done later...)

--

You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/1ec41ddb-c12b-40ac-824a-271df183e03d%40googlegroups.com.