Switching from direct executor to threaded executor after rate limiting

[Krishna Sai Veera Reddy]

Hello All,

I am implementing a gRPC service and have a use-case where I need to rate limit RPC calls but would like the rate limiting server interceptor to run directly on the network thread as it is non-blocking(Using direct executor). But once the requests passes through the rate limiter I would like to use a thread pool to handle the request. Is there a standard way to do this?

The way forward seems to be to use direct executor while building the server, define a ServerCallExecutorSupplier, pass it as a callExecutor to gRPC server and have this executor supplier return null when the request hasn't passed through rate limiter so that it will use direct executor and return a threaded executor when it does pass through the rate limiter. To figure out if a request made it through the rate limiter or not I was thinking I could use the metadata headers for that but this seems overly complicated. Is there a better solution? Thank you!

[Eric Anderson]

On Wed, Nov 8, 2023 at 9:46 AM Krishna Sai Veera Reddy <krishnasaiv...@gmail.com> wrote:

I am implementing a gRPC service and have a use-case where I need to rate limit RPC calls but would like the rate limiting server interceptor to run directly on the network thread as it is non-blocking(Using direct executor).

Is there a problem with running it on another thread? Even for fully-async workloads we would discourage using direct executor as it can significantly impact latencies.

[Krishna Sai Veera Reddy]

Hey Eric,

>Is there a problem with running it on another thread?

This basic rate limiter just increments/decrements a counter and nothing more so it seems wasteful to use a thread to do this. For such cases how do we switch executors? Even if it isn't recommended I would love to know how this can be done theoretically. Thank you for your response!

[Eric Anderson]

On Tue, Nov 14, 2023 at 11:44 AM Krishna Sai Veera Reddy <krishnasaiv...@gmail.com> wrote:

>Is there a problem with running it on another thread?

This basic rate limiter just increments/decrements a counter and nothing more so it seems wasteful to use a thread to do this.

"wasteful." It isn't generally wasteful as the same thread will be used to process the request. It isn't "waste" as much as "what level of rejection rate does my server need to support" and whether doing the processing in another thread is able to satisfy that need. Doing the processing in another thread isn't very expensive, depending on your baseline for cost: it is a few atomic writes and some latency.

The request will be deserialized before calling the interceptor, which can be considerable CPU. To delay that until after your interceptor, use ServerInterceptors.useInputStreamMessages().

For such cases how do we switch executors? Even if it isn't recommended I would love to know how this can be done theoretically. Thank you for your response!

To just use an interceptor: To have the application code run in a different thread than the interceptor, you'd need to wrap all the ServerCall.Listener methods and schedule them on your own executor. You need to make sure the callbacks don't execute concurrently, so probably would use Guava's MoreExecutors.newSequentialExecutor(Executor).

To use ServerCallExecutorSupplier with an interceptor: it's a bit tricky. You could make a ServerCallExecutorSupplier that detects if the rate limit is already exceeded and returns a direct executor. Then the interceptor detects it is running in the direct executor and fails the RPC.

On Tuesday, November 14, 2023 at 10:24:53 AM UTC-8 Eric Anderson wrote:

On Wed, Nov 8, 2023 at 9:46 AM Krishna Sai Veera Reddy <krishnasaiv...@gmail.com> wrote:

I am implementing a gRPC service and have a use-case where I need to rate limit RPC calls but would like the rate limiting server interceptor to run directly on the network thread as it is non-blocking(Using direct executor).

Is there a problem with running it on another thread? Even for fully-async workloads we would discourage using direct executor as it can significantly impact latencies.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/8d96019c-584c-4497-bcb7-821ee48b6b39n%40googlegroups.com.

[Krishna Sai Veera Reddy]

Hey Eric,

Thank you and appreciate your response! It does seem it's not worth it to go through with switching executors this way as we do need to deserialize the request anyway to be able to apply rate limiting.