gRPC server executor pool slowing process execution

[qplc]

Hi,

I've implemented zookeeper balanced grpc server and client.

Following are the execution configuration details:

Grpc Client:

Channel Count: 1

Boss/Acceptor Thread: 1

Nio Threads: 100

Executor/App Threads: 100

Grpc Server:

Nio Threads: 100

Executor/App Threads: 400

Max conucurrent calls per connection: 100

Here, I'm using ForkJoinPool while setting executor. I'm sending messages on RabbitMQ and forwards to gRPC client. Publisher rate is 10k messages per 10sec.

As I've observed each request when reaches server it executes till 10seconds. And as configured most 400 tasks are getting executed concurrently at a time. Due to this piling up other 9600 requests as waiting for application threads to be available.

This is slowing overall process as piling up the requests and I couldn't delegate the tasks to other thread pool as executor has already been dedicated for the same.

Also, I've given 16gb for application. Increasing the thread count could help here as I have already given sufficient configuration.

How do I improve the execution faster?

Thanks,

qplc

[Kun Zhang]

The number of threads, especially the nio threads, look way too many. Too many threads only add overhead.

You may try limiting your number of threads to 2 times the number of CPU cores.

If you are doing blocking operations in your application thread, try switching to a non-blocking asynchronous model which will scale better.

[qplc]

Thanks Kun for your reply.

As you mentioned, I have implemented non streaming async client and server with limited threads configured.

Here are the details:
Machine Cores: 8
Channel Count: 1
Grpc Client NIO Threads: (Cores-2) * 2 = 12
Grpc Client App Threads: 30
Grpc Server NIO Threads: (Cores-2) * 2 = 12
Grpc Server App Threads: 30
Grpc Server Max. Concurrent Calls Per Connection = Grpc Server NIO Threads i.e. 12

RabbitMQ Max. Pool Size = 150
Grpc Client Threadpool size for Submitting RPC Call = 200
Grpc Server Threadpool size to execute business functionality = 200 (Added this threadpool to free Grpc Server App Threads)

I have best tried to finetune grpc client and server but still I have seen a slowness while submitting tasks for business functionality execution.

I have seen many messages piled up in the rabbitmq queue. This shows threads are busy submitting rpc tasks.

Am I wrongly configuring grpc server/client here?

[Robert Engels]

What is the cpu utilization and load factor during the slowdown ? If a single publisher, and the rate is 10k per 10 sec then every task must complete in 1 ms or they will back up, unless you allow out of order handling. 

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/e619cf3a-fc77-486e-ad78-72f9186da003%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[qplc]

Single publisher is sending 1L messages and being distributed over 2mins.

Observed:
load average: 1.24, 0.48, 0.17
%CPU: 600
%MEM: 65

On Thursday, October 25, 2018 at 3:49:07 PM UTC+5:30, qplc wrote:

[Robert Engels]

1L? Still, you might want to look at /proc/schedstat 600% utilization is pretty high, especially since there may be only 4 real cores. 

Also, what is the user cpu % vs system ? You might just be thrashing scheduling all of those threads on 8 cores. 

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/7bc25124-5456-4fcd-9ad3-aff2917337dd%40googlegroups.com.