grpc timout errors on heavy load

[Chirag Pujara]

We have a service that talks to grpc server. We started to see almost 30% requests has timeout errors with increased load. We tried to increase number of threads on grpc server or increase instance count for server as well but error rate is not going anywhere unless number of requests goes down.

NettyServerBuilder .forPort(grpcPort) .executor(Executors.newFixedThreadPool(threadPoolCnt)) .addService(myServiceImpl) .build()

even when we had incraesed thread count to 3 times of original value but most thread were stuck in waiting or time_wait state. Count for active/runnable state went up but count for waiting or time_wait saw much bigger jump. Also cpu usage is less than 2%.

is there any grpc config that could help hear that I am missing. new to grpc.

Thank you.