100K Requests per second with grpc-java using Netty

[nitish bhardwaj]

Hi,

I have a simple grpc service that returns "hello" to the client.  The server is running on 80 core machine. I am using ghz  tool to load test the service. It's able to serve 40k requests per second but the rest of the 20k requests are queued and all grpc threads are in waiting state. CPU is underutilized to 7% and memory is 7-10%.

More details are mentioned here:

https://stackoverflow.com/questions/66385871/grpc-server-in-java-to-support-huge-load-of-requests

What are the required settings to support more load of incoming requests?

Thanks,

Nitish

[sanjay...@google.com]

I have already replied on your StackOverflow thread as follows:

Nothing in the gRPC stack seems to cause this limit. What's the average response time on the server side? It looks like you are limited by the ephemeral ports or TCP connection limit and you may want to tweak your kernel as described at here https://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1 or here https://blog.box.com/ephemeral-port-exhaustion-and-web-services-at-scale