Changing the number of `event_engine` threads for C++ library

[Amirsaman Memaripour]

Hi folks,

I'm facing scalability issues when using the C++ CQ-based server with TLS. In particular, the `event_engine` thread is fully utilizing its core, making it the bottleneck and limiting server's throughput. I was wondering if there is an option to change the number of threads assigned to the `event_engine`?

[AJ Heller]

How did you identify that these threads are a bottleneck for your application? If your application is idle, the event_engine threads will be waiting for work (via condvar wait, which will not tie up your CPU). If on the other hand there's work to be done, those threads will wake up and do that work. The existence of idle/waiting threads should not cause any decrease in performance, and otherwise all active threads are doing their jobs.

If you are using the C++ callback API, then note that your callbacks may run on these `event_engine` threads, so any blocking behavior seen here may also be your own.

[Amirsaman Memaripour]

Thanks for the response AJ. I'm using the CQ API and changed the code to only run `ping` commands on threads polling on the completion queues. I still see lots of cycles consumed by `event_engine` threads (about 28% of total collected samples by `perf`) and wanted to see if that's expected:

I can get close to 240K QPS on 8 cores (AArch64), and 95% of the CPU time is spent running gRPC code (excluding serializing and deserializing of messages as I'm using a custom payload encoding). The public performance dashboard shows similar throughput for the C++ implementation, and I just want to make sure what I see in my implementation (both in terms of resource allocation and peak throughput) are expected.

[AJ Heller]

Yes, EventEngine threads execute a lot of gRPC, and eventually *most* of gRPC will run on EventEngine threads. The performance dashboard is a good basis for comparison.