GRPC Server Threading

[lukem...@gmail.com]

What is the GRPC server threading model?  For example, if I write a GRPC C++ server, will GRPC automatically spawn multiple threads (or use an eventing model) to handle multiple simultaneous client requests?  Are there any configuration parameters I can modify that will impact the number of supported simultaneous client connections?

Also, are there any differences in the GRPC server threading model used in a C++ server vs a Csharp server?

Luke

[Craig Tiller]

Hey,

Great question!

The C++ server has two threading models available: sync and async. Most users will want to use the sync model: the server will have an (internal) threadpool that manages multiplexing requests onto some number of threads (reusing threads between requests). The async model allows you to bring your own threading model, but is a little trickier to use - in that mode you request new calls when your server is ready for them, and block in completion queues while there is no work to do. By arranging when you block on the completion queues, and on which completion queues you make requests, you can arrange a wide variety of threading models.

We're working on exposing some knobs to control the behavior of the synchronous thread pool. I expect to start seeing those changes hit the codebase in the next 4-6 weeks, depending on where our priorities land.

The C# server is similar to the synchronous mode of C++, mixed with C#'s excellent async capabilities so that a request doesn't always take up a thread if there's no processing on it.

Hope this helps,

Craig

[Luke Mauldin]

Craig,

Thank you very much for the answer and for providing a roadmap of what to expect in upcoming releases.  Have there been any medium-scale concurrent tests done on the C# server?  I am concerned that the Grpc C++ server might receive more attention, have greater stability, and offer substantially greater performance than the C# server since Google has preferred C++ to C# in the past.  For Grpc, is C#'s server implementation considered "first class" as compared to C++, Java, etc...?

Luke

[Craig Tiller]

My expectation is that C# should be seeing performance similar to Java. We've not done any serious benchmarking yet, but we expect to soon - and when that happens I expect there'll need to be some tuning made to the stack. But C# performance is something we care about.

[Poojitha A]

Hi Craig,

When you say "The C++ server has two threading models available: sync and async. Most users will want to use the sync model: the server will have an (internal) threadpool that manages multiplexing requests onto some number of threads (reusing threads between requests). "

Could you please be more specific as in how does the multiplexing happen and how the server spaws the threads? 

Any details on this please

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAAvp3oNpg%2ByVCdAJFfcg8s_gUvotdPB4nzV0H8%2BA87wkPjkVzw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Vijay Pai]

Hi there,

The sync server threading model is implemented (currently) in various files in src/cpp/server . The essential idea is that there are many ways to implement the abstract base class ThreadPoolInterface. This is internal to the gRPC C++ implementation and not directly accessible by the user-level API. The implementation in current use is class DynamicThreadPool . This thread pool reacts to new work (RPCs to service) by keeping a certain number of threads in reserve (based on the number of cores), spawning new threads if the currently created threads are all in use, and freeing threads once the work is done and we have too many threads lying around (> the reserved count).

These are the basic ideas, but the details of the thread-pool implementation may change as versions of gRPC C++ evolve.

Best regards,

Vijay

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAE-QhELXVwEAtK%2BiM5Yywx-uk6vC5pjN5Ow1tMWxU2KxMrkZ2g%40mail.gmail.com.

[douy...@gmail.com]

I suppose what you said is about sync server only. What about Async server threading model? Do we need to have our own thread pool when we use Async server? If so, is there an example we can follow? The helloworld greeter_async_server example is single threaded, and when I added multi-threads to handle cq, it doesn't work some times (hitting server.cc:442 assertion error with  GRPC_CALL_ERROR_TOO_MANY_OPERATIONS). 

[Vijay Pai]

Hello,

Yes, you need to provide your own thread pool for the async model if you want multithreaded RPC processing of RPCs. The most straightforward example, IMO, is in test/cpp/thread_stress_test.cc . 

On Friday, February 12, 2016 at 2:14:49 PM UTC-8, douy...@gmail.com wrote:

I suppose what you said is about sync server only. What about Async server threading model? Do we need to have our own thread pool when we use Async server? If so, is there an example we can follow?

Yes, you need to provide your own thread pool for the async model if you want multithreaded processing of RPCs and/or responses. The most straightforward example, IMO, is in test/cpp/end2end/thread_stress_test.cc . There are more complex examples in test/cpp/qps/*async.cc as well.

The helloworld greeter_async_server example is single threaded, and when I added multi-threads to handle cq, it doesn't work some times (hitting server.cc:442 assertion error with  GRPC_CALL_ERROR_TOO_MANY_OPERATIONS). 

I'm going to followup on your other message at https://groups.google.com/forum/#!topic/grpc-io/yCzzroDbPa0 as I think you're describing your issue as a case of mixing sync and async.

[201...@gmail.com]

hi，

why is async server thread model not like sync server thread model that can use threadpool? what is the benefit of let user provide thread model?

thanks~

在 2016年2月17日星期三 UTC+8上午8:14:53，Vijay Pai写道：

[Craig Tiller]

One of the huge advantages is that you can isolate work, so that different threads don't interfere with each other (this is huge for performance).

[201...@gmail.com]

Thank you for your answer~

在 2017年4月5日星期三 UTC+8下午8:51:32，Craig Tiller写道：

[Jessica]

Does the same threading model apply across all supported gRPC supported languages (i.e. Python)? And will gRPC also clean these threads up as well once the RPC calls have finished?

Can anyone expound upon the threading model for Python?

Thanks!

[Nathaniel Manista]

On Fri, Nov 17, 2017 at 12:28 PM, Jessica <thatje...@gmail.com> wrote:

Does the same threading model apply across all supported gRPC supported languages (i.e. Python)?

No. gRPC's threading models are generally per-language and customized to the available libraries and advised best practices of the particular programming language.

[...]

Can anyone expound upon the threading model for Python?

The current service-side threading model for gRPC Python is that the application passes at Server construction time a thread pool in which the application's RPC service behaviors will be called. We've had some users hit some scaling problems with this, particularly in the case of long-lived RPCs that exhaust the pool's threads and prevent service of any other RPCs. We have a few ideas for how to improve the design but nothing in place and supported at this time.

-Nathaniel

[marios...@gmail.com]

Thank you very much for the information on threading. I have a question regarding network and application processing.

Does gRPC have dedicated threads doing network processing and then dispatch work to the thread pools, or net processing is multiplexed on the same threads that do application processing?

[Vijay Pai]

The gRPC C++ core (wrapped by APIs in C++, Python, etc) does not use dedicated threads for the network communication needs of RPCs; these are borrowed from applications or wrapped language implementations when they call into the library. There are a few dedicated threads for purposes like timer management and DNS resolution, but not for actual RPC communication.

[Marios-Evaggelos Kogias]

Thank you very much for your reply Vijay.
So just to make it clear, any thread can call epoll and then do
network and application processing. Is there a way you allocate
connections per threads or all threads have all connections in their
epoll group and connections are protected by locks to avoid serving
the same connection in two threads at the same time?

Thanks

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "grpc.io" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/grpc-io/Cul6fd7cOB0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> grpc-io+u...@googlegroups.com.
> To post to this group, send email to grp...@googlegroups.com.

> Visit this group at https://groups.google.com/group/grpc-io.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/grpc-io/a8be9c89-6984-4f21-ad0e-df896f6909d1%40googlegroups.com.

[Vijay Pai]

Hi there,

@sreecha actually produced a detailed document on this at https://github.com/grpc/grpc/blob/master/doc/epoll-polling-engine.md . I'd highly recommend it if you're interested in how gRPC C++ core's polling works.

Regards,

Vijay

[lwa...@mtu.edu]

Thanks for your tests! I have read through your tests and learn a lot of things beyond documentation! Thanks! I am implementing a pub/sub service and have some problems in threading models used for subscribers service in server side. May I discuss my question with you for that? yia...@gmail.com

[Lei Wang]

Could you kindly help to see this relevant thread: grpc cpp async server mutlple threads (after reading relevant test files) https://groups.google.com/forum/#!topic/grpc-io/CC73-Dr4Ilc ?

[Lei Wang]

Could you help to see this relevant this thread: grpc cpp async server mutlple threads (after reading relevant test files) https://groups.google.com/forum/#!topic/grpc-io/CC73-Dr4Ilc