Async C++ service with multiple methods

[Stephan Menzel]

Hello group,

I have been using grpc since 0.14, so far with a number of services that used C++ as client and server impls. Now I am in the process of transitioning those services from sync to async. I am quite familiar with async services, mostly using boost asio. 

With this in mind, I have looked at the example implementation here: https://github.com/grpc/grpc/blob/v1.15.1/examples/cpp/helloworld/greeter_async_server.cc

I have two questions that I would like to get some hints on.

1) This is my most important issue now. All examples I have seen only use one method for their service. I suppose this is a bit of a theoretical scenario as most real world use cases would have many. Mine have between 10 and 30 methods per service. 

The way the example does its HandleRrpcs() method it always creates one CallData object of fixed type. How can I have more than one? I have tried to do a CRTP template base for such calls which then calls an overloaded work method but I don't know how I can translate this to the Next() call. Also, the way I understand it, each call to service->RequestMyMethod() with another method would override the last, right? Or am I supposed to call  service->RequestFoo() and service->RequestBar() back to back and let the system figure it out? So, is it possible at all to have multiple methods?

2) asio based servers have some kind of mechanism that allows for dispatching an async operation and hand in some callback when it's completed. For example an asio::io_context object. Looking at your example code I can see no equivalent. The example really seems to be synchronous. 

Suppose instead of saying response.set_answer("hello world") my call would really involve some blocking operations that I would like to async 'away'. Like, in a trivial example, I spawn a thread or something to do the work and then wait for a future or some kind of callback. How would that translate to grpc? 

Again, looking at the examples the only thing that would come to mind is having an actual asio io_context running next to the grpc AsyncService and post operations in there, referring to grpc mostly for when the call is done. Is that a reasonable approach or am I missing something here?

Thanks for any suggestions,

Stephan

[Christian Rivasseau]

Hi Stephan,

1: It is indeed very much possible to have multiple methods, you just need to arrange for your own CallData object that will

handle different methods. Depending on your style you could:

  - Have a enum in the CallData constructor that describe which method is handled, and switch on that.

  - Use inheritance (have a CallData subclass for each GRPC method).

  - Takes functors that perform the work as arguments.

Then when your server starts you will need to instantiate the first CallData object for each method.

2: The example is indeed synchronous in its process phase. In real life you will be offloading work to some thread pool,

or calling async services of your own, and then you can call responder_->Finish() once that is done.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/5fab2267-860f-4903-9a13-203d096e531a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christian Rivasseau
Co-founder and CTO @ Lefty
+33 6 67 35 26 74

[Stephan Menzel]

Hi Christian,

thanks for your response.

Am Freitag, 12. Oktober 2018 11:53:21 UTC+2 schrieb Christian Rivasseau:

1: It is indeed very much possible to have multiple methods, you just need to arrange for your own CallData object that will

handle different methods. Depending on your style you could:

  - Have a enum in the CallData constructor that describe which method is handled, and switch on that.

  - Use inheritance (have a CallData subclass for each GRPC method).

  - Takes functors that perform the work as arguments.

OK, this is pretty much what I did already. My problem is, at which point do I know which method is called? Let me use that modified HandleRpcs() example to explain:

Suppose I went for your second choice and have a type per call.

void MyServerImpl::HandleRpcs() {

// Spawn a new CallData instance for each method.  Is that right?

new MyFirstMethod(&m_service_instance, m_cq.get());

new MySecondMethod(&m_service_instance, m_cq.get());

void* tag;  // uniquely identifies a request.

bool ok;

while (true) {

// Block waiting to read the next event from the completion queue. The

// event is uniquely identified by its tag, which in this case is the

// memory address of a CallData instance.

// The return value of Next should always be checked. This return value

// tells us whether there is any kind of event or cq_ is shutting down.

if (!m_cq->Next(&tag, &ok)) break;

MOOSE_ASSERT(ok);

// This here:

// How do I know which call it is? To which type do I cast this tag void ptr? The first or the second?

static_cast<MyFirstMethod *>(tag)->proceed();

}

}

Then when your server starts you will need to instantiate the first CallData object for each method.

2: The example is indeed synchronous in its process phase. In real life you will be offloading work to some thread pool,

or calling async services of your own, and then you can call responder_->Finish() once that is done.

OK, so my separate io_context approach will provide that just fine.

Thanks!

Stephan

[Christian Rivasseau]

In that approach you would need to extend from a base class:

class MyBaseMethod {

 virtual void Proceed() = 0;

};

class MyFirstMethod : MyBaseMethod {

 void Proceed() overrride {

   // do the work of first method.

 }

};

class MySecondMethod : MySecondMethod {

 void Proceed() overrride {

   // do the work of second method.

 }

};

Then case to BaseMethod:

bool ok;

while (true) {

if (!m_cq->Next(&tag, &ok)) break;

MOOSE_ASSERT(ok);

static_cast<MyBaseMethod*>(tag)->proceed();

}

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/89d54d5c-af86-4a17-8e13-3446bc286aa0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Stephan Menzel]

Hello Christian

Am Freitag, 12. Oktober 2018 12:12:16 UTC+2 schrieb Christian Rivasseau:

Then case to BaseMethod:

bool ok;

while (true) {

if (!m_cq->Next(&tag, &ok)) break;

MOOSE_ASSERT(ok);

static_cast<MyBaseMethod*>(tag)->proceed();

}

The pure inheritance solution you suggested was not very practical to me as I needed to have each call with it's own RequestType and ResponseType, which suggested a CRTP solution. I went for this plus some macros, which forced me to add an additional enum to identify the object and then static cast. Gotta say, lots of boilerplate I didn't see coming. Especially since the sync server where kind of neat and in-a-box.

Anyway, it appears to work now and I can start tests next week.

If I may still inquire, in case you or anybody else knows about the thread safety of the functions involved.

The docs say somewhere that Next() is thread safe so multiple workers may serve requests.

But what about the others involved? Specifically AsyncService::RequestMyMethod() and the shutdown methods Server::Shutdown() and CompletionQueue::Shutdown(). Can I safely have multiple threads doing a loop like we have them in the example?

Thank you for your input!

Stephan

[Arpit Baldeva]

Feel free to take a look at this thread - https://groups.google.com/d/topic/grpc-io/T9u2TejYVTc/discussion 

I attached a C++ implementation of the RouteGuide Async server there. That code avoids lot of boiler plate and integrates nicely with any threading architecture you want. The code was written more than 1 year ago. I have a variation of that (mostly for hooking up our custom application) but the fundamentals are the same (meaning that sample code works in production). 

As for boiler plate, yeah, the async grpc version forces lot of it. So the implementation above and a custom code generator plugin can go a long way into making that process nice. For example, with a custom code generator, you can generate the boilerplate like  generating the instances of above templated rpcs and calling them to register with grpc core lib. Note that I am not proposing doing away with grpc_cpp_plugin.exe but simply creating a companion that is able to generate the boilerplate specific to your application. 

Hope that helps. 

[Stephan Menzel]

Am Freitag, 12. Oktober 2018 22:53:15 UTC+2 schrieb Arpit Baldeva:

Feel free to take a look at this thread - https://groups.google.com/d/topic/grpc-io/T9u2TejYVTc/discussion 

I attached a C++ implementation of the RouteGuide Async server there. That code avoids lot of boiler plate and integrates nicely with any threading architecture you want. The code was written more than 1 year ago. I have a variation of that (mostly for hooking up our custom application) but the fundamentals are the same (meaning that sample code works in production). 

Thanks for posting, I had found this already and took a great deal of info from it.

Gotta say though, as much as I like the neat sync interface of gRPC, the async variant comes as a bit of a disappointment to me. Specifically the Next() function. The way I see it, the system already knows the type of call coming in, as it obviously instantiates the correct call object fitting to the RequestMyMethod() call. Why it chooses to discard that type information entirely and hand out a void * without any hint is beyond me. Some kind of gRPC inherent mechanism to know the type seems imperative to me. Perhaps an easy to use base class for the calls? If you excuse my criticism, looking at the code now, I can barely see the advantage of using gRPC at all. I have multiple other interfaces in my system that use pure protobuf request and response objects and send them over all kinds of interfaces. For the network services I have used sync gRPC on top because it gave me a nice interface and the method resolution. Without these advantages, I might as well go for a plain old beast or asio async socket and send protobuf objects back and forth.

Still, thanks for your help tackling this!

Stephan

[Nathan Prat]

Hi,

How can you use CRTP? I tried it this way, but after `cq_->Next` you can't static_cast to a templated class. Or am I missing something?

Anyway, I had the same problem, and using inheritance with a CallData base class, and an abstract method "Process", I have pretty much no boilerplate.
The base class main logic is in a "Process" method, which calls the abstract methods(Process, and some others)
The only boilerplate is the need to call the "Proceed" method of the base class in each derived class ctor(because we can't call a pure virtual from a ctor).

With this, HandleRpcs basically contains only 2 lines:

- cq_->Next(&tag, &ok)

- static_cast<CallBase*>(tag)->Proceed();

[Arpit Baldeva]

@Stephen , yeah, Async model is hard/non-intuitive to use. Only thing I can say is that it allows the application to plug in it's own threading architecture which I found super useful. My application is based off of fibers and that won't work nicely/easily with the sync threading model. As for why to use gRPC, I like that it is based off of Http2  (with a thin shim of protocol on top) so that is a plus rather than having a completely custom messaging protocol. Just like you, my application supports many other protocols (including custom)  and we are trying to get away from that (and they do work off of plain base message type - similar to google::protobuf::Message). 

@Nathan, having to implement a class for every rpc,  derive it from base rpc, implement the "process", add each rpc instance to grpc server to enable it to instantiate incoming request is all the boilerplate code. For lot of monolithic applications that are in transition and adding gRPC support, doing this for hundreds of rpcs is lot of work. A lot of it can be solved by a custom code generator - if your application is in that boat.

Thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/7lCQpAMVUe0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/c434fa5b-76c0-4e1b-a495-9da8f694513a%40googlegroups.com.

[Christopher Warrington - MSFT]

On Friday, October 12, 2018 at 1:53:15 PM UTC-7, Arpit Baldeva wrote:

> As for boiler plate, yeah, the async grpc version forces lot of it. So the
> implementation above and a custom code generator plugin can go a long way
> into making that process nice. For example, with a custom code generator,
> you can generate the boilerplate like generating the instances of above
> templated rpcs and calling them to register with grpc core lib. Note that
> I am not proposing doing away with grpc_cpp_plugin.exe but simply creating
> a companion that is able to generate the boilerplate specific to your
> application.

A project I worked on in the past year took exactly this helper library+code
generation approach. The helper library [1] had a bunch of abstractions,
like one that represented the server-side state of one async, in-progress
method invocation in a well-typed way [2] & [3].

Our custom codegen tool (which was based on a language similar to ProtoBuf)
knew how to generate service base classes that used these abstractions [4] & [5].

The library also imposed a callback-based async model [6] and took care of
reading from completion queues itself [7].

By imposing these restrictions atop the gRPC++ library, we were able to
simplify implementation of async services [8]:

    class GreeterServiceImpl final : public Greeter::Service
    {
    public:
        using Greeter::Service::Service;

    private:
        void SayHello(bond::ext::grpc::unary_call<HelloRequest, HelloReply> call) override
        {
            HelloRequest request = call.request().Deserialize();

            HelloReply reply;
            reply.message = "hello " + request.name;

            call.Finish(reply);
        }
    };

Some of these ideas may help others build abstractions for their projects.

When I left the project, only unary calls had been implemented in C++.
Streaming was in progress--and they appear to not yet have been implemented.
There was also work left to make it possible to share an io_manager among
multiple servers and clients (to reduce the number of threads that were just
idle), and to use multiple completion queues to reduce some lock contention
we were seeing inside of the gRPC completion queue because we were just
using one completion queue for all I/O operations within a server.

[1]: https://microsoft.github.io/bond/manual/bond_over_grpc.html
[2]: https://github.com/Microsoft/bond/blob/bd4b46e78a82dd3a38b52b00233a1a22047014d2/cpp/inc/bond/ext/grpc/unary_call.h#L19-L32
[3]: https://github.com/Microsoft/bond/blob/bd4b46e78a82dd3a38b52b00233a1a22047014d2/cpp/inc/bond/ext/grpc/detail/unary_call_impl.h#L42-L56
[4]: https://github.com/Microsoft/bond/blob/bd4b46e78a82dd3a38b52b00233a1a22047014d2/compiler/tests/generated/service_grpc.h#L415
[5]: https://github.com/Microsoft/bond/blob/bd4b46e78a82dd3a38b52b00233a1a22047014d2/cpp/inc/bond/ext/grpc/detail/service.h#L151-L159
[6]: https://github.com/Microsoft/bond/blob/bd4b46e78a82dd3a38b52b00233a1a22047014d2/cpp/inc/bond/ext/grpc/scheduler.h#L14-L28
[7]: https://github.com/Microsoft/bond/blob/bd4b46e78a82dd3a38b52b00233a1a22047014d2/cpp/inc/bond/ext/grpc/io_manager.h
[8]: https://github.com/Microsoft/bond/blob/bd4b46e78a82dd3a38b52b00233a1a22047014d2/examples/cpp/grpc/helloworld/helloworld.cpp#L18-L33

--
Christopher Warrington
Microsoft Corp.

[Stephan Menzel]

Hello Nathan,

Am Montag, 15. Oktober 2018 17:54:48 UTC+2 schrieb Nathan Prat:

How can you use CRTP? I tried it this way, but after `cq_->Next` you can't static_cast to a templated class. Or am I missing something?

I have a mixture now of virtual inheritance and CRTP.

Basically, one super simple call root class:

// First a macro I use later

#define GRPC_NATIVE_NAME_REQUEST( name )              \

void native_name_request() {                                           \

m_service->Request ## name ## (&m_ctx, &m_request, &m_responder, m_cq, m_cq, this);    \

}

class RpcCallBase {

    public:

        RpcCallBase() {};

        virtual ~RpcCallBase() {};

        virtual void proceed() noexcept {

              MOOSE_ASSERT_MSG(true, "RPC implementation does not overload proceed()");

        };

};

This is what I use to cast the void ptr to in order to get it to RTTI the right type. Then, on top of this, the actual CRTP base. Like this:

// First a macro I use later

#define GRPC_NATIVE_NAME_REQUEST( name )              \

void native_name_request() {                                           \

m_service->Request ## name ## (&m_ctx, &m_request, &m_responder, m_cq, m_cq, this);    \

}

template< class DerivedType, class RequestType, class ResponseType >

class MyServiceCall : public RpcCallBase {

    public:

        typedef MyServiceCall<DerivedType, RequestType, ResponseType> base_type;

MyServiceCall() {

             // constructor with service specific stuff such as parent object and so on

             proceed();  like in the example

        }

        void proceed() noexcept override {

             // Much like the example, except:

             if (m_status == CREATE) {

                   m_status = PROCESS;

                   static_cast<DerivedType *>(this)->native_name_request();

             // this is what the macro injects in order to fake the right type in here. See below.

             } else if (m_status == PROCESS) {

                    // new object of CRTP derived type

                    new base_type(m_service, m_cq, m_parent);

                   

                    // CRTP to the actual work, overloaded by derived class

                    grpc::Status retstat = static_cast<DerivedType *>(this)->work();

              }

    // rest of the stuff pretty much like the example except template types

}

and then, each call can be implemented nicely:

class FooMethodCall : public MyServiceCall<FooMethodCall, FooMethodRequest, FooMethodResponse> {

    public:

        GRPC_NATIVE_NAME_REQUEST( FooMethod )   // I know, not perfect but it does the trick

        FooMethodCall(MyService::AsyncService *n_service, ServerCompletionQueue *n_cq, MyServiceImpl *n_parent)

              : base_type(n_service, n_cq, n_parent) {

        }

        grpc::Status work() {

                      

                    // do the actual work and return a status object

        }     

}

Finally, in the async loop it looks like this:

void MyServiceImpl::HandleRpcs() {

    // Spawn a new CallData instance to serve new clients.

    // The q takes ownership

    new FooMethodCall(&m_service_instance, m_cq.get(), this);

    new BarMethodCall(&m_service_instance, m_cq.get(), this);

    void* tag;  // uniquely identifies a request.

    bool ok;

    while (true) {

        

         if (!m_cq->Next(&tag, &ok)) {

               BOOST_LOG_SEV(logger(), normal) << "service shutting down";

               break;

         }

         RpcCallBase *call = static_cast<RpcCallBase *>(tag);

         if (!ok) {

               // This seems to be the case while the q is draining of events 

               // during shutdown I'm gonna delete them

               delete call;

               continue;

          }

          // hand over to the call object to do the rest

          call->proceed();

     }

}

And it works. This way I don't need any further type guessing or casting beyond the RpcCallBase 

HTH,

Stephan

[Stephan Menzel]

Am Dienstag, 16. Oktober 2018 00:40:58 UTC+2 schrieb Christopher Warrington - MSFT:

By imposing these restrictions atop the gRPC++ library, we were able to
simplify implementation of async services [8]:

    class GreeterServiceImpl final : public Greeter::Service
    {
    public:
        using Greeter::Service::Service;

    private:
        void SayHello(bond::ext::grpc::unary_call<HelloRequest, HelloReply> call) override
        {
            HelloRequest request = call.request().Deserialize();

            HelloReply reply;
            reply.message = "hello " + request.name;

            call.Finish(reply);
        }
    };

Wow, that's pretty neat. It looks almost as tidy as the Sync Service did. Thanks for posting.

Only thing I would have with it is, the fact that the async approach forced me to take the route with the one class per call approach was one of the few things I liked about it. With a service of, say, 50 methods a class containing all the impls can grow tremendously. Even if each call is very much separated from the others. Few years back we had some static analysis run over the code that showed glowing red complexity dots over those files. Probably due to size, because they weren't this complex. I hope this gets better with the one-class-per-call way.

Cheers,

Stephan

PS: Amazed Microsoft uses gRPC. And open sources the results. The world we live in.