Using multiple late-bound services

[Geoff Groos]

Hey everyone

I'm building an API with GRPC which currently looks like this:

serivce OurEndpoint {
   rpc register (RegistrationForFeatureCeeAndDee) returns (stream FeatureCeeOrDeeRequest) {}
     
   rpc featureA (FeatureAyeRequest) returns (FeatureAyeReponse) {}
   rpc featureB (FeatureBeeRequest) returns (FeatureBeeResponse) {}
   
   rpc offerFeatureC(FeatureCeeResponse) returns (Confirmation) {}
   rpc offerFeatureD(FeatureDeeResponse) returns (Confirmation) {}
   rpc offerCeeOrDeeFailed(FailureResponse) returns (Confirmation) {}
}


message FeatureCeeOrDeeRequest {
    oneof request {
        FeatureDeeRequest deeRequest = 1;
        FeatureCeeRequest ceeRequest = 2;      
    }
}


message Confirmation {}

Note that features A and B are fairly traditional client-driven request-response pairs.

Features C and D are callbacks; the client registers with

I can provide answers to C and D, send me a message and I'll call offerFeatureResponse as appropriate.

I don't like this. It makes our application code complex. We effectively have to build our own multiplexer for things like offerCeeOrDeeFailed

What I'd really rather do is this:

serivce OurEndpoint {
   rpc register (RegistrationForFeatureCeeAndDee) returns (Confirmation) {}
     
   rpc featureA (FeatureAyeRequest) returns (FeatureAyeReponse) {}
   rpc featureB (FeatureBeeRequest) returns (FeatureBeeResponse) {}  
}
service EndpointClientMustImplement {
   rpc featureC(FeatureCeeRequest) returns (FeatureCeeResponse) {}
   rpc featureD(FeatureDeeRequest) returns (FeatureDeeResponse) {}
}


message RegistrationForFeatureCeeAndDee {
   ConnectionToken name = 1;
}


message Confirmation {}

The problem here is how to go about implementing ConnectionToken and its handler. Ideally I'd like some code like this:

//kotlin, which is on the jvm.
override fun register(request: RegistrationForFeatureCeeAndDee, response: ResponseObserver<Confirmation>) {
   
    //...
   
    val channel: Channel = ManagedChannelBuilder
            .for("localhost", 5551) // a port shared by the service handling this very response
            .build()
           
    val stub: EndpointClientMustImplement = EndpointClientMustImplement.newBuilder()
            .withServiceNameOrSimilar(request.name)
            .build()
           
    //....
}

What is the best way to go about this?

1. Can I have multiple servers at a single address?

2. Whats the best way to find a service instance by name at runtime rather than by a type-derived (and thus by statically bound) name? I suspect the BindableService and ServerServiceDefinitions will help me here, but I really don't want to mess with the method-table building and the code generating system seems opaque. 

I guess my idea solution would be to ask the code generator to generate code that is open on its service name, --ideally open on a constructor param such that there is no way to instance the service without specifying its service name.

Or, perhalps there's some other strategy I should be using? I could of course specify port numbers and then instance grpc services once-per-port, but that means I'm bounded on the number of ports I'm using by the number of active API users I have, which is very strange.

Many thanks!

-Geoff

[Carl Mastrangelo]

Some comments / questions:

1.  Why doesn't "rpc register" get split into two methods, one per type?  Like "rpc registerCee (CeeRegRequest) returns (CeeRegResponse);"

2.  Being careful with terminology, you have multiple "services" on a singe "server", and the "server" is at one address.   

3.  You can find all services, methods, and types using the reflection api, typically by adding ProtoReflectionService to your Server.  

4.  BindableService and ServerServiceDefinition are standard and stable API, you can make them if you want.  The Protobuf generated code makes its own (and is complicated for other reasons) but you can safely and easily construct one that you prefer.

5.  Multiple ports is usually for something special, like different socket options per port, or different security levels.  That is a more advanced feature less related to API. 

[Geoff Groos]

Thanks Carl,

I think the client-server naming is only causing me problems, so Instead I'll use the real names which are optimizer (server above) and simulator (client above). They are more like peers than client/server, because each offers functionality.

If its the case that one process can start a service, have clients connect to it, and then register services that they offer with that server, then you're correct, and I do only need one server. The key is that the client needs to be able to state "I offer this service, and heres how you can send me messages". I'm just not sure how to implement that in GRPC.

I think I did indeed have the terminology correct: I do want multiple servers, each offering one service. The idea is that a simulator would connect to an already running optimizer, start their own server running a single instance of the service 'EndpointClientMustImplement', bind it with protobuf, then call 'register' on our optimizer with a token that contains details on "heres how you can connect to the service I just started".

The only downside to your suggestion is that it would require multi-threading, because user code would have to call `register`, and then produce two threads (or coroutines) to consume all the messages in both the `featureC` stream and the `featureD` stream. But it does address some of my concerns.

Still, I like the elegance of the solution I was asking for: when a client-simulator connects to a server-optimizer, it starts its own service and tells the optimizer it connects to that it should call back to this service at some token.

Can it be done?

[Carl Mastrangelo]

Inline responses

On Thursday, February 14, 2019 at 3:10:45 PM UTC-8, Geoff Groos wrote:

Thanks Carl,

I think the client-server naming is only causing me problems, so Instead I'll use the real names which are optimizer (server above) and simulator (client above). They are more like peers than client/server, because each offers functionality.

In gRPC, clients always initiate, so they aren't peers so much.   You can use streaming to work around this, by having the client "prime" the RPC by sending a dummy request, and having the server send responses.  These responses are handled by the client, which then sends more "requests", inverting the relationship.  

Alternatively, you could have your client be combined with a local server, and then advertise the ip:port to the actual server, which then is combined with its own client.  

Not clean I know, but bidirectional streaming RPCs are the closest thing to peers that gRPC can offer.  

 

If its the case that one process can start a service, have clients connect to it, and then register services that they offer with that server, then you're correct, and I do only need one server. The key is that the client needs to be able to state "I offer this service, and heres how you can send me messages". I'm just not sure how to implement that in GRPC.

I think I did indeed have the terminology correct: I do want multiple servers, each offering one service. The idea is that a simulator would connect to an already running optimizer, start their own server running a single instance of the service 'EndpointClientMustImplement', bind it with protobuf, then call 'register' on our optimizer with a token that contains details on "heres how you can connect to the service I just started".

The only downside to your suggestion is that it would require multi-threading, because user code would have to call `register`, and then produce two threads (or coroutines) to consume all the messages in both the `featureC` stream and the `featureD` stream. But it does address some of my concerns.

I would not consider threading to be a serious concern here (and I say that as someone who has spent significant time optimizing gRPC).  You will likely need to give up the blocking API anyways, which means you can have requests and responses happen on the same thread, keeping a clear synchronization order between the two.  

[Geoff Groos]

I think we're on the same page.

So if you've got an API user who connect to your API with some kind of functionality that he wants to offer, (thinking in the vein of callbacks in C# or java), then your two best solutions are:

• use a bidirectional stream, such that the client-simulator sends one message, the optimizer-server-to-client-simulator stream is held open until the optimizer-server would invoke the callback, at which point it sends one message to the client-simulator, then the client-simulator sends one message back to the optimizer-server containing the result from the callback's execution. Both sides then "onComplete".
  
  
  rpc callbackBasedFeature(stream FeatureRegistrationOrResult) returns (stream FeatureOneCallbackInvocation)
  
  message FeatureRegistrationOrResult {
    oneof message {
      FeatureRegistration registration = 1 //registers callback
      FeatureResult result = 2 //result from callback evaluation
    }
  }
  message FeatureOneCallbackInvocation {
    //callback parameters
  }
  
  
  
• pro: it allows you to expose callback-based functionality as single service/method end-points.
  
• pro: it allows for a reasonably-trivial and correct error handler in the form of `try { process(inboundStream.next()) } catch(Exception e) { outputStream.onError(e) }`. Assuming callback execution is involved synchronously there is no chance of multiplexing problems (IE: there is no chance you fail to understand which error corresponds to which request).
  
• con: if you have more than one such callback you're requiring that API callers employ some concurrency.
  
• con: Streams are also usually multi-element'd, but if your callback is designed as a one-off rather than an event bus then it you're likely only ever using streams of one element, which is surprising.
  

• use separate servers, a registration message containing enough for the "first-server" (optimizer) to connect to client "servers".
  

service OptimizerServer {
    rpc register(RegistrationRequest) returns (RegistrationResponse);
}
service SimulatorClientServer {
    rpc feature(FeatureRequest) returns (FeatureResponse);
}

message RegistrationRequest {
  string host = 1;
  int port = 2;
  //other meta im missing? routing or DNS or some such?
}

Thus the client-simulator first starts its own GRPC server with a SimulatorClientServer service, then connects to the optimizer-server, calls register(host = getLocalHost(), port = 12345), and waits for invocations of feature

• pro: it requires the minimum of grpc packaging at invocation time. Once setup and running this is two very REST-friendly services talking to each other.
  
• pro: no use of streams on this feature set. Streams are left to their (intended?) purpose of sending back multiple delayed elements on features.
• pro: the problem of errors is trivial since all methods are exposed as simple RPC endpoints: success calls `onNext; onComplete` and errors call `onError; onComplete`. 
  
• pro: clients will end up multi-threaded, but they will leverage the ~elegant thread-pools provided by grpc by default.
  
• con: clients now require a webplatform (netty by default on grpc/java) to function.
  
• con: it requires some fairly strange logic to setup
• con: uses as many ports as there are callback users. 

The original solution I came up with in my question seems clearly inferior to these two solutions.

And I didn't think about the dependency issue until just now: requiring that our API users bundle netty or similar is quite a big ask, and has many implications for deployment.

I think bidirectional streams might be the best way to go.

Thanks for your help!

PS:

Is there any chance you think the GRPC guys themselves would be willing to change the spec to allow this?

maybe allowing you to declare anonymous-ee services on messages as message members? Something like:

service MyCallbackBasedService {
  rpc doCallbackFeature(RequestMessage) returns (ResponseMessage)
}

message RequestMessage {
  string name = 1;
  int otherMeta = 2;
  service callback = 3 {
    rpc invoke(ServerProvidedArgumentsMessage) returns ClientProvidedResponse
  };
}

Resulting in the server side code looking something like:

class Server: MyCallbackBasedService {
  override fun doCallbackFeature(request: RequestMessagee, response: ResponseObserver<ResponseMessage>) {

    val inputs: ServerProvidedArgumentsMessage = makeServerContext()

    val resultFromClient: ClientProvidedResponse = request.callback.bindLocally().invoke(inputs)
    //where the return value from request.callback is actually metadata for service
    //thats already offered on the existing connection that the `request` was made on?

    response.onNext(computeFinalResults(resultsFromClient))
  }
}


and the client side code hopefully being as elegant as

val message = new RequestMessage()
{
  name = "hello-callbacks!",
  otherMeta = 42,
  callback = GrpcServiceBuilder
        .newSAMService<RequestMessage.Services.callback>()
        .invokes((ServerProvidedArgumentsMessage input) => {
    //code that handles server's invocation
    return new ClientProvidedResponse()
    {
      details = "asdf"
    }
  })
}
stub.doCallbackFeature()

Cheers!

[Carl Mastrangelo]

Inline responses

Biggest con (despite me having suggested it): it plays badly with firewalls.  This will make things super complicated.   It's just technically possible.

 

The original solution I came up with in my question seems clearly inferior to these two solutions.

And I didn't think about the dependency issue until just now: requiring that our API users bundle netty or similar is quite a big ask, and has many implications for deployment.

I think bidirectional streams might be the best way to go.

Thanks for your help!

PS:

Is there any chance you think the GRPC guys themselves would be willing to change the spec to allow this?

This has been an ask for a long time, but never really prioritized by the gRPC team.   We called it gRPC-on-gRPC, where a client opens a gRPC Connection, and then the server tunnels gRPC requests on top of it, inverting the relationship.  The main use is command and control, (like IoT devices that want to listen for commands, but may be behind a firewall, or something).  However, the overwhelming majority of RPC users do basic Request-Response pairs and then the RPC is done.  The complication to the gRPC library (which is already super complex) isn't worth it right now.  

The alternative is users making their own state machine on top of the bidi semantics, (which is your first listed solution above).

I have to admit, I have never thought about the callback in a message though.  That's a new one to me.