Understanding RPC in PB

[Noel Fegan]

Hi,

I'm trying to understand the RPC mechanism in PB. I can understand the
concept of separation of the interface of an RPC from any concrete
definition, which is actually great for us, as we would like to be
able to implement an IPC mechanism, based (we think at the moment) on
the use of shared memory on Windows. The lack of a concrete
implementation, however, does mean that the test examples downloaded
with the PB package are pretty much centered around the serialization/
deserialization which PB provides, so I'm confident that PB can
deliver that for us. Before I embark on writing my own RPC
implementation I wanted to check out a few things.

If I use the "service" construct in the proto file it generates a stub
for the RPC mechanism, and it's simply up to me to add my concrete
implementation of two required RPC classes. What will this give me? To
clarify, will this then mean that I write code that invokes the
service "method" on the client end (the one I defined in my service
definition in the proto file) then PB "engine" will handle all the
marshalling etc to have my service method "automagically" invoked. So,
like SOAP, I don't have to do any of the wiring in between? Other than
providing the two RPC classes? I haven't quite digested the details of
the two classes, so maybe I'm missing something.

I've looked at some of the reference implementation and some of them
talk about passing the name of the method as one of the messages you
need to pass in the message if you're doing an RPC. This confused me,
as it seems to suggest that the "magic" I referred to above isn't the
way it works. Perhaps these implementations are bypassing the
"service" feature altogether and merely passing message that require a
complete description of what it is that you want to do, in a more
message driven approach?

Finally, the LITE PB static lib is much smaller than the regular PB
lib, and I understand that if I want to use the service feature I
cannot use the LITE version. Is the "extra" capability in the non-lite
version mainly because of the extra work needed to support the service
feature? Is this to support the magic I mentioned above, that is not
otherwise needed if all you're interested in is serializing and
deserializing?

I hope this makes sense. Any help would be most appreciated.

Noel

[Kenton Varda]

On Wed, Nov 11, 2009 at 1:05 PM, Noel Fegan <nfe...@gmail.com> wrote:

I'm trying to understand the RPC mechanism in PB. I can understand the
concept of separation of the interface of an RPC from any concrete
definition, which is actually great for us, as we would like to be
able to implement an IPC mechanism, based (we think at the moment) on
the use of shared memory on Windows. The lack of a concrete
implementation, however, does mean that the test examples downloaded
with the PB package are pretty much centered around the serialization/
deserialization which PB provides, so I'm confident that PB can
deliver that for us. Before I embark on writing my own RPC
implementation I wanted to check out a few things.

I should note that I am currently working on a change to introduce the ability to extend protoc via plugins.  This will include the ability to insert additional code into the files generated by other code generators.  Once this is done I intend to deprecate all the abstract "service" interfaces in the protobuf implementation -- it will be up to plugins to generate code based on the service definitions in the .proto file.  Thus, your RPC implementation can generate code more appropriate to it and not be constrained to the set of interfaces defined by the protobuf runtime library.

A future change may even make the parser extensible, so that you can define your own language for RPC rather than be limited to the present "service" definitions.  (Though I'm not sure yet if this is something we really want to do.)

 

If I use the "service" construct in the proto file it generates a stub
for the RPC mechanism, and it's simply up to me to add my concrete
implementation of two required RPC classes. What will this give me? To
clarify, will this then mean that I write code that invokes the
service "method" on the client end (the one I defined in my service
definition in the proto file) then PB "engine" will handle all the
marshalling etc to have my service method "automagically" invoked. So,
like SOAP, I don't have to do any of the wiring in between? Other than
providing the two RPC classes? I haven't quite digested the details of
the two classes, so maybe I'm missing something.

Protocol Buffers only provides serialization and deserialization of structured messages.  It does not provide any of the other wiring -- that's what an RPC implementation provides.

 

I've looked at some of the reference implementation and some of them
talk about passing the name of the method as one of the messages you
need to pass in the message if you're doing an RPC. This confused me,
as it seems to suggest that the "magic" I referred to above isn't the
way it works. Perhaps these implementations are bypassing the
"service" feature altogether and merely passing message that require a
complete description of what it is that you want to do, in a more
message driven approach?

The conversation you refer to was discussing what other information the RPC system would need to send over the wire *in addition* to the application message.  This "header" information might as well itself be encoded as a protocol message, unless your RPC system calls for some other way to express metadata.  However, this doesn't mean it has to be part of the application message -- the header message would be a private implementation detail of the RPC system.

 

Finally, the LITE PB static lib is much smaller than the regular PB
lib, and I understand that if I want to use the service feature I
cannot use the LITE version. Is the "extra" capability in the non-lite
version mainly because of the extra work needed to support the service
feature? Is this to support the magic I mentioned above, that is not
otherwise needed if all you're interested in is serializing and
deserializing?

The service interfaces are heavily tied to descriptors.  Descriptors are not part of the lite runtime, therefore to support services we would have had to define a new set of "lite" service interfaces.  This seemed pointless since most users don't even use the service stuff in protobufs.

In fact, the whole services thing exists mainly because inside Google, the protocol compiler has always been responsible for generating RPC stubs in addition to message objects.  Unfortunately, our RPC system is very complicated and non-portable, which means it will be much harder to properly open source than Protocol Buffers*.  In order to allow us to release protobufs without waiting for the entire RPC system to reach a state where we could release it, we decoupled the two by introducing the abstract RPC interfaces you see today.  This actually turned out to be a crappy design -- on one hand it made our internal RPC system less convenient to use, while on the other hand the interfaces are still too specific to our system to be widely adaptable to others.  We really should have done the plugins thing from the start.

In short, I'd recommend that you ignore the whole services thing, or (if you're sure that the services *syntax* matches what you'd have in mind for your system) wait until the plugins feature is released.