proto bufs and sockets

[robert]

hi folks,

i must be missing something, because this looks like exactly the use
case this was meant for, yet I am having difficulties to do it
cleanly: i want to pass one out of a set of messages over a socket and
handle it on the other side. in order to do so, i currently write an
identifier to the socket, and then the actual buffer content. on the
other side, i get the identifier, switch on it, parse the buffer from
the socket and handle it accordingly. this is fairly ok, but there
must be some better way (using the descriptors or so). i am also
having problems understanding how this could be used with a non-
blocking socket. my take on it would be to add a length field after
the identifier, then read up to this length in non-blocking ways, and
then parse from there when everything has been collected. still, this
all feeld very hackish to me...

any suggestions?

regards robert

[Kenton Varda]

I agree with your thought:  You should write a length first, then read that much data into a buffer before attempting to parse it.

Your idea of writing an identifier first also seems fine.  An alternative would be to have a container message like:

message Container {

  // One of the following is filled in.

  optional Foo foo = 1;

  optional Bar bar = 2;

  optional Baz baz = 3;

}

Then pass that over the wire.

[Torbjörn Gyllebring]

For some additional ideas look here:
http://groups.google.com/group/protobuf/browse_thread/thread/b1da210c5476ac3e#

Does the "spec" give any guarantee about the order that fields are
serialized or is that implementation defined?

[Kenton Varda]

On Tue, Jul 15, 2008 at 11:31 AM, Torbjörn Gyllebring <torbjorn....@gmail.com> wrote:

Does the "spec" give any guarantee about the order that fields are
serialized or is that implementation defined?

Parsers are expected to accept fields in any order, but may optimize for the common case that the fields are ordered by number.

[Alek Storm]

On Jul 15, 1:31 pm, Torbjörn Gyllebring
<torbjorn.gyllebr...@gmail.com> wrote:
> For some additional ideas look here:http://groups.google.com/group/protobuf/browse_thread/thread/b1da210c...

>
> Does the "spec" give any guarantee about the order that fields are
> serialized or is that implementation defined?
>
> On 15 Juli, 20:24, "Kenton Varda" <ken...@google.com> wrote:
>
> > I agree with your thought:  You should write a length first, then read that
> > much data into a buffer before attempting to parse it.
> > Your idea of writing an identifier first also seems fine.  An alternative
> > would be to have a container message like:
>
> > message Container {
> >   // One of the following is filled in.
> >   optional Foo foo = 1;
> >   optional Bar bar = 2;
> >   optional Baz baz = 3;
>
> > }

Or do this:

enum Type {
FOO = 1;
BAR = 2;
BAZ = 3;
}

message Foo {
required Type _type = 1 [default = FOO];
...
}

[Jean-Sebastien Stoezel]

Hi,

For the next release, should we expect a turnkey parser to solve the
issue of parsing messages from sockets?

Thanks,
Jean

[Kenton Varda]

Sorry, that's the job of the RPC system, which is outside the scope of protocol buffers.  (But perhaps someone will write and release a good, open source RPC system on top of protocol buffers.)

[Blair Zajac]

It would be interesting to mix the RPC layer of Ice with the serialization
format of Protocol Buffers. Ice has its own serialization mechanism, but it
doesn't deal as well with adding and removing fields as Protocol Buffers does.

Ice is GPL

http://www.zeroc.com/

You write a Slice file, much like the .proto file, that describes the
structures, classes and interfaces for the RPC system and it'll generate C++,
Java, Python, Ruby, C# code (using slice2py, slice2cpp, etc) which you extend to
implement the RPC functionality.

One of the coolest things with it is that with a Slice file, in Python you can do

Ice.loadSlice('path/to/foo.slice')
import foo

and you have all your RPC right there, there's no slice2py and compiling of code
required. The Python Ice module is a wrapper around the C++ library so its
pretty fast. You can write a client or server in Python in Ice.

It handles synchronous and asynchronous method calls.

Anyway, as you can tell I'm really happy with Ice, it's just how to deal with
versioning of the messages after you've deployed and you realize you need to add
an additional field that ones gets a little stuck.

Regards,
Blair

Kenton Varda wrote:
> Sorry, that's the job of the RPC system, which is outside the scope of
> protocol buffers. (But perhaps someone will write and release a good,
> open source RPC system on top of protocol buffers.)
>
> On Tue, Jul 15, 2008 at 1:46 PM, Jean-Sebastien Stoezel
> <js.st...@gmail.com <mailto:js.st...@gmail.com>> wrote:
>
>
> Hi,
>
> For the next release, should we expect a turnkey parser to solve the
> issue of parsing messages from sockets?
>
> Thanks,
> Jean
>
> On 7/15/08, Alek Storm <alek....@gmail.com

> <mailto:alek....@gmail.com>> wrote:
> >
> > On Jul 15, 1:31 pm, Torbjörn Gyllebring
> > <torbjorn.gyllebr...@gmail.com

> <mailto:torbjorn.gyllebr...@gmail.com>> wrote:
> > > For some additional ideas look
> here:http://groups.google.com/group/protobuf/browse_thread/thread/b1da210c...
> > >
> > > Does the "spec" give any guarantee about the order that fields are
> > > serialized or is that implementation defined?
> > >
> > > On 15 Juli, 20:24, "Kenton Varda" <ken...@google.com

[Michi Henning]

On Jul 16, 7:03 am, Blair Zajac <bl...@orcaware.com> wrote:

> Anyway, as you can tell I'm really happy with Ice, it's just how to deal with
> versioning of the messages after you've deployed and you realize you need to add
> an additional field that ones gets a little stuck.

Hi Blair,

you may want to check out "Can a Leopard Change its Spots?" in
http://www.zeroc.com/newsletter/issue8.pdf

It explains how Ice deals with versioning.

Cheers,

Michi.

[gsxr]

Hi Kenton :-)

On Jul 16, 8:51 am, "Kenton Varda" <ken...@google.com> wrote:
> Sorry, that's the job of the RPC system, which is outside the scope of
> protocol buffers. (But perhaps someone will write and release a good, open
> source RPC system on top of protocol buffers.)

Passing responsibility onto another technology such as RPC seems like
a patch over a shortcoming.

I cant imagine maintaining a codebase that includes .proto and .idl
files
for every instance of IPC. The thought of all those extra machine
cycles
being consigned to the bin makes me feel a little uncomfortable.

From past experience I think protocol buffers has missed the mark in
this
area. An on-the-wire representation of an application object should be
self-describing, i.e. detection of end-of-representation is innate.

The only way I have been able to achieve this is by application of
language
techniques (lexical and grammatical).

I joined this group because Google does amazing things and PB looked
like another feather in their cap. It obviously does some things
really
well but its not looking like the silver bullet that I imagined.

Lastly - all this talk of RPC assumes a synchronous model for network
messaging. For me thats just too constraining. Just confirming that
your
intention was to implement sync code.

Cheers.

>
> On Tue, Jul 15, 2008 at 1:46 PM, Jean-Sebastien Stoezel <
>

> js.stoe...@gmail.com> wrote:
>
> > Hi,
>
> > For the next release, should we expect a turnkey parser to solve the
> > issue of parsing messages from sockets?
>
> > Thanks,
> > Jean
>

[Kenton Varda]

Sorry, Scott, but RPC protocols are outside the scope of this project.  Even if Google were to release an RPC system based on protobufs, it would be a separate project.

On Thu, Jul 17, 2008 at 1:31 PM, gsxr <scott....@gmail.com> wrote:

I cant imagine maintaining a codebase that includes .proto and .idl
files
for every instance of IPC.

You can define RPC services in .proto files.  It will generate stubs that are not specific to any particular RPC implementation.

 

Lastly - all this talk of RPC assumes a synchronous model for network
messaging. For me thats just too constraining. Just confirming that
your
intention was to implement sync code.

The word "synchronous" is used to mean a lot of different things.  Can you be more specific about which meaning you're using here?  For example, the services generated by the protocol compiler are asynchronous in that they are non-blocking; you must pass a callback which is called when they finish.  I'm not sure if this is what you were talking about, though.

Regardless, since protobufs are just a way of encoding data, I don't think it really makes sense to say that they are synchronous or asynchronous.

[gsxr]

On Jul 18, 8:50 am, "Kenton Varda" <ken...@google.com> wrote:
> Sorry, Scott, but RPC protocols are outside the scope of this project. Even
> if Google were to release an RPC system based on protobufs, it would be a
> separate project.

Hmmm. OK. Fait accompli.

>
> On Thu, Jul 17, 2008 at 1:31 PM, gsxr <scott.suz...@gmail.com> wrote:
> > I cant imagine maintaining a codebase that includes .proto and .idl
> > files
> > for every instance of IPC.
>
> You can define RPC services in .proto files. It will generate stubs that
> are not specific to any particular RPC implementation.

Ah. That sheds some light.

>
> > Lastly - all this talk of RPC assumes a synchronous model for network
> > messaging. For me thats just too constraining. Just confirming that
> > your
> > intention was to implement sync code.
>
> The word "synchronous" is used to mean a lot of different things. Can you
> be more specific about which meaning you're using here? For example, the
> services generated by the protocol compiler are asynchronous in that they
> are non-blocking; you must pass a callback which is called when they finish.
> I'm not sure if this is what you were talking about, though.
>
> Regardless, since protobufs are just a way of encoding data, I don't think
> it really makes sense to say that they are synchronous or asynchronous.

Understood. PB is amenable to use with any messaging technique. RPC
being
one of those.

The domain that PB operates in is now clearer. Is it not making it
harder than
necessary (i.e. for chosen network messaging technique) if PB
encodings are not
self-describing and hence self-terminating? I raise this question
because that
appears to be a significant issue in this thread. And because I have
been banging
my head against this for years.

[Kenton Varda]

On Thu, Jul 17, 2008 at 2:15 PM, gsxr <scott....@gmail.com> wrote:

The domain that PB operates in is now clearer. Is it not making it

harder than
necessary (i.e. for chosen network messaging technique) if PB
encodings are not
self-describing and hence self-terminating? I raise this question
because that
appears to be a significant issue in this thread. And because I have
been banging
my head against this for years.

I don't think "self-describing" and "self-terminating" are related.  Having the description of a .proto file doesn't make it any more possible to determine where it ends.

It's easy to stick a length in front of a PB to determine where it ends.  This isn't included by default because it would be redundant in cases where a length is already stored separately (e.g. when a file contains a single protobuf, or when storing a protobuf in an std::string).

PBs are intentionally not self-describing because that would add overhead.  You can, however, easily pass a set of FileDescriptorProtos along with your message to describe it:

http://protobuf.googlecode.com/svn/trunk/src/google/protobuf/descriptor.proto

[gsxr]

On Jul 18, 9:34 am, "Kenton Varda" <ken...@google.com> wrote:

> On Thu, Jul 17, 2008 at 2:15 PM, gsxr <scott.suz...@gmail.com> wrote:
> > The domain that PB operates in is now clearer. Is it not making it
> > harder than
> > necessary (i.e. for chosen network messaging technique) if PB
> > encodings are not
> > self-describing and hence self-terminating? I raise this question
> > because that
> > appears to be a significant issue in this thread. And because I have
> > been banging
> > my head against this for years.
>
> I don't think "self-describing" and "self-terminating" are related. Having
> the description of a .proto file doesn't make it any more possible to
> determine where it ends.

We are just misunderstanding each other here.

>
> It's easy to stick a length in front of a PB to determine where it ends.
> This isn't included by default because it would be redundant in cases where
> a length is already stored separately (e.g. when a file contains a single
> protobuf, or when storing a protobuf in an std::string).
>
> PBs are intentionally not self-describing because that would add overhead.
> You can, however, easily pass a set of FileDescriptorProtos along with your
> message to describe it:
>

> http://protobuf.googlecode.com/svn/trunk/src/google/protobuf/descript...

OK. I can see that we are both coming from quite different directions.
I'm
feeling like I've jumped into someone elses thread so I'm happy to
duck
out at this point. The reason I jumped in was because it seemed to be
that others were asking for a PB encoding that was self-terminating.
Perhaps I misunderstood. It also seemed that if that attribute was
present
it would then be possible to implement network messaging quite
readily, i.e. without something like RPC. For me that potential was
compelling.

Cheers.

[Kenton Varda]

Thanks for pointing out your concerns.  It looks like we definitely need some better documentation about this.

[Matthew Newhook]

We thought the same thing! :)

Check out http://www.zeroc.com/labs/protobuf/index.html, and my blog
post http://zeroc.com/blogs/matthew/2008/07/19/google-protocol-buffers-integration/
for some thoughts on this. I'll write more on google protocol buffers
shortly.

Regards, Matthew

On Jul 15, 7:03 pm, Blair Zajac <bl...@orcaware.com> wrote:
> It would be interesting to mix the RPC layer of Ice with the serialization
> format of Protocol Buffers.  Ice has its own serialization mechanism, but it

...