What's the point of Protocol Buffers?

[Tim Acheson]

I generally create web services using WCF or ASP.NET MVC. I don't get
the point of "Protocol Buffers". Am I missing something?

Out of the box, WCF web services and ASP.NET MVC actions serialise my
objects to JSON or XML, using the serialisation libraries provided by
the framework. I don't need to do anything to achieve "encoding
structured data in an efficient yet extensible format" -- I just
define my objects as normal and the .NET framework does everything for
me.

I don't need to write any code to do the serialisation, either. I just
define the return type of the web method in my WCF project, or define
an ASP.NET MVC Action that returns the object. The framework does the
rest.

Also, I rarely come accross a web service that returns anything other
than strings, 32-bit integers and booleans. If I did, I'd probably
question the architecture.

Perhaps somebody could explain why I would want or need to use
Protocol Buffers?

Thanks! :)

[Kenton Varda]

1.  What happens when you need to read/write your messages in Java?  You'd either need to rewrite all your classes or work with ugly generic JSON or XML parse trees.

2.  Protobuf encoding and decoding is much, much faster than JSON or XML, and the encoded messages are much smaller, particularly for non-textual data (numbers, binary blobs, etc.).

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[Daniel Wright]

On Wed, Jul 21, 2010 at 11:04 AM, Kenton Varda <ken...@google.com> wrote:

1.  What happens when you need to read/write your messages in Java?  You'd either need to rewrite all your classes or work with ugly generic JSON or XML parse trees.

And even if you don't want to use Java/C++/Python/whatever, what if someone else wants to work with your system in another language?  What if after they've written some code to work with it, you decide to add some new fields?  Do you log your requests to disk?  How about writing a tool to analyze the logs? etc.

[David Yu]

On Thu, Jul 22, 2010 at 2:04 AM, Kenton Varda <ken...@google.com> wrote:

1.  What happens when you need to read/write your messages in Java?  You'd either need to rewrite all your classes or work with ugly generic JSON or XML parse trees.

2.  Protobuf encoding and decoding is much, much faster than JSON or XML, and the encoded messages are much smaller, particularly for non-textual data (numbers, binary blobs, etc.).

   3.  Schema evolution supported out-of-the-box (backward compatibility) 

--
When the cat is away, the mouse is alone.
- David Yu

[Tim Acheson]

The consensus seems to be that the main rationale for Protocol Buffers
is cross-platform interoperability, and transferring objects between
platforms.

I can already consume JSON objects from a WCF service using JavaScript
in the web browser. I can see that if I used other platforms/languages
I would need to keep schemas in synch between server and client apps,
and it seems Protocol Buffers can help with that. (Incidentally, SOAP
is an excellent way of keeping client applications in synch with
objects and RPC methods on the server/service, and a WCF web service
provides SOAP as well as REST, but I think SOAP is beyond the scope of
this discussion.)

[sheila miguez]

On Thu, Jul 22, 2010 at 3:01 AM, Tim Acheson <tim.a...@gmail.com> wrote:
> The consensus seems to be that the main rationale for Protocol Buffers
> is cross-platform interoperability, and transferring objects between
> platforms.
>
> I can already consume JSON objects from a WCF service using JavaScript
> in the web browser. I can see that if I used other platforms/languages

[...]

But you've left out the 2nd point of Kenton's response about the speed
of serialization compared to json as well as the compactness of the
serialized objects. That was one reason we decided to go with
protobufs.

--
sheila

[Tim Acheson]

If you can show me a format which offers faster serialisation or
deserialisation than JSON in a .NET application, I'd be impressed! :)
Although I haven't heard anybody experiencing problems with the
performance of either direction in .NET with JSON or XML, the
libraries provided by the framework offer excellent performance in
both directions. :)

[Henner Zeller]

On Thu, Jul 22, 2010 at 07:50, Tim Acheson <tim.a...@gmail.com> wrote:
> If you can show me a format which offers faster serialisation or
> deserialisation than JSON in a .NET application, I'd be impressed! :)

If the speed and size of JSON serialization is good enough for you,
stick with it.

> Although I haven't heard anybody experiencing problems with the
> performance of either direction in .NET with JSON or XML, the
> libraries provided by the framework offer excellent performance in
> both directions. :)
>

[David Yu]

On Thu, Jul 22, 2010 at 10:50 PM, Tim Acheson <tim.a...@gmail.com> wrote:

If you can show me a format which offers faster serialisation or
deserialisation than JSON in a .NET application, I'd be impressed! :)

Not for .NET there isn't.   You might wanna try benchmarking it yourself against http://code.google.com/p/protobuf-csharp-port/ (by a googler)

Although I haven't heard anybody experiencing problems with the
performance of either direction in .NET with JSON or XML, the
libraries provided by the framework offer excellent performance in
both directions. :)

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

--

[Liam]

Another feature that protocol buffers offers is the ability to easily
serialize arbitrary binary data. If you were to use JSON or XML, you
would need to encode the data somehow. Base64 is usually used for
this, which of course will represent the data as a string. I noted

that you said:
"Also, I rarely come accross a web service that returns anything other
than strings, 32-bit integers and booleans. If I did, I'd probably
question the architecture."

Have you really never needed anything else? How do you transfer
images or sound files?

On Jul 22, 11:21 am, David Yu <david.yu....@gmail.com> wrote:

> On Thu, Jul 22, 2010 at 10:50 PM, Tim Acheson <tim.ache...@gmail.com> wrote:
> > If you can show me a format which offers faster serialisation or
> > deserialisation than JSON in a .NET application, I'd be impressed! :)
>
> Not for .NET there isn't.   You might wanna try benchmarking it yourself

> againsthttp://code.google.com/p/protobuf-csharp-port/(by a googler)

>
>
>
> > Although I haven't heard anybody experiencing problems with the
> > performance of either direction in .NET with JSON or XML, the
> > libraries provided by the framework offer excellent performance in
> > both directions. :)
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to prot...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > protobuf+u...@googlegroups.com<protobuf%2Bunsu...@googlegroups.com>
> > .

[Marc Gravell]

If you can show me a format which offers faster serialisation or deserialisation than JSON in a .NET application, I'd be impressed! :)

In .NET? Sure: vs which json engine? I'm pretty certain that protobuf-net (especially v2, but probably v1) can thrash it.

This is "v1", which is quite a bit slower than the unreleased "v2":

http://www.servicestack.net/benchmarks/NorthwindDatabaseRowsSerialization.1000000-times.2010-02-06.html

so 8 times faster than NewtonSoft.Json and 10 times faster than Microsoft JsonDataContractSerializer. Smaller output too. A shame there aren't numbers there for JavaScriptSerializer, but the metrics aren't mine.

Marc

--
Regards,

Marc

[cmichael]

>    3.  Schema evolution supported out-of-the-box (backward compatibility)

4. Also, because of the schema, protobufs are strongly typesafe, and
JSON is not.

5. Using the APIs makes programs easier to understand and debug,
versus libraries that support object serialization as an afterthought,
because:
- Unlike a complex class where serialization can have side effects,
you always know how a proto will behave, you know that it will always
serialize (very useful for debugging).
- Much like the MVC model, it actually forces a separation of the
Model (the protobuf) from the Controller (the program using the
protobuf). You can achieve this with any number of methods other than
protobufs, but since the question is why should someone use protobufs
instead of latent serialization, this seems pertinent. Model/
Controller separation just gives you better code, which is why I would
probably use protobufs even if I was writing a program that did very
little I/O with protobufs.

[Timothy Parez]

Hi,

The reason we use it is because we don't just develop software but also hardware solutions.

Hardware solutions which are connected through GPRS or even RS232 connections.

GPRS is slow and in most cases you pay for the amount of data your send,

so we have to keep the packages as small as  possible.

RS232 doesn't work well with large packets, so again size is very important.

Web Services, REST, SOAP, ... they are all very verbose... to expensive/large for our needs.

If you need data to be as small as possible, protocol buffers are a good option.

Timothy

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.

[Timothy Parez]

Also note,

while your computer might have 4 cores running at 2.5Ghz the hardware I'm talking about

has 1 core, runs at 100Mhz and that's it... processing XML on devices like that... a real pain in the ...

But I have to admit, when I write computer to computer software, I go REST or SOAP all the way

[Christopher Smith]

There is also the other end of the spectrum, where I am at. With a
large Hadoop cluster and terabytes of data, the efficient storage and
zippy parsing of protobufs is a huge deal.

In many ways, protobufs allow you do do what XML promised, but much
more efficiently. The other way to look at them is they are ASN.1
reduced to the simplest usefull feature set.

--Chris

On Jul 23, 4:43 pm, Timothy Parez <timothypa...@gmail.com> wrote:
> Also note,
>
> while your computer might have 4 cores running at 2.5Ghz the hardware I'm
> talking about
> has 1 core, runs at 100Mhz and that's it... processing XML on devices like
> that... a real pain in the ...
>
> But I have to admit, when I write computer to computer software, I go REST
> or SOAP all the way
>

> On Sat, Jul 24, 2010 at 1:40 AM, Timothy Parez <timothypa...@gmail.com>wrote:
>
>
>
> > Hi,
>
> > The reason we use it is because we don't just develop software but also
> > hardware solutions.
> > Hardware solutions which are connected through GPRS or even RS232
> > connections.
>
> > GPRS is slow and in most cases you pay for the amount of data your send,
> > so we have to keep the packages as small as  possible.
>
> > RS232 doesn't work well with large packets, so again size is very
> > important.
>
> > Web Services, REST, SOAP, ... they are all very verbose... to
> > expensive/large for our needs.
>
> > If you need data to be as small as possible, protocol buffers are a good
> > option.
>
> > Timothy
>

> > On Wed, Jul 21, 2010 at 12:57 PM, Tim Acheson <tim.ache...@gmail.com>wrote:
>
> >> I generally create web services using WCF or ASP.NET MVC. I don't get
> >> the point of "Protocol Buffers". Am I missing something?
>
> >> Out of the box, WCF web services and ASP.NET MVC actions serialise my
> >> objects to JSON or XML, using the serialisation libraries provided by
> >> the framework. I don't need to do anything to achieve "encoding
> >> structured data in an efficient yet extensible format" -- I just
> >> define my objects as normal and the .NET framework does everything for
> >> me.
>
> >> I don't need to write any code to do the serialisation, either. I just
> >> define the return type of the web method in my WCF project, or define
> >> an ASP.NET MVC Action that returns the object. The framework does the
> >> rest.
>
> >> Also, I rarely come accross a web service that returns anything other
> >> than strings, 32-bit integers and booleans. If I did, I'd probably
> >> question the architecture.
>
> >> Perhaps somebody could explain why I would want or need to use
> >> Protocol Buffers?
>
> >> Thanks! :)
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Protocol Buffers" group.
> >> To post to this group, send email to prot...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> protobuf+u...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c om>
> >> .

[Oliver Jowett]

Christopher Smith wrote:
> There is also the other end of the spectrum, where I am at. With a
> large Hadoop cluster and terabytes of data, the efficient storage and
> zippy parsing of protobufs is a huge deal.
>
> In many ways, protobufs allow you do do what XML promised, but much
> more efficiently. The other way to look at them is they are ASN.1
> reduced to the simplest usefull feature set.

Amusingly enough, we use protocol buffers to transport ASN.1-encoded
data (SS7 TCAP messages). The protobuf API is far better than the API
produced by the commercial ASN.1 compiler we use.

-O

[Christopher Smith]

Yes, one of the nice things about simplicity is it makes it easier to do the few things you do do well. Sometimes we developers tend to forget this. ;-) 

--
Chris

[gsxr]

On Jul 21, 10:57 pm, Tim Acheson <tim.ache...@gmail.com> wrote:
> I generally create web services using WCF or ASP.NET MVC. I don't get
> the point of "Protocol Buffers". Am I missing something?
>

<snip>

> I don't need to write any code to do the serialisation, either. I just
> define the return type of the web method in my WCF project, or define
> an ASP.NET MVC Action that returns the object. The framework does the
> rest.
>
> Also, I rarely come accross a web service that returns anything other
> than strings, 32-bit integers and booleans. If I did, I'd probably
> question the architecture.


>
> Perhaps somebody could explain why I would want or need to use
> Protocol Buffers?

Your question is constrained, i.e. to WCF and .NET. The simple answer
is "you dont need to".

If your entire world is truly constrained in this way then I might
question your
architecture.

OK, that's being flip. What I really mean is that your question might
make PB look unsuccessful when it patently isnt. There is no
technology
in the .NET framework that delivers the same service across the
languages
and platforms that PB does.

If you can pull off all your development within the WCF+.NET island
then
not using PB is (perhaps) an optimal decision.

S.

[Marc Gravell]

And of course, if you *are* happy using WCF and .NET, with all the simplicity that provides, you can *still* use protobuf, since protobuf-net includes a WCF hook that (if you desire) swaps-out the serializer used by WCF. So your *existing* .NET code (bereft of any .proto definition) can enjoy faster, more compact data transfer. Sure, it'll still be in the middle of a WCF packet, but that may give the appropriate mix of protocol services for your scenario.

Marc

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

--
Regards,

Marc

[RogerV]

On Jul 21, 3:57 am, Tim Acheson <tim.ache...@gmail.com> wrote:
> I generally create web services using WCF or ASP.NET MVC. I don't get
> the point of "Protocol Buffers". Am I missing something?

It'd be a similar set of reasons as to why we abandoned XML and JSON
in favor of AMF when marshaling i/o interactions between our Flex RIA
clients and our Java middle-tier.

We don't use plain vanilla simpleton web service calls that just
return a string or a number. Our RIA MVC clients request objects from
the middle-tier and/or object graphs get pushed to them via BlazeDS
Comet.

We needed efficient object graph marshaling that is compact and
performant even when returning very large result sets.

Protocol Buffers addresses that same kind of scenario but with
marshaling standard that is more universal than Adobe's AMF for the
Flex/Flash Player.

Indeed, as we're doing some C# .NET middle-tier interacting with Adobe
Flex RIA client tier, we'll use Protocol Buffers for .NET and Flex bi-
directional object graph marshaling.

Likewise for our middle-tier interacting with our object database.
We'll transition this over to using Protocol Buffers to marshal the
object graphs (.NET interacting bi-directionally with C++).