Handling streaming in Python

[Dan]

Hello,

Does anyone have or can point me to an example of handling a stream of
protobuf messages in python?

Specifically what I'm trying to do is parse a stream that looks like
the following

<HeaderSize PB int32>
<HeaderMessage>
<MessageSize PB int32>
<Message>
...
<MessageSize PB int32>
<Message>

The header just gives the list of field names and their types (int,
double, or string, that's it). There are no nested messages or
anything like that in the streamed messages.

Ultimately what I would like to do is put the values from a message
into a list and pass it over to some other code that writes the data
into another format (csv, xml, json, etc.)

I've got this handled wonderfully using CodedInputStream in the Java
implementation but now I need to do it in python and I don't see
anything equivalent.

[Kenton Varda]

Unfortunately the Python library doesn't provide a nice interface for this.  You have to encode/decode the leading varints by hand.  However, the encoding is pretty simple, so we're talking about maybe a 5-line helper function.

http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints

I would like to improve this at some point, but there's so much code to write and so little time...

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[Gregory Szorc]

You can load the _DecodeVarint and _VarintEncoder modules from
google.protobuf.internal.{decoder,encoder} if you want to risk using an
internal interface. See
https://github.com/indygreg/zippylog/blob/master/lib/py/zippylog/stream.py
for an example (although I can't guarantee the Python code in that project
works at this time).

Greg

Hello,

--

[Kenton Varda]

It's better to copy the functions into your own code than to depend on internal interfaces.

[Dan]

Yep, that's what I'm gonna do. Copy over the VarintDecoder and tweak
it to read from a stream instead of a buffer, thanks.

On Dec 1, 3:15 am, Kenton Varda <ken...@google.com> wrote:
> It's better to copy the functions into your own code than to depend on
> internal interfaces.
>

> On Tue, Nov 30, 2010 at 10:02 PM, Gregory Szorc <gregory.sz...@gmail.com>wrote:
>
> > You can load the _DecodeVarint and _VarintEncoder modules from
> > google.protobuf.internal.{decoder,encoder} if you want to risk using an
> > internal interface. See

> >https://github.com/indygreg/zippylog/blob/master/lib/py/zippylog/stre...an example (although I can't guarantee the Python code in that project

> > protobuf+u...@googlegroups.com<protobuf%2Bunsu...@googlegroups.com>
> > .

> > For more options, visit this group at
> >http://groups.google.com/group/protobuf?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to prot...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > protobuf+u...@googlegroups.com<protobuf%2Bunsu...@googlegroups.com>
> > .