speed - python implementation

[andres]

Hi,

I would like to use protocol buffers in my python code but currently
the serialization and parsing methods are too slow compared to
cPickle. I've read several posts stating that this is because the
python implementation has not been optimized for speed yet. Are there
plans to improve the performance of proto buffers in python? Does
anybody know of a C++ extension/wrapper module which lets you access C+
+ compiled protocol buffers directly from python code?

Thanks,
Andres

[Kenton Varda]

[+petar]

[Jeremy Leader]

Might it be possible to use the XS wrappers generated by protobuf-perlxs
from Python?

--
Jeremy Leader
jle...@oversee.net

[Jeremy Leader]

Jeremy Leader wrote:
> Might it be possible to use the XS wrappers generated by protobuf-perlxs
> from Python?

Aaah, not enough caffeine yet. I somehow confused XS (Perl-specific)
with SWIG (supports Perl, Python, and many others). Never mind!

--
Jeremy Leader
jle...@oversee.net

[Petar Petrov]

On Mon, Oct 27, 2008 at 10:47 AM, andres <andres...@gmail.com> wrote:

Hi,

I would like to use protocol buffers in my python code but currently
the serialization and parsing methods are too slow compared to
cPickle. I've read several posts stating that this is because the
python implementation has not been optimized for speed yet. Are there
plans to improve the performance of proto buffers in python?

Yes, there are plans to improve performance. I have spent a little time on this without significant improvements.
I think performance can hardly get a drastic improvement without a C++ extension module (which we are planning to have).

[codeazure]

On Oct 31, 5:19 am, "Petar Petrov" <pesho.pet...@gmail.com> wrote:
> Yes, there are plans to improve performance. I have spent a little time on
> this without significant improvements.
> I think performance can hardly get a drastic improvement without a C++
> extension module (which we are planning to have).

Are you aware of anyone doing any work on a C++ Boost::Python
interface for PB? This would seem to be a relatively easy thing to
write, implementing the __getattr__/__setattr__ Python methods in
Boost::Python to interface to the reflection mechanism in PB.

If noone else is doing it, I might try this myself & pass it on if it
works.

Regards,
Jeff

[Petar Petrov]

On Sun, Nov 9, 2008 at 5:14 PM, codeazure <code...@gmail.com> wrote:

On Oct 31, 5:19 am, "Petar Petrov" <pesho.pet...@gmail.com> wrote:
> Yes, there are plans to improve performance. I have spent a little time on
> this without significant improvements.
> I think performance can hardly get a drastic improvement without a C++
> extension module (which we are planning to have).

Are you aware of anyone doing any work on a C++ Boost::Python
interface for PB?

No, we aren't aware of such.
 

This would seem to be a relatively easy thing to
write, implementing the __getattr__/__setattr__ Python methods in
Boost::Python to interface to the reflection mechanism in PB.

A few things. The current Python API has to remain pure-Python because some clients aren't able to use C/C++ extensions (like AppEngine).
Boost is generally not accepted in Google, so a Boost::Pythonit interface will have to distribute separately.

We are planning a Python C extension. It will likely consist of a separate python code generator to create Python code which wraps the C++
API and provides Python API similar to the current pure-Python protobuf API.

[daveb]

you could consider wrapping protobuf-c... that will at least save you
the hassle of writing the C wrapper around C++.

- dave

On Nov 12, 10:04 am, "Petar Petrov" <pesho.pet...@gmail.com> wrote:

[codeazure]

On Nov 13, 5:04 am, "Petar Petrov" <pesho.pet...@gmail.com> wrote:
> A few things. The current Python API has to remain pure-Python because some
> clients aren't able to use C/C++ extensions (like AppEngine).
> Boost is generally not accepted in Google, so a Boost::Pythonit interface
> will have to distribute separately.
>
> We are planning a Python C extension. It will likely consist of a separate
> python code generator to create Python code which wraps the C++
> API and provides Python API similar to the current pure-Python protobuf API.

Thanks for your feedback. I can see that for the official Python
support, this makes a lot of sense. You want to make it possible for
as wide a group of users as possible to use it. I'll look forward to
seeing how this development goes.

But I will do some experimentation with Boost::Python as this is a
much more convenient tool for C++ developers. If it works out, I will
make it available for others doing similar development. My current
application uses a very large Python::Boost definition defining my
API, with every variable and function explicitly defined. By changing
it to use a generic Message interface with reflection, I can
drastically simplify this Python/C++ gateway.

One of the things I like about PB is its portability and this is a
great example of how different solutions can be useful in various
situations.

Regards,
Jeff