Python C Implementation - Update Request

[Nicholas Reid]

Hi Petar & Kenton,

Have there been any recent developments on the Python C
implementation? I'd be more than happy to run a prototype through its
paces for testing purposes and provide feedback - it would be great to
have closer-to-C++ speeds in a Python environment.

Kind regards,

Nicholas Reid

[Kenton Varda]

Any comment, Petar?

[Daniel L]

Indeed. Python performance is quite low according to my tests.
Definitely workable for small objects, although very slow for larger
structures. When comparing to C++ or Apache Thrift python bindings on
the same datastructure (basically a 2D array of strings @ >10MB) we
saw several orders of magnitude difference, both serializing and
deserializing.

This was on Python 2.4. I'm sure it's faster on 2.6.

An alternate set of python bindings to the C++ interface would be a
lovely thing.

Cheers,
Daniel Lundin

[Kenton Varda]

Yeah, we've dropped the ball on this.  Unfortunately the people in charge of the Python protocol buffers implementation also have other, apparently very demanding projects, and thus Python protobuf hasn't gotten the attention it deserves.  Sadly even Google doesn't have infinite engineering resources.  I'm trying to find a solution to this -- it's a management issue, though, not a technical one.

[Jon Skeet <skeet@pobox.com>]

On Jun 2, 6:41 pm, Kenton Varda <ken...@google.com> wrote:
> Yeah, we've dropped the ball on this.  Unfortunately the people in charge of
> the Python protocol buffers implementation also have other, apparently very
> demanding projects, and thus Python protobuf hasn't gotten the attention it
> deserves.  Sadly even Google doesn't have infinite engineering resources.
>  I'm trying to find a solution to this -- it's a management issue, though,
> not a technical one.

One positive thing to note is that if I can get the dynamic stuff
working with the C# port (when I've got custom options to actually
work: there's a whole bunch of changes I missed before) it should be
usable from IronPython. I know that .NET 4.0 and IronPython will only
be useful to a subset of our Python "customers" but it may be faster
than a pure Python implementation. (I'd be interested to see.)

Jon

[Jon Skeet]

On Jun 2, 11:56 pm, "Jon Skeet <sk...@pobox.com>" <sk...@pobox.com>
wrote:

> One positive thing to note is that if I can get the dynamic stuff
> working with the C# port (when I've got custom options to actually
> work: there's a whole bunch of changes I missed before) it should be
> usable from IronPython. I know that .NET 4.0 and IronPython will only
> be useful to a subset of our Python "customers" but it may be faster
> than a pure Python implementation. (I'd be interested to see.)

Marc pointed out to me that of course any of the .NET implementations
should be usable from IronPython anyway. Note to self: don't post when
tired.

Jon

[Nicholas Reid]

Hi Kenton,

If you're in the process of reviewing accelerated Python support for
PB2, were you thinking about something the following approach?

1. Add another output format to protoc (say --pylib_out) which
generates both the existing C++ code plus whatever additional C++
necessary to create a compiled Python module from the specified .proto
file.

The kind of thing mentioned here: http://docs.python.org/extending/extending.html#a-simple-example.

2. Ideally, it would even be possible to automatically generate the
Python distutils configuration (http://docs.python.org/extending/
building.html#building) so that

$> python setup.py install

would compile and install the generated C module automatically.

I appreciate this probably means a lot of work, but it will give the
highest possible performance from within Python and may even surpass
Thrift's Python performance.

If you were able to maintain both the --python_out and the --pylib_out
options then users who need pure Python PB support (say for the App
Engine etc) can use one and performance critical applications can use
the other.

Would it also be a good time to look at what would need to be changed
to support Python 3.0?

Cheers,

Nicholas Reid

On Jun 2, 6:41 pm, Kenton Varda <ken...@google.com> wrote:

> Yeah, we've dropped the ball on this.  Unfortunately the people in charge of
> the Python protocol buffers implementation also have other, apparently very
> demanding projects, and thus Python protobuf hasn't gotten the attention it
> deserves.  Sadly even Google doesn't have infinite engineering resources.
>  I'm trying to find a solution to this -- it's a management issue, though,
> not a technical one.
>

[Kenton Varda]

The plan is to modify the current Python implementation so that message objects can wrap C++ protobuf objects via the reflection interface.  This means a few things:

* The Python API will be exactly the same (except for setting an option saying that you want to do this wrapping).

* If we wrap DynamicMessage, then you should not have to compile any C/C++ code specific to your message type at all.  So, once the protocol buffer extension itself is installed, users will not have to install their own extensions.

* There would also be the option of wrapping generated C++ classes specific to your message types, which would squeeze out a few more drops of performance at the expense of having to compile C++ code.