Python Protocol Buffers vs Pickle

Joe K

unread,

Jul 10, 2008, 10:28:47 AM7/10/08

to Protocol Buffers

Hi,

How do Protocol Buffers compare in serialization/deserialization speed
to pickle (not cpickle)?

I'm looking for ways to persist structured data to the App Engine
datastore BlobProperty. Since it's App Engine cpickle isn't an
option.

I realize that protocol buffers has some advantages over pickle
already (from the docs):
1. it doesn't deal well with schema evolution
2. sharing data with application written in C++ or Java

The first one is a concern, the second one not so much. I'm sure I'll
eventually do my own performance testing, but was more curious if
speed optimization was an objective for the python implementation.

Thanks!

Joe

Christian Scholz / Tao Takashi (SL)

unread,

Jul 10, 2008, 12:33:02 PM7/10/08

to Joe K, Protocol Buffers

I mentioned ProtocolBuffers today here at EuroPython and somebody
mentioned that in the future the runtime also might use some C++
routines to do conversions. But then the question might be how it
compares against cpickle I guess ;-)

-- Christian

--
Christian Scholz
Tao Takashi (Second Life name)
taota...@gmail.com
Blog/Podcast: http://mrtopf.de/blog
Planet: http://worldofsl.com

Company: http://comlounge.net
Tech Video Blog: http://comlounge.tv
IRC: MrTopf/Tao_T

Kenton Varda

unread,

Jul 10, 2008, 12:42:49 PM7/10/08

to Joe K, Petar Petrov, Protocol Buffers

I don't think the Python code has been heavily-optimized. Petar (cc'd) may be able to say more about this. Long-term, we're toying with the idea of off-loading serialization to C++ for cases where speed is critical.

If you do performance tests, I'd be interested to hear the results.

Auke Booij

unread,

Jul 10, 2008, 12:59:36 PM7/10/08

to Protocol Buffers

last time i converted a python program to a C++ program (some
relatively simple looping), it only doubled the speed. just fyi.

On Jul 10, 6:42 pm, "Kenton Varda" <ken...@google.com> wrote:
> I don't think the Python code has been heavily-optimized. Petar (cc'd) may
> be able to say more about this. Long-term, we're toying with the idea of
> off-loading serialization to C++ for cases where speed is critical.
> If you do performance tests, I'd be interested to hear the results.
>

Petar Petrov

unread,

Jul 10, 2008, 1:49:45 PM7/10/08

to Kenton Varda, Joe K, Protocol Buffers

We are just beginning to optimize the python API and/or are even looking
for alternatives like wrapping the C++ API.

Right now it will probably be slower than cpickle.

Thanks!

Joe

Reply all

Reply to author

Forward