Speed comparison

[robertiannucci]

I'm working on a project into which protobuf fits perfectly (or so it
would seem). Unfortunately, the project generates some unsightly data
files (in protobuf format, they are about 3MB). Now, a 3MB file is a
bit of a large 'message', but I would think that it should parse in a
reasonable amount of time (around 5 seconds). I ran a really quick
test on this file with the python, cpp, and java targets. The test
program was roughly:

// Begin Code
from log_pb2 import Log
log = Log()
f = open('testfile')
log.ParseFromString(f.read())
print log
// End Code

The results of running this under C++, Java and Python using the
'time' command were:

C++:
real 0m1.440s
user 0m0.982s
sys 0m0.150s

Java:
real 0m3.326s
user 0m2.629s
sys 0m0.248s

Python:
real 0m57.449s
user 0m55.528s
sys 0m0.881s

Now the C++ and Java results I completely understand. I was expecting
something a bit slower for python, but this is really bad... I also
noticed that Java took ~138mb peak for the 3MB logfile, and python
took over 200MB.

I'm running Python 2.5.2 on Leopard, so I don't expect that it's
anything particularly screwy for my configuration. Is the parsing
algorithm for python actually different than for Java/C++? I'm a bit
of a loss to describe this behavior. Can anyone else comment? I'm
going to go dig through the source now, to see if I can find
something.

However, I must say, this protocol is much more slick than XML/
XMLSchema/etc.. I look forward to working with it... I'm thinking
about implementing a 'protoPath' system analogous to XPath. We'll see
what happens :-).

Robbie

[Kenton Varda]

The Python implementation is considerably less mature than C++ or Java.  It was actually sort of rushed out the door because we got sick of delaying the release.  I don't think any optimization work has been done yet, so there is probably a lot of low-hanging fruit.  Sorry, this fact should probably be documented better.

We do hope to make it a lot faster in the future, though!  Petar, any comments?

BTW, did you use optimize_for = SPEED in your test?  I'd actually expect C++ and Java to be considerably faster.  Make sure to add this line to your .proto file:

  option optimize_for = SPEED;

(The default is to optimize for code size, which we've discovered is often a problem with generated code.)

[Kenton Varda]

(Note that optimize_for does not affect Python due to the Python implementation's differing design.)

[robertiannucci]

Ok that makes sense :-).

I hope to run the python version through some profiling soon, so
hopefully I'll be able to find any particularly sticky bits and point
them out/submit patches for them.

Yes, this was using the option optimize_for = SPEED;. I didn't time w/
o it, but it seemed to take about the same amount of time, as I
recall.

Thanks!
Robbie

On Jul 11, 4:49 pm, "Kenton Varda" <ken...@google.com> wrote:
> The Python implementation is considerably less mature than C++ or Java.  It
> was actually sort of rushed out the door because we got sick of delaying the
> release.  I don't think any optimization work has been done yet, so there is
> probably a lot of low-hanging fruit.  Sorry, this fact should probably be
> documented better.
> We do hope to make it a lot faster in the future, though!  Petar, any
> comments?
>
> BTW, did you use optimize_for = SPEED in your test?  I'd actually expect C++
> and Java to be considerably faster.  Make sure to add this line to your
> .proto file:
>   option optimize_for = SPEED;
> (The default is to optimize for code size, which we've discovered is often a
> problem with generated code.)
>

> On Fri, Jul 11, 2008 at 4:19 PM, robertiannucci <viperph...@gmail.com>

[Kenton Varda]

On Fri, Jul 11, 2008 at 5:27 PM, robertiannucci <viper...@gmail.com> wrote:

Ok that makes sense :-).

I hope to run the python version through some profiling soon, so
hopefully I'll be able to find any particularly sticky bits and point
them out/submit patches for them.

Sounds great!

 

Yes, this was using the option optimize_for = SPEED;.  I didn't time w/
o it, but it seemed to take about the same amount of time, as I
recall.

In that case I think your test was I/O bound.  optimize_for = SPEED should be significantly faster.  You should try loading the whole file into a flat array in memory and parsing that.  :)

[Petar Petrov]

On Fri, Jul 11, 2008 at 4:49 PM, Kenton Varda <ken...@google.com> wrote:

The Python implementation is considerably less mature than C++ or Java.  It was actually sort of rushed out the door because we got sick of delaying the release.  I don't think any optimization work has been done yet, so there is probably a lot of low-hanging fruit.  Sorry, this fact should probably be documented better.

We do hope to make it a lot faster in the future, though!  Petar, any comments?

There are a lot of work items for the Python API. Speed optimization is one of them.

[robertiannucci]

Loading the file into a string in memory first doesn't alleviate the
problem... it's only 3MB after all ;-)

Robbie

On Jul 11, 5:33 pm, "Kenton Varda" <ken...@google.com> wrote:
> On Fri, Jul 11, 2008 at 5:27 PM, robertiannucci <viperph...@gmail.com>

[Jon Skeet]

On Jul 12, 12:19 am, robertiannucci <viperph...@gmail.com> wrote:
> I'm working on a project into which protobuf fits perfectly (or so it
> would seem). Unfortunately, the project generates some unsightly data
> files (in protobuf format, they are about 3MB). Now, a 3MB file is a
> bit of a large 'message', but I would think that it should parse in a
> reasonable amount of time (around 5 seconds). I ran a really quick
> test on this file with the python, cpp, and java targets. The test
> program was roughly:

<snip>

Thanks for doing this - I'll let others explain what's going on, as
Kenton seems to have done already.

However, it does suggest to me something that would be really useful:
a benchmarking suite. With all the normal caveats about
microbenchmarks, it would be really nice to have a good set of tests
which can easily be ported to new platforms/languages. Obviously I
have an ulterior motive for this: I would like an easy way of finding
out how my .NET code behaves, when I've got the first version written.

I don't feel qualified to really suggest a good list of test cases,
and I can't contribute significant time to such an effort while I'm
trying to get the .NET stuff out, so this post is really just a matter
of encouraging someone else :)

Jon

[robertiannucci]

That would probably be a good idea.. It seems like an algorithmic
complexity thing somewhere in the reflection system (which,
apparently, is where ALL the code for the python version is). It
behaves just fine for small files, which makes me think that there's
something like O(N^2)-type algorithm somewhere where the other
versions use an O(N) algorithm. I don't believe the slowness is
inherent to Python. I'm trying to compare the versions to find out
the core algorithms used, but it's somewhat difficult because C++
dosn't have the extensive built-in reflection that java does, which,
in turn, isn't anywhere near as dynamic as Python, so it seems that
the core algorithms actually differ per implementation.

That said, yes, it would be quite useful to have a set of benchmarks,
and it would also be good to have some developers notes to outline the
chosen algorithms and their expected runtimes, per implementation.
Both of these would help greatly for developing new language
interfaces. I can draft up some proto's to use as tests, but my
knowledge of protobuffers is still somewhat limited, of course :-).

Robbie

[Kenton Varda]

On Sat, Jul 12, 2008 at 1:21 AM, robertiannucci <viper...@gmail.com> wrote:

Loading the file into a string in memory first doesn't alleviate the
problem... it's only 3MB after all ;-)

I meant that if you do that in C++ or Java, then measure just how long it takes to parse after it's been loaded, I think it will be much faster than your original measurements.  :)

But Python won't be fixed by this, no.

[Kenton Varda]

I have an internal benchmark suite, but unfortunately it is based on internal formats which I'm not at liberty to share.  I'm not sure how to create a realistic test case that could be released publicly, other than to wait for other open source projects (Google or otehrwise) to start using protobufs and then take one of their formats.  Or maybe I could obfuscate one of my test cases by replacing all the data with randomized data that should take the same amount of time to parse (e.g. randomize string contents but keep the same length)...

[Jon Skeet]

On Jul 12, 11:29 pm, "Kenton Varda" <ken...@google.com> wrote:
> I have an internal benchmark suite, but unfortunately it is based on
> internal formats which I'm not at liberty to share.  I'm not sure how to
> create a realistic test case that could be released publicly, other than to
> wait for other open source projects (Google or otehrwise) to start using
> protobufs and then take one of their formats.  Or maybe I could obfuscate
> one of my test cases by replacing all the data with randomized data that
> should take the same amount of time to parse (e.g. randomize string contents
> but keep the same length)...

The latter sounds promising. It may be worth constructing artificial
test cases as well, e.g. just lots of enums, just lots of floats etc.
Apologies for not volunteering to do this myself... if I get the C#
stuff into a suitable state before anyone else takes it on, I may have
a go - with suitable advice from more experienced ProtoBuffer users :)

Jon