Rust vs. C++: performance and safety

[David Renshaw]

Hi everyone,
You may be interested in some posts I wrote comparing the C++ and Rust implementations of Cap'n Proto:

http://dwrensha.github.io/capnproto-rust/2013/11/16/benchmark.html
http://dwrensha.github.io/capnproto-rust/2013/11/17/lifetimes.html

[Kenton Varda]

Nice!

It would be cool to make graphs with Python as well, for comparison.

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.

[Jason Paryani]

Alright, I've been meaning to do this for a while. Writing the benchmarks now.

[Jason Paryani]

Well it seems I can't actually run capnproto-rust on my end. Does it require the latest rust-nightly (I'm only running v0.8)? For now, I've attached output from pycapnp's benchmarks, and here's a chart showing it off. As expected, python is about 20-50x slower than C++.

[Andreas Stenius]

Me too! :)

Wouldn't be nice to have a common benchmark repo, kind of the capnp-test repo, that implements a framework for executing the benchmarks in various implementations and compiling the results in a comprehensible format. That way it would ease new implementations to join in.

2013/11/18 Jason Paryani <jpar...@gmail.com>

[Kenton Varda]

Hmm, is this showing that protobuf with C extensions is actually slower than pure-python protobufs, or is the chart mislabeled?

[Jason Paryani]

Nope, not mislabeled. The protobuf cpp implementation actually runs slower for carsales and catrank, and faster for eval. Also, protobuf runs faster for eval than pycapnp, but I imagine it's from Cap'n Proto using a union and the extra processing that calling which() and doing a string comparison adds.

[Kenton Varda]

Hmm, the difference between pycapnp and pycapnp_packed on catrank seems suspicious given that the packing is done entirely in C++.

[Jason Paryani]

First, I think there is a legitimate slowdown to using packed bytes in python, because I'm not doing it particularly well at the moment. For now, I'm using pipe to write the byte stream and then reading it back into a python string, all using python APIs. Beyond being slower, this could cause problems if a message went over 65KB. I'll be changing it in a bit.

Also, I changed the benchmarks to use a shared seed, so variance should be less of a problem now. I forgot to mention before, but I reduced iterations by 10x from the C++ benchmark, since it was taking forever for python to finish. I also changed from charting wall_time to user_time. Code is up at https://github.com/jparyani/pycapnp/tree/develop/benchmark if you want to check it out.

[Kenton Varda]

Oh, yeah, these benchmarks can vary a lot depending on the seed.  Actually, instead of using random.random(), you should probably copy the RNG code from the C++ benchmark.  It's pretty simple, and it will hopefully mean you generate exactly the same data in Python as is being generated in C++.  Not that it's going to make a significant difference in the Python vs. C++ comparison, obviously, but keeping it consistent across all the languages we benchmark would be good.

-Kenton

[David Renshaw]

On Mon, Nov 18, 2013 at 3:15 AM, Jason Paryani <jpar...@gmail.com> wrote:

Well it seems I can't actually run capnproto-rust on my end. Does it require the latest rust-nightly (I'm only running v0.8)?

You can either use rust-nightly or you can checkout the master branch and build from source.

[Jason Paryani]

I fixed up my benchmarks to now look like the C++ Cap'n Proto benchmarks. I also have a script that will run any benchmarks that follow this form in https://github.com/jparyani/pycapnp/blob/develop/benchmark/bin/run_all.py, and dump all the results as json to stdout.

Also Kenton, I changed to using your rand function. It was a bit of a pain, since you're relying on uint32_t overflow, so I ended up just writing it in c/cython.

[Kenton Varda]

On Mon, Nov 18, 2013 at 1:25 PM, Jason Paryani <jpar...@gmail.com> wrote:

I fixed up my benchmarks to now look like the C++ Cap'n Proto benchmarks. I also have a script that will run any benchmarks that follow this form in https://github.com/jparyani/pycapnp/blob/develop/benchmark/bin/run_all.py, and dump all the results as json to stdout.

Nice work!

 

Also Kenton, I changed to using your rand function. It was a bit of a pain, since you're relying on uint32_t overflow, so I ended up just writing it in c/cython.

Can't you just write something like "n % (2 ** 32)" to implement modulo arithmetic?