[Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

[Victor Stinner]

(Re-post without the two attached files of 100 KB each.)

Hi,

You may know that I'm working on benchmarks. I regenerated all
benchmark results of speed.python.org using performance 0.3.2
(benchmark suite). I started to analyze results.

All results are available online on the website:

https://speed.python.org/


To communicate on my work on benchmarks, I tweeted two pictures:

"sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python
2.7 #python #benchmark":
https://twitter.com/VictorStinner/status/794289596683210760

"Python 3.6 is between 25% and 54% slower than Python 2.7 in the
following benchmarks":
https://twitter.com/VictorStinner/status/794305065708376069


Many people were disappointed that Python 3.6 can be up to 54% slower
than Python 2.7. In fact, I know many reasons which explain that, but
it's hard to summarize them in 140 characters ;-)

For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark
pycrypto_aes. This benchmark tests a pure Python implementation of the
crypto cipher AES. You may know that CPython is slow for CPU intensive
functions, especially on integer and floatting point numbers.

"int" in Python 3 is now "long integers" by default, which is known to
be a little bit slower than "short int" of Python 2. On a more
realistic benchmark (see other benchmarks), the overhead of Python 3
"long int" is negligible.

AES is a typical example stressing integers. For me, it's a dummy
benchmark: it doesn't make sense to use Python for AES: modern CPUs
have an *hardware* implemention which is super fast.


Well, I didn't have time to analyze in depth individual benchmarks. If
you want to help me, here is the source code of benchmarks:
https://github.com/python/performance/blob/master/performance/benchmarks/


Raw results of Python 3.6 compared to Python 2.7:
-------------------
$ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5
Slower (40):
- python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower
- python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower
- unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower
- call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower
- call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower
- call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower
- crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower
- xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower
- logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower
- logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower
- pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower
- spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower
- logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower
- chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower
- go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower
- xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower
- sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower
- xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower
- django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower
- fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower
- hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower
- chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower
- regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower
- json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower
- nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower
- genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower
- raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower
- scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower
- scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower
- deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower
- sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower
- call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower
- scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower
- meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower
- pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower
- richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower
- genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower
- float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower
- scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms:
1.07x slower
- xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower

Faster (15):
- telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster
- unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster
- pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster
- json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster
- pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster
- sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster
- sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster
- regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster
- sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster
- regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster
- mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster
- html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster
- sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster
- pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster
- scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster

Benchmark hidden because not significant (8): 2to3, dulwich_log,
nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle
Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json:
hg_startup, pyflate, spambayes
-------------------

Please ignore call_method, call_method_slots, call_method_unknown
benchmarks: it seems like I had an issue on the benchmark server. I
was unable to reproduce he 70% slowdown on my laptop.

JSON files if you want to analyze them yourself:

http://www.haypocalc.com/tmp/2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
http://www.haypocalc.com/tmp/2016-11-03_15-38-3.6-c4319c0d0131.json.gz

I hope that my work on benchmarks will motive some developers to look
closer at Python 3 performance to find interesting optimizations ;-)

Victor
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Victor Stinner]

The nice thing with benchmarks is that you can always take a subset of
numbers to prove something :-) Another subset where Python 3.6 is
faster:

"Python 3.6 is between 12% (1.14x) and 97% (32x) FASTER than Python

2.7 in the following benchmarks:"

https://twitter.com/VictorStinner/status/794525289623719937

Victor

[Victor Stinner]

I attached the two compressed JSON files to this email if you want to
analyze them yourself.

[Miquel Torres]

Nice! For the record, I'll be giving a talk in PyCon Ireland about Codespeed. Would you mind me citing those tweets and screenshots, to highlight usage on speed.python.org?

You mentioned new more reliable vs old results. How close are we to have an stable setup that gives us benchmarks numbers regularly?

Cheers,
Miquel

_______________________________________________
Speed mailing list
Sp...@python.org
https://mail.python.org/mailman/listinfo/speed

[Victor Stinner]

2016-11-04 20:18 GMT+01:00 Miquel Torres <tob...@gmail.com>:
> Nice! For the record, I'll be giving a talk in PyCon Ireland about
> Codespeed. Would you mind me citing those tweets and screenshots, to
> highlight usage on speed.python.org?

Sure. Keep me in touch in you publish your slides later.

> You mentioned new more reliable vs old results. How close are we to have an
> stable setup that gives us benchmarks numbers regularly?

My plan for the short term is to analyze last (latest?) benchmarks
hiccups and try to fix them.

The fully automated script to run benchmarks is already written:
https://github.com/python/performance/tree/master/scripts

Then, the plan we decided with Zachary Ware is to run a script in a
loop which compiles the default branch of CPython. Later, we may also
do the same for 2.7 and 3.6 branches. And then add PyPy (and PyPy 3).

[Antoine Pitrou]

Hi Victor,

On Fri, 4 Nov 2016 13:53:10 +0100
Victor Stinner <victor....@gmail.com> wrote:
>
> Raw results of Python 3.6 compared to Python 2.7:

That's interesting, but I would be personally more interested in
a performance comparison of 3.5 and 3.6, to know if anything
interesting (or worrying :-)) has happened there.

The performance differences between 2.7 and 3.x are quite well-known by
now, and none of them are really dramatic except for the increase in
startup time.

Regards

Antoine.

[Wolfgang Maier]

On 05.11.2016 10:56, Antoine Pitrou wrote:
>
> Hi Victor,
>
> On Fri, 4 Nov 2016 13:53:10 +0100
> Victor Stinner <victor....@gmail.com> wrote:
>>
>> Raw results of Python 3.6 compared to Python 2.7:
>
> That's interesting, but I would be personally more interested in
> a performance comparison of 3.5 and 3.6, to know if anything
> interesting (or worrying :-)) has happened there.
>

You can get this as well from https://speed.python.org/comparison/
and https://speed.python.org/timeline and looking at this, I think there
is something worrying indeed:
Startup time has increased by ~ 30 % between 3.5 and 3.6 again. More
specifically, all this increase happened between Sep 09 and Sep 15.

I have no clue why that is, but it is definitely the biggest effect far
and wide.

[Guido van Rossum]

Re: https://speed.python.org/timeline/#/?exe=4&ben=python_startup&env=1&revs=50&equid=off&quarts=on&extr=on

That's suspiciously close to the core sprint. Since the -S time stayed roughly the same I suspect that either a new module was added to the startup sequence or one of the (too many) modules already involved grew a lot. My money is on a new module. Using `python -v -c pass` on a Python built from the Sept. 9 tree and one built from Sept. 15, it shouldn't be too hard to figure out which new module(s).

Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

--

--Guido van Rossum (python.org/~guido)

[Victor Stinner]

2016-11-07 20:20 GMT+01:00 Guido van Rossum <gu...@python.org>:
> Re:
> https://speed.python.org/timeline/#/?exe=4&ben=python_startup&env=1&revs=50&equid=off&quarts=on&extr=on
>
> That's suspiciously close to the core sprint. Since the -S time stayed
> roughly the same I suspect that either a new module was added to the startup
> sequence or one of the (too many) modules already involved grew a lot. My
> money is on a new module. Using `python -v -c pass` on a Python built from
> the Sept. 9 tree and one built from Sept. 15, it shouldn't be too hard to
> figure out which new module(s).

I identified the regression, I created:

http://bugs.python.org/issue28637

It's a regression caused by:

https://hg.python.org/cpython/rev/223731925d06/
http://bugs.python.org/issue28082

The change added "import enum" in Lib/re.py. I propose to revert this
change for now, and discuss later a solution which doesn't impact
performances.

It's nice to see that my work of performance was useful to catch a
performance regression.

Victor

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com