Speed tests : Brython vs Skulpt vs pypy.js

[Pierre Quentel]

I passed the same tests as in my previous blog post with Skulpt and pypy.js. For Skulpt I use the online editor on their site, and for pypy.js the online console.

Here are the results I got :

Test                           Execution time (ms)            Times slower than CPython
                     CPython   Brython    Skulpt   pypy.js   Brython    Skulpt   pypy.js 
assignment               105         9      5868      2911      0.09     55.88     27.72
augm_assign              194        34      6713      3667      0.18     34.60     18.90
assignment_float         101       450      6168      2946      4.46     61.06     29.17
build_dict               340      2907     14685      3012      8.55     43.19      8.86
set_dict_item            192        98     23310      4338      0.51    121.39     22.59
build_list               310        49      6741      3051      0.16     21.74      9.84
set_list_item            184        56      8139      2881      0.30     44.23     15.66
add_integers             249        96      6926      3484      0.39     27.81     13.99
add_strings              348       333      8632      3569      0.96     24.80     10.25
str_of_int                54       155      2431       465      2.87     45.01      8.61
create_function          201      3264      7886      3224     16.24     39.23     16.04
function_call            241      1222     22978      2912      5.07     95.33     12.08
                                                        Mean    3.31     51.19     16.14

Even if the mean value can be discussed, these results seem to show that in the same context (an editor in a web page), Brython is much faster than pypy.js (I must say I didn't expect this) and much, much faster than Skulpt ; it is also the only solution that sometimes runs faster than CPython.

I plan to publish another post on my blog with these results.

- Pierre                              

[Kiko]

Really interesting results. Some points:

- pypy.js doesn't claim it is a fast implementation of pypy. In theory, it could be but some tests using the available versions doesn't indicate this and neither the main author.

-Definitely, an arithmetic mean is not the right way to obtain a conclusion in this case. A geometric mean (what pypy uses) show less dispersion when there are some outliers. I think that measure real applications would be more interesting. If I need to create and call 20 functions in my program (an extreme functional case) the brython execution would be slower but if I do a simple program with some loops I will get a big speedup using Brython (on firefox).

-The speed tests for js are very js engine dependent (spidermonkey, node/v8, rhino,... https://en.wikipedia.org/wiki/List_of_ECMAScript_engines) and version dependent so conclusions should be very cautious in this sense in order to don't confuse people..

 

I plan to publish another post on my blog with these results.

I'm ok with a post if tit shows the quirks of this kind of performance tests.

Anyway, results are really amazing and could encourage people to know brython and use it in their projects.

If comments are open it would be a good source of interesting ideas (forget trolls).

And a promotion of the post and the results on reddit, HN, etc, could atract more opinions and ideas.

Good job brython developers!!!!

[Kiko]

Also, it should be very well explained how you get this numbers,... Versions, how you are executing the snippet, the code of the snippet,...

[Pierre Quentel]

Le mercredi 25 mars 2015 10:36:55 UTC+1, kiko (on pybonacci) a écrit :

2015-03-25 9:54 GMT+01:00 Pierre Quentel <pierre....@gmail.com>:

I passed the same tests as in my previous blog post with Skulpt and pypy.js. For Skulpt I use the online editor on their site, and for pypy.js the online console.

Here are the results I got :

Test                           Execution time (ms)            Times slower than CPython
                     CPython   Brython    Skulpt   pypy.js   Brython    Skulpt   pypy.js 
assignment               105         9      5868      2911      0.09     55.88     27.72
augm_assign              194        34      6713      3667      0.18     34.60     18.90
assignment_float         101       450      6168      2946      4.46     61.06     29.17
build_dict               340      2907     14685      3012      8.55     43.19      8.86
set_dict_item            192        98     23310      4338      0.51    121.39     22.59
build_list               310        49      6741      3051      0.16     21.74      9.84
set_list_item            184        56      8139      2881      0.30     44.23     15.66
add_integers             249        96      6926      3484      0.39     27.81     13.99
add_strings              348       333      8632      3569      0.96     24.80     10.25
str_of_int                54       155      2431       465      2.87     45.01      8.61
create_function          201      3264      7886      3224     16.24     39.23     16.04
function_call            241      1222     22978      2912      5.07     95.33     12.08
                                                        Mean    3.31     51.19     16.14

Even if the mean value can be discussed, these results seem to show that in the same context (an editor in a web page), Brython is much faster than pypy.js (I must say I didn't expect this) and much, much faster than Skulpt ; it is also the only solution that sometimes runs faster than CPython.

Really interesting results. Some points:

- pypy.js doesn't claim it is a fast implementation of pypy. In theory, it could be but some tests using the available versions doesn't indicate this and neither the main author.

really ? The author has published an article called "PyPy.js: Now faster than CPython" ; and in the presentation he is going to make at PyCon 2015 he writes : "PyPy.js is an experiment in building a fast, compliant, in-browser python interpreter"

-Definitely, an arithmetic mean is not the right way to obtain a conclusion in this case. A geometric mean (what pypy uses) show less dispersion when there are some outliers.

Ok, I will publish both. The results with the geometric mean is :

Brython Skulpt  pypy.js
1.02    45.01   14.83

-The speed tests for js are very js engine dependent (spidermonkey, node/v8, rhino,... https://en.wikipedia.org/wiki/List_of_ECMAScript_engines) and version dependent so conclusions should be very cautious in this sense in order to don't confuse people..

Of course, in the first blog article the configuration used for the test was given and it will be the same for this comparison. For Python in the browser I think it makes more sense to publish the result of tests run in a browser than on node.js

 

I plan to publish another post on my blog with these results.

I'm ok with a post if tit shows the quirks of this kind of performance tests.

Anyway, results are really amazing and could encourage people to know brython and use it in their projects.

If comments are open it would be a good source of interesting ideas (forget trolls).

And a promotion of the post and the results on reddit, HN, etc, could atract more opinions and ideas.

yes !

Good job brython developers!!!!

[Billy Earney]

Good job.   You might want to state what browser and version you used.

--
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.
To post to this group, send email to bry...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brython/ce5f0c57-168b-4199-b715-75f62cc7e8c6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Pierre Quentel]

Yes, I will give a detailed information on how and on which configuration the tests have been passed.

I am finishing version 3.1.1 and will upload it to brython.info before publishing the blog post, so that the tests can be reproduced there as well as on the skulpt and pypy.js sites

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/CAB1ii-fvNBnnCK1QWHAj3MQab72BMZnhuUWvpnyNKH_%2BhyvC3Q%40mail.gmail.com.

[Billy Earney]

Pierre,

Even though tests/index.html says that all tests pass, I get the following when I run the 'time' test:

======================================================================
ERROR: test_conversions (exec_171.TimeTestCase)
----------------------------------------------------------------------
<class 'Exception'>
le_func is not a function
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()

======================================================================
ERROR: test_ctime (exec_171.TimeTestCase)
----------------------------------------------------------------------
<class 'Exception'>
le_func is not a function
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()
  module 'difflib' line 506
        matching_blocks.sort()

======================================================================
ERROR: test_default_values_for_zero (exec_171.TimeTestCase)
----------------------------------------------------------------------
<class 'AttributeError'>
'module' object has no attribute '_getframe'

======================================================================
ERROR: test_strptime (exec_171.TimeTestCase)
----------------------------------------------------------------------
<class 'TypeError'>
to_struct_time() takes 1 positional argument but more were given

======================================================================
FAIL: test_strptime_exception_context (exec_171.TimeTestCase)
----------------------------------------------------------------------
<class 'AssertionError'>
ValueError not raised

----------------------------------------------------------------------
Ran 20 tests in 8.520s

FAILED (failures=1, errors=4, skipped=1)
----------------------------------------------------------------------
Ran 3 tests in 0.039s

OK (skipped=3)
<completed in 31434.00 ms>

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/CA%2BgUBLQritw0%3Dkd6zB55YrLZEp4dzXQx8-Ag1SngfJAiZXA%3D-A%40mail.gmail.com.

[Pierre Quentel]

To unsubscribe from this group and stop receiving emails from it, send an email to brython+unsubscribe@googlegroups.com.

To post to this group, send email to bry...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/ce5f0c57-168b-4199-b715-75f62cc7e8c6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

I just published the post : https://brythonista.wordpress.com/2015/03/28/comparing-the-speed-of-cpython-brython-skulpt-and-pypy-js/ 

I tried to be as complete and unpartial as possible. Comments are welcome !
- Pierre

[Kiko]

Very nice post.

In the following section the brython link points to an offlline version, Is this correct?:

How the test is run
The test is performed by running a single script made of all the elementary benchmarks. The source code of this script is here http://brython.info/speed/bench_str.py. The online editors or consoles are

• Brython : http://localhost:8000/tests/index.html (with the debug checkbox unchecked)
• Skulpt : http://www.skulpt.org/
• pypy.js : http://pypyjs.org/demo/

To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.

To post to this group, send email to bry...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/9f2996f3-df27-4c44-87c5-3b146aec3d41%40googlegroups.com.

[Pierre Quentel]

oops ! thanks Kiko, it's obviously brython.info instead of localhost !

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/CAB-sx61U-jN%3D6mkM8PJBRE-HG5PaDzW4odAs9SWH-jYezyN5QA%40mail.gmail.com.