(Continuously) Benchmarking Brython

[Jonathan Verner]

I'd like to start a discussion on benchmarking. I am not sure whether there is a standard way to

benchmark Brython or how it is done, if there is one. I do think, that there should be a way,

preferably a very easy way, which can be automated. I've coded up something

(see the pull request) and would be very happy to hear your thoughts on this. There are several

reasons why I think some automated benchmarking should be done:

  -- people evaluating Brython would like to see how it performs; benchmarks are a great way

     to convey this information, but currently most benchmarks out there are outdated and do

     not reflect the current state of python. If there was a regularly updated page with benchmark

     results, people could be pointed to this page.

  -- it will give hard data to see whether "optimizations" are really optimizations and whether

     they are worth it (if one could see the performance difference between different commits,

     it would be much easier to evaluate different approaches to solving speed problems)

  -- the results can help determine where to concentrate our efforts

  -- if its not automated, it will not be done, depending on manually running the benchmarks

     is not feasible, I think :-)

[Pierre Quentel]

I am 100% ok with the objective of being able to continuously show Brython performance.

There is an official Python Benchmark suite, it seems up to date so we could probably include the tests from this suite that are relevant for Brython.

[Kiko]

Could the brython test suite serve for this purpose?

They cover a lot of real life functionality.

 

--
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+unsubscribe@googlegroups.com.
To post to this group, send email to bry...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brython/6b315f2b-50d6-45a3-9829-31e707b142e2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Pierre Quentel]

Le dimanche 1 janvier 2017 19:59:20 UTC+1, kiko (on pybonacci) a écrit :

Could the brython test suite serve for this purpose?

They cover a lot of real life functionality.

They do, but the brython test suite is mostly there to avoid regressions and to check the continuous compliance with Python, not to measure performance.

It's better IMO to use more or less "standard" performance tests such as pystone (to compare with CPython and other Python in the browser implementations), and to keep the elementary tests in folder /speed to identify the weakest points.

[Jonathan Verner]

I'll cross-post parts of this to the discussion on my pull request as I'm not sure where to continue this discussion... What are the proper procedures?

 

It's better IMO to use more or less "standard" performance tests such as pystone (to compare with CPython and other Python in the browser implementations), 
and to keep the elementary tests in folder /speed to identify the weakest points.

I agree with Pierre here. It might also make sense to include some real-world applications written in Brython as tests to measure "real-world" performance.

In reply to Pierre's comment on my pull request:

As much as I agree with the objective, I wonder if a similar result could not be reached with Brython itself. 

The script perf.py I wrote is there mainly for running the tests automatically (either from the console or in the continuous integration environment).

The test runner (benchmark.py) is actually written in Brython. It was very nice that with some trivial modifications (e.g. checking not to use the browser module),

the same file is also usable to run the same tests in CPython. What I also like about this script is its use of the asyncio library which allows me to trivially

make the test page responsive (i.e. the execution of the test suite doesn't block the browser) just by adding a few decorators and changing few function calls to yield froms. In particular there is no need to deal with javascript's callback hell...

I have made minor improvements to the integrated speed test suite, when you "run all tests" a page is opened with the results of Brython vs CPython for all the tests.

Interesting... How do you "run all tests"?

The page is very basic and ugly at the moment , but working on a nicer presentation with graphs would be a good illustration of what can be done in Brython, instead of using Javascript libraries.

I agree, but I guess it would take nontrivial amount of work. I think that, in the meanwhile, it would make sense to use some javascript graphing library like

the one used on arewepython.com or the one from plot.ly. 

Btw, the plot.ly part was written only because I didn't want to go to the trouble of setting up a public server to collect results from continuous integration. 

I don't know if it would be possible to run this in the CI process though.

I assume it could (possibly with some  tweaking to get the output from the browser). The perf.py script I wrote basically does the following:

  - starts a simple http server (based on the microframework bottle.py) to serve the benchmarks and collect the results

  - runs a browser which opens an html file served by this server

  - the html file loads Brython and imports the benchmark.py script to run the tests

  - the results of the tests are posted to the http server and processed by the script

I suppose a similar approach would work with your tests...

[Francisco Javier Vazquez Umbria]

hi. this is my first message in this list.

I think that we must split the benchmarks in three main categories:

1) pure python benchmark: how fast is this 'python implementation in javascript' runs. I bet for 'pystones' in this category. is a standard benchmark and scores the speed of the python implementation.

2) python-javascript interoperatibility benchmark: accessing from python-side to javascript-side and back. this benchmark MUST include javascript canvas rendering from python and DOM manipulation (setting/adding/deleting CSS class to an HTML tag, InnerHTML). LocalStorage, WebGL and ¡ServiceWorker! will be a plus.

3) Solution Overhead: start-up time, warm-up time, dynamic code loading time, size of download of solution overhead (this will be a kill for pypy.js, 8MB the last time that I check out and awful start-up time).

IMHO, any 'python in the browser' implementation (as brython or pypy.js) must aim to completely replace javascript in client, besides the 'python in the browser solution' start-up procedure. if we force the developer to use 'a little of JS here for WebRTC' and 'a bit of JS there for canvas' and so, the developers will never embraces the 'python in browser' solution and sticks to standar javascript. 

We must enforce (and benchmark!) the Brython/Javascript interoperatibility. We have an unique opportunity to uprising brython, like in 2005, after the netscape/explorer browser wars, the incompatilibity of DOM manipulation of the diferents browsers (and the nightmare of IE6) brings 'JQuery', that reigns for 10 years. Today, the actual browser war is HTML5, and every browser is implementing in their own way, sometimes with diferents APIs (WebRTC or WebSockets, for example). If Brython can bring a standard method to address the HMTL5 features, Brython can will be the next 'de-facto' standard in web development.

Brython must be the next generation JQuery!!!

--

You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+unsubscribe@googlegroups.com.
To post to this group, send email to bry...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/brython/f1977a3e-dee1-465e-b051-c135405df7ee%40googlegroups.com.

[Pierre Quentel]

I have made minor improvements to the integrated speed test suite, when you "run all tests" a page is opened with the results of Brython vs CPython for all the tests.

Interesting... How do you "run all tests"?

by clicking the button "run all tests" in the page /speed