web2py benchmark vs other

[Mathieu Clabaut]

Hello,

 I've just stumbled upon the following benchmark : http://www.techempower.com/benchmarks/#section=intro&hw=peak&test=plaintext

 Hope it may be of interest to you…

-Mathieu

[Michele Comitini]

LOL there must be something wrong in their configuration!  2 req per second on plain text?? :D

The code is here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Python/web2py

I will check time permitting

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers@googlegroups.com
unsubscribe: web2py-developers+unsubscribe@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kiran Subbaraman]

Sent a PR their way, with some changes in the model code: https://github.com/TechEmpower/FrameworkBenchmarks/pull/2218

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com

details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.

[Niphlod]

this pop ups once every while. clearly who sets up the app for the benchmark doesn't know anything about performances...

On Monday, August 22, 2016 at 12:15:57 PM UTC+2, Kiran Subbaraman wrote:

Sent a PR their way, with some changes in the model code: https://github.com/TechEmpower/FrameworkBenchmarks/pull/2218

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

On Mon, 22-08-2016 3:08 PM, Michele Comitini wrote:

LOL there must be something wrong in their configuration!  2 req per second on plain text?? :D

The code is here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Python/web2py

I will check time permitting

2016-08-22 11:26 GMT+02:00 Mathieu Clabaut <mat...@clabaut.net>:

Hello,

 I've just stumbled upon the following benchmark : http://www.techempower.com/benchmarks/#section=intro&hw=peak&test=plaintext

 Hope it may be of interest to you…

-Mathieu

--
-- mail from:GoogleGroups "web2py-developers" mailing list

make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michele Comitini]

@kiran so the PRs are 2.  I could not resist to send one as soon as I looked at their db.py.

As @niphlod says they seem not even bother to understand what they are benchmarking!

make speech: web2py-developers@googlegroups.com
unsubscribe: web2py-developers+unsubscribe@googlegroups.com

details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py-developers+unsubscribe@googlegroups.com.

[Kiran Subbaraman]

I proposed similar changes. I'll close my PR, so they can accept yours - I prefer your changes. Also, disagree with the classification of web2py as a 'micro-framework' - maybe I should post that on their google group. Hopefully when they report "numbers" the next time around, web2py numbers look more reasonable.

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

[Anthony]

Here was my initial response with some suggested changes: https://groups.google.com/d/msg/web2py/yk7-Xw4OAuk/yaftkaeCDwAJ.

A while ago I started to make improvements, including switching the server from Rocket to Gunicorn/Meinheld (same as Flask and Django are using in these benchmarks), but didn't quite get everything working before I had to stop. Now it's more difficult for me to proceed because I'm on a new computer using Vagrant with Hyper-V, and setting up the benchmarks VM requires VirtualBox (which cannot be installed when running Hyper-V).

If someone has time to take over, I have attached my repo. I've made the following changes:

• Switch the server to Gunicorn/Meinheld (though I don't think my configuration is quite working -- just tried to copy from another framework).
• Disabled sessions at the framework level via the "web2py_disable_session" flag in routes.py (note, this flag was not actually working properly until a recent bug fix in main -- not sure if that fix has made it to an actual release yet). As a backup, I also added code at the top of the controller to forget the session and unlock the session file (the existing code to forget the session fails to unlock the file). It would be better to disable at the framework level, though.
  
• Removed the db.py model file and instead conditionally define the db and tables directly in the controller only for requests that need it. (Also eliminated the completely unnecessary loading of appconfig.ini.)
• Wrote a simple processor function to generate a list of dictionaries instead of a Rows object when running queries, eliminating the overhead of generating a Row object for each record. This also simplifies some other code.
• Eliminated unused files.
• Updated benchmark_config.json.

I haven't looked at the existing PRs, so perhaps these or better improvements have already been implemented, but hopefully this will help further.

Note, in order to make PRs, it is recommended that you use their VM setup to ensure everything is working properly (you will probably need to do that to test the server setup anyway). Documentation can be found here: http://frameworkbenchmarks.readthedocs.io/en/latest/Development/. It's a bit of extra effort, but helps to ensure the new code will run properly with the benchmarks and to verify that the code changes lead to improved speed.

Anthony

[Anthony]

One more thing I would recommend -- after getting everything working, compile that app (note, this should be done within their VM to ensure the app is compiled with the proper version of Python).

Anthony

[Michele Comitini]

@abastardi, my proposed PR is just the bare minimum especially to avoid DB connecting at each round.

Your proposed changes should give better result.

I suppose much of the delay is during db connection opening handshake.

BTW for the plain text one could write a 00.py model like short cutting a lot:

if request.action == 'plain_text':

  raise HTTP(200, 'Hello World!')

[Anthony]

On Monday, August 22, 2016 at 11:07:00 AM UTC-4, mcm wrote:

@abastardi, my proposed PR is just the bare minimum especially to avoid DB connecting at each round.

Your proposed changes should give better result.

I suppose much of the delay is during db connection opening handshake.

Good idea. I forgot to set the db pool size.
 

BTW for the plain text one could write a 00.py model like short cutting a lot:

if request.action == 'plain_text':

  raise HTTP(200, 'Hello World!')

It would be interesting to see how much effect that has. It may be minimal because in my current version, there are no model files, so only the controller is read and executed (which would be sped up by compiling). If the above approach is taken, though, it should be a conditional model file that executes only on the plain_text requests -- otherwise it will speed up the plain_text benchmark but slow down all the other benchmarks.

Anyone interested in taking my repo and trying to get it finished (should just be a matter of setting up their Vagrant VM and getting the server configuration to work)?

Anthony

[Massimo Di Pierro]

This may be the fastest solution, although it is critical to disable sessions.

Please CC me when you contact me the authors of the benchmark.

[Anthony]

On Monday, August 22, 2016 at 2:16:27 PM UTC-4, Massimo Di Pierro wrote:

This may be the fastest solution, although it is critical to disable sessions.

Please CC me when you contact me the authors of the benchmark.

There's no need to directly contact the authors -- to make changes, you just submit a pull request to the Github repo. Documentation for how to make contributions is here: http://frameworkbenchmarks.readthedocs.io/en/latest/Development/.

Anthony

[mcm]

Follow up from upstream... will se if results improve :-)

> Merged #2217.

[Anthony]

Actually, probably should have removed AppConfig altogether -- it is not actually used anywhere in the code.

[Anthony]

OK, I ended up needing to switch to VirtualBox, so I went ahead and completed work on the updates. Check out the new code and the pull request. I'm not sure the changes will make it into the Round 13 final release (they have already generated the preview data), so we may have to wait until Round 14.

I created two versions of the tests -- "standard" and "optimized". Both versions employ gunicorn+meinheld as the server (instead of Rocket), conditionally define the DAL and the models (only when needed), use connection pooling, turn off migrations, remove AppConfig (which wasn't being used), and compile the applications (also, most of the code is now in two modules, primarily to make it easier to share code between the two applications).

Additionally, the "optimized" app disables sessions completely (via routes.py), foregoes controller execution (by responding directly from a model file), and uses raw SQL instead of DAL models. Note, the session disabling feature is broken in the current stable release but was fixed in master recently, so the setup code clones the master branch and then checks out a specific commit.

Running the tests in the TechEmpower Vagrant VM on a Dell XPS 13 Intel Core i7, I get the following speed-ups relative to the original code:

Standard:

• Database tests: 2.5-3x
• JSON test: 6x
• Plain text test: 300x

Optimized

• Database tests: 3-5x
• JSON test: 10x
• Plain text test: 500x
  

Also, on the database tests, the "standard" application completes about 70% as many requests as Flask and 50% as many as Django (and about 30% for JSON and 15% for plain text). So, still not super speedy, but at least now in the same ballpark.

Anthony

On Monday, August 22, 2016 at 8:59:24 AM UTC-4, Anthony wrote:

[Massimo DiPierro]

thank you. great work.

[Anthony]

OK, looks like this will make it into Round 13: https://github.com/TechEmpower/FrameworkBenchmarks/pull/2263#event-784516350

[Michele Comitini]

Great stuff Anthony!

I see best scoring frameworks in the pack are using postgresql, did you check with it?

Do you know if it is possible to uwsgi http for the test?  I think uwsgi + pg + forking setup would give some more speed on db operation benchmarks.

thanks!

[Paolo Valleri]

@Anthony which py3 version are they using in the tests?

If it's py35, would you check if current master is passing all their benchmark tests? Although web2py's port to py35 is not completed yet, it would be nice to see few comparisons.

Paolo

[Anthony]

On Friday, September 9, 2016 at 5:48:51 PM UTC-4, mcm wrote:

Great stuff Anthony!

I see best scoring frameworks in the pack are using postgresql, did you check with it?

Do you know if it is possible to uwsgi http for the test?  I think uwsgi + pg + forking setup would give some more speed on db operation benchmarks.

The default for all of the Python frameworks is gunicorn+meinheld for the server and MySQL for the database, so I stuck with that to ensure comparability with the other frameworks.

Only Flask and Bottle ran additional tests with Nginx+uWSGI, and in Round 12, I think both frameworks consistently performed better with Meinheld (particularly in the plain text and JSON tests, where the reverse proxy overhead is most noticeable). In the Round 13 preview data, for some reason the Flask and Bottle uWSGI tests are not completing. Anyway, not sure it's worth bothering with uWSGI, particularly given that the Python README discourages having every framework attempt to test every permutation of web server, database, etc.

I think the only framework that tested both MySQL and Postgres was Django. For the tests involving only reads, Postgres was about 10% faster, though for the update test, Postgres was about 25% faster. Maybe worth playing with.

Anthony

[Anthony]

On Saturday, September 10, 2016 at 1:15:02 AM UTC-4, Paolo Valleri wrote:

@Anthony which py3 version are they using in the tests?

If it's py35, would you check if current master is passing all their benchmark tests? Although web2py's port to py35 is not completed yet, it would be nice to see few comparisons.

Python 3.5.1. I'll try to check when I get some time.

Anthony

[Anthony]

On Saturday, September 10, 2016 at 1:15:02 AM UTC-4, Paolo Valleri wrote:

@Anthony which py3 version are they using in the tests?

If it's py35, would you check if current master is passing all their benchmark tests? Although web2py's port to py35 is not completed yet, it would be nice to see few comparisons.

Sorry, no luck. I was able to run the app under Python 3.5.1 when serving via Rocket, but when I tried to run the benchmark tests (using gunicorn+meinheld), web2py returned 500 errors (though no error tickets, so not sure what's wrong). No more time to play with this, so I'll leave it for now.

I did find one incompatibility, though -- at the bottom of /gluon/contrib/aes.py, there are three lines that call .decode on a string, which is not allowed in Python 3: https://github.com/web2py/web2py/blob/b84b47dad252bacedbdd91a52394c628a950162d/gluon/contrib/aes.py#L455.

Anthony

[Paolo Valleri]

Aes hasn't been ported yet. You need to install pycrypto.

--

-- mail from:GoogleGroups "web2py-developers" mailing list

make speech: web2py-developers@googlegroups.com
unsubscribe: web2py-developers+unsubscribe@googlegroups.com

details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py-developers+unsubscribe@googlegroups.com.