gevent/sqlalchemy/psycopg2/gunicorn setup?

[Zsolt Ero]

After reading zzzeek's great blog post: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ and SO answer: https://stackoverflow.com/a/16503103/518169 I would like to use gevent / sqlalchemy / psycopg2 / gunicorn in a new application.

All I've found for Pyramid is the old TicTacToe project, as well as this 5 year old Flask project, https://github.com/kljensen/async-flask-sqlalchemy-example, but I'm afraid Pyramid's transaction handling might need to be taken care of.

Can you point me to any project or snippet which shows how to set it up? I am looking for the required changes I need to make to pyramid-cookiecutter-alchemy. 

[Mikko Ohtamaa]

Hi,

On 31 August 2017 at 17:19, Zsolt Ero <zsol...@gmail.com> wrote:

After reading zzzeek's great blog post: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ and SO answer: https://stackoverflow.com/a/16503103/518169 I would like to use gevent / sqlalchemy / psycopg2 / gunicorn in a new application.

I had a project using SQLAlchemy, Pyramid and gevent (gevent.monkey.patch_all()).

Before engaging to this interesting experience, I suggest you prebook a bed in an asylum. That is the level of problems you need to debug with Python interpreter, stdlib, web server threading, etc. All of those had subtle but hard to debug threading issues that took days and days to debug when you no longer can trust that lower levels of your stack (web server, database connections, etc.) correctly behaving under asyncio. It's especially fun if the problems only appear under a production load.

It was very happy moment when I could finally pip uninstall gevent and move back to well proven threading model.

--

Mikko Ohtamaa
http://opensourcehacker.com
http://twitter.com/moo9000

[Zsolt Ero]

Ok, I suspected it but I'm really puzzled then why does Mike Bayer recommends it? Is concurrency under Python really such an impossible task?

With asyncio / twisted I pretty much cannot use any of my existing code and any of the common libraries. They all need their own tx/aio version. No more import antigravity. SQLAlchemy ORM is one thing which doesn't exist under those worlds.

Also, the second part is the programming style. If I were to write using that async style, I'd just write in Javascript, at least in the Node.js ecosystem every library is designed to work in an async environment. So you don't have to use ORM's with 47 Github stars (https://github.com/fantix/gino) but one with 10.000 (https://github.com/sequelize/sequelize).

So how do you solve slow HTTP endpoints with Pyramid? Just start a lot of workers in a server with lots of RAM?

[Tres Seaver]

On 08/31/2017 10:51 AM, Zsolt Ero wrote:
> Ok, I suspected it but I'm really puzzled then why does Mike Bayer
> recommends it? Is concurrency under Python really such an impossible task?
>
> With asyncio / twisted I pretty much cannot use any of my existing code and
> any of the common libraries. They all need their own tx/aio version. No
> more import antigravity. SQLAlchemy ORM is one thing which doesn't exist
> under those worlds.
>
> Also, the second part is the programming style. If I were to write using
> that async style, I'd just write in Javascript, at least in the Node.js
> ecosystem every library is designed to work in an async environment. So you
> don't have to use ORM's with 47 Github stars
> (https://github.com/fantix/gino) but one with 10.000
> (https://github.com/sequelize/sequelize).
>
> So how do you solve slow HTTP endpoints with Pyramid? Just start a lot of
> workers in a server with lots of RAM?

Mike isn't a big fan of asyncio for RDBMS stuff. E.g., here is his latest
relevant tweet[1]:

> I can't stress enough how inappropriate async IO is for database
> connections. Just spent 8 hrs diagnosing a timeout due to CPU bound ops

[1] https://twitter.com/zzzeek/status/900768397637668865


Tres.
--
===================================================================
Tres Seaver +1 540-429-0999 tse...@palladion.com
Palladion Software "Excellence by Design" http://palladion.com