What driver for SQLAlchemy + PostgresSQL with gevent?

1,089 views
Skip to first unread message

Zsolt Ero

unread,
Aug 31, 2017, 8:12:57 AM8/31/17
to gevent: coroutine-based Python network library

Based on this great answer by zzzeek, I would like to use SQLAlchemy + PostgreSQL with gevent. 


My problem is that I don't know what is an up-to-date PostgreSQL driver for such scenario.


Before I always used psycopg2, as I found it to be extremely stable and reliable.


My problem is that if I understand right psycopg2 needs to be monkey-patched to work under gevent, right? And the only way to do it is to use psycogreen, a project whose latest PyPi release is from 2012, has not been updated since 2015 and doesn't support Python 3.


How do you use SQLAlchemy in a concurrent way today? Is it simply not possible?

Jason Madden

unread,
Aug 31, 2017, 8:20:43 AM8/31/17
to gev...@googlegroups.com
psycogreen is quite stable, as I understand it. The code that it uses to integrate with gevent[1] is nearly trivial. Even if it can't be used as-is under Python 3, it should be a cinch to write equivalent functions that can be.

A pure-python postgresql DBAPI driver is pg8000; I had good luck using it with RelStorage, and it should be out-of-the-box compatible with gevent's monkey-patching. SQLAlchemy can be used with this driver ("postgresql+pg8000://...").

~Jason

[1] https://bitbucket.org/dvarrazzo/psycogreen/src/115d0627da1ac9ff48c0cb9287257cd35868cdf9/psycogreen/gevent.py?at=master&fileviewer=file-view-default

Zsolt Ero

unread,
Aug 31, 2017, 10:06:58 AM8/31/17
to gevent: coroutine-based Python network library
Thanks. If I understand right, the only code in psycogreen is this:

extensions.set_wait_callback(gevent_wait_callback)

def gevent_wait_callback(conn, timeout=None):
    """A wait callback useful to allow gevent to work with Psycopg."""
    while 1:
        state = conn.poll()
        if state == extensions.POLL_OK:
            break
        elif state == extensions.POLL_READ:
            wait_read(conn.fileno(), timeout=timeout)
        elif state == extensions.POLL_WRITE:
            wait_write(conn.fileno(), timeout=timeout)
        else:
            raise psycopg2.OperationalError(
                "Bad result from poll: %r" % state)

And this happens to be the exact same snippet as is included in the official psycopg2_pool.py. 
https://github.com/gevent/gevent/blob/master/examples/psycopg2_pool.py#L19 

So this really should work under Py 3, as there is absolutely nothing Py 2 specific in there, at least I cannot see any.

Now to integrate it with Pyramid, can anyone recommend some snippet or Github example project where I can see some reference implementation? 
All I found online was this 5 year old project integrating Gevent/SQLAlchemy/Flask/psycopg.

https://github.com/kljensen/async-flask-sqlalchemy-example
Reply all
Reply to author
Forward
0 new messages