pyramid async setup

[binadam]

Hello all,

I used the following cookbook for an asynchronous setup:

http://michael.merickel.org/2011/6/21/tictactoe-and-long-polling-with-pyramid/ 

but ran into problems.

First I'd be interested to know if anyone has successfully done this (in production environment) using the following packages (or something similar):

pyramid

gunicorn + gevent

postgresql (with psycopg2 made green)

This is how I made my app async (using pyramid alchemy scaffold):

in __init__.py:

import gevent

import gevent.monkey

gevent.monkey.patch_all()

from .lib import psyco_gevent

psyco_gevent.make_psycopg_green()

from pyramid_sockjs.session import Session

def main(global_config, **settings):

    """ This function returns a Pyramid WSGI application.

    """

    engine = engine_from_config(settings, 'sqlalchemy.')

    DBSession.configure(bind=engine)

    ...etc...

In  development.ini:

[server:main]

use = egg:gunicorn

host = 0.0.0.0

port = 6543

workers = 2

worker_class = gevent

What happens is that after a few DB queries a query seems to hang & the gevent worker  times out. This doesn't happen if I don't run make_psycopg_green (but my understanding is then we're not using green threads)

Thanks.

[binadam]

On Thursday, April 12, 2012 12:24:42 PM UTC-7, binadam wrote:

Hello all,

I used the following cookbook for an asynchronous setup:

http://michael.merickel.org/2011/6/21/tictactoe-and-long-polling-with-pyramid/ 

but ran into problems.

First I'd be interested to know if anyone has successfully done this (in production environment) using the following packages (or something similar):

pyramid

gunicorn + gevent

postgresql (with psycopg2 made green)

I should also add sqlalchemy 

[Michael Bayer]

I'm playing with this right now.

It's working for me, here are the two things I'm observing so far.

1. it might be better to use NullPool with create_engine(), not sure yet.   This eliminates all connection pooling.   I'm not sure if theres's some kind of twinge with using a psycopg2 connection in a greenlet that it wasn't created in, the statement at http://initd.org/psycopg/docs/advanced.html#support-to-coroutine-libraries doesn't seem to say this, but I am seeing it "hang" more often if I don't use NullPool.

2. then it runs great, but watching this go, I can see that there might be a greater chance of old fashioned deadlocks occurring, it's not clear yet.  Try running "ps -ef | grep post" or select from pg_stat_activity to see if anything is just locking.

script is attached

with gevent I can run through about 55K rows of work in 53 seconds, with threads it takes 66 seconds.

[JPe...@ykksnap-america.com]

pylons-...@googlegroups.com wrote on 04/12/2012 05:42:00 PM:

> Michael Bayer <mik...@zzzcomputing.com>
> Sent by: pylons-...@googlegroups.com
>
> 04/12/2012 05:42 PM
>
> Please respond to
> pylons-...@googlegroups.com
>
> To
>
> pylons-...@googlegroups.com
>
> cc
>
> Subject
>
> Re: pyramid async setup

>
> On Apr 12, 2012, at 3:27 PM, binadam wrote:
>
> On Thursday, April 12, 2012 12:24:42 PM UTC-7, binadam wrote:
> Hello all,
>
> I used the following cookbook for an asynchronous setup:
> http://michael.merickel.org/2011/6/21/tictactoe-and-long-polling-with-pyramid/
> but ran into problems.
>
> First I'd be interested to know if anyone has successfully done this
> (in production environment) using the following packages (or
> something similar):
> pyramid
> gunicorn + gevent
> postgresql (with psycopg2 made green)
>
> I should also add sqlalchemy
>
> I'm playing with this right now.
>
> It's working for me, here are the two things I'm observing so far.
>
> 1. it might be better to use NullPool with create_engine(), not sure
> yet.   This eliminates all connection pooling.   I'm not sure if
> theres's some kind of twinge with using a psycopg2 connection in a
> greenlet that it wasn't created in, the statement at http://
> initd.org/psycopg/docs/advanced.html#support-to-coroutine-libraries
> doesn't seem to say this, but I am seeing it "hang" more often if I
> don't use NullPool.

given how slow postgresql is in setting up connections, it still would
seem that you need a pool.  how about using an external pool server like
pgbouncer

>
> 2. then it runs great, but watching this go, I can see that there
> might be a greater chance of old fashioned deadlocks occurring, it's
> not clear yet.  Try running "ps -ef | grep post" or select from
> pg_stat_activity to see if anything is just locking.
>
> script is attached
>
> with gevent I can run through about 55K rows of work in 53 seconds,
> with threads it takes 66 seconds.

> --
> You received this message because you are subscribed to the Google
> Groups "pylons-discuss" group.
> To post to this group, send email to pylons-...@googlegroups.com.
> To unsubscribe from this group, send email to pylons-discuss
> +unsub...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/
> group/pylons-discuss?hl=en.
>

>
> --
> You received this message because you are subscribed to the Google
> Groups "pylons-discuss" group.
> To view this discussion on the web visit https://groups.google.com/
> d/msg/pylons-discuss/-/SvFCNJmz7A0J.
> To post to this group, send email to pylons-...@googlegroups.com.
> To unsubscribe from this group, send email to pylons-discuss
> +unsub...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/
> group/pylons-discuss?hl=en.
> [attachment "stress_sqla.py" deleted by Jim Penny/LBG/UFI/YKKNCA]

[binadam]

thanks for looking into this. with your attached script i get the best performance with gevent & psychopg2 NOT made green (with nullpool); if made green then performance is same as with normal threading (and sometimes slower too). i guess i'll use it that way until psycopg can be made green-thread safe.