Google Groups

SQLAlchemy's QueuePool lock (was [gevent] gevent in production)


Dan Korostelev Jul 15, 2010 10:34 AM
Posted in group: gevent: coroutine-based Python network library
2010/7/11 Marcin Bachry <hege...@gmail.com>:
> Dan Korostelev <nad...@gmail.com> writes:
>> Yep, i'm using this wait callback -
>> http://bitbucket.org/denis/psycogreen/src/tip/gevent/psyco_gevent.py
>>
>> BTW, I used to get some kind of deadlocks with sqlalchemy's connection
>> pool until i specify max_overflow=-1 in engine arguments.
>> Unfortunately I didn't have time to dig it out and understand the
>> reason, so if anyone have an explanation, I'd like to learn about
>> this. :-)
>
> Check out this thread on eventlet mailing list:
>
>  https://lists.secondlife.com/pipermail/eventletdev/2010-March/000793.html

Thanks. I finally took a look at it. It looks to me that using
max_overflow=-1 on uber-high load with expensive db queries, we can
end up in exception about too many open connections. I made a simple
subclass of sqlalchemy's QueuePool that uses gevent's Semaphore
instead of threading.Lock to make it green:

from gevent.coros import Semaphore

class GreenQueuePool(QueuePool):

    def __init__(self, *args, **kwargs):
        super(GreenQueuePool, self).__init__(*args, **kwargs)
        if self._overflow_lock is not None:
            self._overflow_lock = Semaphore()

Now I wonder whether I should actually use it or it's a waste of
performance (for lock aquires/releases) and I should simply make all
my db interactions minimal and fast, so I won't be bothered with
connection pool overflows. Any expirienced opinions, please? :)

--
WBR, Dan Korostelev