[gevent] greenlet thread.local

[me]

i'm trying to setup a simple gevent wsgi server that communicates to
memcached. spawning is set to 'defaut' and monkey.patch_all() is
applied before importing all other modules.

the pure python client for memecached uses thread.local to store
client info including socket.

when benching the wsgi server serially all is fine. when benching with
concurrency memcache communication randomly fails. i added some
logging to memcached client and it looks like the same socket is
always being used by different greenlets and so the communication
fails whenever there is overlapping use. so i guess thread.local is
not greenlet aware?

in eventlib looks like there is a co-routine version of local:
http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/green/thread.py#cl-52

but in gevent it is not there:
http://bitbucket.org/denis/gevent/src/tip/gevent/thread.py#cl-34

am i doing something wrong? does gevent support monkey patching of
thread local?

thanks

[Philip Southam [movieclips.com]]

I was having similar problems when testing heavy concurrency on my app
using siege. I then tried the pylibmc client (http://pypi.python.org/
pypi/pylibmc) with client pooling or pylibmc.ClientPool() (http://
blog.sendapatch.se/category/pylibmc.html) and now, all is working
great. You may want to give that a shot.

Regards,
Philip Southam

On May 3, 3:03 am, me <aab112233...@hotmail.com> wrote:
> i'm trying to setup a simple gevent wsgi server that communicates to
> memcached. spawning is set to 'defaut' and monkey.patch_all() is
> applied before importing all other modules.
>
> the pure python client for memecached uses thread.local to store
> client info including socket.
>
> when benching the wsgi server serially all is fine. when benching with
> concurrency memcache communication randomly fails. i added some
> logging to memcached client and it looks like the same socket is
> always being used by different greenlets and so the communication
> fails whenever there is overlapping use. so i guess thread.local is
> not greenlet aware?
>

> in eventlib looks like there is a co-routine version of local:http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/green/thr...

[Ted Suzman]

gevent does patch threading.local, but it'll only work if the memcache
client you're using actually uses the python threading interface. If
you want the client to work with plain greenlets, you'll need
something like eventlet's corolocal.py -- here's one with a couple
tiny changes to make it work with gevent: http://pastebin.com/W4cyt42i

you'll want your imports to look something like this:

import gevent.monkey
gevent.monkey.patch_all()

import corolocal
import threading
threading.local = corolocal.local

# now thread local storage will work from plain greenlets

Also (perhaps related) there's a bug that can stop the thread local
patch working entirely -
http://code.google.com/p/gevent/issues/detail?id=24

Ted

[me]

thanks. i was not aware of this client (memcached wiki could do with
some updating!). however it wraps libmemcached (a C lib) so i don't
think it will cooperate with greenlet using monkey.patch_socket(). but
in some cases that might be a good thing.

On May 3, 2:14 pm, "Philip Southam [movieclips.com]"

[me]

Ted, i'm using python-libmemcache which does use python's
threading.local:
http://bazaar.launchpad.net/~python-memcached-team/python-memcached/trunk/annotate/head%3A/memcache.py#L96
not sure if you meant something else by "uses the python threading
interface".

so it seems i'm running into the same problem you found? i'll try your
workaround for that bug and your pastebin handiwork to see if that
fixes things.

thanks

On May 3, 6:56 pm, Ted Suzman <t...@suzman.net> wrote:
> gevent does patch threading.local, but it'll only work if the memcache
> client you're using actually uses the python threading interface. If
> you want the client to work with plain greenlets, you'll need
> something like eventlet's corolocal.py -- here's one with a couple
> tiny changes to make it work with gevent:http://pastebin.com/W4cyt42i
>
> you'll want your imports to look something like this:
>
> import gevent.monkey
> gevent.monkey.patch_all()
>
> import corolocal
> import threading
> threading.local = corolocal.local
>
> # now thread local storage will work from plain greenlets
>
> Also (perhaps related) there's a bug that can stop the thread local

> patch working entirely -http://code.google.com/p/gevent/issues/detail?id=24
>
> Ted

>
> On Mon, May 3, 2010 at 3:03 AM, me <aab112233...@hotmail.com> wrote:
> > i'm trying to setup a simple gevent wsgi server that communicates to
> > memcached. spawning is set to 'defaut' and monkey.patch_all() is
> > applied before importing all other modules.
>
> > the pure python client for memecached uses thread.local to store
> > client info including socket.
>
> > when benching the wsgi server serially all is fine. when benching with
> > concurrency memcache communication randomly fails. i added some
> > logging to memcached client and it looks like the same socket is
> > always being used by different greenlets and so the communication
> > fails whenever there is overlapping use. so i guess thread.local is
> > not greenlet aware?
>
> > in eventlib looks like there is a co-routine version of local:

> >http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/green/thr...

[Ted Suzman]

If the memcache client you're using is connecting from C instead of
from python sockets, then I think it won't cooperate with gevent no
matter what you do (like you said). The C memcache client will block
your process whenever you use it -- if you can't have that, you can
switch to a pure python memcache client.

Earlier, I meant that plain greenlets don't count as different
contexts to threading.local. You (or your library) would need to use
the Python threading calls (e.g. threading.Thread) for threading.local
to work. If you want threading.local to work with plain greenlets
instead of just thread-based ones, you can patch threading.local with
something like corolocal.py. But this is only in the case that you're
using pure python sockets -- if you're not, stuff won't cooperate.

Ted

[Andy]

Ted,

Could you clarify what needs to be done to fix the problem?

So the code in http://pastebin.com/W4cyt42i is corolocal.py?

And I need to do the imports as below?

import gevent.monkey
gevent.monkey.patch_all()
import corolocal
import threading
threading.local = corolocal.local

What about the imports described in comment #2 in
http://code.google.com/p/gevent/issues/detail?id=24 --
import gevent.monkey
gevent.monkey.patch_all()
import threading
import thread
threading.local = thread._local

That's slightly different from the imports you described. Do I need
them as well?

And what about the patch mentioned in comment #4:
http://bugs.python.org/issue1522237

Do I need to apply that?

Thanks.

On May 3, 9:56 pm, Ted Suzman <t...@suzman.net> wrote:
> gevent does patch threading.local, but it'll only work if the memcache
> client you're using actually uses the python threading interface. If
> you want the client to work with plain greenlets, you'll need
> something like eventlet's corolocal.py -- here's one with a couple
> tiny changes to make it work with gevent:http://pastebin.com/W4cyt42i
>
> you'll want your imports to look something like this:
>
> import gevent.monkey
> gevent.monkey.patch_all()
>
> import corolocal
> import threading
> threading.local = corolocal.local
>
> # now thread local storage will work from plain greenlets
>
> Also (perhaps related) there's a bug that can stop the thread local

> patch working entirely -http://code.google.com/p/gevent/issues/detail?id=24
>
> Ted

>
> On Mon, May 3, 2010 at 3:03 AM, me <aab112233...@hotmail.com> wrote:
> > i'm trying to setup a simple gevent wsgi server that communicates to
> > memcached. spawning is set to 'defaut' and monkey.patch_all() is
> > applied before importing all other modules.
>
> > the pure python client for memecached uses thread.local to store
> > client info including socket.
>
> > when benching the wsgi server serially all is fine. when benching with
> > concurrency memcache communication randomly fails. i added some
> > logging to memcached client and it looks like the same socket is
> > always being used by different greenlets and so the communication
> > fails whenever there is overlapping use. so i guess thread.local is
> > not greenlet aware?
>
> > in eventlib looks like there is a co-routine version of local:

> >http://bitbucket.org/which_linden/eventlet/src/tip/eventlet/green/thr...

[Ted Suzman]

Hi Andy, this is a messy corner of threading, gevent, and python. I'll
do my best to outline what's going on.

To begin with, python has a feature called thread local storage.
Classes that inherit from threading.local result in objects that
appear 'local' to each thread, i.e. they don't share attributes
between threads. Each thread can read and write its own values to the
local storage object independently. Python ships with two
implementations of thread local storage: the native one, which is used
by default, and a pure-python one, which is in _threading_local.local.

When gevent patches python's threading implementation, it attempts to
switch threading.local to the pure-python _threading_local.local
instead of the native implementation. It does so because the native
thread local storage won't work with gevent's greenlet-based threads.
However, it doesn't seem to patch correctly
(http://code.google.com/p/gevent/issues/detail?id=24). There's a
workaround:

---

import gevent.monkey
gevent.monkey.patch_all()
import threading

# now the workaround: patch threading.local with the pure-python
implementation (gevent should've done this for us in
monkey.patch_threads())
import _threading_local
threading.local = _threading_local.local
---

On top of this, some pythons shipped with a broken
_threading_local.local (http://bugs.python.org/issue1522237). So, you
have to make sure you've got a working one.

Lastly, if you get threading.local patched with a working
_threading_local.local, thread local objects will ONLY work with
greenlet-based threads, and won't work with plain greenlets. If your
library or code makes use of threading.Thread or thread.Thread et al,
then thread local storage will work as expected. If, however, you're
not using threads and you're only using greenlets, you'll find that
thread local objects aren't greenlet local. In other words, plain
greenlets that access an object that mixes in threading.local won't
see separate sets of attributes, they'll be dealing with exactly the
same object. If that's the situation you find yourself in, you can
patch your threading.local with eventlet's corolocal.py, which will
make thread local objects work with all greenlets (whether they
underlie a thread or not). That might look like this:

---

import gevent.monkey
gevent.monkey.patch_all()
import threading

# ok, now replace threading.local with a version that works with all greenlets
import corolocal # find this from http://pastebin.com/W4cyt42i
threading.local = corolocal.local
---

So, in summary, there are three issues:

1. gevent doesn't patch threading.local as it intends to
(http://code.google.com/p/gevent/issues/detail?id=24)
2. even if gevent patches threading.local correctly, your python's
_threading_local.local might be broken
(http://bugs.python.org/issue1522237)
3. even if gevent patches threading.local correctly, and your python's
_threading_local.local works, thread local objects won't work with
plain greenlets (they only work with greenlet-based threads).

Phew.

If you're trying to get something specific working, feel free to email
the list with the particular issue / library / etc and someone might
be able to help.

Ted

[Andy]

> So, in summary, there are three issues:
>
> 1. gevent doesn't patch threading.local as it intends to
> (http://code.google.com/p/gevent/issues/detail?id=24)
> 2. even if gevent patches threading.local correctly, your python's
> _threading_local.local might be broken
> (http://bugs.python.org/issue1522237)
> 3. even if gevent patches threading.local correctly, and your python's
> _threading_local.local works, thread local objects won't work with
> plain greenlets (they only work with greenlet-based threads).
>

Ted,

Thank you very much for your detailed explanations. I now have a
better understanding of the issues.

- Is there anyway to find out if my Python has a working version of
_threading_local.local? I'm using 2.6.2

- How do I find out whether the libraries I'm using are using greenlet-
based threads or plain greenlets?

- The workaround to fix gevent's monkey.patch_threads() involves this:
threading.local = _threading_local.local
while the workaround to make gevent works for plain greenlets
involves this:
threading.local = corolocal.local
Aren't the above 2 workarounds contradicting to each others? The
first workaround assigns threading.local to _threading_local.local but
the second workaround assigns threading.local corolocal.local
Which one should I so? Or is it possible to apply both workarounds?

> If you're trying to get something specific working, feel free to email
> the list with the particular issue / library / etc and someone might
> be able to help.

I'm working on getting my Django project to run on gevent.

- I use monkey.patch_all() just like in this example:
http://bitbucket.org/denis/gevent/src/tip/examples/webchat/run.py
- I use gevent-mysql (http://github.com/mthurlin/gevent-MySQL) for
async mysql driver
- I use python-memcached for memcached client
- I use SQLAlchemy's database pooling to create a pool of DB
connections like described here: http://menendez.com/blog/mysql-connection-pooling-django-and-sqlalchemy/


Thank you. Really appreciate your help.
Andy

[me]

Ted, thanks a lot for the clarification and the code. after applying
the suggested change threading.local based clients work seamlessly
with gevent. i didn't appreciate this greenlet and thread patched
greenlet distinction but do now.

maybe you can submit your gevent adapted corolocal code to be included
in gevent? it seems necessary in some cases.

On May 4, 11:42 am, Ted Suzman <t...@suzman.net> wrote:
> If the memcache client you're using is connecting from C instead of
> from python sockets, then I think it won't cooperate with gevent no
> matter what you do (like you said). The C memcache client will block
> your process whenever you use it -- if you can't have that, you can
> switch to a pure python memcache client.
>
> Earlier, I meant that plain greenlets don't count as different
> contexts to threading.local. You (or your library) would need to use
> the Python threading calls (e.g. threading.Thread) for threading.local
> to work. If you want threading.local to work with plain greenlets
> instead of just thread-based ones, you can patch threading.local with
> something like corolocal.py. But this is only in the case that you're
> using pure python sockets -- if you're not, stuff won't cooperate.
>
> Ted
>

> On Tue, May 4, 2010 at 2:51 AM, me <aab112233...@hotmail.com> wrote:
> > Ted, i'm using python-libmemcache which does use python's
> > threading.local:

> >    http://bazaar.launchpad.net/~python-memcached-team/python-memcached/t...

[Denis Bilenko]

Ted, thanks for the explanation.

I've added a new gevent.local module that is based on the fixed
_threading_local.py from the Python's trunk and evenlet's
corolocal.py. it should work with any greenlet.

gevent.monkey now uses it to patch thread._local. It also explicitly
patches threading.local and _threading_local.local.

In other words, gevent.monkey.patch_thread/patch_all() should now work
as expected.
Anyone who needs this, please update to the latest dev version and let
me know how it goes.

Cheers,
Denis.