Django with memcache not thread-safe

[Samuel]

I ran into a problem with Django's caching backend when using memcache
and threads. Instances of memcache.Client are not thread-safe, and
Django creates a single instance shared by all threads. I wanted to
post this here rather than report a bug since it's not necessarily a
bug in Django; however, memcache.py isn't really to blame either.

The issue came up while running through fastcgi. The keys and values
where being swapped/mixed up which led to seemingly impossible errors.
It wasn't really much of an issue since I just switched to a
preforked.

I also would like to thank everyone who's contributed to Django. I
couldn't imagine web development without it!

- Samuel

[Malcolm Tredinnick]

On Sat, 2007-03-10 at 13:24 -0800, Samuel wrote:
> I ran into a problem with Django's caching backend when using memcache
> and threads. Instances of memcache.Client are not thread-safe, and
> Django creates a single instance shared by all threads. I wanted to
> post this here rather than report a bug since it's not necessarily a
> bug in Django; however, memcache.py isn't really to blame either.
>
> The issue came up while running through fastcgi. The keys and values
> where being swapped/mixed up which led to seemingly impossible errors.
> It wasn't really much of an issue since I just switched to a
> preforked.

Good point. You're right, the code in Django is only going to have the
same thread-safety as memcache.Client (all our accesses to that object
are atomic).

This has to be fixable without too much pain, so if you could file a
ticket, that would be great. That gives us somewhere to point people to
if they come across it before we can fix it (rule of thumb: once one
person spots this, everybody suddenly notices it).

Thanks for pointing this out.

Regards,
Malcolm

[zucchi...@gmail.com]

We encountered the same problem with the memcache client. I submitted
this patch to the memcache client maintainer, but I don't think it's
been released yet:

65c65,73
< class Client:
---
> try:
> # Only exists in Python 2.4+
> from threading import local
> except ImportError:
> # TODO: add the pure-python local implementation
> class local(object):
> pass
>
> class Client(local):
107a116
> local.__init__(self)

On Mar 10, 4:44 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:

[Samuel]

On Mar 10, 1:44 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:

> Good point. You're right, the code in Django is only going to have the
> same thread-safety as memcache.Client (all our accesses to that object
> are atomic).
>
> This has to be fixable without too much pain, so if you could file a
> ticket, that would be great. That gives us somewhere to point people to
> if they come across it before we can fix it (rule of thumb: once one
> person spots this, everybody suddenly notices it).
>
> Thanks for pointing this out.
>
> Regards,
> Malcolm

Here's the ticket:
http://code.djangoproject.com/ticket/3701

I'm not sure what other information to add to the ticket, but it is
linked back to this thread in case any discussion happens on either
side.