Cleaning up memcached connections

[Jacob Kaplan-Moss]

Hi all --

We've noticed that in a few cases Django under mod_python can leave
dangling connections to memcached open. We've had trouble tracking
down the circumstances under which this happens, but when it does it
can lead to memcached servers hitting their connection limits, which
means caching stops. Nasty.

I've fixed the problem, and the patch is at
http://code.djangoproject.com/attachment/ticket/5133/memcached-cleanup-connections.patch,
but I'm not sure about the ramifications of the fix. Thoughts before I
check this in?

Jacob

[Ned Batchelder]

This rang some bells about a problem we'd had with memcached, so I asked Dave St. Germain (our goto guy for issues like this).  Here's what he had to say:

I think this is the wrong fix for the problem.  memcached connections are supposed to be persistent, not torn down after every request.  The client is working exactly as it's supposed to.  They could certainly run out of connections to the memcached server if, for instance, they had apache configured in worker mpm mode, which would mean that there'd be a separate connection per thread (and by default, there are I think 25 threads per apache worker process [times at least 2 processes]).  So, yeah, they could quickly create a bunch of idle connections, but I wouldn't consider it a "bug" in memcached or django.

The patch that I contributed actually "caused" this problem in the apache worker scenario.  Previously, one connection was shared per process (no matter how many threads), but that lead to much worse problems as the memcache client wasn't threadsafe at all.

Instead of closing the connection every request, they could make the memcache client synchronized (or the django wrapper to the client), though that doesn't seem like a great idea, either.

--Ned.

-- 
Ned Batchelder, http://nedbatchelder.com

[ludvig.ericson]

Don't see no harm in adding the close() hook, but as mr. Batchelder
said, I'd say it's the wrong thing to do.

If the connections really are dangling, I'd say there's a much larger
issue to deal with.

Sincerely,
Ludvig Ericson

> >http://code.djangoproject.com/attachment/ticket/5133/memcached-cleanu...,

[Chris Henderson]

I ran into this when I was running benchmarks with ab2 on an Ubuntu
Dapper box. I ended up just using the patch that you provided to fix
the problem, but after I installed gutsy recently, I noticed that the
problem was no longer occurring on a straight svn Django checkout
(without the fix). It seems that whatever the problem was (at least for
me), it was fixed in mod_python 3.3.1. To make sure this was really
what fixed it, I manually compiled 3.3.1 on a dapper machine that
previously had the issue, and it went away. Maybe someone who knows
more than me about mod_python would know why..

chris.

[Graham Dumpleton]

On Oct 26, 3:59 pm, Chris Henderson <chris-

dja...@frogandtoadsoftware.com> wrote:
> I ran into this when I was running benchmarks with ab2 on an Ubuntu
> Dapper box. I ended up just using the patch that you provided to fix
> the problem, but after I installed gutsy recently, I noticed that the
> problem was no longer occurring on a straight svn Django checkout
> (without the fix). It seems that whatever the problem was (at least for
> me), it was fixed in mod_python 3.3.1. To make sure this was really
> what fixed it, I manually compiled 3.3.1 on a dapper machine that
> previously had the issue, and it went away. Maybe someone who knows
> more than me about mod_python would know why..

Ahhhh, no I don't know why. :-)

I'll read these posts and the issue properly and maybe finally look at
memcached to work out what it does and see what I can work out though.
May be a few days before I get a chance though.

BTW, the 25 threads per Apache worker issue may be worse than that
given that there can be multiple sub interpreters in each process. If
a separate handle is created for each thread in each interpreter, that
could be 25*num sub interpreters, in each process in the worst case
scenario.

Graham

> chris.
>
> Jacob Kaplan-Moss wrote:
> > Hi all --
>
> > We've noticed that in a few cases Django under mod_python can leave
> > dangling connections to memcached open. We've had trouble tracking
> > down the circumstances under which this happens, but when it does it
> > can lead to memcached servers hitting their connection limits, which
> > means caching stops. Nasty.
>
> > I've fixed the problem, and the patch is at

> >http://code.djangoproject.com/attachment/ticket/5133/memcached-cleanu...,

[webjunkie]

I had the same problem today. I can confirm that it's fixed with a
newer mod_python.
http://groups.google.com/group/django-users/browse_frm/thread/5718bb2a94ebec2

On Oct 26, 6:59 am, Chris Henderson <chris-

dja...@frogandtoadsoftware.com> wrote:
> I ran into this when I was running benchmarks with ab2 on an Ubuntu
> Dapper box. I ended up just using the patch that you provided to fix
> the problem, but after I installed gutsy recently, I noticed that the
> problem was no longer occurring on a straight svn Django checkout
> (without the fix). It seems that whatever the problem was (at least for
> me), it was fixed in mod_python 3.3.1. To make sure this was really
> what fixed it, I manually compiled 3.3.1 on a dapper machine that
> previously had the issue, and it went away. Maybe someone who knows
> more than me about mod_python would know why..
>
> chris.
>
>
>
> Jacob Kaplan-Moss wrote:
> > Hi all --
>
> > We've noticed that in a few cases Django under mod_python can leave
> > dangling connections to memcached open. We've had trouble tracking
> > down the circumstances under which this happens, but when it does it
> > can lead to memcached servers hitting their connection limits, which
> > means caching stops. Nasty.
>
> > I've fixed the problem, and the patch is at

> >http://code.djangoproject.com/attachment/ticket/5133/memcached-cleanu...,

> > but I'm not sure about the ramifications of the fix. Thoughts before I
> > check this in?
>

> > Jacob- Hide quoted text -
>
> - Show quoted text -

[Jeremy Dunck]

On 10/30/07, webjunkie <web-j...@gmx.net> wrote:
>
> I had the same problem today. I can confirm that it's fixed with a
> newer mod_python.
> http://groups.google.com/group/django-users/browse_frm/thread/5718bb2a94ebec2
>

Err, what was the old version, and what's the new version?

"Upgrade" is kind of vague. :-/

[webjunkie]

On Oct 30, 4:53 pm, "Jeremy Dunck" <jdu...@gmail.com> wrote:
>
> Err, what was the old version, and what's the new version?
>
> "Upgrade" is kind of vague. :-/

Upgraded from 3.2.10 to 3.3.1 ;)

[Graham Dumpleton]

Except that I still can't see why upgrading from mod_python 3.2.10 to
3.3.1 would make a difference. The only remote possibility would be
because the older module importer in mod_python 3.2.10 was causing a
problem because of the way that it overlays a reloaded module on top
of an old one. Issue with this being the reason is that it shouldn't
be applying reloading to code modules in Django and can't therefore
see how it would trigger the problem with the old module loader. So, I
am still mystified.

Graham

[Kenneth Gonsalves]

On 30-Oct-07, at 10:55 PM, webjunkie wrote:

>> Err, what was the old version, and what's the new version?
>>
>> "Upgrade" is kind of vague. :-/
>
> Upgraded from 3.2.10 to 3.3.1 ;)

which is 8 months old ;-)

--

regards
kg
http://lawgon.livejournal.com
http://nrcfosshelpline.in/web/

[webjunkie]

On Oct 31, 3:56 am, Kenneth Gonsalves <law...@thenilgiris.com> wrote:
> On 30-Oct-07, at 10:55 PM, webjunkie wrote:
>
> >> Err, what was the old version, and what's the new version?
>
> >> "Upgrade" is kind of vague. :-/
>
> > Upgraded from 3.2.10 to 3.3.1 ;)
>
> which is 8 months old ;-)

and still not "stable" in Debian.. :(