memcached problems with cache.clear()
Torsten Bronger
Sometimes, we experience a massive increase in active connections to
the memcached server when calling cache.clear(). This causes server
tracebacks because the server cannot open files (e.g. Python
modules) anymore: "error 24: Too many open files". Increasing the
allowed files for www-data in /etc/security/limits didn't help.
Currently, the only solution I see is to replace cache.clear() with
subprocess.call(["/etc/init.d/memcached", "restart"]).
Has anybody had a similar problem?
Tsch�,
Torsten.
--
Torsten Bronger Jabber ID: torsten...@jabber.rwth-aachen.de
or http://bronger-jmp.appspot.com
Torsten Bronger
Torsten Bronger writes:
> Sometimes, we experience a massive increase in active connections
> to the memcached server when calling cache.clear(). This causes
> server tracebacks because the server cannot open files
> (e.g. Python modules) anymore: "error 24: Too many open files".
I found an easy way to reproduce this (with memcached activated in
settings.py):
chantal@mandy:~/chantal$ ./manage.py shell
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.core.cache import cache
>>> for i in xrange(2000): cache.set(str(i), str(i))
...
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/lib/python2.6/dist-packages/django/core/cache/backends/memcached.py", line 64, in set
File "/usr/lib/pymodules/python2.6/memcache.py", line 502, in set
File "/usr/lib/pymodules/python2.6/memcache.py", line 675, in _set
File "/usr/lib/pymodules/python2.6/memcache.py", line 278, in _get_server
File "/usr/lib/pymodules/python2.6/memcache.py", line 883, in connect
File "/usr/lib/pymodules/python2.6/memcache.py", line 897, in _get_socket
File "/usr/lib/python2.6/socket.py", line 182, in __init__
error: [Errno 24] Too many open files
It *seems* (I don't know for sure) that a socket it opened by
cache.set() but not closed. Is this the expected behaviour?
Tom Evans
<bro...@physik.rwth-aachen.de> wrote:
> Hallöchen!
> Torsten.
>
It behaves for me - different OS mind.
Python 2.7.1 (r271:86832, Dec 13 2010, 15:52:15)
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.core.cache import cache
>>> for i in xrange(200000): cache.set(str(i), str(i))
...
>>> cache.get('1')
'1'
>>> from django.conf import settings
>>> settings.CACHES
{'default': {'LOCATION': '127.0.0.1:11211', 'BACKEND':
'django.core.cache.backends.memcached.CacheClass'}}
At a guess I'd say your memcached client library is not closing its socket.
> $ pkg_info | grep memca
libmemcached-0.49 A C and C++ client library to the memcached server
memcached-1.4.5_2 High-performance distributed memory object cache system
> $ pip freeze | grep memc
python-memcached==1.45
Cheers
Tom
Torsten Bronger
Tom Evans writes:
> On Thu, Sep 15, 2011 at 2:31 PM, Torsten Bronger
> <bro...@physik.rwth-aachen.de> wrote:
>
>> [...]
>>
>> I found an easy way to reproduce this (with memcached activated in
>> settings.py):
>>
>> chantal@mandy:~/chantal$ ./manage.py shell
>> Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
>> [GCC 4.4.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> (InteractiveConsole)
>>>>> from django.core.cache import cache
>>>>> for i in xrange(2000): cache.set(str(i), str(i))
>> ...
>> Traceback (most recent call last):
>> �File "<console>", line 1, in <module>
>> �File
> "/usr/lib/python2.6/dist-packages/django/core/cache/backends/memcached.py",
> line 64, in set
>> �File "/usr/lib/pymodules/python2.6/memcache.py", line 502, in set
>> �File "/usr/lib/pymodules/python2.6/memcache.py", line 675, in _set
>> �File "/usr/lib/pymodules/python2.6/memcache.py", line 278, in _get_server
>> �File "/usr/lib/pymodules/python2.6/memcache.py", line 883, in connect
>> �File "/usr/lib/pymodules/python2.6/memcache.py", line 897, in _get_socket
>> �File "/usr/lib/python2.6/socket.py", line 182, in __init__
>> error: [Errno 24] Too many open files
>>
>> It *seems* (I don't know for sure) that a socket it opened by
>> cache.set() but not closed. �Is this the expected behaviour?
>
> It behaves for me - different OS mind.
>
> [...]
>>>> from django.conf import settings
>>>> settings.CACHES
> {'default': {'LOCATION': '127.0.0.1:11211', 'BACKEND':
> 'django.core.cache.backends.memcached.CacheClass'}}
It works for me too if it only accesses localhost. However, if I
say
"LOCATION": ["192.168.26.130:11211", "192.168.26.131:11211"],
I see that fatal behaviour.
Tom Evans
<bro...@physik.rwth-aachen.de> wrote:
> Hallöchen!
>
>
> It works for me too if it only accesses localhost. However, if I
> say
>
> "LOCATION": ["192.168.26.130:11211", "192.168.26.131:11211"],
>
> I see that fatal behaviour.
>
Not for me:
>>> from django.core.cache import cache
>>> for i in xrange(20000): cache.set(str(i), str(i))
...
>>> cache.get('1')
'1'
>>> from django.conf import settings
>>> settings.CACHES
{'default': {'LOCATION': '10.0.12.11:11211', 'BACKEND':
'django.core.cache.backends.memcached.CacheClass'}}
Cheers
Tom
Torsten Bronger
Tom Evans writes:
> On Thu, Sep 15, 2011 at 3:49 PM, Torsten Bronger
> <bro...@physik.rwth-aachen.de> wrote:
>> Hall�chen!
>>
>>
>> It works for me too if it only accesses localhost. �However, if I
>> say
>>
>> � �"LOCATION": ["192.168.26.130:11211", "192.168.26.131:11211"],
>>
>> I see that fatal behaviour.
>>
>
> Not for me:
Okay, thank you for your input; it narrowed the erroneous case a
bit.
But how to proceed? Is Django using the memcached module wrongly,
or is the memcached module buggy?
Torsten Bronger
Tom Evans writes:
> [...]
>
>
>>>> from django.core.cache import cache
>>>> for i in xrange(20000): cache.set(str(i), str(i))
> ...
>>>> cache.get('1')
> '1'
>>>> from django.conf import settings
>>>> settings.CACHES
> {'default': {'LOCATION': '10.0.12.11:11211', 'BACKEND':
> 'django.core.cache.backends.memcached.CacheClass'}}
Could you -- or anybody -- try this again for "93.129.33.177:11211"?
This is the IP of my private PC for the next 8 hours (ping
wilson(dot)homeunix(dot)com). For me, this fails. If it doesn't
fail for you, it is my client's configuration.
Thank you!
Torsten Bronger
Torsten Bronger writes:
> Sometimes, we experience a massive increase in active connections
> to the memcached server when calling cache.clear().
I think https://code.djangoproject.com/ticket/15324 is our problem,
so I will upgrade from SVN-15005 to Django 1.3.