[Django] #29403: TooBig error in memcached
Django
-------------------------------------+-------------------------------------
Reporter: | Owner: nobody
SKisContent |
Type: Bug | Status: new
Component: Core | Version: 1.11
(Cache system) |
Severity: Normal | Keywords: memcached, TooBig
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I'm trying to use the memcached backend cache in a project, and I'm get a
TooBig error on some requests. The error occurs in
''django/core/cache/backends/memcached.py'' on line 86:
{{{
val = self._cache.get(key)
if val is None:
return default
return val
def set(self, key, value, timeout=DEFAULT_TIMEOUT, version=None):
key = self.make_key(key, version=version)
** if not self._cache.set(key, value,
self.get_backend_timeout(timeout)):**
# make sure the key doesn't keep its old value in case of
failure to set (memcached's 1MB limit)
self._cache.delete(key)
def delete(self, key, version=None):
key = self.make_key(key, version=version)
self._cache.delete(key)
}}}
I can reproduce it using the Django shell:
{{{
$ ./manage.py shell
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.core.cache import cache
>>> cache.get('foo')
>>> cache.set('foo', 'bar')
>>> cache.get('foo')
'bar'
>>> big = [i for i in range(1024*1024)]
>>> cache.set('big_foo', big)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/local/lib/python2.7/site-
packages/django/core/cache/backends/memcached.py", line 86, in set
if not self._cache.set(key, value, self.get_backend_timeout(timeout)):
TooBig
}}}
I can replicate the same behavior in pylibmc:
{{{
$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pylibmc.test import make_test_client
>>> mc = make_test_client()
>>> mc.get('foo')
>>> mc.set('foo', 'bar')
True
>>> mc.get('foo')
'bar'
>>> big = [i for i in range(1024*1024)]
>>> mc.set('big_foo', big)
Traceback (most recent call last):
File "<console>", line 1, in <module>
TooBig
}}}
The TooBig exception was added to pylibmc in 2016:
[https://github.com/lericson/pylibmc/commit/736e21276ad04d557b68bd81b7f28be1a4b4e1ec]
The uncaught error breaks the application. Looking at the comment about
the 1MB size limit on the next line (line 87), it seems like the original
intent was to unset the value and fail silently. Therefore, the **if**
block needs to be wrapped in **try...except**.
--
Ticket URL: <https://code.djangoproject.com/ticket/29403>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Core (Cache system) | Version: 1.11
Severity: Normal | Resolution:
Keywords: memcached, TooBig | Triage Stage:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by SKisContent):
To clarify, I'm using **PyLibMCCache**:
{{{
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
'LOCATION': '127.0.0.1:11211',
'TIMEOUT': 1000,
'BINARY': True,
'OPTIONS': {
'tcp_nodelay': True,
'remove_failed': 4
}
}
}
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29403#comment:1>
Django
-------------------------------------+------------------------------------
Reporter: SKisContent | Owner: nobody
Type: Bug | Status: new
Component: Core (Cache system) | Version: 1.11
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+------------------------------------
* stage: Unreviewed => Accepted
Comment:
There's [https://github.com/django/django/pull/9811 a related pull
request].
--
Ticket URL: <https://code.djangoproject.com/ticket/29403#comment:2>
Django
-------------------------------------+------------------------------------
Reporter: SKisContent | Owner: nobody
Component: Core (Cache system) | Version: 1.11
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+------------------------------------
Comment (by Flavio Curella):
I'm not convinced we should fail silently. I understand that some versions
o memcached will, but I'd rather be warn the user that something went
wrong instead of being consistent.
If we know something went wrong we should tell the user. I'd recommend
raising a more descriptive exception, possibly using `raise ... from ...`.
--
Ticket URL: <https://code.djangoproject.com/ticket/29403#comment:3>
Django
Reporter: SKisContent | Owner: Hasan
| Ramezani
Type: Bug | Status: assigned
Component: Core (Cache system) | Version: 1.11
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Hasan Ramezani
* status: new => assigned
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/29403#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: SKisContent | Owner: Hasan
Cleanup/optimization | Status: closed
Component: Core (Cache system) | Version: 1.11
Keywords: memcached, TooBig | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* type: Bug => Cleanup/optimization
* resolution: => needsinfo
Comment:
Like [https://code.djangoproject.com/ticket/29403#comment:3 Flavio
Curella] and
[https://github.com/django/django/pull/9811#issuecomment-387895386 Ed
Morley], I'm not convinced that we should catch this exception, it was
shortly discussed in
[https://code.djangoproject.com/ticket/19914#comment:4 #19914]. See also
arguments in [https://code.djangoproject.com/ticket/28342#comment:8
#28342] (this is not a duplicate but is related). The main question is:
should we force consistency between these two backends? IMO it's better to
respect their different policies of handling errors.
I think we should go back to the DevelopersMailingList and reach a
consensus to move it forward.
--
Ticket URL: <https://code.djangoproject.com/ticket/29403#comment:5>