[Django] #25824: Assertions in libmemcached when multiple django.core.cache.backends.memcached.PyLibMCCache backends used
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
-------------------------------+--------------------
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
We're running Django 1.8.3 via uWSGI with threads enabled (uWSGI 2.0.11.5,
configuration - [https://gist.github.com/mlowicki/9a5042b5411537a51bbf]).
When two backends using
'django.core.cache.backends.memcached.PyLibMCCache' are added and are used
(each box is handling at least hundreds requests per second) then in uWSGI
logs we're seeing lots of
[https://gist.github.com/mlowicki/a8b3dbd98f32dc40c4eb] and application is
misbehaving throwing 500s. When launched in second datacenter with much
less threads then problem is much smaller but still exists (uWSGI
configuration - [https://gist.github.com/mlowicki/cbd18c689c3e2f838bf0])
--
Ticket URL: <https://code.djangoproject.com/ticket/25824>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.8
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by mlowicki):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Old description:
> We're running Django 1.8.3 via uWSGI with threads enabled (uWSGI
> 2.0.11.5, configuration -
> [https://gist.github.com/mlowicki/9a5042b5411537a51bbf]).
>
> When two backends using
> 'django.core.cache.backends.memcached.PyLibMCCache' are added and are
> used (each box is handling at least hundreds requests per second) then in
> uWSGI logs we're seeing lots of
> [https://gist.github.com/mlowicki/a8b3dbd98f32dc40c4eb] and application
> is misbehaving throwing 500s. When launched in second datacenter with
> much less threads then problem is much smaller but still exists (uWSGI
> configuration - [https://gist.github.com/mlowicki/cbd18c689c3e2f838bf0])
New description:
We're running Django 1.8.3 via uWSGI with threads enabled (uWSGI 2.0.11.5,
configuration - [https://gist.github.com/mlowicki/9a5042b5411537a51bbf]).
When two backends using
'django.core.cache.backends.memcached.PyLibMCCache' are added and are used
(each box is handling at least hundreds requests per second) then in uWSGI
logs we're seeing lots of
[https://gist.github.com/mlowicki/a8b3dbd98f32dc40c4eb] and application is
misbehaving throwing 500s. When launched in second datacenter with much
less threads then problem is much smaller but still exists (uWSGI
configuration - [https://gist.github.com/mlowicki/cbd18c689c3e2f838bf0])
Tried with pylibmc 1.4.1 and 1.5.0.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:1>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.8
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
It's not clear to me that Django is at fault. Can you provide any details
about that?
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:2>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.8
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Description changed by mlowicki:
Old description:
> We're running Django 1.8.3 via uWSGI with threads enabled (uWSGI
> 2.0.11.5, configuration -
> [https://gist.github.com/mlowicki/9a5042b5411537a51bbf]).
>
> When two backends using
> 'django.core.cache.backends.memcached.PyLibMCCache' are added and are
> used (each box is handling at least hundreds requests per second) then in
> uWSGI logs we're seeing lots of
> [https://gist.github.com/mlowicki/a8b3dbd98f32dc40c4eb] and application
> is misbehaving throwing 500s. When launched in second datacenter with
> much less threads then problem is much smaller but still exists (uWSGI
> configuration - [https://gist.github.com/mlowicki/cbd18c689c3e2f838bf0])
>
> Tried with pylibmc 1.4.1 and 1.5.0.
New description:
We're running Django 1.8.3 via uWSGI with threads enabled (uWSGI 2.0.11.5,
configuration - [https://gist.github.com/mlowicki/9a5042b5411537a51bbf]).
When two backends using
'django.core.cache.backends.memcached.PyLibMCCache' are added and are used
(each box is handling at least hundreds requests per second) then in uWSGI
logs we're seeing lots of
[https://gist.github.com/mlowicki/a8b3dbd98f32dc40c4eb] and application is
misbehaving throwing 500s. When launched in second datacenter with much
less threads then problem is much smaller but still exists (uWSGI
configuration - [https://gist.github.com/mlowicki/cbd18c689c3e2f838bf0])
Tried with pylibmc 1.4.1 and 1.5.0. Running on Debian Wheezy.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:3>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.8
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by mlowicki):
I'm not 100% sure it's Django's fault but according to
[https://docs.djangoproject.com/en/1.8/topics/cache/#django.core.cache.caches]
it should be thread-safe.
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:4>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => needsinfo
Comment:
Okay, please use TicketClosingReasons/UseSupportChannels and reopen this
ticket with more details if does turn out to be a bug in Django. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:5>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by mlowicki):
FunkyBob told me 1h ago on IRC to file what I've but not it's closed
within couple of minutes...
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:6>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by charettes):
@mlowicki, I'm not familar with `uWSGI`'s threading but Django doesn't
reuse `pylicmc.Client` between threads.
Did you try using `python-memcached` instead?
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:7>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by mlowicki):
Yes, for 2nd backend I'm using python-memcached (and it works) but pylimbc
has much more options to configure and support for binary protocol. I'll
try to debug in pylibmc to see if this will get me somewhere.
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:8>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by charettes):
Alright, keep us updated.
FWIW Django stores aliased cache instances in a `threading.local`
[https://github.com/django/django/blob/81f5d63218851f90bd83c1263d9a427db84d6082/django/core/cache/__init__.py#L65
property] and the wrapped `pylibmc.Client` instances are created in
`PyLibMCCache._cache`.
You might want to investigate if `pylibmc.ThreadMappedPool` does anything
different since it looks like it
[https://stackoverflow.com/questions/12420007/pylibmc-assertion-ptr-query-
id-query-id-1-failed-for-function-memcache solved a similar issue].
#11331 and #5133 might also be related.
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:9>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by funkybob):
I believe the specific quirk of this case is using the same memcache
driver for two separate cache backends ... so it's likely a problem
independent of threading, but possibly that the lib isn't save to use on
two connections concurrently in the same thread.
@mlowicki could you please try to replicate the issue without threads?
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:10>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by apollo13):
Replying to [comment:10 funkybob]:
> @mlowicki could you please try to replicate the issue without threads?
I agree on that. I see no reason why this would be a problem in Django, we
properly use one Connection per thread per alias. Build your
python/memcached/pylibmc with debug infos and throw GDB at it.
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:11>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by apollo13):
* cc: apollo13 (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:12>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by mlowicki):
Just tried without threads and it works just fine.
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:13>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by mlowicki):
It turned out we were assigning cache backend to variable at the top of
the module so all threads were using the same instance.
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:14>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by apollo13):
Jupp, that certainly does cause such behavior.
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:15>
Django
django.core.cache.backends.memcached.PyLibMCCache backends used
Reporter: mlowicki | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 1.8
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by apollo13):
* resolution: needsinfo => invalid
--
Ticket URL: <https://code.djangoproject.com/ticket/25824#comment:16>