Adding SASL authentication support to PyLibMCCache

[Ed Morley]

Hi

For apps running on Heroku, two of the main memcached options are MemCachier and Memcached Cloud [1]. Both of these control access via SASL authentication, which isn't supported by Django's current pylibmc backend [2], even though pylibmc supports it [3]. As such, currently the django-pylibmc backend has to be used instead [4].

I'd like to add SASL auth support to the Django pylibmc backend to make this unnecessary, which is just a case of passing the relevant parameters through to the pylibmc client during its instantiation [5]. A previous newsgroup discussion about this is at [6].

SASL auth requires binary mode, so I'll also need to add support for that - for which there is already a Django ticket filed [7].

I have a couple of questions:
a) Will a PR to add SASL authentication support be accepted?
b) If yes to (a), should the `username`, `password` and `binary` fields be added to the top level `CACHES['foo']` dict, or nested inside `OPTIONS` within that? I'm presuming within OPTIONS? Examples: [8].

Many thanks,

Ed

[1] https://elements.heroku.com/addons#caching
[2] https://github.com/django/django/blob/1.9.1/django/core/cache/backends/memcached.py#L181
[3] https://github.com/lericson/pylibmc/blob/1.5.0/src/pylibmc/client.py#L125
[4] https://github.com/django-pylibmc/django-pylibmc
[5] eg: https://github.com/django-pylibmc/django-pylibmc/blob/v0.6.1/django_pylibmc/memcached.py#L85-L91
[6] https://groups.google.com/forum/#!searchin/django-developers/pylibmc/django-developers/pISp20wuu0E/tamJ1h8zCzsJ
[7] https://code.djangoproject.com/ticket/15815
[8] https://emorley.pastebin.mozilla.org/8858134

[Tim Graham]

I found an old discussion about it, but it didn't attract much interest at the time: https://groups.google.com/d/topic/django-developers/pISp20wuu0E/discussion

[Ed Morley]

Hi Tim,

Thank you for your reply.

Yeah I spotted that discussion (it's [6] in the OP). Back then Heroku (and other PaaS providers) were less common (and I'm not sure how many of those users frequent this list).

Given that django-pylibmc has 8000 downloads a month on PyPI (and this is bearing in mind that Heroku re-uses the virtualenv, so Heroku deployments won't be counted in that figure aside from updates) I think there is a case for including it in the default backend - given it would only be a 5-10 line addition. Even for people who won't be using auth, the new binary mode is faster, so could be useful regardless.

I'm happy to put together a PR to make the impact/complexity easier to judge, if that helps?

Before I do that I would just need to know whether the `username`, `password` and `binary` fields should be added to the top level `CACHES['foo']` dict, or nested inside `OPTIONS` within that? Examples:
https://emorley.pastebin.mozilla.org/8858134

Many thanks,

Ed

[Claude Paroz]

Le lundi 22 février 2016 13:00:31 UTC+1, Ed Morley a écrit :

(...)

I'm happy to put together a PR to make the impact/complexity easier to judge, if that helps?

Yes, it always help to see the code.

 

Before I do that I would just need to know whether the `username`, `password` and `binary` fields should be added to the top level `CACHES['foo']` dict, or nested inside `OPTIONS` within that? Examples:
https://emorley.pastebin.mozilla.org/8858134

I would say that if username and password are not mandatory, put them in options. But no strong opinion here.

Claude

[Tim Graham]

As it so happens, someone else ended up tackling this problem for a different use case in https://github.com/django/django/pull/6233.

There's a small design discussion on the pull request about how to structure OPTIONS (possibly deprecating the current behavior of pylibmc that passes them directly to client.behaviors) if anyone would like to weigh in.

[Ed Morley]

Just to follow up on this - general support for passing options to memcached clients has landed on master [1], so it's now possible to enable binary mode, pass SASL credentials and also control any other settings using OPTIONS.

For examples, see the end of the cache arguments documentation:
https://docs.djangoproject.com/en/dev/topics/cache/#cache-arguments

I'm going to try and backport these changes (plus ticket #27152) to django-pylibmc, so the same Django settings file will work for both, to make transitioning from "older Django+django-pylibmc backend" to "Django 1.11+ with stock backend" easier. My ideal end-state is for django-pylibmc to eventually become redundant for the majority of users, with all but the too-specific (eg compression) features being merged into the native backend.

Best wishes,

Ed

[1] https://github.com/django/django/pull/7160