Redis cache support in core

458 views
Skip to first unread message

Dulmandakh Sukhbaatar

unread,
Jun 17, 2019, 11:14:25 AM6/17/19
to Django developers (Contributions to Django itself)
Hello,

I would like to work on Redis support in core, and I would like to discuss proper solution for that.

Redis is getting so popular and almost every modern backend stack uses it someway, therefore I think that supporting it as a cache backend in core would make Django more appealing. A solution I'm proposing is to extract base KV backend from current Memcached and extend it for both Memcached and Redis, and this won't add many new code to the core. Also we'll have base class for KV storage backends.

Thanks.

Josh Smeaton

unread,
Jun 18, 2019, 11:44:15 PM6/18/19
to Django developers (Contributions to Django itself)
There are already several 3rd party packages that implement redis as a django cache backend, for example https://github.com/niwinz/django-redis

We already have a base class for cache backends - and several implementing it (such as memcache). I don't think there's much benefit taking on another backend when it's already got very good support as an external package.

Ivan Anishchuk

unread,
Jun 19, 2019, 9:00:27 PM6/19/19
to django-d...@googlegroups.com
How about making one of the third-party packages an optional dependency? Celery, for example, does that: you can just install celery[redis] without having to figure out what other packages you need to enable redis support.

Ivan.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bdb84d20-0489-4ecd-b198-fa5878f5c617%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Josh Smeaton

unread,
Jun 19, 2019, 9:02:50 PM6/19/19
to Django developers (Contributions to Django itself)
Celery explicitly document their integration with Redis though. I don't think we want to take over documenting the setup of a 3rd party package in Django.

On Thursday, 20 June 2019 11:00:27 UTC+10, Ivan Anishchuk wrote:
How about making one of the third-party packages an optional dependency? Celery, for example, does that: you can just install celery[redis] without having to figure out what other packages you need to enable redis support.

Ivan.

On Wed, Jun 19, 2019 at 6:44 AM Josh Smeaton <josh....@gmail.com> wrote:
There are already several 3rd party packages that implement redis as a django cache backend, for example https://github.com/niwinz/django-redis

We already have a base class for cache backends - and several implementing it (such as memcache). I don't think there's much benefit taking on another backend when it's already got very good support as an external package.


On Tuesday, 18 June 2019 01:14:25 UTC+10, Dulmandakh Sukhbaatar wrote:
Hello,

I would like to work on Redis support in core, and I would like to discuss proper solution for that.

Redis is getting so popular and almost every modern backend stack uses it someway, therefore I think that supporting it as a cache backend in core would make Django more appealing. A solution I'm proposing is to extract base KV backend from current Memcached and extend it for both Memcached and Redis, and this won't add many new code to the core. Also we'll have base class for KV storage backends.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

Aymeric Augustin

unread,
Jun 20, 2019, 4:35:39 AM6/20/19
to django-d...@googlegroups.com
Hello,

Until now, this feature request was always declined. As a consequence, every user who wants Redis has to choose between django-redis and django-redis-cache. Considering that Redis must be the most popular cache backend these days, I'm in favor of providing an off-the-shelf solution in Django itself. It's unlikely to be a large maintenance burden. It will "just work".

I don't know if factoring out common functionality between the memcached and redis backends will really make them easier to maintain. We'll end up with three modules (key-value, memcached and redis), which will be more complicated than two. If we had three similar backends, that would be a strong argument for factoring out common functionality. With only two backends, it may not be worth the complexity.

To move this forwards, my suggestion would be to write a DEP, to flesh out the rationale for a built-in solution, and to focus on the breadth of functionality the built-in backend would support. My preference would be a basic set of features, like the other cache backends. I believe that's what django-redis-cache does. This will leave room for third-party packages like django-redis to provide more advanced features. To give a concrete example, like other cache and database backends, a redis backend could provide persistent connections but not implement a connection pool. Until now Django has left the management of connection pools to third-party packages.

Best regards,

-- 
Aymeric.



--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Loïc Bistuer

unread,
Jun 20, 2019, 5:11:35 AM6/20/19
to django-d...@googlegroups.com

I’m also +1 on having it as part of core given how prominent Redis is.

 

I agree with Aymeric that it shouldn’t be Django’s responsibility to provide or wrap advanced Redis functionalities but I think it would be lovely if we at least exposed the connection/client as a public API.

 

Regards,

Loïc

Ivan Anishchuk

unread,
Jun 21, 2019, 8:43:13 AM6/21/19
to django-d...@googlegroups.com
I wouldn't say it's that complicated a setup. It would require a single settings snippet -- just like the ones for other backends -- and, I guess, a link to django-redis docs for more details (if django-redis is what we recommend), maybe a quick explanation of what is CLIENT_CLASS and other options. While it would add some maintenance burden (occasionally checking whether any breaking changes were introduced in the 3rd party package that require updating settings) it's still way easier than adding a backend to django core.

While I agree with others about redis being popular and adding such a backend in django being a good idea (I would love if that happened) I understand the reasons for not doing it. A recommendation of a 3rd party package + setup documentation, on the other hand, is pretty simple thing to do.

If we want, it's also not very hard to provide `django.core.cache.backend.redis.Redis Cache` that depends on django-redis and is an alias for `django_redis.cache.RedisCache` -- it's basically the way it works with DB backends, I don't see why it wouldn't be a good idea for cache as well.

Ivan.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Markus Holtermann

unread,
Jun 21, 2019, 9:17:25 AM6/21/19
to Django developers
Hi all,

may I suggest that django-redis may be "promoted" to an official Django package under the Django GitHub organization? This would follow https://github.com/django/deps/blob/master/final/0007-official-projects.rst . The package would be pointed out explicitly in the Django docs but would be shipped outside of Django.

The benefit with 3rd party packets is their shorter release cycle. Which, in the context of django-redis could be beneficial .

/Markus
> >>> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bdb84d20-0489-4ecd-b198-fa5878f5c617%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/bdb84d20-0489-4ecd-b198-fa5878f5c617%40googlegroups.com?utm_medium=email&utm_source=footer>.
> >>> For more options, visit https://groups.google.com/d/optout.
>
> > --
> > You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> > To post to this group, send email to django-d...@googlegroups.com.
> > Visit this group at https://groups.google.com/group/django-developers.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/335b087c-801a-452b-a5b3-a9711e4a00b8%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/335b087c-801a-452b-a5b3-a9711e4a00b8%40googlegroups.com?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To post to this group, send email to
> django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CADPNjZ5rU9wxFg_FCohJaO9%3DaA8wZ1PSyhx580BUZ6-Xr2BVDA%40mail.gmail.com <https://groups.google.com/d/msgid/django-developers/CADPNjZ5rU9wxFg_FCohJaO9%3DaA8wZ1PSyhx580BUZ6-Xr2BVDA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

Josh Smeaton

unread,
Jun 27, 2019, 9:30:16 PM6/27/19
to Django developers (Contributions to Django itself)
Markus, I'd prefer that approach over vendoring a more incomplete solution. Bonus points if we can add packages to extra_requires, so that `pip install Django[redis]` would work nicely.
> >  To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/335b087c-801a-452b-a5b3-a9711e4a00b8%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/335b087c-801a-452b-a5b3-a9711e4a00b8%40googlegroups.com?utm_medium=email&utm_source=footer>.
> >  For more options, visit https://groups.google.com/d/optout.
>
>  --
>  You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
>  To unsubscribe from this group and stop receiving emails from it, send
> an email to django-d...@googlegroups.com.
>  To post to this group, send email to
> django-d...@googlegroups.com.
>  Visit this group at https://groups.google.com/group/django-developers.
>  To view this discussion on the web visit

Carlton Gibson

unread,
Sep 5, 2019, 1:53:59 AM9/5/19
to Django developers (Contributions to Django itself)
A bit delayed but, I'd support a Redis in core. (I still periodically have to go look up the difference between django-redis and django-redis-cache; "Which one did I pick last time?" I think we could help folks out with the first steps here.)

Aymeric's summary seems correct: 


On Thursday, 20 June 2019 10:35:39 UTC+2, Aymeric Augustin wrote:
To move this forwards, my suggestion would be to write a DEP, to flesh out the rationale for a built-in solution, and to focus on the breadth of functionality the built-in backend would support. My preference would be a basic set of features, like the other cache backends. I believe that's what django-redis-cache does. This will leave room for third-party packages like django-redis to provide more advanced features. To give a concrete example, like other cache and database backends, a redis backend could provide persistent connections but not implement a connection pool. Until now Django has left the management of connection pools to third-party packages.

To that I'd just add, maybe reaching out to the maintainers on django-redis and django-redis-cache to see if they'd like to join in might be worth it. 

I'd be happy to help review a preliminary draft for a DEP.

Kind Regards,

Carlton

Jacob Rief

unread,
Sep 5, 2019, 2:33:01 AM9/5/19
to django-d...@googlegroups.com
I'm also in favor on having it as part of the core, since memcache is also supported.

One of the nice features Redis provides, is the possibility to invalidate one or more cached object by using a wildcard key.
It namely is the method delete_pattern() added by django-redis-cache to the given Django caching backend. That
(or a similar method) then should be part of the other Django caching backends as well, such as the dummy cache or in-memory cache.

Jacob

Matthew Pava

unread,
Sep 5, 2019, 9:29:38 AM9/5/19
to django-d...@googlegroups.com

I’d just like to point out that Redis support on Windows is limited at best. All other technologies that Django uses, as far as I can recall, do support Windows.

 

From: django-d...@googlegroups.com [mailto:django-d...@googlegroups.com] On Behalf Of Jacob Rief
Sent: Thursday, September 5, 2019 1:33 AM
To: django-d...@googlegroups.com
Subject: Re: Redis cache support in core

 

I'm also in favor on having it as part of the core, since memcache is also supported.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

Tom Forbes

unread,
Sep 5, 2019, 9:46:07 AM9/5/19
to django-d...@googlegroups.com
That's not necessarily a blocker, and Memcached is not simple to run on Windows either - there are no official prebuilt binaries, as far as I'm aware.

signature.asc

Christian González

unread,
Sep 5, 2019, 10:19:20 AM9/5/19
to django-d...@googlegroups.com

Just as a "new" user of Django channels/Redis:

to install/setup Redis was one of the big obstacles in my mind that kept me from using channels until now. It's not that it would be complicated to install - I just have a "knot in my thoughts" when trying to create a simple channels app and used days of investigation on "isn't that possible to use channels without the redis server setup? Can't I just install django and django-channels and go for it, at least for development?"

Maybe it's easier as I thought, but adding that as a core feature would be a big benefit IMHO.

If I understood all correctly.

Greets, Christian


Am 05.09.19 um 15:45 schrieb Tom Forbes:
pEpkey.asc

Aymeric Augustin

unread,
Sep 5, 2019, 10:40:28 AM9/5/19
to django-d...@googlegroups.com
Hello,

This is an interesting point — and one we tend to forget too often.

The documentation for memcached says that it doesn't support Windows:

The documentation for redis says that it supports Windows, with some limitations, and that there's an effort by Microsoft to lift these limitations:

So the Windows situation seems slightly better with Redis than with memcached, even if it isn't ideal.

Best regards,

-- 
Aymeric.

Jon Dufresne

unread,
Dec 6, 2019, 12:20:20 PM12/6/19
to django-d...@googlegroups.com, ni...@niwi.nz
The django-redis maintainer is currently in the process of stepping down and is moving away from the project. Should we propose moving it to Django's GitHub? For details, see https://github.com/niwinz/django-redis/issues/400

On the other hand, if there is interest in supporting a Redis backend in core, given django-redis's status, now might be a good time to consider it. As long as memcache is in core, I'm +1 on including Redis as well.

Ivan Anishchuk

unread,
Dec 11, 2019, 6:01:54 AM12/11/19
to django-d...@googlegroups.com
Agree, an officially-recomended package plus "extra" dependency (so you don't have to look it up on every install) would probably be ideal. We don't need it in core as long as it's easy to install without looking up and works reliably.

We could also think about adding some options in settings.py automatically (copy-pasting settings is about as bad as looking up packages) but honestly I'm used to having to rewrite settings all the time and don't know much about available built-in automaton.

Ivan.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
Reply all
Reply to author
Forward
0 new messages