GSOC Proposal: Adding Redis Cache Backend To Django Core.

[Girish Sontakke]

Hello, I am Girish Sontakke from India. I am willing to start contributing to Django through GSOC 2021. I have gone through the Idea List available on the Django wiki page for GSOC and select the project of adding Redis Cache Backend To Django Core. For this, I have explored the django.core.cache section of the framework and take some Ideas for Django's Cache Framework from the documentation available at djangoprojects.com.

There are some caching backends already available in the django core but Redis being one the popular caching engines, there should be the implementation of Redis Backend in the core django package.  There are some Greate packages out there for integrating Redis into Django such as django-redis and django-redis-cache, still, there is a need for inner Redis Backend in django.

For this I am writing a proposal here. I request all of the fellow developers and mentors to give some suggestions for this Idea and share your views on this project.

 

With Regards, 

Girish Sontakke.

[Carlton Gibson]

Hi Girish, Welcome. 

Have a search of the history of the mailing list here. There was one, or a couple of (?) threads on adding a Redis backend recently (last year or two). 

From the "Why Redis" section of your proposal, I think the answer is simply that lots of users (most in fact according the recent survey) are using Redis for caching, and we a should have a backend in core for that. 

The scope is JUST to satisfy the cache backend API, and no more. It's a non-goal to deal with any other Redis features, so various data structures and such is not relevant. (Third-party packages would could offer things in this space.) 

For the work breakdown compare the recent PR adding the pymemcache backend. #29887 — the project would be similar in scope.

The ground here is well-trodden, so most of it is documentation — we need to be very clear on requirements, installation and setup. 

I hope that helps. 

Kind Regards,

Carlton

[Girish Sontakke]

Thank you, Carlton,

I search for discussion in the mailing list about Redis cache I found a helpful discussion here. Should I put my proposal there or should I continue here?

In the "Why Redis" section of my proposal, I have mentioned about the different data structures provided by Redis because I thought It is important with respect to the speed and efficiency of Redis cache backend compared to other backends like filesystem and database cache backends which eventually leads to its popularity. I will add the point about the popularity of redis among the developer community.

I have gone through the PR #13310, proposing the patch for adding support for pymemcache in the Django core, as suggested by you at the forum discussion.

Now I have doubt that whether I should remove Provides Many Data-Structures point from the "Why Redis" section or keep it there?.

That helped me a lot.

With Regards,

Girish

[Carlton Gibson]

On 31 Mar 2021, at 11:00, Girish Sontakke <girishs...@gmail.com> wrote:

Now I have doubt that whether I should remove Provides Many Data-Structures point from the "Why Redis" section or keep it there?.

I don’t think it’s vital either way… but it seems a distraction. That Redis has HyperLogLog isn’t something that will matter at all to the cache backend. 

I’d try to focus on the details of the implementation: Look at the backend API, set(), get() &co. Look at the two existing third-party backend, and see what they do there. 

I’d also think about getting the documentation right — there are a few options with installs and connections and … — you don’t have to solve all of those in the proposal but showing that you’re aware of them would be good.

[Girish Sontakke]

Hello Carlton,

As suggested by you I made some revisions to my proposal. Still, it is not a perfect one, I am trying to improve it. Thanks a lot for the help.

Kind Regards,

Girish

[Dan Davis]

Since REDIS is already usable as a cache for Django using 3rd party code, I would rather see a circuit breaker pattern applied to the cache, across all backends.  I'm not that active in maintaining Django right now, but a cache may not be fully redundant, and it also can fail. If the cache is truly used as a cache, then it should be possible to try the cache, and if it fails for several tries, then Django could start to bypass it, and then try to re-enable it later.

This is perhaps a bigger effort than a GSOC project - the hard part is doing this across multiple processes and threads.  What does it mean if the cache is only available for some fraction of the instances/processes hosting a project?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/97c757dd-b09f-49f4-9e3f-0ee4a8e4887cn%40googlegroups.com.

[Girish Sontakke]

Hello,

If I am not wrong, you are talking about the cache incoherence problem.  Currently, I don't have any concrete idea to tackle this problem but I will try to figure it out.

Thanks for bringing this to concern.

Kind Regards,

Girish

[Dan Davis]

cache coherence can be a problem, but I am thinking more about what happens when the cache, which is supposed to make things faster, is not available due to error.  Django ends up waiting for it.  Each request waits for the cache, and this is a hard problem because there is no common shared place to write that the cache is down.  However, introducing anything like that introduces problems also with cache coherence.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/399dbcd7-7127-4715-b2cd-3f5d349d605bn%40googlegroups.com.