Project-wide cache prefix (low-level API)

[Byron]

I ran into a seemingly common problem with cache key conflicts between
two projects that both share an app and the same memcache instance. I
would imagine that being able to optionally set a PROJECT_CACHE_PREFIX
that is prepended to every key when setting/getting would be a nice
transparent way of ensuring there are no conflicts. Obviously if the
behavior is to share cache between projects, then no prefix can be
specified.

I want to get initial feedback, implications, etc. on this before I
start writing a patch.

Thanks,
-Byron

[Gabriel Hurley]

Personally, I solved this by writing a wrapper around
django.utils.cache that can be dropped in transparently. It was a
really simple matter to have it take a cache prefix (in my case from
settings.py) and pass that into the original functions.

It's a pretty easy fix if the core team agrees that it's useful. Seems
like the debate might be more around adding another setting for it/
where/how to manage that prefix.

All the best,

- Gabriel

[Byron]

Yea, I definitely agree that it is a simple solution. I think the the
fact that the setting is optional and would not break any existing
code, it should be included since it "feels" like a common case... we
shall leave it up to the core devs to determine this.

[batiste]

What about the problem that you can, in theory, use the same
application several time (several instance) within the same project?

I don't have a real use-case, but I have heard people doing it.

http://groups.google.com/group/django-users/browse_thread/thread/076f31a1a5479676/8d5f2e0f8f1e1957

Batiste

[stefanw]

On Jun 2, 9:51 am, batiste <batiste.bie...@gmail.com> wrote:
> What about the problem that you can, in theory, use the same
> application several time (several instance) within the same project?

There is a Google Summer of Code Project http://code.djangoproject.com/wiki/SummerOfCode2010
that will hopefully take care of these issues.

Cheers,
Stefan

[lenz]

Could you share your code with us?
Thanks you!

Lenz

On 2 Jun., 01:14, Gabriel Hurley <gab...@gmail.com> wrote:
> Personally, I solved this by writing a wrapper around

> django.utils.cachethat can be dropped in transparently. It was a
> really simple matter to have it take acacheprefix(in my case from

> settings.py) and pass that into the original functions.
>
> It's a pretty easy fix if the core team agrees that it's useful. Seems
> like the debate might be more around adding another setting for it/
> where/how to manage thatprefix.
>
> All the best,
>
>     - Gabriel
>
> On Jun 1, 8:12 am, Byron <bjr...@gmail.com> wrote:
>

> > I ran into a seemingly common problem withcachekey conflicts between

> > two projects that both share an app and the same memcache instance. I
> > would imagine that being able to optionally set a PROJECT_CACHE_PREFIX
> > that is prepended to every key when setting/getting would be a nice
> > transparent way of ensuring there are no conflicts. Obviously if the

> > behavior is to sharecachebetween projects, then noprefixcan be

[Jeremy Dunck]

On Sun, Jun 13, 2010 at 7:18 AM, lenz <lenzh...@googlemail.com> wrote:
> Could you share your code with us?
> Thanks you!

This might help:
http://gist.github.com/425403

[Giuseppe Ciotta]

I have some similar code here: http://gist.github.com/439868

It might be better because it wraps an arbitrary cache backend adding
prefix handling, it's not limited to memcache.

It omits the prefix if the key name starts with the domain of the
current Site. This is important for us because sometimes we want to
share cache keys with other clients, and in this situation they
shouldn't be aware of the particular prefix in use.

It lacks coverage of new cache methods added in 1.2.

[Giuseppe Ciotta]

On Wed, Jun 16, 2010 at 1:04 AM, Giuseppe Ciotta <gci...@gmail.com> wrote:
> On Wed, Jun 16, 2010 at 12:46 AM, Jeremy Dunck <jdu...@gmail.com> wrote:
>> On Sun, Jun 13, 2010 at 7:18 AM, lenz <lenzh...@googlemail.com> wrote:
>>> Could you share your code with us?
>>> Thanks you!
>>
>> This might help:
>> http://gist.github.com/425403
>
> I have some similar code here: http://gist.github.com/439868
>
> It might be better because it wraps an arbitrary cache backend adding
> prefix handling, it's not limited to memcache.

Uhm, http://gist.github.com/425403 does indeed support an arbitrary
backend, there's just a call to _get_memcache_timeout() which should
be changed in order to make it completely backend agnostic. I left
"*args, **kwargs" signatures everywhere to avoid dealing with
parameters.

[Byron]

Opened a ticket and added my patch.. http://code.djangoproject.com/ticket/13795

On Jun 15, 7:15 pm, Giuseppe Ciotta <gcio...@gmail.com> wrote:

> On Wed, Jun 16, 2010 at 1:04 AM, Giuseppe Ciotta <gcio...@gmail.com> wrote:
> > On Wed, Jun 16, 2010 at 12:46 AM, Jeremy Dunck <jdu...@gmail.com> wrote:

> >> On Sun, Jun 13, 2010 at 7:18 AM, lenz <lenzhir...@googlemail.com> wrote:
> >>> Could you share your code with us?
> >>> Thanks you!
>
> >> This might help:
> >>http://gist.github.com/425403
>
> > I have some similar code here:http://gist.github.com/439868
>
> > It might be better because it wraps an arbitrary cache backend adding
> > prefix handling, it's not limited to memcache.
>

> Uhm,http://gist.github.com/425403does indeed support an arbitrary

[Mikhail Korobov]

This is a good feature and I also have my custom cache backend with
project-level key prefixes. It was easy to implement and doesn't
require changes in django core. But if this backend is in django core
(that's a good idea) then some other changes should be done. There
will be 2 similar options after this change:
CACHE_MIDDLEWARE_KEY_PREFIX and PROJECT_CACHE_PREFIX. I think the
patch should deprecate CACHE_MIDDLEWARE_KEY_PREFIX and maybe overhaul
key_prefix arguments (and their defaults) to make the code cleaner.

On Jun 19, 4:46 am, Byron <bjr...@gmail.com> wrote:
> Opened a ticket and added my patch..http://code.djangoproject.com/ticket/13795

>
> On Jun 15, 7:15 pm, Giuseppe Ciotta <gcio...@gmail.com> wrote:
>
>
>
> > On Wed, Jun 16, 2010 at 1:04 AM, Giuseppe Ciotta <gcio...@gmail.com> wrote:
> > > On Wed, Jun 16, 2010 at 12:46 AM, Jeremy Dunck <jdu...@gmail.com> wrote:
> > >> On Sun, Jun 13, 2010 at 7:18 AM, lenz <lenzhir...@googlemail.com> wrote:
> > >>> Could you share your code with us?
> > >>> Thanks you!
>
> > >> This might help:
> > >>http://gist.github.com/425403
>
> > > I have some similar code here:http://gist.github.com/439868
>
> > > It might be better because it wraps an arbitrary cache backend adding
> > > prefix handling, it's not limited to memcache.
>

> > Uhm,http://gist.github.com/425403doesindeed support an arbitrary

[Byron]

Yes I agree. I never quite understood why the
CACHE_MIDDLEWARE_KEY_PREFIX was implemented, but not at the lower
level (did the cache API become available after the middleware
caching?). Of course a custom backend can be written, but then there
would need to be a different one for each backend type. This is a very
simple change to BaseCache that provides a means off prefixing. I also
added a default for `key_prefix' in BaseCache (so current custom
backends don't need to be rewritten and will work as normal).

BTW, the patch passes all previous tests at that revision (with the
exception of a non-related markup test?) and I also subclassed all the
cache tests and added a CACHE_KEY_PREFIX (renamed from
PROJECT_KEY_PREFIX) for those tests -- they all pass as well. I also
added a preliminary snippet of documentation that most likely will
need to be expanded on.

I would appreciate any suggestions, criticisms, etc.

On Jun 19, 1:26 pm, Mikhail Korobov <kmik...@googlemail.com> wrote:
> This is a good feature and I also have my custom cache backend with
> project-level key prefixes. It was easy to implement and doesn't
> require changes in django core. But if this backend is in django core
> (that's a good idea) then some other changes should be done. There
> will be 2 similar options after this change:
> CACHE_MIDDLEWARE_KEY_PREFIX and PROJECT_CACHE_PREFIX. I think the
> patch should deprecate CACHE_MIDDLEWARE_KEY_PREFIX and maybe overhaul
> key_prefix arguments (and their defaults) to make the code cleaner.
>
> On Jun 19, 4:46 am, Byron <bjr...@gmail.com> wrote:
>
>
>
> > Opened a ticket and added my patch..http://code.djangoproject.com/ticket/13795
>
> > On Jun 15, 7:15 pm, Giuseppe Ciotta <gcio...@gmail.com> wrote:
>
> > > On Wed, Jun 16, 2010 at 1:04 AM, Giuseppe Ciotta <gcio...@gmail.com> wrote:
> > > > On Wed, Jun 16, 2010 at 12:46 AM, Jeremy Dunck <jdu...@gmail.com> wrote:
> > > >> On Sun, Jun 13, 2010 at 7:18 AM, lenz <lenzhir...@googlemail.com> wrote:
> > > >>> Could you share your code with us?
> > > >>> Thanks you!
>
> > > >> This might help:
> > > >>http://gist.github.com/425403
>
> > > > I have some similar code here:http://gist.github.com/439868
>
> > > > It might be better because it wraps an arbitrary cache backend adding
> > > > prefix handling, it's not limited to memcache.
>

> > > Uhm,http://gist.github.com/425403doesindeedsupport an arbitrary

[Russell Keith-Magee]

On Sun, Jun 20, 2010 at 6:16 AM, Byron <bjr...@gmail.com> wrote:
> Yes I agree. I never quite understood why the
> CACHE_MIDDLEWARE_KEY_PREFIX was implemented, but not at the lower
> level (did the cache API become available after the middleware
> caching?). Of course a custom backend can be written, but then there
> would need to be a different one for each backend type. This is a very
> simple change to BaseCache that provides a means off prefixing. I also
> added a default for `key_prefix' in BaseCache (so current custom
> backends don't need to be rewritten and will work as normal).
>
> BTW, the patch passes all previous tests at that revision (with the
> exception of a non-related markup test?) and I also subclassed all the
> cache tests and added a CACHE_KEY_PREFIX (renamed from
> PROJECT_KEY_PREFIX) for those tests -- they all pass as well. I also
> added a preliminary snippet of documentation that most likely will
> need to be expanded on.
>
> I would appreciate any suggestions, criticisms, etc.

Apologies for not weighing in sooner -- I think this is a reasonable
feature proposal; I've just marked the ticket as accepted. The patch
looks pretty good too, although I've got a couple of light polish
requests. I've attached comments on the ticket.

Yours,
Russ Magee %-)

[Byron]

Updated the patch http://code.djangoproject.com/ticket/13795

On Jun 23, 11:12 am, Russell Keith-Magee <russ...@keith-magee.com>
wrote:

> On Sun, Jun 20, 2010 at 6:16 AM, Byron <bjr...@gmail.com> wrote:
> > Yes I agree. I never quite understood why the
> > CACHE_MIDDLEWARE_KEY_PREFIX was implemented, but not at the lower

> > level (did thecacheAPI become available after the middleware

> > caching?). Of course a custom backend can be written, but then there
> > would need to be a different one for each backend type. This is a very
> > simple change to BaseCache that provides a means off prefixing. I also
> > added a default for `key_prefix' in BaseCache (so current custom
> > backends don't need to be rewritten and will work as normal).
>
> > BTW, the patch passes all previous tests at that revision (with the
> > exception of a non-related markup test?) and I also subclassed all the

> >cachetests and added a CACHE_KEY_PREFIX (renamed from

[Jacob Kaplan-Moss]

On Wed, Aug 4, 2010 at 8:06 AM, Byron <bjr...@gmail.com> wrote:
> Updated the patch http://code.djangoproject.com/ticket/13795

Looks good. A couple of questions/comments:

* Is there a reason to keep CACHE_MIDDLEWARE_KEY_PREFIX around? I don't
quite see the point of a separate setting, so unless you can think of a
good reason to have both I'd suggest we deprecate
CACHE_MIDDLEWARE_KEY_PREFIX in favor of CACHE_KEY_PREFIX globally.

* In a few places you have functions defined like::

def get_cache(backend_uri, key_prefix=settings.CACHE_KEY_PREFIX):

This is a bad idea: it forces the loading of settings at import time,
meaning that if you've not defined DJANGO_SETTINGS_MODULE or called
settings.configure() you can't even import the cache module.

As a rule, you should avoid evaluating settings at import time, so in
this case you'd do something like::

def get_cache(backend_uri, key_prefix=None):
if key_prefix is None:
key_prefix = settings.CACHE_KEY_PREFIX
...

* Have you considered supporting "versioning" of keys to help with cache
invalidation? Eric Florenzano has been doing some interesting
experimenting along those lines in django-newcache
(http://github.com/ericflo/django-newcache); you may want to look at the
code and/or talk to him about working some of his ideas back into your
key prefix proposal. At the very least, any changes we make to the
caching backend should allow the sorts of things he's doing to keep
working; ideally we should make those sorts of changes really easy to
make.

Thanks for your work - this looks good.

Jacob

[Byron]

Thanks Jacob. I personally don't think the CACHE_MIDDLEWARE_KEY_PREFIX
is necessary anymore. I suggest removing it, but I would imagine it
would have to be deprecated for the next release then removed in 1.4?

> As a rule, you should avoid evaluating settings at import time, so in
> this case you'd do something like::

Ah, good point. I will make the changes.

> Have you considered supporting "versioning" of keys to help with cache invalidation?

Thanks for the link to Eric's work, I will take a look and see how we
can join forces.

On Aug 4, 10:57 am, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
> On Wed, Aug 4, 2010 at 8:06 AM, Byron <bjr...@gmail.com> wrote:

> > Updated the patchhttp://code.djangoproject.com/ticket/13795

[Ned Batchelder]

On 8/4/2010 10:57 AM, Jacob Kaplan-Moss wrote:

On Wed, Aug 4, 2010 at 8:06 AM, Byron <bjr...@gmail.com> wrote:
  

Updated the patch http://code.djangoproject.com/ticket/13795
    

* Have you considered supporting "versioning" of keys to help with cache
  invalidation? Eric Florenzano has been doing some interesting
  experimenting along those lines in django-newcache
  (http://github.com/ericflo/django-newcache); you may want to look at the
  code and/or talk to him about working some of his ideas back into your
  key prefix proposal. At the very least, any changes we make to the
  caching backend should allow the sorts of things he's doing to keep
  working; ideally we should make those sorts of changes really easy to
  make.

  

Couldn't versioning be handled by setting CACHE_KEY_PREFIX to include some varying data like the svn revision?  We've got a cache key prefix hacked into our work environment, and I ensure my dev machine never gets stale cache data like this:

import time
CACHE_KEY_PREFIX = "dev-ned-%s" % time.time()

This uses the start time of the dev server as part of the cache key so each invocation gets fresh data.  In production, you'd use something different, but this illustrates the point.  Or is there some more elaborate versioning being considered here?

--Ned.

[flo...@gmail.com]

The most flexible way is to be able to specify a callable that runs on
each cache key before it is sent to the server. Then it's just up to
Django to provide a sensible default callable, but people could
override it to provide one which matches their own requirements. This
is what I do in django-newcache, at least.

Thanks,
Eric Florenzano

On Aug 4, 11:31 am, Ned Batchelder <n...@nedbatchelder.com> wrote:
> On 8/4/2010 10:57 AM, Jacob Kaplan-Moss wrote:
>
>
>
> > On Wed, Aug 4, 2010 at 8:06 AM, Byron<bjr...@gmail.com>  wrote:
>

> >> Updated the patchhttp://code.djangoproject.com/ticket/13795

[Byron]

Hi Eric - I took a look at django-newcache and I like the overall
design. The whole thought behind my patch was a simple way to provide
a global cache prefix since I had ran into key clashes among my
projects (equivalent to your 'FLAVOR' setting). Your solution provides
a much more flexible and extensible interface (especially with
versioning).

What are your thoughts on adding these implementations to core? Of
course these additional hooks can be implemented for the other
backends as well.

On Aug 4, 3:11 pm, "flo...@gmail.com" <flo...@gmail.com> wrote:
> The most flexible way is to be able to specify a callable that runs on
> each cache key before it is sent to the server.  Then it's just up to
> Django to provide a sensible default callable, but people could
> override it to provide one which matches their own requirements.  This
> is what I do in django-newcache, at least.
>
> Thanks,
> Eric Florenzano
>
> On Aug 4, 11:31 am, Ned Batchelder <n...@nedbatchelder.com> wrote:
>
>
>
> > On 8/4/2010 10:57 AM, Jacob Kaplan-Moss wrote:
>
> > > On Wed, Aug 4, 2010 at 8:06 AM, Byron<bjr...@gmail.com>  wrote:
>
> > >> Updated the patchhttp://code.djangoproject.com/ticket/13795
>
> > > * Have you considered supporting "versioning" of keys to help with cache
> > >    invalidation? Eric Florenzano has been doing some interesting
> > >    experimenting along those lines in django-newcache

> > >    (http://github.com/ericflo/django-newcache);youmay want to look at the

[flo...@gmail.com]

On Aug 4, 1:33 pm, Byron <bjr...@gmail.com> wrote:
> What are your thoughts on adding these implementations to core?

I think that the callable key function would fit well in core, and
debatably the flavor, version, and hashing stuff could fit well in
core too. I don't really think that the thundering herd protection
stuff is necessarily right for core, but that would be for others to
decide.

Thanks,
Eric Florenzano