How to do a counter in DB cache class?

[Mike Lissner]

This might be more of a Python question than a Django one, but in this ticket I'm hoping to make the DB cache a bit more performant by having it not cull stale entries every time somebody adds, changes, or touches a cache key.

The idea is to use a setting or or class attribute so that the cache is instead culled every 50 times or 1000 times or whatever. Most of this is easy, but is there a good way to maintain a counter of how many times a key has been set?

The best I've come up with is just to do a mod of a random number, and let that be good enough. E.g.:

random_number = randint(0, CULL_EVERY_X)

if random_number == CULL_EVERY_X:

    cull()

That's pretty sloppy, but it'd work (on average), and it'd handle things like counting in a multi-process set up.

The other approach, I suppose, would be to keep a counter in the DB and just increment it as needed, but it'd be nice to do something in memory even if it's a little less accurate, since this counter doesn't have to be exact.

Anybody have thoughts on this?

Thanks,

Mike

[Adam Johnson]

Hi Mike!

Probabilistic culling probably is the best we can do in the DB cache, aside from moving culling to a background task.

I wrote an implementation of this probabilistic culling in the django-mysql cache backend (which is mysql only): https://django-mysql.readthedocs.io/en/latest/cache.html#culling . This also shows a way of pushing the cull into a background task by setting the probability to 0 and calling the cull() method.

Counting in the DB is a bad idea since it will put every write through a single hot row, and if transactions are active that will even serialize requests - ouch!

An in-python counter, with a random offset per process (actually process id *plus* thread id), could also work. But it would be trickier imo.

Thanks,

Adm

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7f74e5f0-3206-4956-9ede-79788eda7982n%40googlegroups.com.

[Mike Lissner]

Wow, that's pretty great! Did you consider merging this functionality into django itself? I hadn't seen anything related before now.

You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/qFxdRNLDGuA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM2jSqZNGuBAobuuh%3D6%3DgWyD7SD0TTEzgUxO9CdqRhXotg%40mail.gmail.com.

--

Mike Lissner

Executive Director

Free Law Project

https://free.law

[Adam Johnson]

I would love to make it work on all backends so we could have an improved database cache class, but haven't the time.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAKs1xOEofZLoe7RMCR%2BxT6vkCbiFjGjMrJprfLGsVubXR6n9mQ%40mail.gmail.com.