Redis: what's the proper way to clear all the cache keys of a given app?

[Lisandro]

I'm having some trouble with Redis and I thought some of you could enlighten me :)

I have a web2py instance running with several apps. 

I have a Redis instance working as a cache, with maxmemory set to 20gb and LRU eviction policy.

Every application connects to the same Redis instance.

One of those apps is the main one, from wich I perform some administrative tasks like:

 - getting a list of the cache keys of a specific app

 - clearing all the cache keys of a specific app.

In order to get the list of the keys cached by a specific app I use this custom code:

def get_cache_keys(application):
    import re

    result = []
    regex = ':*'
    prefix = 'w2p:%s' % application
    cache_set = 'w2p:%s:___cache_set' % application
    r = re.compile(regex)
    buckets = redis_conn.smembers(cache_set)
    if buckets:
        keys = redis_conn.sunion(buckets)
        for key in keys:
            if r.match(str(key).replace(prefix, '', 1)):
                result.append(key)
    return result

In the other hand, if I need to clear all the cached keys of a specific app, I get the key list (using the above funciton) and then iterate those keys calling redis_conn.delete(key)
I thought this was ok, but today I've found a problem: I've found a key that was generated by an app but it isn't returned by the code above. 

To be sure, I connected to Redis from the terminal and in fact I was able to get the key and its value, but running the code above the key isn't listed. I'v checked the TTL of the key and it is still valid. 

It's not the only case: I've found several cases.

For example, I have a web2py app called "transmedia".

From redis-cli, I want to know the SETs (buckets) that web2py created to store the keys of that specific app, so I run:

127.0.0.1:6379> SMEMBERS w2p:transmedia:___cache_set
 1) "w2p:transmedia:___cache_set:26383112"
 2) "w2p:transmedia:___cache_set:26384550"
 3) "w2p:transmedia:___cache_set:26383115"
 4) "w2p:transmedia:___cache_set:26383117"
 5) "w2p:transmedia:___cache_set:26383436"
 6) "w2p:transmedia:___cache_set:26383118"
 7) "w2p:transmedia:___cache_set:26383113"
 8) "w2p:transmedia:___cache_set:26383111"
 9) "w2p:transmedia:___cache_set:26383495"
10) "w2p:transmedia:___cache_set:26383440"
11) "w2p:transmedia:___cache_set:26383170"
12) "w2p:transmedia:___cache_set:26383116"

Then I checked all those SETs to get all the keys stored by the app:

127.0.0.1:6379> SMEMBERS w2p:transmedia:___cache_set:26384550
 1) "w2p:cache:transmedia:url_logo_movil"
 2) "w2p:cache:transmedia:menu1"
 3) "w2p:cache:transmedia:url_imagen_default"
 4) "w2p:cache:transmedia:CT"
 5) "w2p:cache:transmedia:url_imagen_logo_newsletter"
 6) "w2p:cache:transmedia:html_head"
 7) "w2p:cache:transmedia:menu0"
 8) "w2p:cache:transmedia:TEMPLATE"
 9) "w2p:cache:transmedia:url_fondo_personalizado"
10) "w2p:cache:transmedia:url_favicon"
11) "w2p:cache:transmedia:C"
12) "w2p:cache:transmedia:CONFIG"
13) "w2p:cache:transmedia:url_logo_grande"
14) "w2p:cache:transmedia:html_body"
15) "w2p:cache:transmedia:url_logo"

There we can see 15 keys. 

All the other sets are "(empty list or set)".

But here is the weird part: the application "transmedia" stored a key that I don't see in the list: w2p:cache:transmedia:url_imagen_publicidad_newsletter

Checking from redis-cli I can see that the key is still there and its TTL is still valid:

127.0.0.1:6379> GET w2p:cache:transmedia:url_imagen_publicidad_newsletter
"\x80\x02U\x00."
127.0.0.1:6379> TTL w2p:cache:transmedia:url_imagen_publicidad_newsletter
(integer) 18711

I can confirm this discrepancy if I perform a SCAN like this:

$ redis-cli --scan --pattern w2p:cache:transmedia:*
w2p:cache:transmedia:url_logo_movil
w2p:cache:transmedia:url_favicon
w2p:cache:transmedia:url_fondo_personalizado
w2p:cache:transmedia:TEMPLATE
w2p:cache:transmedia:menu1
w2p:cache:transmedia:html_head
w2p:cache:transmedia:url_imagen_logo_newsletter
w2p:cache:transmedia:html_body
w2p:cache:transmedia:url_logo
w2p:cache:transmedia:CT
w2p:cache:transmedia:url_logo_grande
w2p:cache:transmedia:CONFIG
w2p:cache:transmedia:C
w2p:cache:transmedia:url_imagen_publicidad_newsletter
w2p:cache:transmedia:url_imagen_default
w2p:cache:transmedia:menu0

Notice the output now includes the 16 keys, not 15.

So why isn't the key listed when scanning all the SETS (buckets) that web2py created for the app? Is there something wrong with my code?

My main concern is about clearing all the keys of a given app. 

Right now I can't trust that my code will delete all the keys, because the function that gets the key list doesn't always include all the keys that the app stored. 

So another question: is there a better way to delete all the keys cached by an app? 

I thought I could use redis-cli from the commandline like this:

$ redis-cli --scan --pattern w2p:cache:transmedia:* | xargs redis-cli del

This would work, but in order to run it from within my main app I would have to use subprocess.
What do you think?

[Lisandro]

I have a theory about what's going on.

The implementation of Redis in web2py stores the keys in "buckets" or sets, and the name of those sets are stored in cache with no expiration time accordingly to the code documentation: 

"all buckets are indexed in a fixed key that never expires"

For what I see, this is how web2py stores keys and buckets for a given appname:

127.0.0.1:6379> SMEMBERS w2p:appname:___cache_set
 1) "w2p:appname:___cache_set:26319660"
 2) "w2p:appname:___cache_set:26377475"
 3) "w2p:appname:___cache_set:26336873"
 4) "w2p:appname:___cache_set:26318136"

Every key set can be accessed with the same command, for example:

127.0.0.1:6379> SMEMBERS w2p:appname:___cache_set:26319660
 1) "w2p:cache:appname:url_logo_movil"
 2) "w2p:cache:appname:menu1"
 3) "w2p:cache:appname:url_imagen_default"
 4) "w2p:cache:appname:CT"
 5) "w2p:cache:appname:url_imagen_logo_newsletter"

At the other hand:

• My Redis instance is configured to use no more than 20 GB of RAM, with an eviction policy of "allkeys-lru", that is evict keys lest recently used. 
• The limit of 20gb has been reached a few times in the past.

Accordingly to the Redis docs, one alternative to allkeys-lru is "volatile-lru". This option only evict keys that have an expiration time. So I deduce that my confiuguration (allkeys-lru) evics keys no matter they have or not an expiration time. And considering web2py sets the bucket names with keys that never expire, my theory is that some keys storing the name of the buckets were evicted. That's why my code doesn't find some keys that are still indeed cached: that's because my code does the search from bucket names.

That's my best guess.. 

Anyway, I'll try to adapt the code of my app that clears the cache of an application: I think I'll go for subprocess with a command like:

$ redis-cli --scan --pattern w2p:cache:appname:* | xargs redis-cli del

Any comment or suggestion will be much appreciated.

Warm regards,

Lisandro.