How to support GC like functionality in redis

[Sonali Palkar]

I want to implement a garbage collector like feature for my
application which uses redis db. I am using sets, sorted sets, lists
and hashes of redis to store my data. I have a real time app which is
inserting/updating contents in redis. I want to cleanup selective
data from those structures based on few conditions without stopping my
real time application which is inserting /updating values.

I want to ensure that my delete operations are atomic and running
application will not get affected whenever my GC will be cleaning up
unwanted data from redis memory. Any inputs how to go about?


Thanks.
Sonali

[Didier Spezia]

Hi,

there is no efficient way to iterate on the whole database with Redis.

The KEYS command will block the instance when you have a

significant number of keys. So you should not use it if your

application is supposed to be always responsive.

If you have the memory space, you can start a slave and iterate

using KEYS on the slave while still updating the master, but

it can be tricky to guarantee the consistency.

Rather than scanning the whole database like a garbage collector,

it is probably more sensible to evaluate whether the objects should

be discarded at set/update time, by posting events to a queue

(i.e. a Redis list) A pool of worker processes can dequeue those

events and then analyze the objects and eventually delete them ...

Regards,

Didier.

[Josiah Carlson]

Use something like the manual expiration recipe at the end of:
http://dr-josiah.blogspot.com/2011/02/some-redis-use-cases.html

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[Santosh Kumar]

Hi,

You could look into using the "expire" command to just let redis expire out your keys. If you are looking to gc entries in a sorted set, you could look at either the zremrangebyrank and zremrangebyscore. Lists have a ltrim method that you can use to simulate a capped collection. Since redis is single threaded all of these operations are atomic, but vary in terms of time complexity (the redis docs are super useful for this info).

Best,

Santosh