getting notified when a key expires
Amit Rathore
Is there any way to get notified when a key expires in Redis? If not
directly, then are there any common practices to approaching this?
Regards,
Amit Rathore,
Runa, Inc.
Salvatore Sanfilippo
2.2 to issue a PUBLISH against some channel named like
__keyspace__::expires for every expired key.
I know a few guys that could be very happy with this simple change.
Cheers,
Salvatore
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
--
Salvatore 'antirez' Sanfilippo
http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele
Jeremy Zawodny
teleo
notifications, if they were supported?
Amit Rathore
out after 1 hour of inactivity. We accomplish this using the Redis key-
expiration mechanism. If we can get notified when a session expires,
we would like to run some reports. This will allow us to show real-
time metrics elsewhere.
Do you need more information?
Regards,
Amit.
Pedro Melo
On Thu, Sep 16, 2010 at 9:40 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> No way currently... but actually I'm tempted of putting an option in
> 2.2 to issue a PUBLISH against some channel named like
> __keyspace__::expires for every expired key.
>
> I know a few guys that could be very happy with this simple change.
Make that two guys now, and you have a set of free <insert favorite
beverage here> waiting for you at Codebits 2010 :)
Bye,
--
Pedro Melo
http://www.simplicidade.org/
xmpp:me...@simplicidade.org
mailto:me...@simplicidade.org
Salvatore Sanfilippo
> Yeah, it has come up several times over the last year.
> Jeremy
Yeah... I try to resist to see if something is *really* needed in
order to avoid bloating Redis, but it is possible that now is about
time?
Now I need a collective effort to give me the power :) That is, please
give me use cases that you guys are experimenting at work where this
could be useful.
I'll try with a few:
1) You can use it as a timer to do expiring of other things. For
instance you want to expire set elements, what you can do is having
two keys for every unix time in the future where you have at least one
"set element expire" active: one of the keys is just a dummy key with
an expire set at the same time (see EXPIREAT). The other key is a list
of set elements I want to expire when this happens. So when I get the
notification, I read the list, and remove the set elements. It's wise
to also set a (more far in time) expire in the list key as well, so
that if you miss the notification, the cleanup of the two keys still
happen.
2) You have workers updating your cache, that is composed of elements
that should not be invalidated for five minutes. You want to rebuild
the expired entries ASAP once they expire. So you can subscribe to the
expires channel and put what you get in the stream into a list: other
workers will BLPOP from this list and rebuild the cache.
3) There is some need to collect stats about the most used items in
your cache, that are probably the most used items in your application.
It's simply possible to subscribe to the expires and every time one
entry expires we can increment a counter or likewise.
4) I need to track the number of users online. Every time I see an
user I add the ID into a Set. Every time I see the session expiring I
remove it from the Set.
Ok honestly this use cases are weak. I need something better to go forward :)
Cheers,
Salvatore
Aníbal Rojas
What I am doing now doesn't really require this feature, but I still
think that Redis need to be able to externalize this aspect of its
behaviour, because from an operational view, you need to be able to
track down what is going on your server.
--
Aníbal Rojas - @anibalrojas
Ruby / Rails focused Devops
More info: http://www.google.com/profiles/anibalrojas
My free/busy schedule: http://tungle.me/anibal
Salvatore Sanfilippo
> Umm, I think I should line up with the 'Usual Suspects' :)
>
> What I am doing now doesn't really require this feature, but I still
> think that Redis need to be able to externalize this aspect of its
> behaviour, because from an operational view, you need to be able to
> track down what is going on your server.
Not convincing Anibal ;) Because stated in this way sounds like "for
debugging purposes".
We need something more than this... otherwise we could add this in the
DEBUG command or alike.
Cheers,
Salvatore
Aníbal Rojas
Choosing between DEBUG and EXPIRE subscription, DEBUG gets my
money for sure ;)
The problem with EXPIRE subscription, in the end is that you can't
be sure that the process handling it will be always there, so there is
some opportunity for an inconsistent state to arise. So in the end a
better solution, is having the appropiate structures for tracking what
you want to get rid off, if it is really required.
But from a Data Structure Server point of view, nothing happens in
Redis without knowing the result, _except_ expiration of KEYS, I have
been faced with the duty of keeping a Redis instance, that was pushed
far beyond its limits, up and running and from a operational point of
view, I should be able to collect as much information as possible
about what is going on in the server minimizing the impact of
gathering it.
--
Aníbal Rojas - @anibalrojas
Ruby / Rails focused Devops
More info: http://www.google.com/profiles/anibalrojas
My free/busy schedule: http://tungle.me/anibal
Amit Rathore
that use case on my project (I don't just count the number of users
online, but a bunch of other reporting).
How can I implement this without getting notified on key expiry?
I'm asking because if there's some way of doing this without polling
of some kind, I'd love to know.
Regards,
Amit.
Salvatore Sanfilippo
> Why do you say that use case #4 is weak? I have pretty much exactly
> that use case on my project (I don't just count the number of users
> online, but a bunch of other reporting).
> How can I implement this without getting notified on key expiry?
>
> I'm asking because if there's some way of doing this without polling
> of some kind, I'd love to know.
Hello Amit,
there is a very simple way to get this function just using the
features we already have.
Use a DB just for this (SELECT it). Every time you see an user perform
a SETEX of user:<id>, with the timeout you like.
To check if an user exists just do EXISTS user:<id>. To count the
number of users just check with DBSIZE. Get the list of all the online
users with "KEYS *".
Amit Rathore
Your approach will tell us how many users we have logged into the
system at any given time.
We need to run some reports WHENEVER a user LOGS OUT (in other words,
when the key expires).
That is not possible with your solution, since I will not be notified
when a user:<id> key expires...
On Sep 17, 4:19 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> > Why do you say that use case #4 is weak? I have pretty much exactly
> > that use case on my project (I don't just count the number of users
> > online, but a bunch of other reporting).
> > How can I implement this without getting notified on key expiry?
>
> > I'm asking because if there's some way of doing this without polling
> > of some kind, I'd love to know.
>
> Hello Amit,
>
> there is a very simple way to get this function just using the
> features we already have.
> Use a DB just for this (SELECT it). Every time you see an user perform
> a SETEX of user:<id>, with the timeout you like.
> To check if an user exists just do EXISTS user:<id>. To count the
> number of users just check with DBSIZE. Get the list of all the online
> users with "KEYS *".
>
> Cheers,
> Salvatore
>
> --
mamu
may be that tells how useful it could be in various real life problems
for me. Below are few that could be used and it's must have for us.
- Application start some task which is kind fire and forget, but it's
critical so application need to make sure it happens. To support that
application need to go back after X amount of time(5 min) to verify
that it really happened. This could be achieved with redis key expire
notification.
- Application need to fire scheduled task everyday at 5 am that could
be achieved with key expire. This is like using it scheduler but it's
added benefit who want to implement it.
Above can be achieved with other solutions but most don't scale well.
Josiah Carlson
> I joined this thread to share my point of view regarding this topic,
> may be that tells how useful it could be in various real life problems
> for me. Below are few that could be used and it's must have for us.
Must-haves are much easier to accept when they include a patch ;)
> - Application start some task which is kind fire and forget, but it's
> critical so application need to make sure it happens. To support that
> application need to go back after X amount of time(5 min) to verify
> that it really happened. This could be achieved with redis key expire
> notification.
The application is going to check after 5 minutes, and you want to use
Redis as a timer to signal that it should check? One thing to
remember about Redis expiration is that it's not a hard 5 minutes.
Keys that are supposed to expire in 5 minutes may never expire, or
they could expire in one second (if there is memory pressure). You're
better off using an explicit event queue with timing. RabbitMQ,
ActiveMQ, and other Redis-based solutions all have the ability to say
"execute command X in Y seconds".
> - Application need to fire scheduled task everyday at 5 am that could
> be achieved with key expire. This is like using it scheduler but it's
> added benefit who want to implement it.
Not really. Redis key expiration may occur before it's desired
expiration, and it could also expire never. It's not a hard deadline,
it's not a scheduler, and it's not cron.
> Above can be achieved with other solutions but most don't scale well.
Actually, many scale pretty well. For example, ActiveMQ will cluster
if necessary, but it will also move tens of thousands of items through
the system each second on modern hardware. If you just need a simple
work queue, ZeroMQ claims to be able to move upwards of 2.8 million
messages/second (for 8 byte messages), can do replication, tree
structures, etc.
Heck, a Redis ZSET is arguably close to perfect for scheduled tasks,
though you need to have one process that pulls tasks off of the
scheduled list and puts it on a regular LIST for multiple readers.
Multiple writers are trivial (put stuff in the list that you want to
happen *now*, stuff in the ZSET that you want to happen in the
future). You could easily do a few tens of thousands of events/second
without breaking a sweat. And also *actually* get what you want
(check in 5 minutes/check at 5AM, vs check some time between now and
the future).
What scales are you talking about that "don't scale",
- Josiah
> On Sep 16, 6:36 pm, teleo <lev.ko...@gmail.com> wrote:
>> Hi Amit, I am curious about the use case. How would you use the
>> notifications, if they were supported?
>>
>> On Sep 16, 9:41 pm, Amit Rathore <amitrath...@gmail.com> wrote:
>>
>> > Hi folks,
>>
>> > Is there any way to get notified when a key expires in Redis? If not
>> > directly, then are there any common practices to approaching this?
>>
>> > Regards,
>> > Amit Rathore,
>> > Runa, Inc.
>>
>>
>