Set expiring items

[Adrián Navarro]

Hello,

I'm trying to do the following: have a set of short strings (as a
«queue») with expiration times (have items get dropped if they aren't
processed fast enough).

It's probably because I'm well tired now, but I can't seem to find a
way to do that: SADD queue v4lu3 and then EXPIRE that new item.

Is there a way, a workaround or just plain nothing?

Thank you for your time and good night.

[Josiah Carlson]

You can only expire entire keys (a whole set, hash, zset, list, or
string) at a time.

Generally, sets are not terribly good queues, as fast read/remove
access patterns for individual items involve randomly popping items.

Most people choose to use lists as queues, as they have obvious
first-in-first-out (or last-in-first-out) behavior.


To get a queue with expiration behavior, I can think of a two
different ways of building it with Redis.

First way:
1. Store your queue items in a list or set or whatever
2. When you add an item X to the queue, also set a key 'item:X' to an
empty string with an expiration.
3. When you fetch an item from the queue, delete the representative
key. If the deletion occurred, then execute the queue item, otherwise
discard it (because it expired)

Second way:
1. Store your queue items in a sorted set, with members being your
queue items, and the score being your expiration time
2. You manually expire old items by removing items with expiration
times in the past
3. You can fetch items based on their expiration times with "ZRANGE
QUEUE 0 0", or randomly with "count = ZCARD QUEUE; index =
floor(random() * count); ZRANGE QUEUE index index", but you need to
ZREM items once you've pulled them from the queue.

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/s7ExkvtSKlwJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

[Adrián Navarro]

Got it thanks!

I'm also thinking about adding a timestamp in the value… SADD queue
1335971908:itemslug and just do processing on the client. Not worried
about performance, just a few queries a minute.

SADD queue time:value, SPOP queue, and process or discard depending on
situation. Sounds good to me.

Thank you for your ideas and making clear that there is really no
built-in way to do it.

Have a nice day!

--
Adrián Navarro / (+34) 608 831 094

[Josiah Carlson]

If you are adding timestamps, then uniqueness doesn't matter.

If uniqueness doesn't matter, why are you using a set? Are you looking
to get random queue ordering? Why not use a list?

Regards,
- Josiah

[Adrián Navarro]

Uniqueness did matter, sorting did not… And I didn't think about it.

I've 'forgotten' the expiring times for now, I'll go back into that later :(