Synchronized tasks queue using Redis

[Miko95]

Hi,

I'm trying to implement a synchronized tasks queue using Redis as follows:

• I use a sorted set to store the tasks sorted by timestamp. The timestamp indicates when the task should be executed.
• Multiple instances of my application (and possibly the different worker threads inside each instance) share the queue so that the load of executing task is shared among these instances (and threads inside instances). I started with an architecture where each instance was responsible of its task but it is problematic in case an instance crashed. Thus, my goal is to have all the instances to be logically equivalent so that if one instance crashes, the processing of ALL tasks continues as normal.
• A new task is added using the ZADD command.
• Each worker thread periodically execute the ZRANGEBYSCORE command (with LIMIT) to select the tasks to execute. Each task is then executed and each task's score is updated for the next execution.
• At some point, due to some condition evaluated using the task's data, a task may be removed from the sorted set.

The problem is that during task execution by one worker thread, another worker thread would get the same tasks when executing the ZRANGEBYSCORE command. So, I'm looking for a way to lock the sorted set (ideally, part of it).

A transaction doesn't seem to solve my problem since the commands to execute for each task depend on the results of previous commands. The following pseudo code illustrates the issue:

1. Get the tasks to execute using the ZRANGEBYSCORE command.
2. For each task to execute:
1. Decrement a counter (also stored in redis) using DECRBY command.
2. If the counter reached a certain value, schedule a notification and remove the task from the sorted set. Otherwise, update the score of the current task.

How can I solve this design issue?

Is Redis appropriate for this kind of design?

Thanks, Mickael

[Matt Stancliff]

On May 8, 2014, at 10:10 AM, Miko95 <mickael...@gmail.com> wrote:

> I'm trying to implement a synchronized tasks queue using Redis as follows:
>

> Is Redis appropriate for this kind of design?

Redis is great for queuing systems, but it takes a lot of design to get it right.

I’m a fan of https://github.com/seomoz/qless-py which uses Lua scripting to maintain the job queue: https://github.com/seomoz/qless-core

There’s also the very popular https://github.com/resque/resque

Your approach with timestamps may be tricky. The qless system doesn’t have timestamps, but it does offer recurring jobs (every N seconds) and a “delay” option you can set on jobs you insert (e.g. add job now, but delay it for execution by a minimum of 300 seconds). The resque system has a scheduler add-on that’ll let you run jobs at specific times: https://github.com/resque/resque-scheduler


-Matt

[Josiah Carlson]

In RPQueue for Python (https://github.com/josiahcarlson/rpqueue/), delayed, scheduled, and periodic task execution is one of the central design ideas.

What we do there is to use a ZSET for the scheduling of items, where a Lua script moves (or copies and reschedules) scheduled items from the ZSET schedule to LIST-based FIFO queues. All workers use BLPOP calls to find items to execute from the queues, with one thread on every task runner daemon running the Lua script to handle schedule updates. I will typically run a single daemon on a server, which spawns as many worker processes as there are cores in the machine, and which spawns a user-configurable number of threads per process - for all of this, there is 1 thread handling schedule updates.

There should already be a queue library that offers this support for Java, though I don't follow the Java dev community, so I don't know where to point you.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Miko95]

Hi Matt and Josiah,

I finally managed to implement it using an LUA script. I just discovered this amazing feature of Redis.

Thanks for your help,

Mickael