Mapping tasks to workers using PubSub

[mixdev]

I have a job with multiple tasks. I need to split the job and map the
tasks to multiple workers using pubsub so that tasks get done in
parellel. What's the best way to do it so that I can avoid two workers
working on the same message?

I have read that the WATCH along with a proper TRANSACTON flow can
create a locking effect in other datatypes (like you create another
list with the corresponding keys to denote that the key is locked by a
process). But in pubsub it is messages (not keys) and the messages are
of arbitrary size in my case. I also thought about creating a MD5 hash
of the message and use it as a key for the WATCH technique. But this
will create significant delay in the cycle and probably negate the
speed Redis offers IMO.

What's the best way to deal with this scenario?

[Didier Spezia]

Hi,

I would not use pubsub functions for this, but rather simple lists
used
as queues. Your main process can split the job and post the tasks to
one or several lists (RPUSH). Your worker processes can wait and fetch
incoming traffic using BLPOP.

Because Redis is single threaded there is no risk for two workers
to process the same item.

The main process can post to one or several queues. The workers can
listen to one or several queues, so it is quite flexible.

Regards,
Didier.

[Ezra Zygmuntowicz]

I would not use pub/sub for this either. I have been using a little peice of code called Redactor to make little actors based on blpop, looks like this:

https://gist.github.com/937648

You can define mailboxes and then define receieve callbacks to proceass the messages. Even if you run a huge pool of the same workers you will never run the same job twice.

Cheers-

Ezra Zygmuntowicz
ezmo...@gmail.com