Blocking producers with list based queues.

[mai...@gmail.com]

I'm implementing a worker queue based on the list data type in redis and the blocking BRPOP/BLPOP operations to block the client when there are no items to process.

My concern is how to block the producers in order to not crash the server by adding too many items to the list.

The simplest solution is to use LLEN to query the list length and if its too large block or sleep but then i need another mechanism to unblock when items were processed.

Ideally think I need some sort of blocking LPUSH the gets a length as an argument and blocks if the list is longer than the argument, understanding there is no such command I would like to know is a similar behavior can be implemented?

I know I can check the return value of LPUSH but then I'll have to remove the item if the list is too long and the item I'm removing might not be the item I added if another client inserted an item between the checks.

Perhaps a lua script to execute LLEN and conditionally LPUSH is the way to go?

[mai...@gmail.com]

I've implemented the bounded enqueue using a lua script [1]. For now I ended up not blocking on enqueue but i'll be happy to hear opinions about the idea of blocking the producer vs returning with an error.
Also feedback on the implementation will be great.

[1] https://github.com/someuser77/redis-queue

[Josiah Carlson]

The chance of Redis getting a blocking RPUSH/LPUSH if the destination list is too long, in my opinion, is close enough to zero to be considered measurement error. Why? Salvatore (the creator of Redis) recently released Disque [1], which is meant to be a highly-available queue system with many of the same semantics as the simple list-based queues that have been used in Redis for 5+ years (with additional options for 1+ execution semantics). And in particular, Disque will drop messages and return an error if the queue would grow beyond the user-provided maximum queue size as part of the ADDJOB call (see the MAXLEN argument).

Regarding whether or not it makes sense to block on RPUSH/LPUSH, I would say no. In roughly 100% of the cases that I've seen queues being used in practice (I am the author of a queueing library for Python + Redis, have used tasks/queues in the last 3 startups I've worked for, have advised several companies on the use/implementation of queues using Redis, ...), the only delay that you want to see is network latency and/or replication delay (to ensure that your task gets to replicas).

If you can drop queue items if your queues are too long, then drop the queue items. If you can't drop the queue items if your queues are too long, then make sure you have enough memory and add monitoring with alerts, or even monitoring that auto-scales up if your queues get too long.

[1] https://github.com/antirez/disque/

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.