Help with queue design for long-running tasks

[Jeff F]

I'm struggling to arrive at a design that fits our situation just right, and wonder if I'm approaching it wrong.

We want to use RabbitMQ as a task queue. The tasks take a long time, from one second to two minutes, with an average time of around, say, 15 seconds. The workers use a lot of resources, and we can only have a handful running concurrently; a few will be slower than the rest, and can have lower consumer priority.

Some jobs an end-user will be waiting on, and arrive a few per minute; they can come across with a routing key ("app.job.sync") and/or a priority that distinguishes them. The rest of the jobs tend to come in large batches—as many as a couple hundred at a time.

Priority queues help, but leave too much likelihood that all workers will have become newly occupied with two-minute jobs when a "sync" job request comes in at the front of the line.

I thought I could have a couple workers consuming only from a sync queue, and all the rest pulling from both the sync and the background queue. The only problem with this is that Rabbit will feed a consumer subscribed to multiple queues alternately from each queue. Ideally it would somehow know to drain the sync queue before sending anything from the other one to a dual-subscribed consumer. (Message priority has no effect across queues.)

Thank you for your time and any ideas you can offer!

[Jeff F]

Would anyone be willing to take a look at my post and offer an opinion?

My specific question is: if a consumer is subscribed to two queues, is there a way to only receive messages from queue B when queue A is empty?

If that is indeed not possible, can someone suggest another approach?

Thank you!

[Luke Bakken]

Hi Jeff,

I'm re-visiting your original post as well. It feels like you'll have to coordinate your workers, or leave several workers dedicated to only the "sync" queue to ensure some are available to process those requests.

By "coordinate your workers" I mean that there could be shared state as to the number that are processing high vs low priority tasks at any given time. If not enough are available for the high priority tasks, some would cancel consuming from the low-priority queue to ensure they are available for high-priority tasks. They could re-consume low-priority tasks once enough workers are available.

Querying the state would happen before acknowledging the last message processed, because that would be the appropriate time to cancel or re-consume from queues.

At least, this seems like a potential solution with about 5 minutes of thought.

Thanks,

Luke

[Jeff F]

I appreciate the response Luke.

I have been trying to hold out against making a controlling or managing process on the consumer side. Seems like I might avoid that with workers that are just a little bit more active and make more of their own decisions on the fly about which queues they're consuming from. That opens up another avenue for fine-tuning the routing that I really hadn't explored.

Thank you!

[Luke Bakken]

Hi Jeff,

I'm glad I could help out. Don't hesitate to use this mailing list as a resource going forward, and have fun with RabbitMQ.

Luke