The dead of the Single Consumer pattern

18 views
Skip to first unread message

Rene Nejsum

unread,
Nov 1, 2020, 1:34:34 AM11/1/20
to rabbitmq-discuss
I am looking for input, ideas and "ammunition" for an ongoing discussion we have at work. I am working with Django using Celery (soon moving to Dramatiq) for async tasks. 

In my previous jobs I have often used the SPSC (and MPSC) pattern, which "guarantees" that a message is handled in the same order as it arrives. I find it very useful to know that if a client (could be a Django request handler) is calling:

def requesthandler(request):
  ...
  calculate_new_state.send(args)
  ...
  operate_on_new_state.send(args)
  ...

Is run in the same order as the are sent (.send() is Dramatiq syntax for sending a message on a RabbitMQ queue)

But as soon as the message hits Dramatiq (or Celery) the order is randomised because of multiple workers on the queue. None of the two tools have an easy way to configure that on a given queue I need a single consumer.  

RabbitMQ guaranties the order of the messages is preserved, but it's like no one needs/uses that order for anything? (At least in the Python/Django/web world?)

Am I too old school? Is there something I am missing? Is there another way to guarantee order? My coworkers suggest a table in the DB to "lock" the queue, but that just seams weird to me, since RabbitMQ already handles this, if only we could work out to implement Single Consumer?
Reply all
Reply to author
Forward
0 new messages