Competing Consumers in Clustered RabbitMQ Deployment

[James Renaud]

Hey folks,

I've been researching this for a couple days, and not really getting the result that I'm looking for with Masstransit, I'm hoping someone can help point me in the right direction.

Here's the setup:

- I have three servers all configured as part of a RabbitMQ cluster (let's call them 1, 2, and 3).

- Each server connects to rabbitmq://localhost/Some_queue_name

- I have an application running on Server1 which is publishing messages

- I have an application running on Server2 & Server3 which is consuming messages

Within my real-world implementation, any server (1, 2, 3...or ServerN) could publish a message, and all other servers should consume it, thus keeping each server in lock-step. There could be multiple consumers and publishers on a single box, and then that set up would be replicated across each server.

Here's a quick view of what this looks like:

What I'm observing:

- Server1 publishes the message

- Server2 & Server3 are round-robin receiving the published messages (Message1 goes to Server2, Message2 goes to Server3 and back around).

What I can't seem to get an answer on, is getting the following to work:

- Server1 publishes the message

- Server2 & Server3 each consume the message

As far as I've been able to tell, the only way to do this is if the RabbitMQ queue exists on Server1, Server2 and Server3 have to connect to RabbitMQ on Server1 which completely negates the use of the RabbitMQ cluster, and it doesn't seem as though MassTransit has any knowledge of RabbitMQ clustering (nor do I particularly expect it to).

Any help anyone can offer, as I seem to be spinning my wheels, would be great.

[Alexey Zimarev]

I am not sure why do you expect both servers to consume this message since you want competing consumers? What you describe looks exactly as it should work.

[James Renaud]

Maybe competing isn't what I'm looking for, I'll admit messaging is pretty new to me and I'm still learning as I go.

I need each consumer to be equal, not one winning over another for a published message.

In reading up on RabbitMQ, the suggestion is that each consumer has it's own queue, and the message is published through an exchange, and the exchange publishes to each of those individual queues. If that's the case, how does the exchange know about each queue, is that something I would configure, or that Rabbit/MT takes care of based on subscribing to a certain message type?

[Alexey Zimarev]

This is exactly right, you do not want competing consumers but you just want regular pub-sub. Each endpoint should have its own queue. MassTransit will scan consumers and configure RabbitMQ to have an exchange for your endpoint, configure the binding between the message type topic and endpoint change, as well as binding between the endpoint exchange and its queue. This happens automatically under the hood.

[Alexey Zimarev]

It is actually described in the documentation http://masstransit.readthedocs.io/en/master/overview/underthehood.html#starting-a-bus

[James Renaud]

Sure enough, leaving it up to the smoke and mirrors of MT solved my problem.

I hadn't quite grasped the concept of the exchange, and that it has a linkage to the object being published for the appropriate endpoints to consume.

Thanks for the help Alexey, seems to be working the way I want it to now.

[Alexey Zimarev]

Great :)

Just to clarify the competing consumers - as the name suggest, such consumers compete to consume messages from the same queue. This is a very useful pattern when you want to scale your process by distributing consumers to different machines. As your test shows, it works perfectly well also in clustered scenario although clustering is not required here since you can have competing consumers going to the same queue on one broker and it will have the same behaviour.