RabbitMQ same queue name on different machine

[Izak Smit]

Hi RabbitMQ

I've been really trying to get around this one queue on a server in a cluster mode for weeks now - the problem with clustering nodes is that the physical queue sits on one server and can become a problem when load start to increase

My idea is to use the same queue on different servers and distribute the load over the servers

The only way I can do this is if I use none clustered servers and create the same queue name "myworker" and with a load balancer in front of rabbitMQ split the load across - but the nice feature of a cluster is again, you don't need to keep track of things - so I am in between clusters and queue on all servers :)

What I am trying to do is really simple

 - Devices in field report to rabbit on their own queue

 - Workers listen for message on worker-queue

 - Device post any info to the worker-queue

 - worker can respond to deivce-queue

In short 

  - Device listen on own queue for message

  - Workers listen on worker-queue for mesg from devices

  - Workers can post request to device using device-queue

Wtih thousands and thousands of devices sending messages to a single queue on a single server will kill the system - so we need to run same queue on all servers and allow the workers to dequeu messages

So if we use a single server this works. If use lots of servers not in cluster mode it will work, but then we need to keep track of where the device reported the first time - aka the device must be routed to a dedicated broker and never talk to any other server but that server - need to keep a track record almost like ldap 

If we use clustering, the cluster keeps track of where this dedicated queue for the device will be - but then you can not run same queue on all nodes - :|

What I did find is the federation queues will work for running "clustered", but clustering and federation is becoming so complex setup for me, I worried I not going to keep everything in sync and in long term keeping this up with upgrades ext will become tricky - I already worried about upgrading a clustered RMQ :) - adding federation queues is just another puzzle to keep in mind when doing things like upgrading or when things goes down

My final short version is

  - I am now looking at running stand alone nodes and build a dedicated routing system for devices

But maybe somebody have done something like this - with worker queues and clusters with devices own queues keeping things easy to scale, upgrade and without 2 much worries if a node goes down

[Michael Klishin]

Izak,

The typical answer is "use more queues". This has benefits of multiple kids, including
better utilisation of multiple CPU cores and horizontal scalability when queue masters
are distributed among cluster nodes.

Take a look at https://github.com/rabbitmq/rabbitmq-sharding. It may be helpful but I'd
recommend investigating how you can use more than 1 queue, which *will* become a throughput bottleneck
eventually.

Even in cases where you seemingly depend on message order there are techniques to work around it:
http://blog.travis-ci.com/2013-08-08-solving-the-puzzle-of-scalable-log-processing/
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

 +rabbitmq-users — please CC the list.

On 23 February 2015 at 20:42:45, Oegma2 (oeg...@gmail.com) wrote:
> Thanks for the links and info regarding sharding - I was actually
> wonder about that since we're talking about sharding our datastores
> as well :) but this will provide some different angle on running
> "more" queues and scaling as we go along

We haven't heard any major issues reported for it. Give it a try.

> Is this in production already (sharding RMQ)?
> How is this different from federations,...(still new to federations,
> so need o read up a lot still)

Queue federation has similarities. It favours local consumers when deciding how to distribute messages.
The queue sharding plugin is less smarter and uses a hashing function to partition messages
and assumes you automatically want to shard to all nodes in the cluster.

[Michael Klishin]

 On 23 February 2015 at 20:42:45, Oegma2 (oeg...@gmail.com) wrote:
> We haven't heard any major issues reported for it. Give it a try.

Ugh, I've noticed we don't offer binaries for 3.4.x. You'd need to build from source.

[Michael Klishin]

On 23 February 2015 at 23:27:03, Michael Klishin (mkli...@pivotal.io) wrote:
> Ugh, I've noticed we don't offer binaries for 3.4.x. You'd need
> to build from source.

Nope:
http://www.rabbitmq.com/community-plugins.html
http://www.rabbitmq.com/community-plugins/v3.4.x/rabbitmq_sharding-3.4.x-bccd6750.ez 

[Oegma2]

Hi Michael

Thanks for the link regarding the community plugins - once I figured out federated queues, I will check out sharding

I've setup 2 RMQ instances in a clustered mode and then enabled the rabbitmq federation plugin on both servers to do federated queues on a clustered setup

Hostname for 2 machines (Rabbit1=server1, Rabbit2=server2)

Now is my federated queue setup correct if I use the same URI (amqp://Rabbit1) as the upstream server...? Looks like you tell the federation plugin that the upstream server is Rabbit1?

​

​

[Oegma2]

Very very nice article on use for federations - http://jaxenter.com/distributed-log-aggregation-with-rabbitmq-federation-107287.html