Using 1 IP to access HA Cluster

[choonsi...@onapp.com]

Hi,

I am working on setting up a HA rabbitmq cluster by referring to this document <http://www.rabbitmq.com/ha.html>, the setup works fine as expected. However, there's still another part I am stuck with and would need some ideas.

The goal of this setup is to provide a single IP for our services to connect to it. There are two options that I am considering (am open to more suggestions though), which are setting up floating IP through corosync+pacemaker, as well as a HAproxy (which I have no experience with currently).

The reason I am hesitate about setting up floating IP is that, as far as I know rabbitMQ HA cluster is a master/slave setup. While the queues can be configured to be fully replicated, however most transactions are handled by the main master node.

> As discussed, for each mirrored queue there is one master and several slaves, each on a different node. The slaves apply the operations that occur to the master in exactly the same order as the master and thus maintain the same state. All actions other than publishes go only to the master, and the master then broadcasts the effect of the actions to the slaves. Thus clients consuming from a mirrored queue are in fact consuming from the master.

I don't feel comfortable about setting floating IP with corosync+pacemaker knowing this is a master/slave set up. As the setup is only dealing with dead server at the moment, the access to the HA cluster is expected to switch to the slave (and promote it to master) when the server running the master is completely dead. In other words, I want the floating IP to always direct to the master node if possible. (Please correct me if I am correct, if I have multiple queues, they don't necessarily all share the same master right? I mean some queue may have another node as master?)

It would be nice if I can somehow set up a master/master replication.

Anyway, can anyone advise what can I do next to achieve my goal of having only 1 point of entry to this rabbitMQ cluster?

best wishes,

Choon-Siang Lai

[choonsi...@onapp.com]

ouch, i feel dumb now

s/Please correct me if I am correct/Please correct me if I am wrong/

[Michael Klishin]

 On 25 September 2014 at 13:45:16, choonsi...@onapp.com (choonsi...@onapp.com) wrote:
> The reason I am hesitate about setting up floating IP is that,
> as far as I know rabbitMQ HA cluster is a master/slave setup. While
> the queues can be configured to be fully replicated, however
> most transactions are handled by the main master node.

To clarify: those are queue masters, not node masters. In other words,
if you distributed connections that do queue.declare across the cluster,
there will be multiple "master nodes".

> > As discussed, for each mirrored queue there is one master and
> several slaves, each on a different node. The slaves apply the
> operations that occur to the master in exactly the same order
> as the master and thus maintain the same state. All actions other
> than publishes go only to the master, and the master then broadcasts
> the effect of the actions to the slaves. Thus clients consuming
> from a mirrored queue are in fact consuming from the master.

Correct.

> I don't feel comfortable about setting floating IP with corosync+pacemaker
> knowing this is a master/slave set up. As the setup is only dealing
> with dead server at the moment, the access to the HA cluster is
> expected to switch to the slave (and promote it to master) when
> the server running the master is completely dead. In other words,
> I want the floating IP to always direct to the master node if possible.
> (Please correct me if I am correct, if I have multiple queues,
> they don't necessarily all share the same master right? I mean
> some queue may have another node as master?)

See above. Queue master is on the node where that node was declared
(barring slave promotions.

> It would be nice if I can somehow set up a master/master replication.

Exchange federation can do that. One thing that it's missing is deletion of
consumed messages over federation links.

> Anyway, can anyone advise what can I do next to achieve my goal
> of having only 1 point of entry to this rabbitMQ cluster?

Set up an HAproxy node in front of your cluster and make all clients connect
to it. Start with the "leastconn" strategy. Now your problem is largely shifted to how
to make this node available. Having multiple proxy nodes and round-robin DNS
should work quite well.

See https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts if you need an HAproxy intro.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[choonsi...@onapp.com]

Hi MK,

Thanks for the reply and the link. Will try to set up a HAproxy for the cluster and draft a simulation plan later.

[Paulo Almeida]

In my setup I have haproxy running on the application server. The application connects to localhost, as if rabbitmq was running locally. 

Adding this to /etc/haproxy/haproxy.conf was enough:

listen rabbitmq 127.0.0.1:5672

       mode               tcp

       balance            leastconn

       option             tcplog

       option             contstats

       server rabbit1 rabbit1:5672 weight 1 maxconn 1024  check

       server rabbit2 rabbit2:5672 weight 1 maxconn 1024  check

I then edited /etc/hosts to have rabbit1 and rabbit2 resolve to the correct IP addresses.