Cluster Nodes with mirrored Queues

[AG]

Hi

I have a cluster that replicates all Queues on all Nodes. While I didn't have explicit params for cluster config, the queues are replicated on all nodes as I can be seen from ADMIN

a) How can it be confirmed if Queues are in indeed in mirrored mode (Either by looking at ADMIN or some other way)

b) I had single Publisher and Consumer to a single Queue on this cluster and message activity can be seen on both NODES in cluster in ADMIN WEB Page. Does the consumer connect to a always the master node?

c) Does the Publisher connect to Master or BOTH nodes? 

Thanks
AG

[Simon MacMullen]

On 02/10/2014 23:20, AG wrote:
> I have a cluster that replicates all Queues on all Nodes. While I didn't
> have explicit params for cluster config, the queues are replicated on
> all nodes as I can be seen from ADMIN

Are you saying you did not set up a mirroring policy but all queues can
be seen from the management plugin on all nodes? That means the queues
are unmirrored. Queues will always be visible to all nodes, mirrored or not.

> a) How can it be confirmed if Queues are in indeed in mirrored mode
> (Either by looking at ADMIN or some other way)

In the web UI, you should see "<master node> +1" (or +2, +3 etc) in the
Node column. The +N part tells you how many slaves the queue has.

Alternately, "rabbitmqctl list_queues name pid slave_pids" will show the
master node and slave nodes from the command line.

> b) I had single Publisher and Consumer to a single Queue on this cluster
> and message activity can be seen on both NODES in cluster in ADMIN WEB
> Page. Does the consumer connect to a always the master node?

From the consumer's point of view it connects to whichever cluster node
you told it to via AMQP. If that's not the node which contains the
queue, messages are then routed across the cluster to the consumer. This
is transparent but can have a performance cost.

> c) Does the Publisher connect to Master or BOTH nodes?

Again, the publisher connects to whichever node you tell it to. Messages
are then routed from that node to all nodes containing the queue.

See http://www.rabbitmq.com/ha.html#behaviour for more details.

Cheers, Simon

[Arvind Gupta]

Thanks Simon

So I must have mirrored queues.

If I do make them mirrored, and I have two nodes A, B and B was asked to Join A Cluster. Then

a) Mirrored node means contents are copied from A to B from the point B joined according to semantics of mirrored queues. But just to make sure, does that mean if I connect to A, I *definitely* read A queue and if I connect to B, then I read B Queue?

b) If the ans is yes, then we can have mirrored queues and have a load balancer round robin to each node to achieve true load balancing across nodes or not?

AG

[Simon MacMullen]

On 03/10/14 17:08, Arvind Gupta wrote:
> Thanks Simon
>
> So I must have mirrored queues.
> If I do make them mirrored, and I have two nodes A, B and B was asked to
> Join A Cluster. Then
> a) Mirrored node means contents are copied from A to B from the point B
> joined according to semantics of mirrored queues. But just to make sure,
> does that mean if I connect to A, I *definitely* read A queue and if I
> connect to B, then I read B Queue?
> b) If the ans is yes, then we can have mirrored queues and have a load
> balancer round robin to each node to achieve true load balancing across
> nodes or not?

From http://www.rabbitmq.com/ha.html:

"Messages published to the queue are replicated to all slaves. Consumers
are connected to the master regardless of which node they connect to,
with slaves dropping messages that have been acknowledged at the master.
Queue mirroring therefore enhances availability, but does not distribute
load across nodes (all participating nodes each do all the work)."

Cheers, Simon

[AG]

Thanks again.

I am curious if nodes A and B are in cluster and A is Master, and B is slave (unmirrored Queues - please see reference call for creating), then does the slave just show the same stats (message rates etc) in the management plugin? What might be a reason for them to be different, if any

>>>>>queue)channel.queueDeclare(QUEUE_NAME, false, false, false, null);

Thanks

AG

[AG]

Hi Simon

I am not seeing any of the nodes in Management Web UI that show +1 etc,
even after running the following command (the node name is "Hello"

"./rabbitmqctl set_policy ha-all "^He\." '{"ha-mode":"all"}'

So cluster nodes are up and I applied the policy and no changes. Do I
have to restart the nodes for them to appear in mirrored mode?

Also, Is it only in mirrored Mode that I can expect that my Consumer
connected to Node B on Cluster "AB", will pick up messages when Node A
was rabbit process was killed?

Thanks
AG

[Simon MacMullen]

On 09/10/14 21:14, AG wrote:
> Hi Simon
>
> I am not seeing any of the nodes in Management Web UI that show +1 etc,
> even after running the following command (the node name is "Hello"

I assume you mean "queue name" not "node name".

> "./rabbitmqctl set_policy ha-all "^He\." '{"ha-mode":"all"}'

Why are you escaping the "."? The regexp "^He\." will match "He.llo" but
not "Hello".

The policy will appear against the queue name in the queue list; until
you have a working regexp it won't do anything.

> So cluster nodes are up and I applied the policy and no changes. Do I
> have to restart the nodes for them to appear in mirrored mode?

No.

> Also, Is it only in mirrored Mode that I can expect that my Consumer
> connected to Node B on Cluster "AB", will pick up messages when Node A
> was rabbit process was killed?

It's not very clear what you're asking here.

If there was a queue on A then it will be unavailable when A goes down,
unless the queue is mirrored.

If the queue was on B then mirroring doesn't matter.

Cheers, Simon

[Arvind Gupta]

That does it, I do have mirrored queue now and yes I meant Queue (not node)

On the last question,where it was not clear, I meant cluster AB, where A master goes down and we have mirrored queue. Does the client connected still have to ignore Consumer_Cancellation_Notification or not to continue receiving messages on B

Also, a question related to config files. I put in rabbitmq-env.conf and rabbitma.config in  etc/rabbitmq

a) As long as rabbitmq-env.conf was there, I didn't see variables being set and

b) cluster config info wasn't read from rabbitma.config

when I removed rabbitmq-env.conf, then it did read rabbitma.config, but

a) never started in cluster mode,

NODENAME=FQDN OF SLAVE

CONFIG_FILE=/x/y/z/rabbit (Does the full path work here, or do we have set the home directory)

[

 {rabbit,

  [{cluster_nodes,

        {['rabbit@{FQDN OF MASTER}', 'rabbit@{FQDN OF SLAVE}'], disc}

  }]

 }

].

I was wondering if someone could give an example of setting the cluster via cluster file as well mirroring of a queue

Thanks
AG

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/Rq69NwfZ_a8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send an email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

On 10 October 2014 at 17:49:25, Arvind Gupta (arv9...@gmail.com) wrote:
> On the last question,where it was not clear, I meant cluster
> AB, where A master goes down and we have mirrored queue. Does the
> client connected still have to ignore Consumer_Cancellation_Notification
> or not to continue receiving messages on B

It depends on what node the client was connected to.

If the client was connected to A, consumer cancel notification doesn't matter
because your entire connection will go down. So it will have to connect to B and
use basic.consume again.

If the client was connected to B, starting with 3.3 by default it won't even get
the notification, RabbitMQ will automatically re-register its consumer after queue
failover is complete.

In other words, your client won't have to do anything. 

> Also, a question related to config files. I put in rabbitmq-env.conf
> and rabbitma.config in etc/rabbitmq
> a) As long as rabbitmq-env.conf was there, I didn't see variables
> being set and

What variables?

> b) cluster config info wasn't read from rabbitma.config
>
> when I removed rabbitmq-env.conf, then it did read rabbitma.config,
> but
> a) never started in cluster mode,
>
>
> NODENAME=FQDN OF SLAVE
> CONFIG_FILE=/x/y/z/rabbit (Does the full path work here, or
> do we have set the home directory)

It is meant to be a full path w/o the ".config", what exactly is the value you have?

By the way, this is mentioned on http://www.rabbitmq.com/configure.html#configuration-file:

«The Erlang runtime automatically appends the .config extension to the value of this variable.»
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Arvind Gupta]

Please see my ans below in []

On Fri, Oct 10, 2014 at 6:55 AM, Michael Klishin <mkli...@pivotal.io> wrote:

On 10 October 2014 at 17:49:25, Arvind Gupta (arv9...@gmail.com) wrote:
> On the last question,where it was not clear, I meant cluster
> AB, where A master goes down and we have mirrored queue. Does the
> client connected still have to ignore Consumer_Cancellation_Notification
> or not to continue receiving messages on B

It depends on what node the client was connected to.

[ yes I meant connected to Node B. Good to know, there is no notification.  ] 

If the client was connected to A, consumer cancel notification doesn't matter
because your entire connection will go down. So it will have to connect to B and
use basic.consume again.

If the client was connected to B, starting with 3.3 by default it won't even get
the notification, RabbitMQ will automatically re-register its consumer after queue
failover is complete.

In other words, your client won't have to do anything. 

> Also, a question related to config files. I put in rabbitmq-env.conf
> and rabbitma.config in etc/rabbitmq
> a) As long as rabbitmq-env.conf was there, I didn't see variables
> being set and

What variables?

[ NODENAME and CONFIG_FILE as indicated below ]

> b) cluster config info wasn't read from rabbitma.config
>
> when I removed rabbitmq-env.conf, then it did read rabbitma.config,
> but
> a) never started in cluster mode,
>
>
> NODENAME=FQDN OF SLAVE
> CONFIG_FILE=/x/y/z/rabbit (Does the full path work here, or
> do we have set the home directory)

It is meant to be a full path w/o the ".config", what exactly is the value you have? I

By the way, this is mentioned on http://www.rabbitmq.com/configure.html#configuration-file:

[ I had /etc/rabbitmq/rabbitmq for full path ] 

«The Erlang runtime automatically appends the .config extension to the value of this variable.»

[ The LOG does indicate that it is reading CONFIG_FILE, but still doesn't start the node in cluster mode. My cluster config was included in previous post. Any comments why? If I do the cluster_join command, it works. A sample cluster config file would be great help. The one included doesn't have I believe with two nodes - are we supposed to just point to master going by that example 

Thanks so much,

AG 

]