Federation between 2 clusters not working

[Ankit Gupta]

Hi Team,

I am trying to upgrade my rabbitmq cluster using the blue/green method documented here . The export/import works fine but I run into issues while setting up the federation upstream. The command mentioned in document work fine with no errors but on the UI under federation status, I see the message "no links"

[rabbitmq]# rabbitmqctl set_parameter federation-upstream blue '{"uri":"amqp://old-cluster"}'

Setting runtime parameter "federation-upstream" for component "blue" to "{"uri":"amqp://old-cluster"}" in vhost "/" ...

[rabbitmq]# rabbitmqctl set_policy blue-green-migration ".*" '{"federation-upstream":"blue"}' --apply-to queues

Setting policy "blue-green-migration" for pattern ".*" to "{"federation-upstream":"blue"}" with priority "0" for vhost "/" ...

There is no error in logs of the old cluster or new cluster. This is bit strange so seeking help on how to investigate/troubleshoot this.

Old cluster details:

RabbitMQ version: 3.5.6

Erlarng version: 19.1-1

OS: CentOS 7

New cluster details:

RabbitMQ version: 3.7.3

Erlang version: 20.1-1

OS: CentOS 7

Both these clusters are running in AWS on c4.xlarge instance type. 

Please let me know if you need any other information from my side.

Thanks

Ankit

[Michael Klishin]

Links will be started for matching queues and exchanges. If there are none

(or none matching) in the vhost in question, there will be no links (and no errors

in the logs since having a policy that matches no entities is not an error)

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ankit Gupta]

As I  mentioned, I am doing an upgrade using blue/green upgrade method. As part of that I have already exported and imported definitions. As part of that, there are around 180 queues which are same in source and destination clusters. Even before posting this reply, I confirmed that same queues and exchanges which are present in both clusters

[Michael Klishin]

This is not terribly scientific but 99% of questions about federation links come down to

1. Policies that do not match anything

2. Policies and entities being in the wrong (different) virtual hosts

All other issues either produce errors and exceptions in the logs or require a running link by definition.

I don’t know what else can be suggested with the amount of information you posted.

Feel free to disable and re-enable the plugin and watch a video from Pivotal where the blue/green deployment strategy with federated queues is demo’ed. It’s up somewhere on the Pivotal blog and YouTube.

[anup anand]

Very recently I encountered a variation of this issue. I had federated exchanges working fine from past 2 weeks. All of a sudden federation links vanished. I tried re-configuring the upstream but it did not work. All this while I could see federation policy active on my exchanges. When i removed and recreated the policy, links were established and things got back to normal.

You might want to cross check if your queues are having the federation policy applied.  Priority of the policy also plays a role when it comes to resolution in case of multiple policies having same regex.

Thanks

Anup

[Ankit Gupta]

There is only one virtual host (default) on both clusters

[root@oldcluster]# rabbitmqctl list_vhosts

Listing vhosts ...

/

[root@oldcluster oldcluster]#

[root@newcluster ~]# rabbitmqctl list_vhosts

Listing vhosts ...

/

[root@newcluster]#

Policy is define for all queues like this - 

/       blue-green-migration    queues  .*      {"federation-upstream":"blue"}  0

Parameter is defined as -

[root@newcluster ~]# rabbitmqctl list_parameters

Listing runtime parameters ...

federation-upstream     blue    {"uri":"amqp://guest:password@IP_OLD_CLUSTER"}

[root@newcluster ~]#

[anup anand]

Is there any other policy defined with priority 0?

Can you try creating federation policy with priority 9999. Idea is to have federation policy supersede any other qualifying policy.

Thanks

Anup

[Michael Klishin]

Anup has a good point: policies also have priorities and only one applies at a time.

I tried following the Blue/Green deployment guide with 2 blank nodes on one of which ("blue")

I declared two durable queues. I then  made a mistake in the process: the policy was set on the

blue node. Nonetheless it is possible to observe that the policy matches my queues:

https://monosnap.com/file/XxI2UwiyxuGific9pWaJg7pyZDWx3f

there are no other policies, so priority is not an issue. We will update the docs to make it clear that the priority

must have a high priority.

I then corrected my mistake:

 * Deleted the policy

 * Deleted the upstream

 * Redeclared the upstream on the green node

 * Redeclared the policy on the green node

At that point I could see the links in the management UI (I have rabbitmq_federation_management enabled) and

that both ends have 2 queues with a policy applied to them.

I then published 3 messages to durable.q1 on the blue node and added a consumer to a queue with the same name on the green node.

The consumer consumed them almost immediately as queue federation moved them to the blue node which had consumers.

Of course during a live system there would be consumers in the green cluster. Consumer migration will be done gradually,

one by one, and then the same with publishers.

So besides the missing note about policy priority — which does exist in the policy guide — I could not find any obvious issues in the

Blue/Green deployment guide.

HTH.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Ankit Gupta]

Thanks Anup! That worked.

Thanks Michael and Anup for prompt response on this issue. You have unblocked me and really appreciate your response on this forum. It would be good to update the documentation for blue/green upgrade accordingly.

Thanks

Ankit