Federation plugin setup producing no logs or output

[Noonan, Tom]

Hello:

I’m attempting to set up federation but I’m running into a wall with
logging. My setup isn’t working, as shown by "rabbitmqctl eval
'rabbit_federation_status:status().’". However, more concerningly, it
isn’t logging anything about any errors. This is making it
extraordinarily difficult to troubleshoot as I have no clues as to why it
isn¹t working. Starting from Rabbit servers with the sensu vhost already
set up my steps are as follows:

On Both Upstream and Downstream
------------------------------

# Set federation logging to debug level
echo "[{rabbit, [ {log_levels, [{connection,
info},{federation,debug}]}]}]." >> /etc/rabbitmq/rabbitmq.config

# Enable federation plugin
rabbitmq-plugins enable rabbitmq_federation

# Restart Rabbit wiping the logs
/etc/init.d/rabbitmq-server stop
rm /var/log/rabbitmq/*
/etc/init.d/rabbitmq-server start

On the downstream node
---------------------

# Define the upstream
rabbitmqctl set_parameter federation-upstream upstreamA
'{"uri":"amqp://sensu:password@host:5672","max-hops":1}'

# Define the policy
rabbitmqctl set_policy -p /sensu --apply-to exchanges sensu-result
"^(results|keepalives)" '{"federation-upstream-set":"all"}'

The first block works as expected. The plugin enable prints a message to
the logs that it is enabled. The downstream commands report success on the
command line, but don't seem to do anything. No login attempt is seen on
the upstream, and neither rabbit server logs anything related to
federation.


Can someone please advise as to what I¹m doing wrong and why it isn¹t
logging anything? I¹m unable to figure this out on my own without logs.

Here is the log off the downstream node:

root@56c4b544d8f8:/# cat /var/log/rabbitmq/rabbit\@56c4b544d8f8.log

=INFO REPORT==== 10-Feb-2016::23:05:03 ===
Starting RabbitMQ 3.6.0 on Erlang 17.3
Copyright (C) 2007-2015 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 10-Feb-2016::23:05:03 ===
node : rabbit@56c4b544d8f8
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : 5y7SA4yuUvTkkwRXSBGkUw==
log : /var/log/rabbitmq/rab...@56c4b544d8f8.log
sasl log : /var/log/rabbitmq/rab...@56c4b544d8f8-sasl.log
database dir : /var/lib/rabbitmq/mnesia/rabbit@56c4b544d8f8

=INFO REPORT==== 10-Feb-2016::23:05:03 ===
Memory limit set to 95MB of 238MB total.

=INFO REPORT==== 10-Feb-2016::23:05:04 ===
Disk free limit set to 50MB

=INFO REPORT==== 10-Feb-2016::23:05:04 ===
Limiting to approx 524188 file handles (471767 sockets)

=INFO REPORT==== 10-Feb-2016::23:05:04 ===
FHC read buffering: OFF
FHC write buffering: ON

=INFO REPORT==== 10-Feb-2016::23:05:04 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 10-Feb-2016::23:05:05 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 10-Feb-2016::23:05:05 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 10-Feb-2016::23:05:05 ===
started TCP Listener on [::]:5672

=INFO REPORT==== 10-Feb-2016::23:05:05 ===
Server startup complete; 2 plugins started.
* rabbitmq_federation
* amqp_client

=INFO REPORT==== 10-Feb-2016::23:05:09 ===
accepting AMQP connection <0.280.0> (192.168.253.197:50469 ->
172.17.0.5:5672)

=INFO REPORT==== 10-Feb-2016::23:05:09 ===
accepting AMQP connection <0.277.0> (172.17.0.1:41230 -> 172.17.0.5:5672)

=INFO REPORT==== 10-Feb-2016::23:19:56 ===
accepting AMQP connection <0.392.0> (192.168.253.159:48544 ->
172.17.0.5:5672)

=INFO REPORT==== 10-Feb-2016::23:22:10 ===
accepting AMQP connection <0.416.0> (192.168.253.182:56361 ->
172.17.0.5:5672)

=INFO REPORT==== 10-Feb-2016::23:24:25 ===
accepting AMQP connection <0.441.0> (192.168.253.233:41257 ->
172.17.0.5:5672)

=INFO REPORT==== 10-Feb-2016::23:26:40 ===
accepting AMQP connection <0.466.0> (192.168.253.219:42905 ->
172.17.0.5:5672)

=INFO REPORT==== 10-Feb-2016::23:28:56 ===
accepting AMQP connection <0.496.0> (192.168.253.192:41494 ->
172.17.0.5:5672)




Tom Noonan II | Corvisa, a ShoreTel company | DevOps Engineer
1610 N. Second Street, Suite 101, Milwaukee, WI 53212
p: 720.966.8896 | w: www.corvisa.com <http://www.corvisa.com/>

[Michael Klishin]

On 11 February 2016 at 02:59:10, Noonan, Tom (thomas...@corvisa.com) wrote:
> The first block works as expected. The plugin enable prints
> a message to
> the logs that it is enabled. The downstream commands report success
> on the
> command line, but don't seem to do anything. No login attempt
> is seen on
> the upstream, and neither rabbit server logs anything related
> to
> federation.
>
>
> Can someone please advise as to what I¹m doing wrong and why it
> isn¹t
> logging anything? I¹m unable to figure this out on my own without
> logs.

Do you have at least one exchange that matches ^(results|keepalives)?
Federation links are only opened when there's something that would use them
(e.g. an exchange that needs to be federated) ,
otherwise federation doesn't do anything, whether exchange or queue.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Noonan, Tom]

Yes, Sensu sets up a results and keepalive exchange in the /sensu vhost.

On downstream:
root@56c4b544d8f8:/# rabbitmqctl list_exchanges -p /sensu
Listing exchanges ...
amq.rabbitmq.trace topic
amq.fanout fanout
amq.headers headers
roundrobin:remotecheckers direct
amq.topic topic
amq.match headers
results direct
keepalives direct
direct
amq.direct direct


My understanding is the exchange needs to exist on the downstream to set
up federation, and will be created on the upstream if it doesn’t already
exist.

Tom Noonan II | Corvisa, a ShoreTel company | DevOps Engineer
1610 N. Second Street, Suite 101, Milwaukee, WI 53212
p: 720.966.8896 | w: www.corvisa.com <http://www.corvisa.com/>

On 2/10/16, 5:13 PM, "rabbitm...@googlegroups.com on behalf of Michael
Klishin" <rabbitm...@googlegroups.com on behalf of

>--
>You received this message because you are subscribed to the Google Groups
>"rabbitmq-users" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to rabbitmq-user...@googlegroups.com.
>To post to this group, send an email to rabbitm...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

On 11 February 2016 at 03:24:39, Noonan, Tom (thomas...@corvisa.com) wrote:
> Yes, Sensu sets up a results and keepalive exchange in the /sensu vhost.
>
> On downstream:
> root@56c4b544d8f8:/# rabbitmqctl list_exchanges -p /sensu
> Listing exchanges ...
> amq.rabbitmq.trace topic
> amq.fanout fanout
> amq.headers headers
> roundrobin:remotecheckers direct
> amq.topic topic
> amq.match headers
> results direct
> keepalives direct
> direct
> amq.direct direct
>
>
> My understanding is the exchange needs to exist on the downstream to set
> up federation, and will be created on the upstream if it doesn’t already
> exist.

Your understanding is correct. There is a federation management plugin and federated
exchanges stand out on exchange list in management UI, so take a look if you have
any upstreams running and any exchanges are marked as federated. If not, then I suspect exchange
names do not match .

[Noonan, Tom]

So upon further investigation my issues appear to be due to the fact that
I¹m trying to federate a vhost. Starting from a blank system I add a test
vhost and enable the plugins and users I need:

# /etc/init.d/rabbitmq-server start
# rabbitmqctl add_vhost test
# rabbitmqctl add_user test secret
# rabbitmqctl set_permissions -p test test ".*" ".*" ".*"
# rabbitmq-plugins enable rabbitmq_management
# rabbitmq-plugins enable rabbitmq_federation
# rabbitmq-plugins enable rabbitmq_federation_management
# rabbitmqctl add_user admin admin
# rabbitmqctl set_user_tags admin administrator
# rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
# rabbitmqctl set_permissions -p test admin ".*" ".*" ".*²

I then restart rabbit to ensure everything is loaded correctly and stable:
# /etc/init.d/rabbitmq-server stop
# /etc/init.d/rabbitmq-server start


So now at this point I have a fresh system with the default / vhost and a
test vhost (named test) with the default amq exchanges. I can confirm
this via the management UI.

At this point I add my upstream and add the policy example by copying it
directly from https://www.rabbitmq.com/federation-reference.html

# rabbitmqctl set_parameter federation-upstream upstreamA

'{"uri":"amqp://sensu:password@host:5672","max-hops":1}'

# rabbitmqctl set_policy federate-me '^amq\.'
'{"federation-upstream-set":"all"}'
Setting policy "federate-me" for pattern "^amq\\." to
"{\"federation-upstream-set\":\"all\"}" with priority "0" ...
# rabbitmqctl eval 'rabbit_federation_status:status().¹
(Returns an array of results, omitted for brevity)


So far everything is working as expected. In the UI I see the upstream,
the amq exchanges on the / vhost show the federate-me policy, and
federation status reports an expected error as the user I¹m using for my
upstream doesn¹t have access to the / vhost on the upstream server.

So far so good. So at this point I clear the policy from the / vhost and
try and apply it to the test vhost. Note that this is the same set_policy
command as above, only with the ³-p test² argument added to set the vhost:

# rabbitmqctl clear_policy federate-me
Clearing policy "federate-me" ...
# rabbitmqctl -p test set_policy federate-me '^amq\.'
'{"federation-upstream-set":"all"}'
Setting policy "federate-me" for pattern "^amq\\." to
"{\"federation-upstream-set\":\"all\"}" with priority "0" ...
# rabbitmqctl eval 'rabbit_federation_status:status().'
[]


This is where the wheels fall off and I see the same behavior as in my
initial message. The management UI reports the federate-me policy being
applied to the test vhost amq exchanges, but federation status reports ³Š
No links Š². I¹m at a loss as to why this is happening, as far as I know
there is no special setup needed to prep a vhost for federation. Per the
exchanges tab in the management UI I believe my policy is being applied to
the exchanges, but something else is broken. Am I still overlooking
something?

Tom Noonan II | Corvisa, a ShoreTel company | DevOps Engineer
1610 N. Second Street, Suite 101, Milwaukee, WI 53212
p: 720.966.8896 | w: www.corvisa.com <http://www.corvisa.com/>

[Michael Klishin]

On 12 February 2016 at 01:40:02, Noonan, Tom (thomas...@corvisa.com) wrote:
> # rabbitmqctl set_parameter federation-upstream upstreamA
> '{"uri":"amqp://sensu:password@host:5672","max-hops":1}'
> # rabbitmqctl set_policy federate-me '^amq\.'
> '{"federation-upstream-set":"all"}'

The above upstream connects using default vhost. Vhosts are specified as percent-encoded
paths in AMQP URIs: http://www.rabbitmq.com/uri-spec.html.

which is not a deal breaker for federation per se but a mismatch that's not present in
your original transcript where you use the default vhost intentionally. 

[Noonan, Tom]

Aha, the vhost was the error. Something had given me the impression that
the upstream declaration was independent of the vhosts which was
incorrect, and I overlooked it in the upstream tab of the management UI
when troubleshooting. I needed to add the -p <vhost> flag when setting my
upstream. Once I did this federation status lit up. Thanks for the tips,
they were a huge help.

Tom Noonan II | Corvisa, a ShoreTel company | DevOps Engineer
1610 N. Second Street, Suite 101, Milwaukee, WI 53212
p: 720.966.8896 | w: www.corvisa.com <http://www.corvisa.com/>