RabbitMQ cluster crash with stuck queues (RabbitMQ 3.6.9, Erlang 19.3)

6,840 views

Skip to first unread message

Aneci Adrian

unread,

Apr 27, 2017, 5:28:05 AM4/27/17

to rabbitmq-users

Hi team,

We have some issues in a 2-nodes rabbitmq cluster. After a master node restart some of the queues got stuck being unresponsive(NaN value in UI and no ha policy applied on them).

Also we cannot delete them from the UI, the browser freeze after delete button is pressed.

From the logs I've found a lot of errors after the master started up.

Below you can find some snipped logs:

=ERROR REPORT==== 19-Apr-2017::09:21:28 ===

** Generic server <0.919.0> terminating

** Last message in was {'$gen_cast',go}

** When Server state == {not_started,

{amqqueue,

{resource,<<"sensu">>,queue,

<<"QUEUE_NAME">>},

false,true,none,[],<7225.2872.0>,[],[],

['rabbit@node-1'],

[{vhost,<<"sensu">>},

{name,<<"hapolicy">>},

{pattern,<<"^.">>},

{'apply-to',<<"all">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[{<7225.2873.0>,<7225.2872.0>}],

[],live,0}}

** Reason for termination ==

** {noproc,

{gen_server2,call,

[msg_store_transient,

{new_client_state,

<<74,182,53,1,204,63,175,161,65,87,1,84,46,29,215,150>>,

<0.919.0>,undefined,#Fun<rabbit_variable_queue.20.118228399>},

infinity]}}

=ERROR REPORT==== 19-Apr-2017::09:43:04 ===

Discarding message {'$gen_call',{<0.16596.1>,#Ref<0.0.2.48068>},stat} from <0.16596.1> to <0.6006.0> in an old incarnation (1) of this node (3)

=ERROR REPORT==== 26-Apr-2017::05:47:38 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.3095>},stat} from <0.11897.0> to <0.499.0> in an old incarnation (1) of this node (3)

=WARNING REPORT==== 26-Apr-2017::05:47:38 ===

msg_store_persistent: rebuilding indices from scratch

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2687.5> (10.104.100.242:51852 -> 172.18.0.2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue 'QUEUE_NAME' in vhost 'sensu' due to timeout

Can you please give us a hist what can be the problem? Can be a bug?

Thank you!

Regards,

Adrian.

Karl Nilsson

unread,

Apr 27, 2017, 6:23:50 AM4/27/17

to rabbitm...@googlegroups.com

Hi,

It looks like the message store might have gone down and restarted. Could you share a more complete log from both nodes and the steps you have already taken to try to resolve it?

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Karl Nilsson

Pivotal/RabbitMQ

Aneci Adrian

unread,

Apr 27, 2017, 8:34:55 AM4/27/17

to rabbitmq-users

Hi Karl,

Here are some detailed logs form the node that was restarted and also the one that use to be the master:

{code}

=INFO REPORT==== 26-Apr-2017::05:47:31 ===

Starting RabbitMQ 3.6.9 on Erlang 19.3

Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 26-Apr-2017::05:47:31 ===

node : rabbit@sensu-prod-rabbit-1

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : kwA70MfN15Db51BmexXZQg==

log : /data/rabbitmq_sensu-prod-rabbit-1.log

sasl log : tty

database dir : /data/mnesia

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Memory limit set to 12702MB of 31756MB total.

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Disk free limit set to 50MB

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Limiting to approx 65436 file handles (58890 sockets)

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

FHC read buffering: OFF

FHC write buffering: ON

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Starting rabbit_node_monitor

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Management plugin: using rates mode 'basic'

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=ERROR REPORT==== 26-Apr-2017::05:47:38 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.3095>},stat} from <0.11897.0> to <0.499.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 26-Apr-2017::05:47:38 ===

msg_store_persistent: rebuilding indices from scratch

=ERROR REPORT==== 26-Apr-2017::05:47:38 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.3108>},stat} from <0.11897.0> to <0.638.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Mirrored queue 'QUEUE_NAME' in vhost 'sensu': Adding mirror on node 'rabbit@sensu-prod-rabbit-1': <0.11982.0>

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Mirrored queue 'QUEUE_NAME' in vhost 'sensu': Adding mirror on node 'rabbit@sensu-prod-rabbit-1': <0.11986.0>

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Mirrored queue 'aliveness-test' in vhost 'sensu': Adding mirror on node 'rabbit@sensu-prod-rabbit-1': <0.11990.0>

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Mirrored queue 'QUEUE_NAME' in vhost 'sensu': Adding mirror on node 'rabbit@sensu-prod-rabbit-1': <0.11994.0>

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Mirrored queue 'QUEUE_NAME' in vhost 'sensu': Adding mirror on node 'rabbit@sensu-prod-rabbit-1': <0.11997.0>

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Adding mirror on node 'rabbit@sensu-prod-rabbit-1': <0.12002.0>

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Adding mirror on node 'rabbit@sensu-prod-rabbit-1': <0.12006.0>

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

Mirrored queue 'results' in vhost 'sensu': Adding mirror on node 'rabbit@sensu-prod-rabbit-1': <0.12010.0>

=ERROR REPORT==== 26-Apr-2017::05:47:38 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.3197>},stat} from <0.11897.0> to <0.403.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

started TCP Listener on [::]:5672

=INFO REPORT==== 26-Apr-2017::05:47:38 ===

rabbit on node 'rabbit@sensu-prod-rabbit-2' up

=ERROR REPORT==== 26-Apr-2017::05:47:38 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.3333>},stat} from <0.11897.0> to <0.387.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:47:40 ===

Management plugin started. Port: 15672

=INFO REPORT==== 26-Apr-2017::05:47:40 ===

Statistics database started.

=ERROR REPORT==== 26-Apr-2017::05:47:40 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.3991>},stat} from <0.11897.0> to <0.455.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:47:40 ===

Server startup complete; 6 plugins started.

* rabbitmq_management

* rabbitmq_web_dispatch

* rabbitmq_management_agent

* cowboy

* cowlib

* amqp_client

=ERROR REPORT==== 26-Apr-2017::05:47:40 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.1.7615>},stat} from <0.11897.0> to <0.751.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2674.5> (ip1:51846 -> ip2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2674.5> (ip1:51846 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.1.151430>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2687.5> (IP1:51852 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2687.5> (IP1:51852 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.1.151467>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2699.5> (IP1:51858 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2699.5> (IP1:51858 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.1.151504>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2713.5> (IP1:51862 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2713.5> (IP1:51862 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.1.151541>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2724.5> (IP1:51868 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.103276>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2724.5> (ip1:51868 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103285>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2737.5> (IP1:51872 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2737.5> (ip1:51872 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.103291>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2750.5> (ip1:51876 -> ip2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2753.5> (ip1:51880 -> ip2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2750.5> (ip1:51876 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2753.5> (ip1:51880 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2772.5> (ip1:51884 -> ip2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2772.5> (ip1:51884 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103311>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

accepting AMQP connection <0.2785.5> (IP1:51890 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:18 ===

connection <0.2785.5> (ip1:51890 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:18 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103324>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2796.5> (ip1:51894 -> ip2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.103328>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2800.5> (ip1:51898 -> ip2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2796.5> (ip1:51894 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2800.5> (ip1:51898 -> ip2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2820.5> (ip1:51902 -> ip2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2823.5> (IP1:51906 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103348>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2820.5> (IP1:51902 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2823.5> (IP1:51906 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.103374>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2843.5> (IP1:51910 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2847.5> (IP1:51914 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2850.5> (IP1:51918 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2847.5> (IP1:51914 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2856.5> (IP1:51922 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2850.5> (IP1:51918 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2856.5> (IP1:51922 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2878.5> (IP1:51926 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2843.5> (IP1:51910 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2878.5> (IP1:51926 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2895.5> (IP1:51930 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2895.5> (IP1:51930 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2906.5> (IP1:51934 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103490>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2913.5> (IP1:51938 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2913.5> (IP1:51938 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.103512>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2906.5> (IP1:51934 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2929.5> (IP1:51942 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2932.5> (IP1:51946 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2940.5> (IP1:51950 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2929.5> (IP1:51942 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2947.5> (IP1:51954 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2951.5> (IP1:51958 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2932.5> (IP1:51946 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2961.5> (IP1:51962 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2940.5> (IP1:51950 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.2968.5> (IP1:51966 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2961.5> (IP1:51962 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2968.5> (IP1:51966 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103646>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2947.5> (IP1:51954 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.2951.5> (IP1:51958 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3003.5> (IP1:51970 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3006.5> (IP1:51974 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.103674>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3006.5> (IP1:51974 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3003.5> (IP1:51970 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3031.5> (IP1:51978 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3034.5> (IP1:51982 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3031.5> (IP1:51978 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3034.5> (IP1:51982 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3049.5> (IP1:51986 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3049.5> (IP1:51986 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3068.5> (IP1:51990 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3068.5> (IP1:51990 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103776>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3081.5> (IP1:51994 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3081.5> (IP1:51994 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103791>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3093.5> (IP1:51998 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3093.5> (IP1:51998 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3105.5> (IP1:52002 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3105.5> (IP1:52002 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103823>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3120.5> (IP1:52010 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3120.5> (IP1:52010 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3132.5> (IP1:52014 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103864>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3132.5> (IP1:52014 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3145.5> (IP1:52018 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103895>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3145.5> (IP1:52018 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.103904>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3158.5> (IP1:52022 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3158.5> (IP1:52022 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3169.5> (IP1:52026 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103955>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3169.5> (IP1:52026 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.103965>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3181.5> (IP1:52030 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11921.0>,#Ref<0.0.2.103985>},stat} from <0.11921.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

accepting AMQP connection <0.3185.5> (IP1:52034 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3185.5> (IP1:52034 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_cast',{deliver,{delivery,false,false,<0.2682.5>,{basic_message,{resource,<<"sensu">>,exchange,<<"keepalives">>},[<<>>],{content,60,{'P_basic',<<"application/octet-stream">>,undefined,undefined,1,0,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},<<152,0,24,97,112,112,108,105,99,97,116,105,111,110,47,111,99,116,101,116,45,115,116,114,101,97,109,1,0>>,rabbit_framing_amqp_0_9_1,[<<"{\"name\":\"SERVER\",\"address\":\"IP\",\"subscriptions\":[\"prod\",\"ams01\",\"SERVER\",\"client:SERVER\"],\"safe_mode\":false,\"keepalive\":{},\"socket\":{\"bind\":\"127.0.0.1\",\"port\":3030},\"version\":\"0.26.5\",\"timestamp\":1493185759}">>]},<<156,71,103,183,190,97,206,41,79,2,221,90,69,198,161,100>>,false},undefined,flow},false}} from <0.2682.5> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:19 ===

connection <0.3181.5> (IP1:52030 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:19 ===

Discarding message {'$gen_call',{<0.11897.0>,#Ref<0.0.2.104028>},stat} from <0.11897.0> to <0.631.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:21 ===

accepting AMQP connection <0.4328.5> (IP1:52050 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:21 ===

Discarding message {'$gen_call',{<0.2980.5>,#Ref<0.0.1.153824>},stat} from <0.2980.5> to <0.547.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:22 ===

connection <0.4328.5> (IP1:52050 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:22 ===

Discarding message {'$gen_call',{<0.3011.5>,#Ref<0.0.1.154440>},stat} from <0.3011.5> to <0.439.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:27 ===

accepting AMQP connection <0.3116.5> (IP1:52006 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:27 ===

Discarding message {'$gen_call',{<0.11910.0>,#Ref<0.0.4.20437>},stat} from <0.11910.0> to <0.719.0> in an old incarnation (1) of this node (3)

=WARNING REPORT==== 26-Apr-2017::05:49:27 ===

closing AMQP connection <0.3116.5> (IP1:52006 -> IP2:5672):

client unexpectedly closed TCP connection

=INFO REPORT==== 26-Apr-2017::05:49:28 ===

accepting AMQP connection <0.10304.5> (IP1:52122 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:28 ===

Discarding message {'$gen_call',{<0.3023.5>,#Ref<0.0.3.61146>},stat} from <0.3023.5> to <0.626.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:28 ===

connection <0.10304.5> (IP1:52122 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:28 ===

Discarding message {'$gen_call',{<0.3202.5>,#Ref<0.0.1.160012>},stat} from <0.3202.5> to <0.383.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:43 ===

accepting AMQP connection <0.22055.5> (IP1:52284 -> IP2:5672)

=ERROR REPORT==== 26-Apr-2017::05:49:43 ===

Discarding message {'$gen_call',{<0.2708.5>,#Ref<0.0.3.70321>},stat} from <0.2708.5> to <0.619.0> in an old incarnation (1) of this node (3)

=INFO REPORT==== 26-Apr-2017::05:49:43 ===

connection <0.22055.5> (IP1:52284 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=ERROR REPORT==== 26-Apr-2017::05:49:43 ===

Discarding message {'$gen_call',{<0.2922.5>,#Ref<0.0.2.122628>},stat} from <0.2922.5> to <0.667.0> in an old incarnation (1) of this node (3)

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2674.5> (IP1:51846 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2674.5> (IP1:51846 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2687.5> (IP1:51852 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME in vhost 'sensu' due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2687.5> (IP1:51852 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2699.5> (IP1:51858 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2699.5> (IP1:51858 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2724.5> (IP1:51868 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2724.5> (IP1:51868 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2750.5> (IP1:51876 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2750.5> (IP1:51876 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2713.5> (IP1:51862 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME in vhost 'sensu' due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2713.5> (IP1:51862 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2753.5> (IP1:51880 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2753.5> (IP1:51880 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2737.5> (IP1:51872 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2737.5> (IP1:51872 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2847.5> (IP1:51914 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2796.5> (IP1:51894 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2847.5> (IP1:51914 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2772.5> (IP1:51884 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2796.5> (IP1:51894 -> IP2:5672):

client unexpectedly closed TCP connection

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2772.5> (IP1:51884 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2800.5> (IP1:51898 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2823.5> (IP1:51906 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2800.5> (IP1:51898 -> IP2:5672):

client unexpectedly closed TCP connection

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2823.5> (IP1:51906 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2850.5> (IP1:51918 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2878.5> (IP1:51926 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2878.5> (IP1:51926 -> IP2:5672):

client unexpectedly closed TCP connection

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2850.5> (IP1:51918 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2940.5> (IP1:51950 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2820.5> (IP1:51902 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2856.5> (IP1:51922 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2785.5> (IP1:51890 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2820.5> (IP1:51902 -> IP2:5672):

client unexpectedly closed TCP connection

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2856.5> (IP1:51922 -> IP2:5672):

client unexpectedly closed TCP connection

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2940.5> (IP1:51950 -> IP2:5672):

client unexpectedly closed TCP connection

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2785.5> (IP1:51890 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2895.5> (IP1:51930 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2895.5> (IP1:51930 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2913.5> (IP1:51938 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2913.5> (IP1:51938 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2843.5> (IP1:51910 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2843.5> (IP1:51910 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2968.5> (IP1:51966 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2968.5> (IP1:51966 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.3003.5> (IP1:51970 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2947.5> (IP1:51954 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.3003.5> (IP1:51970 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2951.5> (IP1:51958 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME in vhost 'sensu' due to timeout

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.2906.5> (IP1:51934 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2947.5> (IP1:51954 -> IP2:5672):

client unexpectedly closed TCP connection

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2951.5> (IP1:51958 -> IP2:5672):

client unexpectedly closed TCP connection

=ERROR REPORT==== 26-Apr-2017::05:50:22 ===

Channel error on connection <0.3031.5> (IP1:51978 -> IP2:5672, vhost: 'sensu', user: 'sensu'), channel 1:

operation queue.declare caused a channel exception not_found: failed to perform operation on queue QUEUE_NAME due to timeout

=WARNING REPORT==== 26-Apr-2017::05:50:22 ===

closing AMQP connection <0.2906.5> (IP1:51934 -> IP2:5672):

client unexpectedly closed TCP connection

{code}

And also some logs from the second node(the slave one):

{code}

=WARNING REPORT==== 26-Apr-2017::05:45:24 ===

closing AMQP connection <0.4993.0> (IP1:33630 -> IP2:5672):

client unexpectedly closed TCP connection

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

rabbit on node 'rabbit@sensu-prod-rabbit-1' down

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

node 'rabbit@sensu-prod-rabbit-1' down: connection_closed

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'aliveness-test' in vhost 'sensu': Slave <rab...@sensu-prod-rabbit-2.2.403.0> saw deaths of mirrors <rab...@sensu-prod-rabbit-1.1.411.0>

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'aliveness-test' in vhost 'sensu': Promoting slave <rab...@sensu-prod-rabbit-2.2.403.0> to master

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'aliveness-test' in vhost 'sensu': Synchronising: 0 messages to synchronise

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'aliveness-test' in vhost 'sensu': Synchronising: batch size: 4096

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Slave <rab...@sensu-prod-rabbit-2.2.603.0> saw deaths of mirrors <rab...@sensu-prod-rabbit-1.1.615.0>

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Promoting slave <rab...@sensu-prod-rabbit-2.2.603.0> to master

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'aliveness-test' in vhost 'sensu': Synchronising: all slaves already synced

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Synchronising: 0 messages to synchronise

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Synchronising: batch size: 4096

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Synchronising: all slaves already synced

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'results' in vhost 'sensu': Slave <rab...@sensu-prod-rabbit-2.2.363.0> saw deaths of mirrors <rab...@sensu-prod-rabbit-1.1.375.0>

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'results' in vhost 'sensu': Promoting slave <rab...@sensu-prod-rabbit-2.2.363.0> to master

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'results' in vhost 'sensu': Synchronising: 0 messages to synchronise

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'results' in vhost 'sensu': Synchronising: batch size: 4096

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue 'results' in vhost 'sensu': Synchronising: all slaves already synced

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Slave <rab...@sensu-prod-rabbit-2.2.698.0> saw deaths of mirrors <rab...@sensu-prod-rabbit-1.1.711.0>

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Promoting slave <rab...@sensu-prod-rabbit-2.2.698.0> to master

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Synchronising: 0 messages to synchronise

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Synchronising: batch size: 4096

=INFO REPORT==== 26-Apr-2017::05:45:24 ===

Mirrored queue QUEUE_NAME in vhost 'sensu': Synchronising: all slaves already synced

=INFO REPORT==== 26-Apr-2017::05:45:30 ===

accepting AMQP connection <0.8213.0> (IP1:56026 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:45:30 ===

connection <0.8213.0> (IP1:56026 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:47:08 ===

accepting AMQP connection <0.8485.0> (IP1:48664 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:47:08 ===

accepting AMQP connection <0.8488.0> (IP1:48668 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:47:08 ===

accepting AMQP connection <0.8482.0> (IP1:48660 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:47:08 ===

connection <0.8485.0> (IP1:48664 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:47:08 ===

connection <0.8482.0> (IP1:48660 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:47:08 ===

connection <0.8488.0> (IP1:48668 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'

=INFO REPORT==== 26-Apr-2017::05:47:08 ===

accepting AMQP connection <0.8544.0> (IP1:48670 -> IP2:5672)

=INFO REPORT==== 26-Apr-2017::05:47:08 ===

connection <0.8544.0> (IP1:48670 -> IP2:5672): user 'sensu' authenticated and granted access to vhost 'sensu'