3.6.12 got stuck after upgrade

[Dmitry Kurbatov]

Hi, All.

I have failed production upgrade from 3.5.7 to 3.6.12 (OTP 18.1) :(

I have 5 nodes cluster: 4 disc + 1 ram. Storage upgrade was successful (by logs). Cluster worked fine for a ~30 minutes and after that one of the nodes become unavailable - just disappeared from the radars. There are no any records in the logs of this node after disappearance. Other nodes has only these messages:

=ERROR REPORT==== 30-Oct-2017::23:05:09 ===

** Node rabbit@rabbitmq3 not responding **
** Removing (timedout) connection **

=INFO REPORT==== 30-Oct-2017::23:05:09 ===
rabbit on node rabbit@rabbitmq3 down

=INFO REPORT==== 30-Oct-2017::23:05:16 ===
node rabbit@rabbitmq3 down: net_tick_timeout

I tried to get status:

[ro...@rabbitmq3.ssd:~]# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq3
Error: unable to connect to node rabbit@rabbitmq3: nodedown
^C
Session terminated, terminating shell... ...terminated.

but process was in `ps` list and consume cpu:

rabbitmq  8845 54.8  2.6 2697244 442056 ?      Sl   22:58   4:42 /usr/lib/erlang/erts-7.1/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -A 128 -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rab...

I have restart whole cluster, but later other node disappeared. Restart cluster - some time - down - repeat.

Finally I've reduced cluster to only one single node, but it still becoming stuck over some random time. I've decide to fallback to 3.5.7.

I have 2 questions:

1. (naive) What was that?

2. (pactical) What is the commands I can run to gather runtime/debug information of stuck node?

--

Thanks in advance,

Dmitry

[Michael Klishin]

Start with server logs and continuous process memory and CPU consumption monitoring.

It could be that the node has a lot of data to recover/migrate but that's just a guess at this point.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Then there is `rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'`but on a node starting up

there usually will be processes that "make no progress" as far as process stack trace changes go.

Another thing I'd check is what the queue master distribution is between the nodes since other nodes

do boot successfully. It could be that queue master rebalancing might help.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Dmitry Kurbatov]

Start with server logs and continuous process memory and CPU consumption monitoring.

There was 12Gb free RAM and low CPU consumption - node has 8 CPU and only beam.smp works.

 

It could be that the node has a lot of data to recover/migrate but that's just a guess at this point.

As I said earlier - I have reduced cluster to one single node but the problem has not away.

What is the commands (rabbimqctl eval ...) shoud I run for gathering debug info on next upgrade attempt?

[Dmitry Kurbatov]

> Then there is `rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'`but on a
> node starting up

At the latest attempt to run `rabbitmqctl cluster_status` I got error:

Error: unable to connect to node rabbit@rabbitmq3: nodedown

Will `rabbitmqctl eval` work in this case?

[Michael Klishin]

No, because rabbitmqctl says it could not contact the node.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

"Reducing" cluster to 1 node is not something I'd recommend when investigating this.

While nodes can be restarted in arbitrary order starting with 3.6.7, they still will try to contact

other cluster peers to sync internal database schema and data from.

If CPU and RAM consumption during recovery is low it is perfectly possible that the node tries to reach

its last known peer, that times out, it retries, and so on before it completely gives up (there are 10 retries every 30 seconds

by default).

All of that leaves very clear log messages so instead of focussing on finding a debug command to run, consider

taking a long careful look at the logs (and posting them here).

Looking at server logs is almost always the first step that should be taken.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dmitry Kurbatov]

> "Reducing" cluster to 1 node is not something I'd recommend when investigating this.

Why not? On the node1 I have ran `rabbitmqctl forget_cluster_node` and
remove others from cluster. I mean that I had eliminate possible
"clustering errors" like sync blocks and so on. I have single node
rabbitmq and that node had the same problems as a 5-node cluster -
stucking over some time after start.

> While nodes can be restarted in arbitrary order starting with 3.6.7, they still will try to contact other cluster peers to sync internal database schema and data from.

I repeat - there was not cluster - there was just single node.

> Looking at server logs is almost always the first step that should be taken.

I repeat second time - there are no any records in log on stuck node.
That node just stoped write to log after stuck.

Are any other debug commands exist to get additional info about
rabbit's current state, that may help to understand the problem?

>> email to rabbitmq-user...@googlegroups.com.
>> To post to this group, send an email to rabbitm...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>

> --
> You received this message because you are subscribed to the Google Groups
> "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.

[Michael Klishin]

Dmitry,

You haven't exactly provided a step by step transcript of what has been done so I had to guess

what might be happening. No need to stress your point over and over and treat members of this list as 5 year olds.

Failing to follow this simple rule will get you permanently banned from this list (and I am not going

to repeat this more than one).

Now with that out of the way…

RabbitMQ 3.6.x doesn't have a debug log level but there is no shortage of Erlang tools

that can trace what's going on in the system or let you inspect stack traces:

http://erlang.org/doc/man/observer.html

http://ferd.github.io/recon/recon_trace.html

https://github.com/massemanet/redbug

as well as `rabbitmqctl eval` which can do many things, including sample stack traces of

running processes.

strace and similar tools can provide plenty of information about what the VM may be doing,

be it disk I/O or a hostname resolution query that takes a long time.

>> email to rabbitmq-users+unsubscribe@googlegroups.com.
>> To post to this group, send an email to rabbitmq-users@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
> --
> You received this message because you are subscribed to the Google Groups
> "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to rabbitmq-users+unsubscribe@googlegroups.com.

> To post to this group, send email to rabbitmq-users@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send an email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Sections 4, 5, 8 and 9 in Erlang in Anger can also be helpful:

https://s3.us-east-2.amazonaws.com/ferd.erlang-in-anger/text.v1.1.0.pdf.

>> To post to this group, send an email to rabbitm...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
> --
> You received this message because you are subscribed to the Google Groups
> "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to rabbitmq-users+unsubscribe@googlegroups.com.

> To post to this group, send email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

The "sample stack frames of running processes" part

was previously mentioned in https://groups.google.com/d/msg/rabbitmq-users/hK323-O6-tw/L5dNhs0BCAAJ.

>> To post to this group, send an email to rabbitm...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
> --
> You received this message because you are subscribed to the Google Groups
> "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to rabbitmq-users+unsubscribe@googlegroups.com.

> To post to this group, send email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Dmitry Kurbatov]

Michael,

plz re-read my initial question. There is detailed explanation of the
problem and my steps.

I was not aiming to offend members of this list or reproach someone. I
tried to indicate that your understanding of the issue is incorrect.
I'm not an English native, maybe it was a difficulties of translation.

In any case - lot of thanks for you cooperation and help during solve
of my problems (now and earlier). Special thanks for the links to
debugging erlang - this was my main question.

>>> >> email to rabbitmq-user...@googlegroups.com.

>>> >> To post to this group, send an email to
>>> >> rabbitm...@googlegroups.com.
>>> >> For more options, visit https://groups.google.com/d/optout.
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > MK
>>> >
>>> > Staff Software Engineer, Pivotal/RabbitMQ
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "rabbitmq-users" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an

>>> > email to rabbitmq-user...@googlegroups.com.

>>> > To post to this group, send email to rabbitm...@googlegroups.com.
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "rabbitmq-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to rabbitmq-user...@googlegroups.com.

>>> To post to this group, send an email to rabbitm...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>> --
>> MK
>>
>> Staff Software Engineer, Pivotal/RabbitMQ
>
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>

> --
> You received this message because you are subscribed to the Google Groups
> "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

Detailed or not (more like not), I don't have much to add.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

To get this thread somewhere here some specific steps that have to happen:

 * `rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'` is executed on a node that does not finish booting
 * Its process is straced
 * Output of both things is posted to this list (strace is likely to produce a ton of output, so only the last few pages should be fine)
 * Logs of that last boot are posted to this list (feel free to edit out hostnames, usernames and such)

if the data directory is small enough to share, it should be shared with us off-list. Together with logs it should give our team

enough information to try node recovery ourselves.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Dmitry Kurbatov]

It happened again on other cluster.

Here some logs:

[ro...@r630-1.qqs:~]# rabbitmqctl eval 'rabbitmq_diagnostics:maybe_stuck().'
Error: unable to connect to node 'rabbit@r630-1': nodedown

^C
Session terminated, terminating shell... ...terminated.

[ro...@r630-1.qqs:~]# ps axu | grep rabbit
rabbitmq 80817 0.0 0.0 4448 1736 ? S Oct31 0:00
/bin/sh /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 81276 82.8 0.5 11573392 2063380 ? Sl Oct31 5710:21
/usr/lib/erlang/erts-7.1/bin/beam.smp -W w -A 768 -P 1048576 -t

5000000 -stbt db -zdbbl 128000 -K true -A 128 -B i -- -root

/usr/lib/erlang -progname erl -- -home /var/lib rabbitmq -- -pa
/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.12/ebin -noshell -noinput -s
rabbit boot -sname rabbit@r630-1 -boot start_sasl -config
/etc/rabbitmq/rabbitmq -kernel inet_default_connect_options
[{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false
-rabbit error_logger {file,"/var/log/rabbitmq/rab...@r630-1.log"}
-rabbit sasl_error_logger
{file,"/var/log/rabbitmq/rab...@r630-1-sasl.log"} -rabbit
enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit
plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.12/plugins"
-rabbit plugins_expand_dir
"/var/lib/rabbitmq/mnesia/rabbit@r630-1-plugins-expand" -os_mon
start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup
false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@r630-1" -kernel
inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 81547 0.0 0.0 7464 1492 ? Ss Oct31 0:11 inet_gethost 4
rabbitmq 81548 0.0 0.0 13788 2068 ? S Oct31 0:25 inet_gethost 4
rabbitmq 81699 0.0 0.0 13788 1712 ? S Oct31 0:00 inet_gethost 4
rabbitmq 105602 0.0 0.0 9548 3576 ? S 2016 21:45
/usr/lib/erlang/erts-7.1/bin/epmd -daemon
root 112054 0.0 0.0 11752 2172 pts/13 S+ 20:28 0:00 grep
--color=auto rabbit


[ro...@r630-1.qqs:~]# strace -f -p 81276
Process 81276 attached with 181 threads
[pid 81531] futex(0x7fdfd32c2dd0, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81530] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 81529] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 81525] futex(0x7fdfd32c2c50, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81528] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 81527] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 81526] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 81524] futex(0x7fdfd32c2c10, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81523] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 81518] futex(0x7fdfd32c2a90, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81522] futex(0x7fdfd32c2b90, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
...omited same messages...
[pid 81484] futex(0x7fdfd32c2210, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81483] futex(0x7fdfd32c21d0, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81482] wait4(-1, <unfinished ...>
[pid 81480] futex(0x7fdfd32c2110, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81479] futex(0x7fdfd32c20d0, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
...omited same messages...
[pid 81356] futex(0x7fdfd32c0210, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81355] futex(0x7fdfd32c01d0, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81354] futex(0x7fdfd32c0190, FUTEX_WAIT_PRIVATE, 4294967295, NULL
<unfinished ...>
[pid 81353] futex(0x8860a4, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished ...>
[pid 81352] read(8, <unfinished ...>
[pid 81276] select(0, NULL, NULL, NULL, NULL <unfinished ...>
[pid 81493] futex(0x7fdfd32c2450, FUTEX_WAIT_PRIVATE, 4294967295,
NULL^CProcess 81276 detached
Process 81352 detached
Process 81353 detached
Process 81354 detached
Process 81355 detached
...omited same messages...







There are no useful records in logs of r630-1 and r630-2 nodes. Here
log from third quorum node that hasn't any load except web-management.


=ERROR REPORT==== 4-Nov-2017::20:06:39 ===
Ranch listener rabbit_web_dispatch_sup_15672 had connection process
started with cowboy_protocol:start_link/4 at <0.22469.41> exit with
reason: [{reason,{timeout,{gen_server,call,[<0.22465.0>,{fetch,#Fun<rabbit_mgmt_db.22.104142656>,[[[{name,<<"CD_9416@xxx11_1509822212199">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.22229.162>},{state,live}],[{name,<<"xtrader.odd.XXX02-16948-vran-13d95112-7e9a-416c-ad62-e6a828aaaff2">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.30940.218>},{state,live}],[{name,<<"WEB_...@web-demo1.qqs_1509710846487">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.16447.157>},{state,live}],[{name,<<"totalSelectionQueue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.797.0>},{state,live}],[{name,<<"profiling.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"profilingDLX">>},{<<"x-message-ttl">>,30000}]}},{pid,<8005.806.0>},{state,live}],[{name,<<"xxx.mailservice.emails">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"xxx.mailservice.dead">>}]}},{pid,<8005.824.0>},{state,live}],[{name,<<"wep-queue-web-demo1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.21483.122>},{arguments,{struct,[]}},{pid,<8005.13392.122>},{state,live}],[{name,<<"CD_9300@XXX09_1509802529729">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.10350.217>},{state,live}],[{name,<<"f1be5e71-7e5d-48fa-9b11-f2f2ee56efeb">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.11331.15>},{arguments,{struct,[{<<"x-message-ttl">>,120000}]}},{pid,<8005.7360.88>},{state,live}],[{name,<<"xxx.ticket.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.563.0>},{state,live}],[{name,<<"xtrader.odd.XXX04-25552-vran2-5cbeeeb4-97f8-4c2c-a8ca-3f03edd8c0cb">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.13835.230>},{state,live}],[{name,<<"queue-webmaster2.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.3459.161>},{arguments,{struct,[]}},{pid,<8005.9895.161>},{state,live}],[{name,<<"xxx.alfa.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.602.0>},{state,live}],[{name,<<"profiling.data.queue-dead">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.608.0>},{state,live}],[{name,<<"CD_30772@xxx06_1509782636659">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.109.144>},{state,live}],[{name,<<"scccQueue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"reports.exchange.dead">>}]}},{pid,<8005.617.0>},{state,live}],[{name,<<"xtrader.changes.XXX10-8428-vran3-8a0dd19b-128d-425a-acb6-6c175ff20719">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.5060.233>},{state,live}],[{name,<<"CD_13780@XXX07_1509813676685">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.27201.158>},{state,live}],[{name,<<"replication.live.AAA">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.659.0>},{state,live}],[{name,<<"xxx.catch.cache....@catchservice3.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.22100.0>},{state,live}],[{name,<<"I_54_1509676270819">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.13056.114>},{state,live}],[{name,<<"CD_31552@XXX13_1509802900141">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.24859.217>},{state,live}],[{name,<<"I_91_1509676270822">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.14514.114>},{state,live}],[{name,<<"replication.archive.AAA">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.731.0>},{state,live}],[{name,<<"xtrader.changes.XXX02-20496-vbondarenko-17624092-6dbe-40f9-a77f-8014ae3af91d">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,'...'}]}},{pid,<8004.6393.212>},{state,live}],[{name,<<"WEB_...@webmaster1.qqs_1509676447716">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8004.6092.138>},{state,live}],[{name,<<"CD_37600@xxx11_1509802566857">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8005.5668.153>},{state,live}],[{name,<<"productSelfExcludeMsgQueue">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.14356.114>},{state,live}],[{name,<<"WEB_...@web-demo1.qqs_1509710846486">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8004.17018.157>},{state,live}],[{name,<<"CD_4696@xxx06_1509781996333">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'},'...']}},{pid,<8004.18132.201>},{state,live}],[{name,<<"I_23_1509676270836">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8005.16207.114>},{state,live}],[{name,<<"xxxCancelQueue.Dead">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.800.0>},{state,live}],[{name,<<"xtrader.odd.XXX10-8428-vran3-2a04cf61-04f7-4fe1-86e9-44a745777660">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8004.22724.232>},{state,live}],[{name,<<"I_82_1509676270843">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8005.21228.132>},{state,live}],[{name,<<"WEB_...@webmaster2.qqs_1509818680697">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...'],'...'}},{pid,<8004.18528.230>},{state,live}],[{name,<<"xtrader.changes.XXX02-16948-vran-1f80a629-4d27-47d1-816a-28d4b5ab0bf1">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,'...'}},{pid,<8004.2245.219>},{state,live}],[{name,<<"event-actions-web-web-demo1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8004.17194.157>},{arguments,{'...'}},{pid,<8004.17393.157>},{state,live}],[{name,<<"event-actions-web-webmaster1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.16656.114>},{arguments,{'...'}},{pid,<8005.16644.114>},{state,live}],[{name,<<"xtrader.abr.XXX08-7712-vran-c055635e-9d09-43ba-8d82-36d81d261d42">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{'...'}},{pid,<8005.8735.163>},{state,live,'...'}],[{name,<<"CD_7624@PB-PAY-PC04_1509716351755">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{'...'}},{pid,<8005.23935.124>,'...'},{state,'...'}],[{name,<<"WEB_...@webmaster2.qqs_1509818680698">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{'...'},'...'},{pid,'...'},{'...'}],[{name,<<"xtrader.changes.XXX06-39636-vran3-b03733e9-c61c-40f3-bb98-5d9a32beb487">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none,'...'},{arguments,'...'},{'...'},{'...'}],[{name,<<"CD_38108@XXX07_1509800742732">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false,'...'},{owner_pid,'...'},{'...'},{'...'},{'...'}],[{name,<<"wep-work-queue-webmaster1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true,'...'},{exclusive,'...'},{'...'},{'...'},{'...'},{'...'}],[{name,<<"CD_22304@xxx06_1509814353364">>},{vhost,<<"/">>},{durable,true,'...'},{auto_delete,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,<<"xxxPlaceQueue.Dead">>},{vhost,<<"/">>,'...'},{durable,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,<<"xxx.tmp.queue">>,'...'},{vhost,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},'...'],[{'...'},'...'],['...'],['...'],['...'],['...'],['...'],'...']]},60000]}}},{mfa,{rabbit_mgmt_wm_queues,to_json,2}},{stacktrace,[{gen_server,call,3,[{file,"gen_server.erl"},{line,212}]},{rabbit_mgmt_db,submit_cached,4,[{file,"src/rabbit_mgmt_db.erl"},{line,712}]},{rabbit_mgmt_util,augment,2,[{file,"src/rabbit_mgmt_util.erl"},{line,392}]},{rabbit_mgmt_util,run_augmentation,2,[{file,"src/rabbit_mgmt_util.erl"},{line,369}]},{rabbit_mgmt_util,augment_resources0,6,[{file,"src/rabbit_mgmt_util.erl"},{line,358}]},{rabbit_mgmt_util,with_valid_pagination,3,[{file,"src/rabbit_mgmt_util.erl"},{line,282}]},{rabbit_mgmt_wm_queues,to_json,2,[{file,"src/rabbit_mgmt_wm_queues.erl"},{line,54}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,976}]}]},{req,[{socket,#Port<0.1504195>},{transport,ranch_tcp},{connection,close},{pid,<0.22469.41>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{172,16,41,44},53560}},{host,<<"rabbitmq.qqs">>},{host_info,undefined},{port,15672},{path,<<"/api/queues/">>},{path_info,undefined},{qs,<<>>},{qs_vals,[]},{bindings,[]},{headers,[{<<"te">>,<<"deflate,gzip;q=0.3">>},{<<"connection">>,<<"TE,
close">>},{<<"authorization">>,<<"Basic
emFiYml4OnphYmJpeG1vbg==">>},{<<"host">>,<<"rabbitmq.qqs:15672">>},{<<"user-agent">>,<<"libwww-perl/6.05">>},{<<"x-forwarded-for">>,<<"192.168.41.95">>}]},{p_headers,[{<<"if-modified-since">>,undefined},{<<"if-none-match">>,undefined},{<<"if-unmodified-since">>,undefined},{<<"if-match">>,undefined},{<<"accept">>,undefined},{<<"connection">>,[<<"te">>,<<"close">>]}]},{cookies,undefined},{meta,[{media_type,{<<"application">>,<<"json">>,[]}},{charset,undefined}]},{body_state,waiting},{buffer,<<>>},{multipart,undefined},{resp_compress,true},{resp_state,waiting},{resp_headers,[{<<"vary">>,[<<"accept">>,[<<",
">>,<<"accept-encoding">>],[<<",
">>,<<"origin">>]]},{<<"content-type">>,[<<"application">>,<<"/">>,<<"json">>,<<>>]},{<<"vary">>,<<"origin">>}]},{resp_body,<<>>},{onresponse,#Fun<rabbit_cowboy_middleware.onresponse.4>}]},{state,{context,{user,<<"berta">>,[monitoring],[{rabbit_auth_backend_internal,none}]},<<"bertamon">>,undefined}}]

=ERROR REPORT==== 4-Nov-2017::20:06:44 ===
Ranch listener rabbit_web_dispatch_sup_15672 had connection process
started with cowboy_protocol:start_link/4 at <0.22490.41> exit with
reason: [{reason,{timeout,{gen_server,call,[<0.22465.0>,{fetch,#Fun<rabbit_mgmt_db.22.104142656>,[[[{name,<<"CD_9416@xxx11_1509822212199">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.22229.162>},{state,live}],[{name,<<"xtrader.odd.XXX02-16948-vran-13d95112-7e9a-416c-ad62-e6a828aaaff2">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.30940.218>},{state,live}],[{name,<<"WEB_...@web-demo1.qqs_1509710846487">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.16447.157>},{state,live}],[{name,<<"totalSelectionQueue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.797.0>},{state,live}],[{name,<<"profiling.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"profilingDLX">>},{<<"x-message-ttl">>,30000}]}},{pid,<8005.806.0>},{state,live}],[{name,<<"xxx.mailservice.emails">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"xxx.mailservice.dead">>}]}},{pid,<8005.824.0>},{state,live}],[{name,<<"wep-queue-web-demo1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.21483.122>},{arguments,{struct,[]}},{pid,<8005.13392.122>},{state,live}],[{name,<<"CD_9300@XXX09_1509802529729">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.10350.217>},{state,live}],[{name,<<"f1be5e71-7e5d-48fa-9b11-f2f2ee56efeb">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.11331.15>},{arguments,{struct,[{<<"x-message-ttl">>,120000}]}},{pid,<8005.7360.88>},{state,live}],[{name,<<"xxx.ticket.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.563.0>},{state,live}],[{name,<<"xtrader.odd.XXX04-25552-vran2-5cbeeeb4-97f8-4c2c-a8ca-3f03edd8c0cb">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.13835.230>},{state,live}],[{name,<<"queue-webmaster2.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.3459.161>},{arguments,{struct,[]}},{pid,<8005.9895.161>},{state,live}],[{name,<<"xxx.alfa.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.602.0>},{state,live}],[{name,<<"profiling.data.queue-dead">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.608.0>},{state,live}],[{name,<<"CD_30772@xxx06_1509782636659">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.109.144>},{state,live}],[{name,<<"scccQueue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"reports.exchange.dead">>}]}},{pid,<8005.617.0>},{state,live}],[{name,<<"xtrader.changes.XXX10-8428-vran3-8a0dd19b-128d-425a-acb6-6c175ff20719">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.5060.233>},{state,live}],[{name,<<"CD_13780@XXX07_1509813676685">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.27201.158>},{state,live}],[{name,<<"replication.live.AAA">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.659.0>},{state,live}],[{name,<<"xtrader.abr.XXX06-39636-vran3-68310b63-dbb0-4c1d-9fa8-2909e004175f">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.18348.235>},{state,live}],[{name,<<"xxx.catch.cache....@catchservice3.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.22100.0>},{state,live}],[{name,<<"I_54_1509676270819">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.13056.114>},{state,live}],[{name,<<"CD_31552@XXX13_1509802900141">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.24859.217>},{state,live}],[{name,<<"I_91_1509676270822">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000,'...'}]}},{pid,<8005.14514.114>},{state,live}],[{name,<<"replication.archive.AAA">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.731.0>},{state,live}],[{name,<<"xtrader.changes.XXX02-20496-vbondarenko-17624092-6dbe-40f9-a77f-8014ae3af91d">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8004.6393.212>},{state,live}],[{name,<<"WEB_...@webmaster1.qqs_1509676447716">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8004.6092.138>},{state,live}],[{name,<<"CD_37600@xxx11_1509802566857">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8005.5668.153>},{state,live}],[{name,<<"productSelfExcludeMsgQueue">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.14356.114>},{state,live}],[{name,<<"WEB_...@web-demo1.qqs_1509710846486">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'},'...']}},{pid,<8004.17018.157>},{state,live}],[{name,<<"CD_4696@xxx06_1509781996333">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8004.18132.201>},{state,live}],[{name,<<"I_23_1509676270836">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8005.16207.114>},{state,live}],[{name,<<"xxxCancelQueue.Dead">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.800.0>},{state,live}],[{name,<<"xtrader.odd.XXX10-8428-vran3-2a04cf61-04f7-4fe1-86e9-44a745777660">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8004.22724.232>},{state,live}],[{name,<<"I_82_1509676270843">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...'],'...'}},{pid,<8005.21228.132>},{state,live}],[{name,<<"WEB_...@webmaster2.qqs_1509818680697">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,'...'}},{pid,<8004.18528.230>},{state,live}],[{name,<<"xtrader.changes.XXX02-16948-vran-1f80a629-4d27-47d1-816a-28d4b5ab0bf1">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{'...'}},{pid,<8004.2245.219>},{state,live}],[{name,<<"event-actions-web-web-demo1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8004.17194.157>},{arguments,{'...'}},{pid,<8004.17393.157>},{state,live}],[{name,<<"event-actions-web-webmaster1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.16656.114>},{arguments,{'...'}},{pid,<8005.16644.114>},{state,live,'...'}],[{name,<<"CD_7624@PB-PAY-PC04_1509716351755">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{'...'}},{pid,<8005.23935.124>,'...'},{state,'...'}],[{name,<<"WEB_...@webmaster2.qqs_1509818680698">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{'...'},'...'},{pid,'...'},{'...'}],[{name,<<"xtrader.changes.XXX06-39636-vran3-b03733e9-c61c-40f3-bb98-5d9a32beb487">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none,'...'},{arguments,'...'},{'...'},{'...'}],[{name,<<"CD_38108@XXX07_1509800742732">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false,'...'},{owner_pid,'...'},{'...'},{'...'},{'...'}],[{name,<<"wep-work-queue-webmaster1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true,'...'},{exclusive,'...'},{'...'},{'...'},{'...'},{'...'}],[{name,<<"CD_22304@xxx06_1509814353364">>},{vhost,<<"/">>},{durable,true,'...'},{auto_delete,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,<<"xxxPlaceQueue.Dead">>},{vhost,<<"/">>,'...'},{durable,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,<<"xxx.tmp.queue">>,'...'},{vhost,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},'...'],[{'...'},'...'],['...'],['...'],['...'],['...'],['...'],'...']]},60000]}}},{mfa,{rabbit_mgmt_wm_queues,to_json,2}},{stacktrace,[{gen_server,call,3,[{file,"gen_server.erl"},{line,212}]},{rabbit_mgmt_db,submit_cached,4,[{file,"src/rabbit_mgmt_db.erl"},{line,712}]},{rabbit_mgmt_util,augment,2,[{file,"src/rabbit_mgmt_util.erl"},{line,392}]},{rabbit_mgmt_util,run_augmentation,2,[{file,"src/rabbit_mgmt_util.erl"},{line,369}]},{rabbit_mgmt_util,augment_resources0,6,[{file,"src/rabbit_mgmt_util.erl"},{line,358}]},{rabbit_mgmt_util,with_valid_pagination,3,[{file,"src/rabbit_mgmt_util.erl"},{line,282}]},{rabbit_mgmt_wm_queues,to_json,2,[{file,"src/rabbit_mgmt_wm_queues.erl"},{line,54}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,976}]}]},{req,[{socket,#Port<0.1504230>},{transport,ranch_tcp},{connection,keepalive},{pid,<0.22490.41>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{172,16,41,44},54913}},{host,<<"rabbitmq.qqs">>},{host_info,undefined},{port,15672},{path,<<"/api/queues">>},{path_info,undefined},{qs,<<>>},{qs_vals,[]},{bindings,[]},{headers,[{<<"authorization">>,<<"Basic
xxx==">>},{<<"user-agent">>,<<"curl/7.35.0">>},{<<"host">>,<<"rabbitmq.qqs:15672">>},{<<"accept">>,<<"*/*">>},{<<"x-forwarded-for">>,<<"192.168.41.95">>}]},{p_headers,[{<<"if-modified-since">>,undefined},{<<"if-none-match">>,undefined},{<<"if-unmodified-since">>,undefined},{<<"if-match">>,undefined},{<<"accept">>,[{{<<"*">>,<<"*">>,[]},1000,[]}]}]},{cookies,undefined},{meta,[{media_type,{<<"application">>,<<"json">>,[]}},{charset,undefined}]},{body_state,waiting},{buffer,<<>>},{multipart,undefined},{resp_compress,true},{resp_state,waiting},{resp_headers,[{<<"vary">>,[<<"accept">>,[<<",
">>,<<"accept-encoding">>],[<<",
">>,<<"origin">>]]},{<<"content-type">>,[<<"application">>,<<"/">>,<<"json">>,<<>>]},{<<"vary">>,<<"origin">>}]},{resp_body,<<>>},{onresponse,#Fun<rabbit_cowboy_middleware.onresponse.4>}]},{state,{context,{user,<<"berta">>,[monitoring],[{rabbit_auth_backend_internal,none}]},<<"bertamon">>,undefined}}]

=ERROR REPORT==== 4-Nov-2017::20:06:58 ===
** Node 'rabbit@r630-1' not responding **

** Removing (timedout) connection **

=INFO REPORT==== 4-Nov-2017::20:06:58 ===
rabbit on node 'rabbit@r630-1' down

=WARNING REPORT==== 4-Nov-2017::20:06:58 ===
Management delegate query returned errors:
[{<8005.21210.0>,{exit,{nodedown,'rabbit@r630-1'},[]}}]
=INFO REPORT==== 4-Nov-2017::20:07:01 ===
node 'rabbit@r630-1' down: net_tick_timeout

=ERROR REPORT==== 4-Nov-2017::20:07:11 ===
Ranch listener rabbit_web_dispatch_sup_15672 had connection process
started with cowboy_protocol:start_link/4 at <0.22586.41> exit with
reason: [{reason,{timeout,{gen_server,call,[<0.22522.0>,{fetch,#Fun<rabbit_mgmt_db.21.104142656>,[]},60000]}}},{mfa,{rabbit_mgmt_wm_connections,to_json,2}},{stacktrace,[{gen_server,call,3,[{file,"gen_server.erl"},{line,212}]},{rabbit_mgmt_db,submit_cached,2,[{file,"src/rabbit_mgmt_db.erl"},{line,707}]},{rabbit_mgmt_wm_connections,augmented,2,[{file,"src/rabbit_mgmt_wm_connections.erl"},{line,55}]},{rabbit_mgmt_wm_connections,to_json,2,[{file,"src/rabbit_mgmt_wm_connections.erl"},{line,43}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,976}]},{cowboy_rest,set_resp_body,2,[{file,"src/cowboy_rest.erl"},{line,858}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]},{req,[{socket,#Port<0.1504353>},{transport,ranch_tcp},{connection,keepalive},{pid,<0.22586.41>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{172,16,41,44},62387}},{host,<<"rabbitmq.qqs">>},{host_info,undefined},{port,15672},{path,<<"/api/connections">>},{path_info,undefined},{qs,<<>>},{qs_vals,[]},{bindings,[]},{headers,[{<<"authorization">>,<<"Basic
emFiYml4OnphYmJpeG1vbg==">>},{<<"user-agent">>,<<"curl/7.35.0">>},{<<"host">>,<<"rabbitmq.qqs:15672">>},{<<"accept">>,<<"*/*">>},{<<"x-forwarded-for">>,<<"192.168.41.95">>}]},{p_headers,[{<<"if-modified-since">>,undefined},{<<"if-none-match">>,undefined},{<<"if-unmodified-since">>,undefined},{<<"if-match">>,undefined},{<<"accept">>,[{{<<"*">>,<<"*">>,[]},1000,[]}]}]},{cookies,undefined},{meta,[{media_type,{<<"application">>,<<"json">>,[]}},{charset,undefined}]},{body_state,waiting},{buffer,<<>>},{multipart,undefined},{resp_compress,true},{resp_state,waiting},{resp_headers,[{<<"vary">>,[<<"accept">>,[<<",
">>,<<"accept-encoding">>],[<<",
">>,<<"origin">>]]},{<<"content-type">>,[<<"application">>,<<"/">>,<<"json">>,<<>>]},{<<"vary">>,<<"origin">>}]},{resp_body,<<>>},{onresponse,#Fun<rabbit_cowboy_middleware.onresponse.4>}]},{state,{context,{user,<<"berta">>,[monitoring],[{rabbit_auth_backend_internal,none}]},<<"bertamon">>,undefined}}]

=WARNING REPORT==== 4-Nov-2017::20:07:12 ===
Management delegate query returned errors:
[{<8005.21210.0>,{exit,{nodedown,'rabbit@r630-1'},[]}}]
=ERROR REPORT==== 4-Nov-2017::20:11:58 ===
** Node 'rabbit@r630-2' not responding **

** Removing (timedout) connection **

=INFO REPORT==== 4-Nov-2017::20:11:58 ===
rabbit on node 'rabbit@r630-2' down

=WARNING REPORT==== 4-Nov-2017::20:11:58 ===
Management delegate query returned errors:
[{<8004.21253.0>,{exit,{nodedown,'rabbit@r630-2'},[]}}]
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
node 'rabbit@r630-2' down: net_tick_timeout

=WARNING REPORT==== 4-Nov-2017::20:11:59 ===
Cluster minority/secondary status detected - awaiting recovery

=INFO REPORT==== 4-Nov-2017::20:11:59 ===
RabbitMQ is asked to stop...

=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping RabbitMQ applications and their dependencies in the following order:
rabbitmq_shovel_management
rabbitmq_management
rabbitmq_web_dispatch
cowboy
cowlib
rabbitmq_management_agent
rabbitmq_auth_backend_ldap
rabbitmq_shovel
rabbitmq_stomp
rabbit
mnesia
amqp_client
rabbit_common
os_mon

=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbitmq_shovel_management'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbitmq_management'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbitmq_web_dispatch'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'cowboy'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'cowlib'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbitmq_management_agent'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbitmq_auth_backend_ldap'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbitmq_shovel'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbitmq_stomp'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
stopped STOMP TCP Listener on [::]:61613

=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbit'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
stopped TCP Listener on [::]:5672

=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping message store for directory
'/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-quorum/msg_store_persistent'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Message store for directory
'/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-quorum/msg_store_persistent'
is stopped
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping message store for directory
'/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-quorum/msg_store_transient'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Message store for directory
'/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-quorum/msg_store_transient'
is stopped
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'mnesia'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'amqp_client'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'rabbit_common'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Stopping application 'os_mon'
=INFO REPORT==== 4-Nov-2017::20:11:59 ===
Successfully stopped RabbitMQ and its dependencies




You asked me for boot log also. It's from r630-2:

=INFO REPORT==== 4-Nov-2017::20:32:12 ===
Starting RabbitMQ 3.6.12 on Erlang 18.1
Copyright (C) 2007-2017 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 4-Nov-2017::20:32:12 ===
node : rabbit@r630-2
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : R2aPVpWWXldfHkoS5sjbiA==
log : /var/log/rabbitmq/rab...@r630-2.log
sasl log : /var/log/rabbitmq/rab...@r630-2-sasl.log
database dir : /var/lib/rabbitmq/mnesia/rabbit@r630-2

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Memory high watermark set to 328846 MiB (344820236902 bytes) of 386877
MiB (405670866944 bytes) total

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Enabling free disk space monitoring

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Disk free limit set to 40567MB

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Limiting to approx 10140 file handles (9124 sockets)

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
FHC read buffering: OFF
FHC write buffering: ON

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Starting rabbit_node_monitor

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
Management plugin: using rates mode 'basic'

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 4-Nov-2017::20:32:14 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 4-Nov-2017::20:32:14 ===
msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 4-Nov-2017::20:32:44 ===
rabbit on node 'rabbit@r630-1' up

=INFO REPORT==== 4-Nov-2017::20:32:45 ===
rabbit on node 'rabbit@rabbitmq-quorum' up

=WARNING REPORT==== 4-Nov-2017::20:32:49 ===
Transport option binary unknown or invalid.

=WARNING REPORT==== 4-Nov-2017::20:32:49 ===
Transport option {packet,raw} unknown or invalid.

=WARNING REPORT==== 4-Nov-2017::20:32:49 ===
Transport option {reuseaddr,true} unknown or invalid.

=INFO REPORT==== 4-Nov-2017::20:32:49 ===
started TCP Listener on [::]:5672

=INFO REPORT==== 4-Nov-2017::20:32:49 ===
rabbit on node 'rabbit@rabbitmq-quorum' up

=INFO REPORT==== 4-Nov-2017::20:32:49 ===
rabbit_stomp: default user 'guest' enabled

=INFO REPORT==== 4-Nov-2017::20:32:49 ===
started STOMP TCP Listener on [::]:61613

=INFO REPORT==== 4-Nov-2017::20:32:50 ===
rabbit on node 'rabbit@r630-1' up

=INFO REPORT==== 4-Nov-2017::20:32:50 ===
Management plugin started. Port: 15672

=INFO REPORT==== 4-Nov-2017::20:32:50 ===
Statistics database started.

=INFO REPORT==== 4-Nov-2017::20:32:50 ===
Server startup complete; 10 plugins started.
* rabbitmq_shovel_management
* rabbitmq_management
* rabbitmq_management_agent
* rabbitmq_stomp
* rabbitmq_auth_backend_ldap
* rabbitmq_web_dispatch
* rabbitmq_shovel
* cowboy
* amqp_client
* cowlib

=WARNING REPORT==== 4-Nov-2017::20:36:21 ===
STOMP detected missed client heartbeat(s) on connection
192.168.41.44:22164 -> 192.168.10.71:61613, closing it

=WARNING REPORT==== 4-Nov-2017::20:36:46 ===
STOMP detected missed client heartbeat(s) on connection
192.168.41.44:29520 -> 192.168.10.71:61613, closing it

=WARNING REPORT==== 4-Nov-2017::20:37:29 ===
STOMP detected missed client heartbeat(s) on connection
192.168.41.44:31760 -> 192.168.10.71:61613, closing it

...many stomp warnings...


And config, just in case:

[
{kernel, [
{inet_default_connect_options, [{nodelay, true}]},
{inet_default_listen_options, [{nodelay, true}]}
]},

{rabbit, [
{auth_backends, [rabbit_auth_backend_internal,
rabbit_auth_backend_ldap]},
{log_levels, [{channel, warning},{connection,
warning},{mirroring, warning}]},
{vm_memory_high_watermark, 0.85},
{vm_memory_high_watermark_paging_ratio, 0.90},
{hipe_compile, false},
{disk_free_limit, {mem_relative, 0.1}},
{tcp_listen_options, [
binary,
{packet, raw},
{reuseaddr, true},
{backlog, 256},
{nodelay, true}
]},
{cluster_partition_handling, pause_minority}
]},

{rabbitmq_auth_backend_ldap, [
{servers, ["ldap.qqs.local"]},
{user_dn_pattern, "uid=${username},ou=People,dc=corp,dc=com"},
{use_ssl, false},
{port, 389},
{log, false},
{vhost_access_query,
{in_group, "cn=RabbitMQ,ou=Group,dc=corp,dc=com"}
},
{resource_access_query,
{for, [
{permission, configure, {in_group,
"cn=RabbitMQ_Admin,ou=Group,dc=corp,dc=com"}},
{permission, write, {in_group,
"cn=RabbitMQ_Admin,ou=Group,dc=corp,dc=com"}},
{permission, read, {in_group,
"cn=RabbitMQ_Admin,ou=Group,dc=corp,dc=com"}}
]}
},
{tag_queries, [
{administrator, {in_group,
"cn=RabbitMQ_Admin,ou=Group,dc=corp,dc=com"}},
{monitoring, {in_group, "cn=RabbitMQ,
ou=Group,dc=corp,dc=com"}},
{management, {in_group, "cn=RabbitMQ,
ou=Group,dc=corp,dc=com"}}
]}
]}
].


After cluster restart all works well.


ps: Unfortunately I can't send you mnesia dump because of company restrictions.

[Michael Klishin]

> Error: unable to connect to node 'rabbit@r630-1': nodedown

doesn't tell you much (the node may actually be down or not, see http://www.rabbitmq.com/cli.html).

Other messages come down to management plugin trying to contact other nodes

and not receiving a response. Which strongly suggests the nodes were not upgraded as 3.6.7 release notes

require (full cluster shutdown).

There is nothing else that stands out.

reason: [{reason,{timeout,{gen_server,call,[<0.22465.0>,{fetch,#Fun<rabbit_mgmt_db.22.104142656>,[[[{name,<<"CD_9416@xxx11_1509822212199">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.22229.162>},{state,live}],[{name,<<"xtrader.odd.XXX02-16948-vran-13d95112-7e9a-416c-ad62-e6a828aaaff2">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.30940.218>},{state,live}],[{name,<<"WEB_27...@web-demo1.qqs_1509710846487">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.16447.157>},{state,live}],[{name,<<"totalSelectionQueue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.797.0>},{state,live}],[{name,<<"profiling.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"profilingDLX">>},{<<"x-message-ttl">>,30000}]}},{pid,<8005.806.0>},{state,live}],[{name,<<"xxx.mailservice.emails">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"xxx.mailservice.dead">>}]}},{pid,<8005.824.0>},{state,live}],[{name,<<"wep-queue-web-demo1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.21483.122>},{arguments,{struct,[]}},{pid,<8005.13392.122>},{state,live}],[{name,<<"CD_9300@XXX09_1509802529729">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.10350.217>},{state,live}],[{name,<<"f1be5e71-7e5d-48fa-9b11-f2f2ee56efeb">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.11331.15>},{arguments,{struct,[{<<"x-message-ttl">>,120000}]}},{pid,<8005.7360.88>},{state,live}],[{name,<<"xxx.ticket.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.563.0>},{state,live}],[{name,<<"xtrader.odd.XXX04-25552-vran2-5cbeeeb4-97f8-4c2c-a8ca-3f03edd8c0cb">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.13835.230>},{state,live}],[{name,<<"queue-webmaster2.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.3459.161>},{arguments,{struct,[]}},{pid,<8005.9895.161>},{state,live}],[{name,<<"xxx.alfa.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.602.0>},{state,live}],[{name,<<"profiling.data.queue-dead">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.608.0>},{state,live}],[{name,<<"CD_30772@xxx06_1509782636659">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.109.144>},{state,live}],[{name,<<"scccQueue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"reports.exchange.dead">>}]}},{pid,<8005.617.0>},{state,live}],[{name,<<"xtrader.changes.XXX10-8428-vran3-8a0dd19b-128d-425a-acb6-6c175ff20719">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.5060.233>},{state,live}],[{name,<<"CD_13780@XXX07_1509813676685">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.27201.158>},{state,live}],[{name,<<"replication.live.AAA">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.659.0>},{state,live}],[{name,<<"xxx.catch.cache.reset.queue.17997@catchservice3.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.22100.0>},{state,live}],[{name,<<"I_54_1509676270819">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.13056.114>},{state,live}],[{name,<<"CD_31552@XXX13_1509802900141">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.24859.217>},{state,live}],[{name,<<"I_91_1509676270822">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.14514.114>},{state,live}],[{name,<<"replication.archive.AAA">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.731.0>},{state,live}],[{name,<<"xtrader.changes.XXX02-20496-vbondarenko-17624092-6dbe-40f9-a77f-8014ae3af91d">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,'...'}]}},{pid,<8004.6393.212>},{state,live}],[{name,<<"WEB_13281@webmaster1.qqs_1509676447716">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8004.6092.138>},{state,live}],[{name,<<"CD_37600@xxx11_1509802566857">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8005.5668.153>},{state,live}],[{name,<<"productSelfExcludeMsgQueue">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.14356.114>},{state,live}],[{name,<<"WEB_27298@web-demo1.qqs_1509710846486">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8004.17018.157>},{state,live}],[{name,<<"CD_4696@xxx06_1509781996333">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'},'...']}},{pid,<8004.18132.201>},{state,live}],[{name,<<"I_23_1509676270836">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8005.16207.114>},{state,live}],[{name,<<"xxxCancelQueue.Dead">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.800.0>},{state,live}],[{name,<<"xtrader.odd.XXX10-8428-vran3-2a04cf61-04f7-4fe1-86e9-44a745777660">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8004.22724.232>},{state,live}],[{name,<<"I_82_1509676270843">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8005.21228.132>},{state,live}],[{name,<<"WEB_21849@webmaster2.qqs_1509818680697">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...'],'...'}},{pid,<8004.18528.230>},{state,live}],[{name,<<"xtrader.changes.XXX02-16948-vran-1f80a629-4d27-47d1-816a-28d4b5ab0bf1">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,'...'}},{pid,<8004.2245.219>},{state,live}],[{name,<<"event-actions-web-web-demo1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8004.17194.157>},{arguments,{'...'}},{pid,<8004.17393.157>},{state,live}],[{name,<<"event-actions-web-webmaster1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.16656.114>},{arguments,{'...'}},{pid,<8005.16644.114>},{state,live}],[{name,<<"xtrader.abr.XXX08-7712-vran-c055635e-9d09-43ba-8d82-36d81d261d42">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{'...'}},{pid,<8005.8735.163>},{state,live,'...'}],[{name,<<"CD_7624@PB-PAY-PC04_1509716351755">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{'...'}},{pid,<8005.23935.124>,'...'},{state,'...'}],[{name,<<"WEB_21849@webmaster2.qqs_1509818680698">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{'...'},'...'},{pid,'...'},{'...'}],[{name,<<"xtrader.changes.XXX06-39636-vran3-b03733e9-c61c-40f3-bb98-5d9a32beb487">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none,'...'},{arguments,'...'},{'...'},{'...'}],[{name,<<"CD_38108@XXX07_1509800742732">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false,'...'},{owner_pid,'...'},{'...'},{'...'},{'...'}],[{name,<<"wep-work-queue-webmaster1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true,'...'},{exclusive,'...'},{'...'},{'...'},{'...'},{'...'}],[{name,<<"CD_22304@xxx06_1509814353364">>},{vhost,<<"/">>},{durable,true,'...'},{auto_delete,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,<<"xxxPlaceQueue.Dead">>},{vhost,<<"/">>,'...'},{durable,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,<<"xxx.tmp.queue">>,'...'},{vhost,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},'...'],[{'...'},'...'],['...'],['...'],['...'],['...'],['...'],'...']]},60000]}}},{mfa,{rabbit_mgmt_wm_queues,to_json,2}},{stacktrace,[{gen_server,call,3,[{file,"gen_server.erl"},{line,212}]},{rabbit_mgmt_db,submit_cached,4,[{file,"src/rabbit_mgmt_db.erl"},{line,712}]},{rabbit_mgmt_util,augment,2,[{file,"src/rabbit_mgmt_util.erl"},{line,392}]},{rabbit_mgmt_util,run_augmentation,2,[{file,"src/rabbit_mgmt_util.erl"},{line,369}]},{rabbit_mgmt_util,augment_resources0,6,[{file,"src/rabbit_mgmt_util.erl"},{line,358}]},{rabbit_mgmt_util,with_valid_pagination,3,[{file,"src/rabbit_mgmt_util.erl"},{line,282}]},{rabbit_mgmt_wm_queues,to_json,2,[{file,"src/rabbit_mgmt_wm_queues.erl"},{line,54}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,976}]}]},{req,[{socket,#Port<0.1504195>},{transport,ranch_tcp},{connection,close},{pid,<0.22469.41>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{172,16,41,44},53560}},{host,<<"rabbitmq.qqs">>},{host_info,undefined},{port,15672},{path,<<"/api/queues/">>},{path_info,undefined},{qs,<<>>},{qs_vals,[]},{bindings,[]},{headers,[{<<"te">>,<<"deflate,gzip;q=0.3">>},{<<"connection">>,<<"TE,

close">>},{<<"authorization">>,<<"Basic
emFiYml4OnphYmJpeG1vbg==">>},{<<"host">>,<<"rabbitmq.qqs:15672">>},{<<"user-agent">>,<<"libwww-perl/6.05">>},{<<"x-forwarded-for">>,<<"192.168.41.95">>}]},{p_headers,[{<<"if-modified-since">>,undefined},{<<"if-none-match">>,undefined},{<<"if-unmodified-since">>,undefined},{<<"if-match">>,undefined},{<<"accept">>,undefined},{<<"connection">>,[<<"te">>,<<"close">>]}]},{cookies,undefined},{meta,[{media_type,{<<"application">>,<<"json">>,[]}},{charset,undefined}]},{body_state,waiting},{buffer,<<>>},{multipart,undefined},{resp_compress,true},{resp_state,waiting},{resp_headers,[{<<"vary">>,[<<"accept">>,[<<",
">>,<<"accept-encoding">>],[<<",
">>,<<"origin">>]]},{<<"content-type">>,[<<"application">>,<<"/">>,<<"json">>,<<>>]},{<<"vary">>,<<"origin">>}]},{resp_body,<<>>},{onresponse,#Fun<rabbit_cowboy_middleware.onresponse.4>}]},{state,{context,{user,<<"berta">>,[monitoring],[{rabbit_auth_backend_internal,none}]},<<"bertamon">>,undefined}}]

=ERROR REPORT==== 4-Nov-2017::20:06:44 ===
Ranch listener rabbit_web_dispatch_sup_15672 had connection process
started with cowboy_protocol:start_link/4 at <0.22490.41> exit with

reason: [{reason,{timeout,{gen_server,call,[<0.22465.0>,{fetch,#Fun<rabbit_mgmt_db.22.104142656>,[[[{name,<<"CD_9416@xxx11_1509822212199">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.22229.162>},{state,live}],[{name,<<"xtrader.odd.XXX02-16948-vran-13d95112-7e9a-416c-ad62-e6a828aaaff2">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.30940.218>},{state,live}],[{name,<<"WEB_27...@web-demo1.qqs_1509710846487">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.16447.157>},{state,live}],[{name,<<"totalSelectionQueue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.797.0>},{state,live}],[{name,<<"profiling.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"profilingDLX">>},{<<"x-message-ttl">>,30000}]}},{pid,<8005.806.0>},{state,live}],[{name,<<"xxx.mailservice.emails">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"xxx.mailservice.dead">>}]}},{pid,<8005.824.0>},{state,live}],[{name,<<"wep-queue-web-demo1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.21483.122>},{arguments,{struct,[]}},{pid,<8005.13392.122>},{state,live}],[{name,<<"CD_9300@XXX09_1509802529729">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.10350.217>},{state,live}],[{name,<<"f1be5e71-7e5d-48fa-9b11-f2f2ee56efeb">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.11331.15>},{arguments,{struct,[{<<"x-message-ttl">>,120000}]}},{pid,<8005.7360.88>},{state,live}],[{name,<<"xxx.ticket.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.563.0>},{state,live}],[{name,<<"xtrader.odd.XXX04-25552-vran2-5cbeeeb4-97f8-4c2c-a8ca-3f03edd8c0cb">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.13835.230>},{state,live}],[{name,<<"queue-webmaster2.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.3459.161>},{arguments,{struct,[]}},{pid,<8005.9895.161>},{state,live}],[{name,<<"xxx.alfa.queue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.602.0>},{state,live}],[{name,<<"profiling.data.queue-dead">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.608.0>},{state,live}],[{name,<<"CD_30772@xxx06_1509782636659">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.109.144>},{state,live}],[{name,<<"scccQueue">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-dead-letter-exchange">>,<<"reports.exchange.dead">>}]}},{pid,<8005.617.0>},{state,live}],[{name,<<"xtrader.changes.XXX10-8428-vran3-8a0dd19b-128d-425a-acb6-6c175ff20719">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.5060.233>},{state,live}],[{name,<<"CD_13780@XXX07_1509813676685">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.27201.158>},{state,live}],[{name,<<"replication.live.AAA">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.659.0>},{state,live}],[{name,<<"xtrader.abr.XXX06-39636-vran3-68310b63-dbb0-4c1d-9fa8-2909e004175f">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-max-length">>,2000}]}},{pid,<8004.18348.235>},{state,live}],[{name,<<"xxx.catch.cache.reset.queue.17997@catchservice3.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.22100.0>},{state,live}],[{name,<<"I_54_1509676270819">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8005.13056.114>},{state,live}],[{name,<<"CD_31552@XXX13_1509802900141">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000}]}},{pid,<8004.24859.217>},{state,live}],[{name,<<"I_91_1509676270822">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{<<"x-expires">>,300000,'...'}]}},{pid,<8005.14514.114>},{state,live}],[{name,<<"replication.archive.AAA">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.731.0>},{state,live}],[{name,<<"xtrader.changes.XXX02-20496-vbondarenko-17624092-6dbe-40f9-a77f-8014ae3af91d">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8004.6393.212>},{state,live}],[{name,<<"WEB_13...@webmaster1.qqs_1509676447716">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8004.6092.138>},{state,live}],[{name,<<"CD_37600@xxx11_1509802566857">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'}]}},{pid,<8005.5668.153>},{state,live}],[{name,<<"productSelfExcludeMsgQueue">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.14356.114>},{state,live}],[{name,<<"WEB_27298@web-demo1.qqs_1509710846486">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[{'...'},'...']}},{pid,<8004.17018.157>},{state,live}],[{name,<<"CD_4696@xxx06_1509781996333">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8004.18132.201>},{state,live}],[{name,<<"I_23_1509676270836">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8005.16207.114>},{state,live}],[{name,<<"xxxCancelQueue.Dead">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,[]}},{pid,<8005.800.0>},{state,live}],[{name,<<"xtrader.odd.XXX10-8428-vran3-2a04cf61-04f7-4fe1-86e9-44a745777660">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{struct,['...']}},{pid,<8004.22724.232>},{state,live}],[{name,<<"I_82_1509676270843">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,['...'],'...'}},{pid,<8005.21228.132>},{state,live}],[{name,<<"WEB_21...@webmaster2.qqs_1509818680697">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{struct,'...'}},{pid,<8004.18528.230>},{state,live}],[{name,<<"xtrader.changes.XXX02-16948-vran-1f80a629-4d27-47d1-816a-28d4b5ab0bf1">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none},{arguments,{'...'}},{pid,<8004.2245.219>},{state,live}],[{name,<<"event-actions-web-web-demo1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8004.17194.157>},{arguments,{'...'}},{pid,<8004.17393.157>},{state,live}],[{name,<<"event-actions-web-webmaster1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,true},{owner_pid,<8005.16656.114>},{arguments,{'...'}},{pid,<8005.16644.114>},{state,live,'...'}],[{name,<<"CD_7624@PB-PAY-PC04_1509716351755">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{'...'}},{pid,<8005.23935.124>,'...'},{state,'...'}],[{name,<<"WEB_21849@webmaster2.qqs_1509818680698">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false},{owner_pid,none},{arguments,{'...'},'...'},{pid,'...'},{'...'}],[{name,<<"xtrader.changes.XXX06-39636-vran3-b03733e9-c61c-40f3-bb98-5d9a32beb487">>},{vhost,<<"/">>},{durable,false},{auto_delete,true},{exclusive,false},{owner_pid,none,'...'},{arguments,'...'},{'...'},{'...'}],[{name,<<"CD_38108@XXX07_1509800742732">>},{vhost,<<"/">>},{durable,true},{auto_delete,false},{exclusive,false,'...'},{owner_pid,'...'},{'...'},{'...'},{'...'}],[{name,<<"wep-work-queue-webmaster1.qqs">>},{vhost,<<"/">>},{durable,false},{auto_delete,true,'...'},{exclusive,'...'},{'...'},{'...'},{'...'},{'...'}],[{name,<<"CD_22304@xxx06_1509814353364">>},{vhost,<<"/">>},{durable,true,'...'},{auto_delete,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,<<"xxxPlaceQueue.Dead">>},{vhost,<<"/">>,'...'},{durable,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,<<"xxx.tmp.queue">>,'...'},{vhost,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{name,'...'},{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},{'...'},'...'],[{'...'},{'...'},'...'],[{'...'},'...'],['...'],['...'],['...'],['...'],['...'],'...']]},60000]}}},{mfa,{rabbit_mgmt_wm_queues,to_json,2}},{stacktrace,[{gen_server,call,3,[{file,"gen_server.erl"},{line,212}]},{rabbit_mgmt_db,submit_cached,4,[{file,"src/rabbit_mgmt_db.erl"},{line,712}]},{rabbit_mgmt_util,augment,2,[{file,"src/rabbit_mgmt_util.erl"},{line,392}]},{rabbit_mgmt_util,run_augmentation,2,[{file,"src/rabbit_mgmt_util.erl"},{line,369}]},{rabbit_mgmt_util,augment_resources0,6,[{file,"src/rabbit_mgmt_util.erl"},{line,358}]},{rabbit_mgmt_util,with_valid_pagination,3,[{file,"src/rabbit_mgmt_util.erl"},{line,282}]},{rabbit_mgmt_wm_queues,to_json,2,[{file,"src/rabbit_mgmt_wm_queues.erl"},{line,54}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,976}]}]},{req,[{socket,#Port<0.1504230>},{transport,ranch_tcp},{connection,keepalive},{pid,<0.22490.41>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{172,16,41,44},54913}},{host,<<"rabbitmq.qqs">>},{host_info,undefined},{port,15672},{path,<<"/api/queues">>},{path_info,undefined},{qs,<<>>},{qs_vals,[]},{bindings,[]},{headers,[{<<"authorization">>,<<"Basic

log            : /var/log/rabbitmq/rabbit@r630-2.log
sasl log       : /var/log/rabbitmq/rabbit@r630-2-sasl.log

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fabiano Modos]

Hello,

I am facing similar issue with RabbitMQ 3.6.11 on a Linux Centos 6 (Linux version 2.6.32-696.10.1.el6.x86_64 (mock...@x86-029.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC) ))

There was 2 cluster node working great for around 3 months, all the sudden it started crashing with the same symptoms as described in this post. So we removed the cluster and just kept one single node active, but the issue is still happening.

Overall analysis:

-There is no relevant error/warning info in the rabbitmq logs, it just stop printing logs.

-The rabbitmq process show as running in the htop, but doesn't respond to any http,amqp, rabbitmctl, service stop command... need to kill and start it again

-CPU/Memory/Disk are OK.

-I did strace analysis into the Running and Locked RabbitMQ process, it seems to have a Deadlock into the select call

Strace Running RabbitMQ

7506  futex(0x8e8644, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished ...>

7505  read(8,  <unfinished ...>

7473  select(0, NULL, NULL, NULL, NULL <unfinished ...>

7575  <... timerfd_settime resumed> )   = 0

7573  <... restart_syscall resumed> )   = -1 ETIMEDOUT (Connection timed out)

 

Strace Locked RabbitMQ

24440 1511031147.134693 futex(0x8e8644, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished ...>

24439 1511031147.134706 read(8,  <unfinished ...>

24407 1511031147.134723 select(0, NULL, NULL, NULL, NULL <unfinished ...>

24475 1511031147.134784 futex(0x7fbd2f6c0a10, FUTEX_WAIT_PRIVATE, 4294967295, NULL <unfinished ...>

24508 1511031147.134809 futex(0x7fbd2f6c1250, FUTEX_WAIT_PRIVATE, 4294967295, NULL <unfinished ...>

-epmd process is always locked into

24123 select(7, [3 4], NULL, NULL, {3, 819114}) = 0 (Timeout)

24123 select(7, [3 4], NULL, NULL, {5, 0} <detached ...>

-There are a lot of Bad File Descriptor error in the Running RabbitMQ process, I don't know if this is a normal behavior     

11078 close(15515 <unfinished ...>

11080 close(14770 <unfinished ...>

7572  clock_gettime(CLOCK_MONOTONIC_RAW,  <unfinished ...>

11079 <... close resumed> )             = -1 EBADF (Bad file descriptor)

11078 <... close resumed> )             = -1 EBADF (Bad file descriptor)

Any help will be really appreciated, we are facing this issue for 3 days already into a really critical system. We are even considering to move to another MQ solution.

Please let me know if that are any other information about this issue that you would like me to share here. 

ps: We are upgrading to version 3.6.14, hope that solves the problem :)

Best Regards

Fabiano Modos

[Luke Bakken]

Hi Fabiano -

What version of Erlang are you using? That is an extremely important detail. If it is version 18 I recommend upgrading to the latest version of 19 or 20.

Thanks,

Luke

[Dmitry Kurbatov]

After few disasters we upgraded Erlang from 18.1 to 20.1 and stucks
has gone. I suspect reason of this issue is some bug in Erlang VM.

> --
> You received this message because you are subscribed to the Google Groups
> "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.

[Michael Klishin]

Repeated select(2) calls are not necessarily an indication of a deadlock. Some services — such as epmd — first and foremost

read data from the network and do nothing else most of the time.

Erlang 18.x is known to have issues that lead the VM into a state where it does nothing but also doesn't shut down.

That's one of the reasons why RabbitMQ will soon require 19.3.6.3 or later.

> email to rabbitmq-users+unsubscribe@googlegroups.com.
> To post to this group, send email to rabbitmq-users@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fabiano Modos]

I appreciate your quick response.

We were using version 18.1 and now upgrade to version 20.

The service is up already for more than 24 hours, so we are confident this solved our problems.

> email to rabbitmq-users+unsubscribe@googlegroups.com.
> To post to this group, send email to rabbitmq-users@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/hK323-O6-tw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send an email to rabbitmq-users@googlegroups.com.

[Michael Klishin]

Thank you for reporting back.

> email to rabbitmq-users+unsubscribe@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/hK323-O6-tw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send an email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--