rabbitmqctl list_queues gets timeout

[Raul Kaubi]

Hi

Centos7

erlang 21.0.8

rabbitmq 3.7.7

I installed new 2 node cluster.

Opened ports: 4369, 5672, 15672, 25672 between two nodes

Added another node into cluster. All works fine, but I cannot seem to list_queues via rabbitmqctl

On primary node:

[root@dc2-rabbit1 ~]# /usr/sbin/rabbitmqctl list_queues consumers messages message_bytes name
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
0       0       0       ha.pris.sign
0       0       0       ha.KIS2.EtoimikuTeavitused
0       0       0       ha.digitoimik.sign
0       0       0       ha.pris.sign_error
0       0       0       ha.KIS2.Cache.TulevikuSyndmused
.....

.... 

<until the end, so in primary node, it shows me all the queues, and then after 60 seconds, timeout>

Error: operation list_queues on node rabbit@dc2-rabbit1 timed out. Timeout value used: 60.0. Some queue(s) are unresponsive, use list_unresponsive_queues command.

On Slave node:

# /usr/sbin/rabbitmqctl list_queues consumers messages message_bytes name
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
Error: operation list_queues on node rabbit@rabbitnode1 timed out. Timeout value used: 60.0. Some queue(s) are unresponsive, use list_unresponsive_queues command.

<it does not show me any queues>

list_unresponsive_queues also gets timeout.

I also tried deleteing all queues from rabbit, and still this command gets timeout.

I am guessing there is some other port, which is not opened.

Regards
Raul

[Raul Kaubi]

This works fine on both nodes.

rabbitmqadmin list queues

Raul

[Raul Kaubi]

Also, if I stop one node, then on other node this command executes successfully.

But if both nodes are running, then this command will hang.

Raul

[Michael Klishin]

This has been discussed several times in the last few months. `rabbitmqctl list_•` commands contact all cluster members

in parallel to stream results. Should one of them be inaccessible (due to a firewall configuration), it will block.

Data is your friend. Use `rabbitmq-diagnostics maybe_stuck` to see if there's anything relevant that's not making progress [according to stack trace sampling].

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Raul Kaubi]

Hi

According to this, it is normal:

https://groups.google.com/forum/#!topic/rabbitmq-users/oyvREbOPydU

# rabbitmq-diagnostics maybe_stuck

Asking node rabbit@dc1-rabbit1 to detect potentially stuck Erlang processes...

2018-09-17 08:35:57 There are 613 processes.

2018-09-17 08:35:57 Investigated 1 processes this round, 5000ms to go.

2018-09-17 08:35:58 Investigated 1 processes this round, 4500ms to go.

2018-09-17 08:35:58 Investigated 1 processes this round, 4000ms to go.

2018-09-17 08:35:59 Investigated 1 processes this round, 3500ms to go.

2018-09-17 08:35:59 Investigated 1 processes this round, 3000ms to go.

2018-09-17 08:36:00 Investigated 1 processes this round, 2500ms to go.

2018-09-17 08:36:00 Investigated 1 processes this round, 2000ms to go.

2018-09-17 08:36:01 Investigated 1 processes this round, 1500ms to go.

2018-09-17 08:36:01 Investigated 1 processes this round, 1000ms to go.

2018-09-17 08:36:02 Investigated 1 processes this round, 500ms to go.

2018-09-17 08:36:02 Found 1 suspicious processes.

2018-09-17 08:36:02 [{pid,<9747.1.0>},

                     {registered_name,erts_code_purger},

                     {current_stacktrace,

                         [{erts_code_purger,wait_for_request,0,[]}]},

                     {initial_call,{erts_code_purger,start,0}},

                     {message_queue_len,0},

                     {links,[]},

                     {monitors,[]},

                     {monitored_by,[]},

                     {heap_size,233}]

Have I missed some ports, that need to be opened..? (all default ports are used)

I have opened ports: 4369, 5672, 15672, 25672 between two nodes

Raul

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Code purger has been mentioned before on this list. It's a part of Erlang/OTP that sits there

and waits for on disk module files to change so that it can prune their old versions.

It therefore will show up in maybe_stuck output since its stack trace doesn't change 99.99% of the time.

The list of ports used for various purposes [1], network troubleshooting [2] and by far the most useful tools for that (tcpdump and its GUI cousin Wireshark)

are all documented. In this case you want to see traffic on port 25672 from the host on which rabbitmqctl is run to all cluster member nodes.

Again, it's the CLI tool that contacts all cluster nodes, not cluster nodes contact each other on behalf of the client (the way HTTP API requests are handled, for example).

Please take this from here.

1. https://www.rabbitmq.com/networking.html#ports

2. https://www.rabbitmq.com/troubleshooting-networking.html

3. https://www.rabbitmq.com/amqp-wireshark.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Raul Kaubi]

Ok, problem got solved.

It needed 35672-35682 port range also between clusters for cli tools to work.

Raul

[Michael Klishin]

Not for CLI tools in general but for some commands.

Thank you for reporting back to the list. We'll try to emphasize this in the docs.