RabbitMQ is not responding in the Management Browser and timeoute in spring boot

[Carl Adrian Adap]

Hi my rabbitmq usually fine, but few days ago it suddenly become slow. The management tool after logging in doesnt fully load up, and during spring bootup there are timeouts for my project that uses rabbitmq.

I have attach the image of my browser while accessing the management tool. it just get stuck there.

And here is the exception i get when starting my spring application:

ERROR: org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer - Failed to check/redeclare auto-delete queue(s).

java.util.concurrent.TimeoutException

at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:77)

at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:111)

at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37)

at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:367)

at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:293)

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:621)

at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:665)

at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:208)

at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:444)

at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:80)

at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:130)

at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)

at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1035)

at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1028)

at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1004)

at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueProperties(RabbitAdmin.java:254)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.redeclareElementsIfNecessary(SimpleMessageListenerContainer.java:963)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$300(SimpleMessageListenerContainer.java:83)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1081)

at java.lang.Thread.run(Thread.java:745)

Here are the logs of rabbitmq:

=INFO REPORT==== 9-May-2017::21:25:47 ===

Starting RabbitMQ 3.6.9 on Erlang 19.3

Copyright (C) 2007-2016 Pivotal Software, Inc.

Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 9-May-2017::21:25:47 ===

node           : rabbit@localhost

home dir       : /Users/carl.adrian

config file(s) : /usr/local/etc/rabbitmq/rabbitmq.config (not found)

cookie hash    : p9A7M/eY+67/JmrMfzGvGA==

log            : /usr/local/var/log/rabbitmq/rab...@localhost.log

sasl log       : /usr/local/var/log/rabbitmq/rab...@localhost-sasl.log

database dir   : /usr/local/var/lib/rabbitmq/mnesia/rabbit@localhost

=INFO REPORT==== 9-May-2017::21:25:49 ===

Memory limit set to 2606MB of 6515MB total.

=INFO REPORT==== 9-May-2017::21:25:49 ===

Disk free limit set to 50MB

=INFO REPORT==== 9-May-2017::21:25:49 ===

Limiting to approx 156 file handles (138 sockets)

=INFO REPORT==== 9-May-2017::21:25:49 ===

FHC read buffering:  OFF

FHC write buffering: ON

=INFO REPORT==== 9-May-2017::21:25:49 ===

Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 9-May-2017::21:25:49 ===

Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 9-May-2017::21:25:49 ===

Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 9-May-2017::21:25:49 ===

Starting rabbit_node_monitor

=INFO REPORT==== 9-May-2017::21:25:49 ===

Management plugin: using rates mode 'basic'

=INFO REPORT==== 9-May-2017::21:25:49 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 9-May-2017::21:25:49 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 9-May-2017::21:25:49 ===

msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 9-May-2017::21:25:54 ===

started TCP Listener on 127.0.0.1:5672

=INFO REPORT==== 9-May-2017::21:25:54 ===

rabbit_stomp: default user 'guest' enabled

=INFO REPORT==== 9-May-2017::21:25:54 ===

started STOMP TCP Listener on [::]:61613

=WARNING REPORT==== 9-May-2017::21:25:59 ===

Could not obtain lsof output

=INFO REPORT==== 9-May-2017::21:26:04 ===

Management plugin started. Port: 15672

=INFO REPORT==== 9-May-2017::21:26:04 ===

Statistics database started.

=INFO REPORT==== 9-May-2017::21:26:04 ===

MQTT retained message store: rabbit_mqtt_retained_msg_store_dets

=INFO REPORT==== 9-May-2017::21:26:04 ===

started MQTT TCP Listener on [::]:1883

=INFO REPORT==== 9-May-2017::21:26:09 ===

Server startup complete; 10 plugins started.

 * rabbitmq_management_visualiser

 * rabbitmq_mqtt

 * rabbitmq_management

 * rabbitmq_management_agent

 * rabbitmq_amqp1_0

 * rabbitmq_web_dispatch

 * rabbitmq_stomp

 * amqp_client

 * cowboy

 * cowlib

And here is the output for rabbitmqctl status:

Status of node rabbit@localhost ...

[{pid,571},

 {running_applications,

     [{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.6.9"},

      {rabbitmq_mqtt,"RabbitMQ MQTT Adapter","3.6.9"},

      {rabbitmq_management,"RabbitMQ Management Console","3.6.9"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.9"},

      {rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.6.9"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.9"},

      {rabbitmq_stomp,"RabbitMQ STOMP plugin","3.6.9"},

      {rabbit,"RabbitMQ","3.6.9"},

      {os_mon,"CPO  CXC 138 46","2.4.2"},

      {amqp_client,"RabbitMQ AMQP Client","3.6.9"},

      {rabbit_common,

          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",

          "3.6.9"},

      {xmerl,"XML parser","1.3.13"},

      {compiler,"ERTS  CXC 138 10","7.0.4"},

      {cowboy,"Small, fast, modular HTTP server.","1.0.4"},

      {ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},

      {ssl,"Erlang/OTP SSL application","8.1.1"},

      {public_key,"Public key infrastructure","1.4"},

      {cowlib,"Support library for manipulating Web protocols.","1.0.2"},

      {inets,"INETS  CXC 138 49","6.3.6"},

      {asn1,"The Erlang ASN1 compiler version 4.0.4","4.0.4"},

      {crypto,"CRYPTO","3.7.3"},

      {sasl,"SASL  CXC 138 11","3.0.3"},

      {mnesia,"MNESIA  CXC 138 12","4.14.3"},

      {syntax_tools,"Syntax tools","2.1.1"},

      {stdlib,"ERTS  CXC 138 10","3.3"},

      {kernel,"ERTS  CXC 138 10","5.2"}]},

 {os,{unix,darwin}},

 {erlang_version,

     "Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:4:4] [async-threads:64] [hipe] [kernel-poll:true] [dtrace]\n"},

 {memory,

     [{total,65533376},

      {connection_readers,0},

      {connection_writers,0},

      {connection_channels,0},

      {connection_other,2848},

      {queue_procs,1981584},

      {queue_slave_procs,0},

      {plugins,2282472},

      {other_proc,19914688},

      {mnesia,342792},

      {metrics,330200},

      {mgmt_db,831544},

      {msg_index,216736},

      {other_ets,2986072},

      {binary,100864},

      {code,25292787},

      {atom,1041593},

      {other_system,10536548}]},

 {alarms,[]},

 {listeners,

     [{clustering,25672,"::"},

      {amqp,5672,"127.0.0.1"},

      {stomp,61613,"::"},

      {http,15672,"::"},

      {mqtt,1883,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,2712184422},

 {disk_free_limit,50000000},

 {disk_free,190712520704},

 {file_descriptors,

     [{total_limit,156},{total_used,2},{sockets_limit,138},{sockets_used,0}]},

 {processes,[{limit,1048576},{used,484}]},

 {run_queue,0},

 {uptime,2820},

 {kernel,{net_ticktime,60}}]

I HOPE SOMEONE COULD HELP ME PLEASE :(

[dfed...@pivotal.io]

Hi,

Can you also attach rab...@localhost-sasl.log and see if any of these commands timeout:
- `rabbitmqctl list_connections -t 20`
- `rabbitmqctl list_channels -t 20`

- `rabbitmqctl list_queues -t 20`

And if it does, please execute `rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck()' ` to see which process causes timeouts.

[Michael Klishin]

According to the logs and metrics, the node doesn't seem

to be under any kind of pressure.

There is only one thing that stands out:

> Could not obtain lsof output

but that seems to come from the management agent and should not

prevent e.g. new connections from being accepted in theory.

You are running with 156 file descriptors

available to the node and it's really easy to run into that, see http://www.rabbitmq.com/production-checklist.html

and bump it to, say, 50K.

See the other log that ends with "-sasl.log", it may have clues.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ