RabbitMQ hanged up

[Alexander Tarasov]

Hello everyone!

I have single RabbitMQ 3.6.1-1 installed on CentOS 7.2.1511, with Erlang R16B-03.16.el7. Today I cleaned it up:

rabbitmqctl stop_app
rabbitmqctl force_reset
rabbitmqctl start_app

It worked for some time and now (maybe, after 8 hours of data processing) it doesn't respond to application requests and status check:

# rabbitmqctl status
Status of node 'rabbit@RabbitMQ-Server' ...
^C
Session terminated, killing shell... ...killed.

Also, I have a lot of messages in /var/log/rabbitmq/startup_log. These messages are repeating in infinite loop: http://pastebin.com/38AXyw4n

There are nothing in other logs:

# ls -lS
total 6481664
-rw-r--r-- 1 root     root     2365375398 Apr 30 21:34 startup_log
-rw-r--r-- 1 rabbitmq rabbitmq          0 Apr 30 14:18 rab...@RabbitMQ-Server-sasl.log
-rw-r--r-- 1 rabbitmq rabbitmq          0 Apr 30 20:45 rab...@RabbitMQ-Server.log
-rw-r--r-- 1 root     root              0 Apr 30 14:18 startup_err

What's wrong with RabbitMQ? How I could restore it back to working state?

Any hints will be much appreciated.

[Michael Klishin]

The easiest way to know "what's wrong" with a running but seemingly stuck node is via `rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'`

However, according to the log it's the error logger that fails, e.g. it could be that the log file has incorrect permissions or similar.

Dysfunctional logging seems to terminate channel writers (processes that send data over TCP sockets to clients) enough times

so that they are no longer restarted.

If that is the case, using a heartbeat interval of 5-10 seconds is a good idea, then clients should be able to

close connections in this state fairly quickly:

http://www.rabbitmq.com/heartbeats.html

HTH.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alexander Tarasov]

Thank you Michael. For the future, I took note of maybe_stuck() function.

Seems like it was some inconstistency between versions of RabbitMQ and Erlang. I've installed Erlang package from RabbitMQ maintainers (https://rabbitmq.com/releases/erlang/), after that logging and AMQP handshakes began working as expected. 

    systemctl stop rabbitmq-server

    killall epmd

 

    yum remove rabbitmq-server

    yum remove erlang

    yum autoremove

 

    wget https://www.rabbitmq.com/releases/erlang/erlang-18.3-1.el7.centos.x86_64.rpm

    wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.1/rabbitmq-server-3.6.1-1.noarch.rpm

    yum install erlang-18.3-1.el7.centos.x86_64.rpm

    yum install rabbitmq-server-3.6.1-1.noarch.rpm

воскресенье, 1 мая 2016 г., 12:17:40 UTC+3 пользователь Michael Klishin написал:

The easiest way to know "what's wrong" with a running but seemingly stuck node is via `rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'`

However, according to the log it's the error logger that fails, e.g. it could be that the log file has incorrect permissions or similar.

Dysfunctional logging seems to terminate channel writers (processes that send data over TCP sockets to clients) enough times

so that they are no longer restarted.

If that is the case, using a heartbeat interval of 5-10 seconds is a good idea, then clients should be able to

close connections in this state fairly quickly:

http://www.rabbitmq.com/heartbeats.html

HTH.

On Sat, Apr 30, 2016 at 5:18 PM, Alexander Tarasov <oioki....@gmail.com> wrote:

Hello everyone!

I have single RabbitMQ 3.6.1-1 installed on CentOS 7.2.1511, with Erlang R16B-03.16.el7. Today I cleaned it up:

rabbitmqctl stop_app
rabbitmqctl force_reset
rabbitmqctl start_app

It worked for some time and now (maybe, after 8 hours of data processing) it doesn't respond to application requests and status check:

# rabbitmqctl status
Status of node 'rabbit@RabbitMQ-Server' ...
^C
Session terminated, killing shell... ...killed.

Also, I have a lot of messages in /var/log/rabbitmq/startup_log. These messages are repeating in infinite loop: http://pastebin.com/38AXyw4n

There are nothing in other logs:

# ls -lS
total 6481664
-rw-r--r-- 1 root     root     2365375398

 Apr 30 21:34 startup_log
-rw-r--r-- 1 rabbitmq rabbitmq          0 Apr 30 14:18 rabbit@RabbitMQ-Server-sasl.log
-rw-r--r-- 1 rabbitmq rabbitmq          0 Apr 30 20:45 rab...@RabbitMQ-Server.log

-rw-r--r-- 1 root     root              0 Apr 30 14:18 startup_err

What's wrong with RabbitMQ? How I could restore it back to working state?

Any hints will be much appreciated.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.