RabbitMQ crashing with "no_exists,[rabbit_runtime_parameters,cluster_name]"

[kelan...@vmware.com]

Hi,

We use RabbitMQ in our java application and I see the below crash report in rabbitMQ. We do not use rabbitMQ cluster, but the error seems to be related to cluster_name.
We use SpringAMQP and I see also see exceptions in our application logs.
Can someone help to understand the root cause of this issue?

RabbitMQ errors:

=CRASH REPORT==== 2-Sep-2016::17:30:47 ===
  crasher:
    initial call: rabbit_reader:init/4
    pid: <0.18421.1242>
    registered_name: []
    exception exit: {aborted,
                        {no_exists,[rabbit_runtime_parameters,cluster_name]}}
      in function  mnesia:abort/1 (mnesia.erl, line 318)
      in call from rabbit_runtime_parameters:lookup0/2 (src/rabbit_runtime_parameters.erl, line 268)
      in call from rabbit_runtime_parameters:value0/2 (src/rabbit_runtime_parameters.erl, line 264)
      in call from rabbit_reader:server_properties/1 (src/rabbit_reader.erl, line 278)
      in call from rabbit_reader:start_connection/3 (src/rabbit_reader.erl, line 1087)
      in call from rabbit_reader:handle_input/3 (src/rabbit_reader.erl, line 1037)
      in call from rabbit_reader:recvloop/4 (src/rabbit_reader.erl, line 442)
      in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 424)
    ancestors: [<0.13989.1242>,<0.343.0>,<0.342.0>,<0.341.0>,rabbit_sup,
                  <0.154.0>]
    messages: []
    links: [<0.13989.1242>]
    dictionary: [{process_name,
                      {rabbit_reader,
                          <<"10.155.184.107:54004 -> 10.155.191.91:5671">>}}]
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 1480
  neighbours:

=SUPERVISOR REPORT==== 2-Sep-2016::17:30:47 ===
     Supervisor: {<0.13989.1242>,rabbit_connection_sup}
     Context:    child_terminated
     Reason:     {aborted,{no_exists,[rabbit_runtime_parameters,
                                      cluster_name]}}
     Offender:   [{pid,<0.18421.1242>},
                  {name,reader},
                  {mfargs,
                      {rabbit_reader,start_link,
                          [<0.14370.1242>,
                           {acceptor,{0,0,0,0,0,0,0,0},5671},
                           {sslsocket,
                               {gen_tcp,#Port<0.16317565>,tls_connection,
                                   <0.345.0>},
                               <0.14011.1242>}]}},
                  {restart_type,intrinsic},
                  {shutdown,30000},
                  {child_type,worker}]



Spring AMQP exception:

2016-09-02 18:08:04.380 GMT  INFO SimpleAsyncTaskExecutor-19413 SimpleMessageListenerContainer$AsyncMessageProcessingConsumer:601 - Restarting Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
2016-09-02 18:08:05.697 GMT  WARN SimpleAsyncTaskExecutor-19413 SimpleMessageListenerContainer$AsyncMessageProcessingConsumer:572 - Consumer raised exception, processing can restart if the connection factory supports it
org.springframework.amqp.AmqpIOException: java.io.IOException
        at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:54)
        at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:185)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:228)
        at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:77)
        at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:125)
        at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)
        at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:200)
        at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:527)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException
        at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
        at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
        at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:360)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)
        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)
        at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:181)
        ... 7 more
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
        at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
        at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
        at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:313)
        ... 10 more
Caused by: java.io.EOFException
        at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
        at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
        at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
        at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:515)

[Michael Klishin]

RabbitMQ doesn't crash, a particular connection process does.

What RabbitMQ version is used? Can you please post *full* logs?

The `rabbit_runtime_parameters` table is initialized on boot

and I couldn't find any known issues on GitHub or in the legacy bug tracker

about this.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Wyraz]

I get the same issue here. I'm running rabbitmq 3.6.1 in a 2-node-cluster. After a certain time (a few days), one of the nodes behaves the way described above, each time a client connects. I use both, java and python clients - same behaviour.

Here's a stack trace from the failing node (it runs on 5671, no SSL, replaced some private information with X):

=INFO REPORT==== 5-Sep-2016::11:41:24 ===

accepting AMQP connection <0.15422.82> (194.X.X.X:50160 -> 194.X.X.X:5671)

==> node_...@XXX-sasl.log <==

=CRASH REPORT==== 5-Sep-2016::11:41:24 ===

  crasher:

    initial call: rabbit_reader:init/4

    pid: <0.15422.82>

    registered_name: []

    exception exit: {aborted,

                        {no_exists,[rabbit_runtime_parameters,cluster_name]}}

      in function  mnesia:abort/1 (mnesia.erl, line 309)

      in call from rabbit_runtime_parameters:lookup0/2 (src/rabbit_runtime_parameters.erl, line 272)

      in call from rabbit_runtime_parameters:value0/2 (src/rabbit_runtime_parameters.erl, line 268)

      in call from rabbit_reader:server_properties/1 (src/rabbit_reader.erl, line 282)

      in call from rabbit_reader:start_connection/3 (src/rabbit_reader.erl, line 1091)

      in call from rabbit_reader:handle_input/3 (src/rabbit_reader.erl, line 1041)

      in call from rabbit_reader:recvloop/4 (src/rabbit_reader.erl, line 446)

      in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 428)

    ancestors: [<0.15413.82>,<0.929.0>,<0.927.0>,<0.925.0>,rabbit_sup,

                  <0.207.0>]

    messages: [{'EXIT',#Port<0.427822>,normal}]

    links: [<0.15413.82>]

    dictionary: [{process_name,

                      {rabbit_reader,

                          <<"194.X.X.X:50160 -> 194.X.X.X:5671">>}}]

    trap_exit: true

    status: running

    heap_size: 2586

    stack_size: 27

    reductions: 1434

  neighbours:

=SUPERVISOR REPORT==== 5-Sep-2016::11:41:24 ===

     Supervisor: {<0.15413.82>,rabbit_connection_sup}

     Context:    child_terminated

     Reason:     {aborted,{no_exists,[rabbit_runtime_parameters,

                                      cluster_name]}}

     Offender:   [{pid,<0.15422.82>},

                  {name,reader},

                  {mfargs,

                      {rabbit_reader,start_link,

                          [<0.15418.82>,

                           {acceptor,{0,0,0,0,0,0,0,0},5671},

                           #Port<0.427822>]}},

                  {restart_type,intrinsic},

                  {shutdown,4294967295},

                  {child_type,worker}]

=SUPERVISOR REPORT==== 5-Sep-2016::11:41:24 ===

     Supervisor: {<0.15413.82>,rabbit_connection_sup}

     Context:    shutdown

     Reason:     reached_max_restart_intensity

     Offender:   [{pid,<0.15422.82>},

                  {name,reader},

                  {mfargs,

                      {rabbit_reader,start_link,

                          [<0.15418.82>,

                           {acceptor,{0,0,0,0,0,0,0,0},5671},

                           #Port<0.427822>]}},

                  {restart_type,intrinsic},

                  {shutdown,4294967295},

                  {child_type,worker}]

[Michael Wyraz]

Hello,

we identified and solved the issue. One of our amqp clients had a bug that opened lots of connections (one every few minutes) withou closing it. This caused rabbit mq to run out of available file descriptors, causing this strange error. Increasing the descriptors and fixing the client solved the problem.

Regards,

Michael.

[Michael Klishin]

Thank you for reporting back!

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thuyet Tran]

Thank you

Vào 17:47:26 UTC+7 Thứ Năm, ngày 10 tháng 11 năm 2016, Michael Wyraz đã viết: