Rabbitmq stops accept new connection

[hitesh]

Hello There, 

  I am using rabbitmq in my online multiplayer games, i am making queue for individual clients. (Here client means android, ios and windows mobile.), I used rabbitmq with nodejs programming. 

My problem is my my server will up 6-7 days continues then after rabbitmq stops to accepting connection. now i have to stop all my servers of game and then i am restarting my rabbitmq service. Can you guys please suggest me what is the reason behind it. It pain for me to stop all game server and restarting RBMQ service.

Please help.

[Michael Klishin]

This exact question was asked before:

https://groups.google.com/forum/#!searchin/rabbitmq-users/Dileep/rabbitmq-users/kE5YFGZxAbs/an61rauinKAJ
https://groups.google.com/forum/#!searchin/rabbitmq-users/Dileep/rabbitmq-users/kE5YFGZxAbs/an61rauinKAJ

Please check RabbitMQ logs for alarms. You are likely running out of file descriptors.
Which suggests it's worth investigating and monitoring the number of active connections.
If it's growing monotonically, your app likely does not close connections it no longer needs.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Hitesh Dobariya]

no i am not running out of descriptor, i checked this all stuffs all are ok. I found that erlang processes continues grows up thats why my memory consumption continuously growing up. 

--

Thanks,

Hitesh Dobariya,

Artoon Solutions PVT. LTD

[Michael Klishin]

On 2 February 2015 at 07:27:11, Hitesh Dobariya (hit...@artoonsolutions.com) wrote:
> no i am not running out of descriptor, i checked this all stuffs
> all are ok. 

> I found that erlang processes continues grows up thats
> why my memory consumption continuously growing up.

Both don't matter in this case. Please post both RabbitMQ log files
(regular and SASL) and `rabbitmqctl status` output.

If you have any proxies between your clients and RabbitMQ node(s), see
if they may have run out of file descriptors or log anything suspicious. 

[Hitesh Dobariya]

Here is the "RabbitMQ Status"

[{pid,16798},

 {running_applications,

     [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},

      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.5"},

      {webmachine,"webmachine","1.10.3-rmq3.3.5-gite9359c7"},

      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.5-git680dba8"},

      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.5"},

      {rabbit,"RabbitMQ","3.3.5"},

      {os_mon,"CPO  CXC 138 46","2.2.14"},

      {inets,"INETS  CXC 138 49","5.9.7"},

      {mnesia,"MNESIA  CXC 138 12","4.11"},

      {amqp_client,"RabbitMQ AMQP Client","3.3.5"},

      {xmerl,"XML parser","1.3.5"},

      {sasl,"SASL  CXC 138 11","2.3.4"},

      {stdlib,"ERTS  CXC 138 10","1.19.4"},

      {kernel,"ERTS  CXC 138 10","2.16.4"}]},

 {os,{unix,linux}},

 {erlang_version,

     "Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:30]                                                                                                              [kernel-poll:true]\n"},

 {memory,

     [{total,1120946416},

      {connection_procs,753374928},

      {queue_procs,10807880},

      {plugins,667848},

      {other_proc,7411384},

      {mnesia,932248},

      {mgmt_db,106707552},

      {msg_index,7963248},

      {other_ets,25358528},

      {binary,138061448},

      {code,19901432},

      {atom,703377},

      {other_system,49056543}]},

 {alarms,[]},

 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,3137319731},

 {disk_free_limit,50000000},

 {disk_free,35611340800},

 {file_descriptors,

     [{total_limit,102300},

      {total_used,512},

      {sockets_limit,92068},

      {sockets_used,510}]},

 {processes,[{limit,1048576},{used,186135}]},

 {run_queue,0},

 {uptime,173939}]

...done.

Here is my Live server logs, (Currently rabbitmq running in good condition)

=INFO REPORT==== 2-Feb-2015::04:38:22 ===

accepting AMQP connection <0.21361.41> 

=WARNING REPORT==== 2-Feb-2015::04:38:26 ===

closing AMQP connection <0.21361.41>

connection_closed_abruptly

=WARNING REPORT==== 2-Feb-2015::04:38:26 ===

closing AMQP connection <0.20464.41>

connection_closed_abruptly

=INFO REPORT==== 2-Feb-2015::04:38:27 ===

closing AMQP connection <0.7901.41> 

=INFO REPORT==== 2-Feb-2015::04:38:28 ===

accepting AMQP connection <0.21093.41> 

=INFO REPORT==== 2-Feb-2015::04:38:29 ===

accepting AMQP connection <0.16433.41> 

=WARNING REPORT==== 2-Feb-2015::04:38:32 ===

closing AMQP connection <0.20274.41>

connection_closed_abruptly

=WARNING REPORT==== 2-Feb-2015::04:38:33 ===

closing AMQP connection <0.20736.41>

connection_closed_abruptly

=INFO REPORT==== 2-Feb-2015::04:38:33 ===

accepting AMQP connection <0.8993.41> 

=WARNING REPORT==== 2-Feb-2015::04:38:37 ===

closing AMQP connection <0.8993.41> 

connection_closed_abruptly

=INFO REPORT==== 2-Feb-2015::04:38:37 ===

accepting AMQP connection <0.21508.41> 

=INFO REPORT==== 2-Feb-2015::04:38:39 ===

accepting AMQP connection <0.21541.41>

=INFO REPORT==== 2-Feb-2015::04:38:39 ===

accepting AMQP connection <0.21296.41> 

=INFO REPORT==== 2-Feb-2015::04:38:40 ===

accepting AMQP connection <0.9221.41>

[Hitesh Dobariya]

Some times i am getting closing AMQP connection <0.27737.21>:{inet_error,etimedout}

[Michael Klishin]

On 2 February 2015 at 07:41:08, Hitesh Dobariya (hit...@artoonsolutions.com) wrote:
> Here is my Live server logs, (Currently rabbitmq running in
> good condition)

We need logs from when it was rejecting connections. Please include the  SASL log.

[Hitesh Dobariya]

---------- Forwarded message ----------
From: Hitesh Dobariya <hit...@artoonsolutions.com>
Date: Mon, Feb 2, 2015 at 10:32 AM
Subject: Re: [rabbitmq-users] Rabbitmq stops accept new connection
To: Michael Klishin <mkli...@pivotal.io>
Cc: rabbitm...@googlegroups.com

Please find attached logs..  Let me know if you want more logs from me.

--

Thanks,

Hitesh Dobariya,

Artoon Solutions PVT. LTD

[Hitesh Dobariya]

Another attachment .. 

[Michael Klishin]

On 2 February 2015 at 08:05:46, Hitesh Dobariya (hit...@artoonsolutions.com) wrote:
> Some times i am getting closing AMQP connection <0.27737.21>:{inet_error,etimedout}

This means a TCP connection has timed out.
There's also a bunch of warnings like this:

=WARNING REPORT==== 31-Jan-2015::06:09:02 ===
closing AMQP connection <0.14393.1> ([ip]:54964 -> [ip]:5672):
connection_closed_abruptly

which means that either TCP connection was lost or a client decided to close it
without closing RabbitMQ connection properly (this is harmless).

You seem to have these connectivity issues and reconnects in groups several times a minute:
this means either a pretty unreliable network or a misbehaving intermediary (e.g. a proxy).

You also have some of these:
=ERROR REPORT==== 31-Jan-2015::06:06:47 ===
AMQP connection <0.20023.1> (running), channel 1 - error:
{amqp_error,frame_error,
            "type 1, first 16 octets = <<0,60,0,40,0,0,2,112,101,16,104,98,77,254,46,115>>: {invalid_frame_end_marker,\n                                                                                102}",
            none}

which more likely suggest a network issue than a buggy client (I hope you use amqp.lib and not node.amqp,
which is not maintained and has known bugs).

In the SASL log, only one error report that seems to be due to connectivity loss
from a plugin such as Federation (or direct Erlang client, in case you have any Erlang apps
in your system):

=SUPERVISOR REPORT==== 1-Feb-2015::03:38:00 ===
     Supervisor: {<0.31343.19>,amqp_channel_sup_sup}
     Context:    shutdown_error
     Reason:     shutdown
     Offender:   [{nb_children,1},
                  {name,channel_sup},
                  {mfargs,
                      {amqp_channel_sup,start_link,
                          [direct,<0.31342.19>,
                           <<"<rabbit@ip-…>">>]}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,supervisor}]

There are no alarms in status output or logs.

I'd recommend investigating what may be the cause for so many network
timeouts and abrupt TCP connection closures. This may be that connections in your
app are short lived (less than a minute?) but the problem is pretty systemic according
to the log provided.

So far I don't see any errors that may suggest it is a server bug.