keep running out of Socket descriptors on my server!

[Omid Bimo]

I have latest Rabbitmq installed on my debian server with mqtt+amqp+webManagement plugins enabled. The server has 64GB of RAM.

my problem is that every like 8 to 10 hours, the File descriptors reaches near the limit (which i increased several times, now standing at 200000) and Socket descriptors becomes full (according to web managerment interface - its red) and at that time, i have like 20000 connections/consumers and 30000 queues. why is this happening? its like rabbitmq hold every file descriptor/socket descriptor that it creates and never tires to close them!

any help would be appreciated.

[Simon MacMullen]

On 17/10/14 17:12, Omid Bimo wrote:
> and at that time, i have like 20000
> connections/consumers and 30000 queues.

Are you saying you see large numbers of connections /channels / queues
in the management UI under those tabs?

> why is this happening?

By far the most likely reason is that some client out there is creating
connections, channels and queues and not disconnecting.

Cheers, Simon

[Omid Bimo]

yes. after 8 to 10 hours, connections/sockets build up and 

File descriptors (?)	Socket descriptors (?)	Erlang processes

in the management UI become yellow and red, then accepting no more connections.

the topology here is 2 kinds of clients, android mobile clients connect to 1883 port for receiving mqtt notifications and local web application server pushes notifications to them.

[Michael Klishin]

On 17 October 2014 at 20:21:13, Omid Bimo (omid...@gmail.com) wrote:
> yes. after 8 to 10 hours, connections/sockets build up and
>
> File descriptors (?)
> Socket descriptors (?)
>

> in the management UI become yellow and red, then accepting no
> more connections.
> the topology here is 2 kinds of clients, android mobile clients
> connect to 1883 port for receiving mqtt notifications and local
> web application server pushes notifications to them.

Do you see any error reports in the log files, in particular SASL? What version do
you run? Do your clients (both MQTT and regular ones)  typically disconnect cleanly or just close TCP connections?
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Omid Bimo]

I dont see any SASL in logs. 

Version is Rabbitmq 3.3.5 , Erlang 17.

here is a sample of what shows up in log:

=INFO REPORT==== 17-Oct-2014::23:55:59 ===
accepting MQTT connection <0.3762.1192> (2.178.69.78:53364 -> server-valid-ip:1883)
=INFO REPORT==== 17-Oct-2014::23:55:59 ===
accepting MQTT connection <0.11552.1192> (37.129.50.137:55652 -> server-valid-ip:1883)
=INFO REPORT==== 17-Oct-2014::23:55:59 ===
accepting AMQP connection <0.14345.1212> (5.9.8.67:50472 -> server-valid-ip:5672)

 =ERROR REPORT==== 17-Oct-2014::23:55:59 ===
closing MQTT connection "2.190.177.192:13314 -> server-valid-ip:1883" (keepalive timeout)
=INFO REPORT==== 17-Oct-2014::23:55:59 ===
MQTT detected network error for "5.117.78.19:48359 -> server-valid-ip:1883": peer closed TCP connection
=INFO REPORT==== 17-Oct-2014::23:55:59 ===
closing AMQP connection <0.6527.925> (5.9.8.67:49885 -> server-valid-ip:5672)
=INFO REPORT==== 17-Oct-2014::23:55:59 ===
MQTT detected network error for "5.122.64.93:60666 -> server-valid-ip:1883": peer closed TCP connection

[Michael Klishin]

On 18 October 2014 at 00:30:40, Omid Bimo (omid...@gmail.com) wrote:
> I dont see any SASL in logs.

There is a file named rabbit@{hostname}-sasl.log next to
rabbit@{hostname}.log.

[Omid Bimo]

oh yeah. just saw it.
it contains logs like this:

=SUPERVISOR REPORT==== 18-Oct-2014::09:12:58 ===
     Supervisor: {<0.14914.159>,amqp_channel_sup_sup}
     Context:    shutdown_error
     Reason:     shutdown
     Offender:   [{nb_children,1},
                  {name,channel_sup},
                  {mfargs,
                      {amqp_channel_sup,start_link,
                          [direct,<0.7243.144>,
                           <<"2.187.81.175:46012 -> server-valid-ip:1883">>]}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,supervisor}]


=SUPERVISOR REPORT==== 18-Oct-2014::09:13:01 ===
     Supervisor: {<0.28267.122>,amqp_channel_sup_sup}
     Context:    shutdown_error
     Reason:     shutdown
     Offender:   [{nb_children,1},
                  {name,channel_sup},
                  {mfargs,
                      {amqp_channel_sup,start_link,
                          [direct,<0.28536.122>,
                           <<"5.118.66.114:41723 -> server-valid-ip:1883">>]}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,supervisor}]

[Michael Klishin]

What broker version do you run? 

[Omid Bimo]

Rabbitmq 3.3.5 - Erlang 17

[Omid Bimo]

so anyone any ideas please?

[Michael Klishin]

On 20 October 2014 at 21:59:32, Omid Bimo (omid...@gmail.com) wrote:
> so anyone any ideas please?

This sounds like an MQTT plugin issue but we will be able to verify this assumption  only
after RabbitMQ 3.4.0 release, should be later this week.

[Omid Bimo]

just upgraded to 3.4.0! lets see how it works.

[Omid Bimo]

Still the same issue. Anyone can help?

[Michael Klishin]

On 28 October 2014 at 22:10:30, Omid Bimo (omid...@gmail.com) wrote:
> > =SUPERVISOR REPORT==== 18-Oct-2014::09:12:58 ===
> Supervisor: {<0.14914.159>,amqp_channel_sup_sup}
> Context: shutdown_error
> Reason: shutdown
> Offender: [{nb_children,1},
> {name,channel_sup},
> {mfargs,
> {amqp_channel_sup,start_link,
> [direct,<0.7243.144>,
> <<"2.187.81.175:46012 -> server-valid-ip:1883">>]}},
> {restart_type,temporary},
> {shutdown,brutal_kill},
> {child_type,supervisor}]

Is there any other kind of error or supervisor report in the logs?
This one provides little information to form a hypothesis.

Also, how can we attempt to reproduce this? 

[kelvin]

在 2014年10月18日星期六UTC+8上午12时12分56秒，Omid Bimo写道：

I have latest Rabbitmq installed on my debian server with mqtt+amqp+webManagement plugins enabled. The server has 64GB of RAM.

my problem is that every like 8 to 10 hours, the File descriptors reaches near the limit (which i increased several times, now standing at 200000) and Socket descriptors becomes full (according to web managerment interface - its red) and at that time, i have like 20000 connections/consumers and 30000 queues. why is this happening? its like rabbitmq hold every file descriptor/socket descriptor that it creates and never tires to close them!

any help would be appreciated.

Have you checked socket status using netstat? I guess maybe it has something to do with the keep alive timer. Probably I was wrong.

[Andreas Ländle]

Michael Klishin <michael@...> writes:

>
>
> On 28 October 2014 at 22:10:30, Omid Bimo (omid8bimo@...) wrote:
> > > =SUPERVISOR REPORT==== 18-Oct-2014::09:12:58 ===
> > Supervisor: {<0.14914.159>,amqp_channel_sup_sup}
> > Context: shutdown_error
> > Reason: shutdown
> > Offender: [{nb_children,1},
> > {name,channel_sup},
> > {mfargs,
> > {amqp_channel_sup,start_link,
> > [direct,<0.7243.144>,
> > <<"2.187.81.175:46012 -> server-valid-ip:1883">>]}},
> > {restart_type,temporary},
> > {shutdown,brutal_kill},
> > {child_type,supervisor}]
>
> Is there any other kind of error or supervisor report in the logs?
> This one provides little information to form a hypothesis.
>
> Also, how can we attempt to reproduce this? 

Since I've encountered the same problem. Where you able to find a solution?
Thanks in advance for any hint.

Best regards,

Andreas

=SUPERVISOR REPORT==== 11-May-2016::07:18:39 ===
Supervisor: {<0.14037.0>,amqp_channel_sup_sup}

Context: shutdown_error
Reason: shutdown
Offender: [{nb_children,1},
{name,channel_sup},
{mfargs,
{amqp_channel_sup,start_link,

[direct,<0.14036.0>,
<<"10.41.220.51:54533 -> 10.41.219.50:1883">>]}},

[Marco Lettieri]

i've the same issue and seems not solved with3.7.17Erlang 20.3

[Michael Klishin]

We cannot guess what is happening in your system. Monitoring data can tell a pretty complete story, though [1][2].

Overprovisioning the open file handle limit [3] is generally recommended (as not every connection count peak is due to a bug somewhere).

Applications can leak connections and for some reason RabbitMQ is always the suspect  first [1].

The amqp_channel_sup_sup exception is just connection shutdown noise that has been explained many times

on this list and in GitHub issues (which we do not use for questions but they still happen there).

1. https://www.rabbitmq.com/connections.html#monitoring

2. https://www.rabbitmq.com/monitoring.html

3. https://www.rabbitmq.com/networking.html#open-file-handle-limit

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.