`rabbitmqctl stop` hangs if RabbitMQ is out of FDs

[vitaly numenta]

I am using RabbitMQ v3.5.1 on MacBookPro/Mac OS X.

Add the line

ulimit -n 115 

in file /etc/rabbitmq/rabbitmq-env.conf on linux or /usr/local/etc/rabbitmq/rabbitmq-env.conf on Mac OS X.

Restart RabbitMQ to pick up the new configuration changes.

Then run `rabbitmqctl stop` (without the backticks). This command hangs.

$ rabbitmqctl status

Status of node rabbit@localhost ...

[{pid,29418},

 {running_applications,[{os_mon,"CPO  CXC 138 46","2.2.15"},

                        {inets,"INETS  CXC 138 49","5.10.2"},

                        {mnesia,"MNESIA  CXC 138 12","4.12.1"},

                        {amqp_client,"RabbitMQ AMQP Client","3.5.1"},

                        {xmerl,"XML parser","1.3.7"},

                        {sasl,"SASL  CXC 138 11","2.4"},

                        {stdlib,"ERTS  CXC 138 10","2.1"},

                        {kernel,"ERTS  CXC 138 10","3.0.1"}]},

 {os,{unix,darwin}},

 {erlang_version,"Erlang/OTP 17 [erts-6.1] [source] [64-bit] [smp:8:8] [async-threads:30] [hipe] [kernel-poll:true]\n"},

 {memory,[{total,49238128},

          {connection_readers,0},

          {connection_writers,0},

          {connection_channels,0},

          {connection_other,2808},

          {queue_procs,3296280},

          {queue_slave_procs,0},

          {plugins,0},

          {other_proc,16983184},

          {mnesia,104680},

          {mgmt_db,0},

          {msg_index,382424},

          {other_ets,1272176},

          {binary,179624},

          {code,20519410},

          {atom,711569},

          {other_system,5785973}]},

 {alarms,[]},

 {listeners,[]},

 {vm_memory_high_watermark,0.4},

 {vm_memory_limit,5928263680},

 {disk_free_limit,50000000},

 {disk_free,354038005760},

 {file_descriptors,[{total_limit,15},

                    {total_used,15},

                    {sockets_limit,11},

                    {sockets_used,0}]},

 {processes,[{limit,1048576},{used,230}]},

 {run_queue,0},

 {uptime,540}]

[Michael Klishin]

Sorry but that ulimit value is insane for any server. Try with 50K.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vitaly numenta]

Michael, I agree that it's an insane ulimit value. I was following up on https://github.com/pika/pika/issues/647, where this ulimit was used to reproduce a hang in pika.BlockingConnection along the same lines as with `rabbitmqctl stop`. In this case, pika is able to establish a socket connection with RabbitMQ broker, but the connection handshake never completes. I am inclined to conclude that pika shouldn't need to worry about this malicious-like under-resourcing of RabbitMQ  either.

Should RabbitMQ exit with non-zero error code if configured with insufficient FDs? It might be better than hanging in `rabbitmqctl stop`, etc.

[Michael Klishin]

On 21 Sep 2015 at 09:32:07, vitaly numenta (vitaly.kru...@gmail.com) wrote:
> Should RabbitMQ exit with non-zero error code if configured
> with insufficient FDs? It might be better than hanging in `rabbitmqctl
> stop`, etc.

There are a few problems with this suggestion.

1. Running out of file descriptors is often a temporary condition. Should we terminate
   a node when this happens? I’m pretty sure not everybody subscribes to this opinion.
2. What if a different process consumes most of the FDs on a node? Should RabbitMQ terminate
   because of that?
3. Connections eventually time out in most clients. You can also configure this at the
   proxy level, if you use it. If Pika doesn’t have a handshake or socket timeouts, this is
   an issue in Pika.

RabbitMQ tries calculating the file handle descriptors it uses (on the platforms it can do so),
reuse them where possible and block inbound connections when it’s reached the limit. This is
the lesser  of possible evils when running out of file descriptors if you ask me.

We also mention OS limits in the production checklist guide:
http://www.rabbitmq.com/production-checklist.html.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

 On 21 Sep 2015 at 12:36:06, Michael Klishin (mkli...@pivotal.io) wrote:
> Should RabbitMQ exit with non-zero error code if configured
> with insufficient FDs?

Also, defining “sufficient” above is impossible. We don’t know the kind of workload
expected, and tuning for different needs has to be done differently, as explained
in http://www.rabbitmq.com/networking.html.

[vitaly numenta]

A related question that concerns https://github.com/pika/pika/issues/647 "pika.BlockingConnection hangs when RabbitMQ server does not have free file/socket descriptors":

If the connection/handshake with RabbitMQ API (port 5672) succeeds, can this connection be affected by the out-of-file-descriptors issue subsequently when processing commands, such as publish, queue-declare, etc. I suspect that it can, just need confirmation. Thank you.