Unexplained rabbitmq errors - error on amqp connection

[Michael Stevens]

Hi,

I'm running RabbitMQ 3.6.5 on CentOS 7.

My setup is mostly working, but RabbitMQ seems to occasionally get into a weird state after it has been running for a while.

The most recent one seems to involve INTERNAL_ERROR when I try to queue new messages, and a rabbitmq server log filled with errors of which this seems to be the most common:

Error on AMQP connection <0.30401.5222> (172.30.0.38:33506 -> 172.30.0.38:5672, vhost: '/', user: 'guest', state: blocking), chan
nel 1:
{{{{badmatch,{error,emfile}},
   [{rabbit_queue_index,append_journal_to_segment,1,
                        [{file,"src/rabbit_queue_index.erl"},{line,815}]},
    {rabbit_queue_index,'-flush_journal/1-fun-0-',2,
                        [{file,"src/rabbit_queue_index.erl"},{line,800}]},
    {lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
    {rabbit_queue_index,segment_fold,3,
                        [{file,"src/rabbit_queue_index.erl"},{line,985}]},
    {rabbit_queue_index,flush_journal,1,
                        [{file,"src/rabbit_queue_index.erl"},{line,792}]},
    {rabbit_variable_queue,ui,1,
                           [{file,"src/rabbit_variable_queue.erl"},
                            {line,2602}]},
    {rabbit_variable_queue,push_betas_to_deltas1,4,
                           [{file,"src/rabbit_variable_queue.erl"},
                            {line,2582}]},
    {rabbit_variable_queue,push_betas_to_deltas,2,
                           [{file,"src/rabbit_variable_queue.erl"},
                            {line,2540}]}]},
  {gen_server2,call,[<0.588.0>,stat,infinity]}},
 [{gen_server2,call,3,[{file,"src/gen_server2.erl"},{line,325}]},
  {rabbit_misc,with_exit_handler,2,[{file,"src/rabbit_misc.erl"},{line,498}]},
  {rabbit_channel,handle_method,3,
                  [{file,"src/rabbit_channel.erl"},{line,1302}]},
  {rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,457}]},
  {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1032}]},
  {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

Please help :)

[Michael Klishin]

RabbitMQ queue index tried to create a new segment file and got an "emfile" back.

This likely means the OS cannot allocate it any more file descriptors.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Stevens]

In normal use I've got about 50-100 of 4096 file descriptors used. Something does seem to go mad and use them all occasionally but I've not been able to identify what.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/wSFm5NnG9NE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

[Michael Klishin]

I could find any other reasons for EMFILE on Linux specifically or even in general. Most posts and mailing list

threads (including on erlang-users) come down to [1].

4096 is still quite low for a data service. We recommend at least 50K in the docs [2]

but I bump it to 500K on any node that hosts a data service (RabbitMQ or not).

There isn't any significant downside that I can think of.

1. http://blog.izs.me/post/56827866110/wtf-is-emfile-and-why-does-it-happen-to-me

2. http://www.rabbitmq.com/production-checklist.html

[Michael Stevens]

I've put it up to 50k, will see how things go.