Need suggestions to find the root cause of AMQP crash

[Bala Krishnan]

Hi Team,

One of my Customer's rabbitmq got crashed and i am finding the root cause for the issue.

I checked if any erl_crash.dump file was created, but not created.

Please give me some troubleshooting  steps on how to find the root cause.

Below logs I have excerpted from the log file:

• Reason for termination ==
• Unknown macro: {emfile,[Unknown macro}
  ,
  Unknown macro:
  Unknown macro: {os,cmd,2,[ {file,"os.erl"},{line,275}]},{rabbit_disk_monitor,get_disk_free,2,[\\\\\{file,"src/rabbit_disk_monitor.erl"},\\\\\\\\\\{line,218}]|file://%7Bfile%2C/][},{rabbit_disk_monitor,handle_info,2,{file,"src/rabbit_disk_monitor.erl"},////{line,190}" class="external-link" target="_blank" rel="nofollow noopener">\\\\\\\\{file,"src/rabbit_disk_monitor.erl"},\\\\\\\\{line,241}},{rabbit_disk_monitor,internal_update,1,|file://%7Bfile%2C/] [\\\\{file,"src/rabbit_disk_monitor.erl"},\\\\\\\\\\{line,218}},{rabbit_disk_monitor,handle_info,2,[{file,"src/rabbit_disk_monitor.erl"},\\\\\\\\\\{line,190}},{gen_server,try_dispatch,4,\\\\\{file,"gen_server.erl"},\\\\\\\\{line,637}%7D,%7Bgen_server,handle_msg,6,[/////%7Bfile,]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"}, \{line,249}]}]}
  2022-02-07 19:54:46.822 [error] <0.577.0> CRASH REPORT Process rabbit_disk_monitor with 0 neighbours crashed with reason: maximum number of file descriptors exhausted, check ulimit -n
  2022-02-07 19:54:46.822 [error] <0.576.0> Supervisor rabbit_disk_monitor_sup had child rabbit_disk_monitor started with rabbit_disk_monitor:start_link(50000000) at <0.577.0> exit with reason maximum number of file descriptors exhausted, check ulimit -n in context child_terminated
  2022-02-07 19:54:46.824 [info] <0.21534.1208> Free disk space monitor encountered an error (e.g. failed to parse output from OS tools): {{'EXIT',{emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd"},[binary,stderr_to_stdout,stream,in,hide,out]],[]},{os,cmd,2,[{file,"os.erl"}

  ,{line,275}
  ]},
  Unknown macro:
  Unknown macro:
  Unknown macro:
  Unknown macro:

  Unknown macro: {rabbit_disk_monitor,get_disk_free,2,[{file,"src/rabbit_disk_monitor.erl"},{line,241}]},{rabbit_disk_monitor,enable,1,[{file,"src/rabbit_disk_monitor.erl"},{line,315}

  ]},
  Unknown macro:
  Unknown macro:
  Unknown macro:
  Unknown macro:

  Unknown macro: {rabbit_disk_monitor,init,1,[{file,"src/rabbit_disk_monitor.erl"},{line,144}

  ]},
  Unknown macro:
  Unknown macro:
  Unknown macro:
  Unknown macro:

  Unknown macro: {gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}

  ]},
  Unknown macro:
  Unknown macro:
  Unknown macro:
  Unknown macro:

  Unknown macro: {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}

  ]}]}},33737097216}, retries left: 10

• 
  

2022-02-07 20:04:11.635 [warning] <0.1066.0> Ranch acceptor reducing accept rate: out of file descriptors
2022-02-07 20:04:11.640 [warning] <0.1059.0> Ranch acceptor reducing accept rate: out of file descriptors
2022-02-07 20:04:11.737 [error] <0.443.0> ** Generic server vm_memory_monitor terminating
** Last message in was update
** When Server state == {state,33737097216,13494838886,1276346368,0.4,1000,#Ref<0.2938654514.2465726467.83181>,false,{#Fun<rabbit_alarm.0.52593784>,#Fun<rabbit_alarm.1.52593784>},{unix,linux},"15289",4096,[47,112,114,111,99,47,"15289",47,115,116,97,116,109]}
** Reason for termination ==
** {{badmatch,{error,emfile}},[{vm_memory_monitor,read_proc_file,1,[{file,"src/vm_memory_monitor.erl"},{line,579}]},{vm_memory_monitor,get_process_memory_using_strategy,2,[{file,"src/vm_memory_monitor.erl"},{line,291}]},{vm_memory_monitor,update_process_memory,1,[{file,"src/vm_memory_monitor.erl"},{line,274}]},{vm_memory_monitor,internal_update,1,[{file,"src/vm_memory_monitor.erl"},{line,408}]},{vm_memory_monitor,handle_info,2,[{file,"src/vm_memory_monitor.erl"},{line,238}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2022-02-07 20:04:11.737 [error] <0.443.0> CRASH REPORT Process vm_memory_monitor with 0 neighbours crashed with reason: no match of right hand value {error,emfile} in vm_memory_monitor:read_proc_file/1 line 579
2022-02-07 20:04:11.738 [error] <0.442.0> Supervisor vm_memory_monitor_sup had child vm_memory_monitor started with vm_memory_monitor:start_link(0.4, #Fun<rabbit_alarm.0.52593784>, #Fun<rabbit_alarm.1.52593784>) at <0.443.0> exit with reason no match of right hand value {error,emfile} in vm_memory_monitor:read_proc_file/1 line 579 in context child_terminated
2022-02-07 20:04:11.738 [error] <0.442.0> Supervisor vm_memory_monitor_sup had child vm_memory_monitor started with vm_memory_monitor:start_link(0.4, #Fun<rabbit_alarm.0.52593784>, #Fun<rabbit_alarm.1.52593784>) at <0.443.0> exit with reason maximum number of file descriptors exhausted, check ulimit -n in context start_error
2022-02-07 20:04:11.738 [error] <0.582.1365> CRASH REPORT Process <0.582.1365> with 0 neighbours crashed with reason: maximum number of file descriptors exhausted, check ulimit -n

--

Regards
N.Balakrishnan MSc
(+91-9788844520)

[Wes Peng]

Your system’s file handles have exceeded. Please increase it.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAH6Ceb-ag5n%2BZiyi0r4DE7k7_O7ngFO3oAoZ4iO9bmhsLSXkJg%40mail.gmail.com.

[Bala Krishnan]

May I know how to find allowed limit by OS and how to change it?

Thank you

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAPiMwJKM1eZKDM-CNKKBTS21Q%3DYUKke8hDUKdZziwTmW_oSw7g%40mail.gmail.com.

[Wes Peng]

If you were using linux please check this one

https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Thanks 

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAH6Ceb-pjSzv59D6iaAQQLTzJ5-KB4%3DDxkg%2BQu32K5Ses5%2Bqwg%40mail.gmail.com.