eaccess, open_port crash

[Pieter Roodt]

Hi,

Our one Rabbit server is becoming unresponsive occasionally with the following error report. Can anyone shed some light on what is wrong and how to fix it? 

=CRASH REPORT==== 2-Feb-2016::13:51:03 ===

  crasher:

    initial call: rabbit_mgmt_external_stats:init/1

    pid: <0.4789.1>

    registered_name: rabbit_mgmt_external_stats

    exception exit: {eacces,

                        [{erlang,open_port,

                             [{spawn,

                                  "C:\\Windows\\system32\\cmd.exe /chandle.exe /accepteula -s -p 103536 2> nul"},

                              [stream,in,eof,hide]],

                             []},

                         {os,cmd,1,[{file,"os.erl"},{line,175}]},

                         {rabbit_mgmt_external_stats,get_used_fd,1,[]},

                         {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,

                             []},

                         {rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,

                             []},

                         {rabbit_mgmt_external_stats,emit_update,1,[]},

                         {rabbit_mgmt_external_stats,handle_info,2,[]},

                         {gen_server,handle_msg,5,

                             [{file,"gen_server.erl"},{line,597}]}]}

      in function  gen_server:terminate/6 (gen_server.erl, line 737)

    ancestors: [rabbit_mgmt_agent_sup,<0.324.0>]

    messages: []

    links: [<0.325.0>]

    dictionary: []

    trap_exit: false

    status: running

    heap_size: 610

    stack_size: 24

    reductions: 282

  neighbours: 

[Daryn Wilkinson]

I'm having the exact same issue. Was hoping someone might have seen something like this before.

[Michael Klishin]

On 2 February 2016 at 16:31:26, Pieter Roodt (pieter...@gmail.com) wrote:
> Can anyone shed some light on what is wrong and how to fix it?

RabbitMQ tries to run

C:\\Windows\\system32\\cmd.exe /chandle.exe /accepteula -s -p 103536 2> nul

and gets an EOF.

That should not affect the rest of the broker. What version do you use? 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Pieter Roodt]

Thanks for the reply.

We are using version 3.1.1 (Erlang R15B)

This might be a red herring then. Looking at the log file now I can see there are other exceptions too. Could this be the trouble maker?  

=CRASH REPORT==== 1-Feb-2016::12:03:30 ===

  crasher:

    initial call: rabbit_disk_monitor:init/1

    pid: <0.16336.112>

    registered_name: rabbit_disk_monitor

    exception exit: {eacces,

                        [{erlang,open_port,

                             [{spawn,

                                  "C:\\Windows\\system32\\cmd.exe /cdir /-C /W \"e:/Services/RabbitMQ/Base/db/rabbit@CPTPRODAPP02-mnesia\""},

                              [stream,in,eof,hide]],

                             []},

                         {os,cmd,1,[{file,"os.erl"},{line,175}]},

                         {rabbit_disk_monitor,get_disk_free,2,[]},

                         {rabbit_disk_monitor,internal_update,1,[]},

                         {rabbit_disk_monitor,handle_info,2,[]},

                         {gen_server,handle_msg,5,

                             [{file,"gen_server.erl"},{line,597}]},

                         {proc_lib,init_p_do_apply,3,

                             [{file,"proc_lib.erl"},{line,227}]}]}

      in function  gen_server:terminate/6 (gen_server.erl, line 737)

    ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.162.0>]

    messages: []

    links: [<0.83.0>,<0.190.0>]

    dictionary: []

    trap_exit: false

    status: running

    heap_size: 6765

    stack_size: 24

    reductions: 8934

  neighbours:

[Michael Klishin]

On 2 February 2016 at 16:50:12, Pieter Roodt (pieter...@gmail.com) wrote:
> "C:\\Windows\\system32\\cmd.exe /cdir /-C /W \"e:/Services/RabbitMQ/Base/db/rabbit@CPTPRODAPP02-mnesia\""}

The error is similar but the command checks for free disk space and gets EACCESS, not an EOF.

Up to 3.5.7 this could affect the rest of RabbitMQ:
https://github.com/rabbitmq/rabbitmq-server/issues/91

however, I don't recall anything similar with the chandle.exe, which was in your original issue.

[Michael Klishin]

On 2 February 2016 at 16:50:12, Pieter Roodt (pieter...@gmail.com) wrote:

> We are using version 3.1.1 (Erlang R15B)

There were quite a few releases since then ;)

http://www.rabbitmq.com/changelog.html 

[Michael Klishin]

On 2 February 2016 at 16:31:26, Pieter Roodt (pieter...@gmail.com) wrote:

> initial call: rabbit_mgmt_external_stats:init/1

Sigh, so the exception happens in init/1, this is exactly the same issue (just in a different module/process) as
https://github.com/rabbitmq/rabbitmq-server/issues/91 and it can affect the rest of RabbitMQ,
and definitely produce more noise than it deserves.

I've filed https://github.com/rabbitmq/rabbitmq-management-agent/issues/12,
should be resolved by 3.6.1.

[Pieter Roodt]

Alright, so an upgrade is probably best. It's a 3 year old server and it's only now giving this issue, so kudos to you guys for the robustness so far ;) 

Is there a way to switch off this module in the meanwhile? 

[Michael Klishin]

On 2 February 2016 at 17:04:40, Pieter Roodt (pieter...@gmail.com) wrote:
> Is there a way to switch off this module in the meanwhile?

It's part of the rabbitmq-management-agent plugin but I'm afraid disabling it would lead
to rabbitmq-management having unmet dependencies. 

[dfed...@pivotal.io]

Can you provide more info on how it becomes unresponsive? Does node crush or there are some delays in message processing? Also some system monitoring info will be helpful if it's available (like cpu/memory/disk limits and utilisation).