rabbitmq start failed
527 views
Skip to first unread message
Sergey Kazanskiy
Apr 1, 2015, 7:58:37 AM4/1/15
to rabbitm...@googlegroups.com
Hello guys, after power-loss situation at last friday in Amsterdam i got this situation:
root@queue02:/etc# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise
root@queue02:/etc# service rabbitmq-server start
* Starting message broker rabbitmq-server
* FAILED - check /var/log/rabbitmq/startup_\{log, _err\} [fail]
root@queue02:/etc# cat /var/log/rabbitmq/startup_*
RabbitMQ 3.5.0. Copyright (C) 2007-2014 GoPivotal, Inc.
## ## Licensed under the MPL. See http://www.rabbitmq.com/
## ##
########## Logs: /var/log/rabbitmq/rab...@queue02.log
###### ## /var/log/rabbitmq/rab...@queue02-sasl.log
##########
Starting broker... completed with 6 plugins.
root@queue02:/etc# cat /var/log/rabbitmq/rab...@queue02-sasl.log
=CRASH REPORT==== 1-Apr-2015::11:28:23 ===
crasher:
initial call: rabbit_disk_monitor:init/1
pid: <0.739.0>
registered_name: []
exception exit: unsupported_platform
in function gen_server:init_it/6
ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.569.0>]
messages: []
links: [<0.738.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 4181
stack_size: 24
reductions: 13928
neighbours:
=SUPERVISOR REPORT==== 1-Apr-2015::11:28:23 ===
Supervisor: {local,rabbit_disk_monitor_sup}
Context: start_error
Reason: unsupported_platform
Offender: [{pid,undefined},
{name,rabbit_disk_monitor},
{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
{restart_type,{transient,1}},
{shutdown,4294967295},
{child_type,worker}]
Right after that:
root@queue02:/etc# service rabbitmq-server status
Status of node rabbit@queue02 ...
[{pid,11182},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.5.0"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.0"},
{rabbit,"RabbitMQ","3.5.0"},
{os_mon,"CPO CXC 138 46","2.2.7"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.0"},
{webmachine,"webmachine","1.10.3-rmq3.5.0-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.0-git680dba8"},
{amqp_client,"RabbitMQ AMQP Client","3.5.0"},
{xmerl,"XML parser","1.2.10"},
{inets,"INETS CXC 138 49","5.7.1"},
{mnesia,"MNESIA CXC 138 12","4.5"},
{sasl,"SASL CXC 138 11","2.1.10"},
{stdlib,"ERTS CXC 138 10","1.17.5"},
{kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:24:24] [rq:24] [async-threads:30] [kernel-poll:true]\n"},
{memory,
[{total,35501416},
{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,5408},
{queue_procs,2704},
{queue_slave_procs,0},
{plugins,436632},
{other_proc,9363744},
{mnesia,59824},
{mgmt_db,140400},
{msg_index,46864},
{other_ets,1091144},
{binary,18664},
{code,17718222},
{atom,1618897},
{other_system,4998913}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,13482210099},
{disk_free_limit,50000000},
{disk_free,13430390784},
{file_descriptors,
[{total_limit,3996},
{total_used,3},
{sockets_limit,3594},
{sockets_used,1}]},
{processes,[{limit,1048576},{used,200}]},
{run_queue,0},
{uptime,262}]
root@queue02:/etc# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise
root@queue02:/etc# service rabbitmq-server start
* Starting message broker rabbitmq-server
* FAILED - check /var/log/rabbitmq/startup_\{log, _err\} [fail]
root@queue02:/etc# cat /var/log/rabbitmq/startup_*
RabbitMQ 3.5.0. Copyright (C) 2007-2014 GoPivotal, Inc.
## ## Licensed under the MPL. See http://www.rabbitmq.com/
## ##
########## Logs: /var/log/rabbitmq/rab...@queue02.log
###### ## /var/log/rabbitmq/rab...@queue02-sasl.log
##########
Starting broker... completed with 6 plugins.
root@queue02:/etc# cat /var/log/rabbitmq/rab...@queue02-sasl.log
=CRASH REPORT==== 1-Apr-2015::11:28:23 ===
crasher:
initial call: rabbit_disk_monitor:init/1
pid: <0.739.0>
registered_name: []
exception exit: unsupported_platform
in function gen_server:init_it/6
ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.569.0>]
messages: []
links: [<0.738.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 4181
stack_size: 24
reductions: 13928
neighbours:
=SUPERVISOR REPORT==== 1-Apr-2015::11:28:23 ===
Supervisor: {local,rabbit_disk_monitor_sup}
Context: start_error
Reason: unsupported_platform
Offender: [{pid,undefined},
{name,rabbit_disk_monitor},
{mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
{restart_type,{transient,1}},
{shutdown,4294967295},
{child_type,worker}]
Right after that:
root@queue02:/etc# service rabbitmq-server status
Status of node rabbit@queue02 ...
[{pid,11182},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.5.0"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.0"},
{rabbit,"RabbitMQ","3.5.0"},
{os_mon,"CPO CXC 138 46","2.2.7"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.0"},
{webmachine,"webmachine","1.10.3-rmq3.5.0-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.0-git680dba8"},
{amqp_client,"RabbitMQ AMQP Client","3.5.0"},
{xmerl,"XML parser","1.2.10"},
{inets,"INETS CXC 138 49","5.7.1"},
{mnesia,"MNESIA CXC 138 12","4.5"},
{sasl,"SASL CXC 138 11","2.1.10"},
{stdlib,"ERTS CXC 138 10","1.17.5"},
{kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:24:24] [rq:24] [async-threads:30] [kernel-poll:true]\n"},
{memory,
[{total,35501416},
{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,5408},
{queue_procs,2704},
{queue_slave_procs,0},
{plugins,436632},
{other_proc,9363744},
{mnesia,59824},
{mgmt_db,140400},
{msg_index,46864},
{other_ets,1091144},
{binary,18664},
{code,17718222},
{atom,1618897},
{other_system,4998913}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,13482210099},
{disk_free_limit,50000000},
{disk_free,13430390784},
{file_descriptors,
[{total_limit,3996},
{total_used,3},
{sockets_limit,3594},
{sockets_used,1}]},
{processes,[{limit,1048576},{used,200}]},
{run_queue,0},
{uptime,262}]
Michael Klishin
Apr 1, 2015, 8:10:02 AM4/1/15
to rabbitm...@googlegroups.com, Sergey Kazanskiy
On 1 April 2015 at 14:58:38, Sergey Kazanskiy (aete...@gmail.com) wrote:
> root@queue02:/etc# cat /var/log/rabbitmq/startup_*
>
> RabbitMQ 3.5.0. Copyright (C) 2007-2014 GoPivotal, Inc.
> ## ## Licensed under the MPL. See http://www.rabbitmq.com/
> ## ##
> ########## Logs: /var/log/rabbitmq/rab...@queue02.log
> ###### ## /var/log/rabbitmq/rab...@queue02-sasl.log
> ##########
> Starting broker... completed with 6 plugins.
>
> root@queue02:/etc# cat /var/log/rabbitmq/rab...@queue02-sasl.log
>
> =CRASH REPORT==== 1-Apr-2015::11:28:23 ===
> crasher:
> initial call: rabbit_disk_monitor:init/1
> pid: <0.739.0>
> registered_name: []
> exception exit: unsupported_platform
This does not mean that RabbitMQ is failing to start. This means its disk monitor fails (so free disk space
> root@queue02:/etc# cat /var/log/rabbitmq/startup_*
>
> RabbitMQ 3.5.0. Copyright (C) 2007-2014 GoPivotal, Inc.
> ## ## Licensed under the MPL. See http://www.rabbitmq.com/
> ## ##
> ########## Logs: /var/log/rabbitmq/rab...@queue02.log
> ###### ## /var/log/rabbitmq/rab...@queue02-sasl.log
> ##########
> Starting broker... completed with 6 plugins.
>
> root@queue02:/etc# cat /var/log/rabbitmq/rab...@queue02-sasl.log
>
> =CRASH REPORT==== 1-Apr-2015::11:28:23 ===
> crasher:
> initial call: rabbit_disk_monitor:init/1
> pid: <0.739.0>
> registered_name: []
> exception exit: unsupported_platform
alarms won't be available).
It shells out to
/bin/df -kP [RabbitMQ node database directory]
so please see (and post to the list) what that command outputs (and its exit code).
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
Message has been deleted
Sergey Kazanskiy
Apr 1, 2015, 8:27:37 AM4/1/15
to rabbitm...@googlegroups.com, aete...@gmail.com
root@queue02:/etc# /bin/df -kP /var/lib/rabbitmq/
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/sys-var 15376784 1481964 13113716 11% /var
root@queue02:/etc# echo $?
0
среда, 1 апреля 2015 г., 15:10:02 UTC+3 пользователь Michael Klishin написал:
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/sys-var 15376784 1481964 13113716 11% /var
root@queue02:/etc# echo $?
0
среда, 1 апреля 2015 г., 15:10:02 UTC+3 пользователь Michael Klishin написал:
On 1 April 2015 at 14:58:38, Sergey Kazanskiy (aete...@gmail.com) wrote:
> root@queue02:/etc# cat /var/log/rabbitmq/startup_*
>
> RabbitMQ 3.5.0. Copyright (C) 2007-2014 GoPivotal, Inc.
> ## ## Licensed under the MPL. See http://www.rabbitmq.com/
> ## ##
> ########## Logs: /var/log/rabbitmq/rabbit@queue02.log
> ###### ## /var/log/rabbitmq/rabbit@queue02-sasl.log
> ##########
> Starting broker... completed with 6 plugins.
>
> root@queue02:/etc# cat /var/log/rabbitmq/rabbit@queue02-sasl.log
Michael Klishin
Apr 1, 2015, 8:29:24 AM4/1/15
to rabbitm...@googlegroups.com, Sergey Kazanskiy
On 1 April 2015 at 15:15:06, Sergey Kazanskiy (aete...@gmail.com) wrote:
> Filesystem 1024-blocks Used Available Capacity Mounted on
> /dev/mapper/sys-var 15376784 1481528 13114152 11% /var
> Filesystem 1024-blocks Used Available Capacity Mounted on
That looks reasonable at first glance.
A quick way to try restarting disk monitor is to run
rabbitmqctl eval '{ok, DiskLimit} = application:get_env(rabbit, disk_free_limit), rabbit_sup:start_delayed_restartable_child(rabbit_disk_monitor, [DiskLimit]).'
but I'm afraid in your case (when the monitor exited "voluntarily") it may not be effective.
It's harmless to try, though.
Michael Klishin
Apr 1, 2015, 8:31:35 AM4/1/15
to rabbitm...@googlegroups.com, Sergey Kazanskiy
On 1 April 2015 at 15:29:22, Michael Klishin (mkli...@pivotal.io) wrote:
> That looks reasonable at first glance.
>
> A quick way to try restarting disk monitor is to run
>
> rabbitmqctl eval '{ok, DiskLimit} = application:get_env(rabbit,
> disk_free_limit), rabbit_sup:start_delayed_restartable_child(rabbit_disk_monitor,
> [DiskLimit]).'
>
> but I'm afraid in your case (when the monitor exited "voluntarily")
> it may not be effective.
>
> It's harmless to try, though.
That all assumes that the node (the VM) has started.
> That looks reasonable at first glance.
>
> A quick way to try restarting disk monitor is to run
>
> rabbitmqctl eval '{ok, DiskLimit} = application:get_env(rabbit,
> disk_free_limit), rabbit_sup:start_delayed_restartable_child(rabbit_disk_monitor,
> [DiskLimit]).'
>
> but I'm afraid in your case (when the monitor exited "voluntarily")
> it may not be effective.
>
> It's harmless to try, though.
Was that the case ?
Sergey Kazanskiy
Apr 1, 2015, 9:04:26 AM4/1/15
to rabbitm...@googlegroups.com, aete...@gmail.com
root@queue02:/etc# rabbitmqctl eval '{ok, DiskLimit} = application:get_env(rabbit, disk_free_limit), rabbit_sup:start_delayed_restartable_child(rabbit_disk_monitor, [DiskLimit]).'
{error,{already_started,<5066.193.0>}}
there is no any usefull data and thats secondary server, so i try to reinstall it. But purge and reinstall do nothing. and no default queues was created.
среда, 1 апреля 2015 г., 15:29:24 UTC+3 пользователь Michael Klishin написал:
{error,{already_started,<5066.193.0>}}
there is no any usefull data and thats secondary server, so i try to reinstall it. But purge and reinstall do nothing. and no default queues was created.
среда, 1 апреля 2015 г., 15:29:24 UTC+3 пользователь Michael Klishin написал:
Michael Klishin
Apr 1, 2015, 9:09:39 AM4/1/15
to rabbitm...@googlegroups.com, Sergey Kazanskiy
On 1 April 2015 at 16:04:28, Sergey Kazanskiy (aete...@gmail.com) wrote:
> there is no any usefull data and thats secondary server, so i
> try to reinstall it. But purge and reinstall do nothing. and no
> default queues was created.
Sergey,
> there is no any usefull data and thats secondary server, so i
> try to reinstall it. But purge and reinstall do nothing. and no
> default queues was created.
RabbitMQ does not create any queues by default.
Did the VM start in your case? Do you still have logs? If so, can you post them to the list or
send to me directly?
Michael Klishin
Apr 1, 2015, 9:25:53 AM4/1/15
to rabbitm...@googlegroups.com, Sergey Kazanskiy
On 1 April 2015 at 16:18:18, Sergey Kazanskiy (aete...@gmail.com) wrote:
> Did the VM start in your case? Do you still have logs? If so, can
> you post them to the list or
> send to me directly?
FTR, Sergey reports that RabbitMQ seem to start, as I expected.
> Did the VM start in your case? Do you still have logs? If so, can
> you post them to the list or
> send to me directly?
We filed a couple of issues, no clues on what in the output may have tripped disk monitor, though :
https://github.com/rabbitmq/rabbitmq-server/issues/91
https://github.com/rabbitmq/rabbitmq-server/issues/92
Reply all
Reply to author
Forward
0 new messages