Rabbitmq-server not starting.

1,148 views
Skip to first unread message

bluc...@gmail.com

unread,
Jan 6, 2016, 1:42:39 PM1/6/16
to rabbitmq-users
Hi,

I am running ubuntu 12.04. The port PORT_NO is open. RMQ version is RabbitMQ 3.3.5. I am slightly at my wits end reading docs and trying things out on googling. I have quite a few delayed tasks in a umber of queues and I don't want to lost them. So I have not tried deleting any directories as was suggested in some of the answers.

Recently, my rabbitmq-server crashed. When I try to start it with 'sudo service rabbitmq-server start', I get:

################################

BOOT FAILED
===========

Error description:
   {badmatch,
       {error,
           {{{function_clause,
                 [{rabbit_queue_index,journal_minus_segment1,
                      [{no_pub,del,no_ack},
                       {{<<255,41,36,5,29,222,107,210,211,164,52,21,54,62,248,8>>,
                         {message_properties,undefined,false},
                         true},
                        del,ack}],
                      []},
                  {rabbit_queue_index,'-journal_minus_segment/2-fun-0-',4,[]},
                  {array,sparse_foldl_3,7,[{file,"array.erl"},{line,1689}]},
                  {array,sparse_foldl_2,9,[{file,"array.erl"},{line,1683}]},
                  {rabbit_queue_index,'-recover_journal/1-fun-0-',1,[]},
                  {lists,map,2,[{file,"lists.erl"},{line,1237}]},
                  {rabbit_queue_index,segment_map,2,[]},
                  {rabbit_queue_index,recover_journal,1,[]}]},
             {gen_server2,call,[<0.201.0>,out,infinity]}},
            {child,undefined,msg_store_persistent,
                {rabbit_msg_store,start_link,
                    [msg_store_persistent,
                     "/var/lib/rabbitmq/mnesia/rab...@ip-1.1.1.1",[],
                     {#Fun<rabbit_queue_index.2.111155628>,
                      {start,
                          [{resource,<<"/">>,queue,<<"priority_photos">>},
                           {resource,<<"/">>,queue,<<"celery">>},
                           {resource,<<"/">>,queue,<<"photos">>},
                           {resource,<<"/">>,queue,<<"priority_tasks">>}]}}]},
                transient,4294967295,worker,
                [rabbit_msg_store]}}}}

Log files (may contain more information):
   /var/log/rabbitmq/rab...@ip-1.1.1.1.log
   /var/log/rabbitmq/rab...@ip-1.1.1.1-sasl.log

Stack trace:
   [{rabbit_variable_queue,start_msg_store,2,[]},
    {rabbit_variable_queue,start,1,[]},
    {rabbit_amqqueue,recover,0,[]},
    {rabbit,recover,0,[]},
    {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
    {rabbit,run_boot_step,1,[]},
    {rabbit,'-start/2-lc$^0/1-0-',1,[]},
    {rabbit,start,2,[]}]



BOOT FAILED
===========

Error description:
   {could_not_start,rabbit,
    {bad_return,
     {{rabbit,start,[normal,[]]},
      {'EXIT',
       {rabbit,failure_during_boot,
        {badmatch,
         {error,
          {{{function_clause,
             [{rabbit_queue_index,journal_minus_segment1,
               [{no_pub,del,no_ack},
                {{<<255,41,36,5,29,222,107,210,211,164,52,21,54,62,248,8>>,
                  {message_properties,undefined,false},
                  true},
                 del,ack}],
               []},
              {rabbit_queue_index,'-journal_minus_segment/2-fun-0-',4,[]},
              {array,sparse_foldl_3,7,[{file,"array.erl"},{line,1689}]},
              {array,sparse_foldl_2,9,[{file,"array.erl"},{line,1683}]},
              {rabbit_queue_index,'-recover_journal/1-fun-0-',1,[]},
              {lists,map,2,[{file,"lists.erl"},{line,1237}]},
              {rabbit_queue_index,segment_map,2,[]},
              {rabbit_queue_index,recover_journal,1,[]}]},
            {gen_server2,call,[<0.201.0>,out,infinity]}},
           {child,undefined,msg_store_persistent,
            {rabbit_msg_store,start_link,
             [msg_store_persistent,
              "/var/lib/rabbitmq/mnesia/rab...@ip-1.1.1.1",[],
              {#Fun<rabbit_queue_index.2.111155628>,
               {start,
                [{resource,<<"/">>,queue,<<"priority_photos">>},
                 {resource,<<"/">>,queue,<<"celery">>},
                 {resource,<<"/">>,queue,<<"photos">>},
                 {resource,<<"/">>,queue,<<"priority_tasks">>}]}}]},
            transient,4294967295,worker,
            [rabbit_msg_store]}}}}}}}}}

Log files (may contain more information):
   /var/log/rabbitmq/rab...@ip-1.1.1.1.log
   /var/log/rabbitmq/rab...@ip-1.1.1.1-sasl.log

{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{badmatch,{error,{{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{no_pub,del,no_ack},{{<<16 bytes>>,{message_properties,undefined,false},true},del,ack}],[]},{rabbit_queue_index,'-journal_minus_segment/2-fun-0-',4,[]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1689}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1683}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[]},{lists,map,2,[{file,"lists.erl"},{line,1237}]},{rabbit_queue_index,segment_map,2,[]},{rabbit_queue_index,recover_journal,1,[]}]},{gen_server2,call,[<0.201.0>,out,infinity]}},{child,undefined,msg_store_persistent,{rabbit_msg_store,start_link,[msg_store_persistent,"/var/lib/rabbitmq/mnesia/rab...@ip-1.1.1.1",[],{#Fun<rabbit_queue_index.2.111155628>,{start,[{resource,<<1 byte>>,queue,<<15 bytes>>},{resource,<<1 byte>>,queue,<<6 bytes>>},{resource,<<1 byte>>,queue,<<6 bytes>>},{resource,<<1 byte>>,queue,<<14 bytes>>}]}}]},transient,4294967295,worker,[rabbit_msg_store]}}}}}}}}}}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()

#################################

$hostname -s
> ip-1.1.1.1

$cat /etc/hosts
> 127.0.0.1 localhost ip-1.1.1.1

$cat /etc/hostname
> ip-1.1.1.1

$sudo rabbitmqctl status
Status of node 'rab...@ip-1.1.1.1' ...
Error: unable to connect to node 'rab...@ip-1.1.1.1': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rab...@ip-1.1.1.1']

rab...@ip-1.1.1.1:
  * connected to epmd (port PORT_NO) on ip-1.1.1.1
  * epmd reports: node 'rabbit' not running at all
                  other nodes on ip-1.1.1.1: [rabbitmqctl22229]
  * suggestion: start the node

current node details:
- node name: rabbitmqctl22229@localhost
- home dir: /var/lib/rabbitmq
- cookie hash: VbIFRZPh6w4/yz1XTEn40w==


$ls /var/lib/rabbitmq/mnesia/
rab...@ip-3.3.3.3                 rab...@ip-2.2.2.2-plugins-expand  rab...@ip-1.1.1.1-plugins-expand
rab...@ip-3.3.3.3-plugins-expand  rab...@ip-1.1.1.1
rab...@ip-2.2.2.2                rab...@ip-1.1.1.1.pid


$ps aux | grep erl
rabbitmq 21866  0.0  0.0   7412   316 ?        S    18:23   0:00 /usr/local/lib/erlang/erts-6.1.1/bin/epmd -daemon

Michael Klishin

unread,
Jan 6, 2016, 2:02:45 PM1/6/16
to rabbitm...@googlegroups.com, bluc...@gmail.com
On 6 January 2016 at 21:42:42, bluc...@gmail.com (bluc...@gmail.com) wrote:
> I am running ubuntu 12.04. The port PORT_NO is open. RMQ version
> is RabbitMQ 3.3.5. I am slightly at my wits end reading docs and
> trying things out on googling. I have quite a few delayed tasks
> in a umber of queues and I don't want to lost them. So I have not tried
> deleting any directories as was suggested in some of the answers.

Queue index cannot be recovered. Try a more recent version first, indices can be rebuilt
from message segment files. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ


Haris Ibrahim K. V.

unread,
Jan 6, 2016, 10:26:11 PM1/6/16
to Michael Klishin, rabbitm...@googlegroups.com
Hi Michael,
I've upgraded the version to 3.6.0.

I tried:

$sudo service rabbitmq-server start
* Starting message broker rabbitmq-server
[ OK ]

However,

$sudo rabbitmqctl status
Status of node 'rabbit@ip-1-1-1-1' ...
Error: unable to connect to node 'rabbit@ip-1-1-1-1': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@ip-1-1-1-1']

rabbit@ip-10-218-184-8:
* connected to epmd (port PORT_NO) on ip-1-1-1-1
* epmd reports: node 'rabbit' not running at all
other nodes on ip-1-1-1-1: ['rabbitmq-cli-27202']
* suggestion: start the node

current node details:
- node name: 'rabbitmq-cli-27202@localhost'
- home dir: /var/lib/rabbitmq
- cookie hash: VbIFRZPh6w4/yz1XTEn40w==

$ps aux | grep erl
rabbitmq 24034 0.0 0.0 7412 320 ? S 03:14 0:00
/usr/local/lib/erlang/erts-6.1.1/bin/epmd -daemon

The startup_err log file is empty. Here is the entry in startup_log log file:

#####################
BOOT FAILED
===========

Error description:
{could_not_start,rabbit,
{{badmatch,
{error,
{{{function_clause,
[{rabbit_queue_index,journal_minus_segment1,
[{no_pub,del,no_ack},
{{true,
<<255,41,36,5,29,222,107,210,211,164,52,21,54,
62,248,8,0,0,0,0,0,0,0,0,0,0,0,0>>,
<<>>},
del,ack}],
[{file,"src/rabbit_queue_index.erl"},{line,1195}]},
{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',
4,
[{file,"src/rabbit_queue_index.erl"},{line,1172}]},
{array,sparse_foldl_3,7,
[{file,"array.erl"},{line,1690}]},
{array,sparse_foldl_2,9,
[{file,"array.erl"},{line,1684}]},
{rabbit_queue_index,'-recover_journal/1-fun-0-',1,
[{file,"src/rabbit_queue_index.erl"},{line,880}]},
{lists,map,2,[{file,"lists.erl"},{line,1238}]},
{rabbit_queue_index,segment_map,2,
[{file,"src/rabbit_queue_index.erl"},{line,1004}]},
{rabbit_queue_index,recover_journal,1,
[{file,"src/rabbit_queue_index.erl"},{line,871}]}]},
{gen_server2,call,[<0.243.0>,out,infinity]}},
{child,undefined,msg_store_persistent,
{rabbit_msg_store,start_link,
[msg_store_persistent,
"/var/lib/rabbitmq/mnesia/rab...@ip-1.1.1.1",
[],
{#Fun<rabbit_queue_index.2.56515753>,
{start,
[{resource,<<"/">>,queue,<<"celery">>},
{resource,<<"/">>,queue,<<"priority_tasks">>},
{resource,<<"/">>,queue,<<"priority_photos">>},
{resource,<<"/">>,queue,<<"photos">>}]}}]},
transient,4294967295,worker,
[rabbit_msg_store]}}}},
[{rabbit_variable_queue,start_msg_store,2,
[{file,"src/rabbit_variable_queue.erl"},{line,458}]},
{rabbit_variable_queue,start,1,
[{file,"src/rabbit_variable_queue.erl"},{line,440}]},
{rabbit_priority_queue,start,1,
[{file,"src/rabbit_priority_queue.erl"},{line,92}]},
{rabbit_amqqueue,recover,0,
[{file,"src/rabbit_amqqueue.erl"},{line,234}]},
{rabbit,recover,0,[{file,"src/rabbit.erl"},{line,538}]},
{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,
[{file,"src/rabbit_boot_steps.erl"},{line,49}]},
{rabbit_boot_steps,run_step,2,
[{file,"src/rabbit_boot_steps.erl"},{line,49}]},
{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,
[{file,"src/rabbit_boot_steps.erl"},{line,26}]}]}}

Log files (may contain more information):
/var/log/rabbitmq/rab...@ip-1.1.1.1.log
/var/log/rabbitmq/rab...@ip-1.1.1.1-sasl.log

{"init terminating in
do_boot",{could_not_start,rabbit,{{badmatch,{error,{{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{no_pub,del,no_ack},{{true,<<255,41,36,5,29,222,107,210,211,164,52,21,54,62,248,8,0,0,0,0,0,0,0,0,0,0,0,0>>,<<>>},del,ack}],[{file,"src/rabbit_queue_index.erl"},{line,1195}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1172}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1690}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1684}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,880}]},{lists,map,2,[{file,"lists.erl"},{line,1238}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1004}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,871}]}]},{gen_server2,call,[<0.243.0>,out,infinity]}},{child,undefined,msg_store_persistent,{rabbit_msg_store,start_link,[msg_store_persistent,"/var/lib/rabbitmq/mnesia/rab...@ip-1.1.1.1",[],{#Fun<rabbit_queue_index.2.56515753>,{start,[{resource,<<"/">>,queue,<<"celery">>},{resource,<<"/">>,queue,<<"priority_tasks">>},{resource,<<"/">>,queue,<<"priority_photos">>},{resource,<<"/">>,queue,<<"photos">>}]}}]},transient,4294967295,worker,[rabbit_msg_store]}}}},[{rabbit_variable_queue,start_msg_store,2,[{file,"src/rabbit_variable_queue.erl"},{line,458}]},{rabbit_variable_queue,start,1,[{file,"src/rabbit_variable_queue.erl"},{line,440}]},{rabbit_priority_queue,start,1,[{file,"src/rabbit_priority_queue.erl"},{line,92}]},{rabbit_amqqueue,recover,0,[{file,"src/rabbit_amqqueue.erl"},{line,234}]},{rabbit,recover,0,[{file,"src/rabbit.erl"},{line,538}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,"src/rabbit_boot_steps.erl"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,"src/rabbit_boot_steps.erl"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,"src/rabbit_boot_steps.erl"},{line,26}]}]}}}
#######################

Any pointers would be extremely helpful at this point.

Thank you.
--
Haris Ibrahim K. V.
http://sosaysharis.wordpress.com
@harisibrahimkv

Haris Ibrahim K. V.

unread,
Jan 11, 2016, 1:27:09 AM1/11/16
to rabbitm...@googlegroups.com
Just in case someone finds this thread and wonders what I did (or for
my own reference) - Based on one of the suggestions that I got, I
tried removing everything under /var/lib/rabbitmq/mnesia/ directory
and tried restarting. The error was same as above.

Since I had anyway made up my mind to lose data and get this working,
I apt purged my rabbitmq installation and re-installed to get it
working again.

Cheers,

Michael Klishin

unread,
Jan 11, 2016, 4:20:22 AM1/11/16
to rabbitm...@googlegroups.com, Haris Ibrahim K. V.
On 11 January 2016 at 09:27:08, Haris Ibrahim K. V. (bluc...@gmail.com) wrote:
> I
> tried removing everything under /var/lib/rabbitmq/mnesia/
> directory
> and tried restarting. The error was same as above.

I find it hard to believe: deleting node's data directory means there will be nothing
to recover. So perhaps it was under a different path that purging also removed. 
Reply all
Reply to author
Forward
0 new messages