Rabbitmq server cannot start while rabbitmq_management plugin enabled

[Shafeek VK]

Hi,

I have a cluster setup of two nodes. Node 1 is started, but node 2 cannot be started if the management plugin is enabled.

I can start the server if the plugin is disabled. (Then web API is not available).

Could you please help me?

startup_log content is here

[root@ip-10-11-40-37 bin]# cat  /var/log/rabbitmq/startup_log

              RabbitMQ 3.3.4. Copyright (C) 2007-2014 GoPivotal, Inc.

  ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/

  ##  ##

  ##########  Logs: /var/log/rabbitmq/rab...@ip-10-11-40-37.log

  ######  ##        /var/log/rabbitmq/rab...@ip-10-11-40-37-sasl.log

  ##########

              Starting broker...

BOOT FAILED

===========

Error description:

   {could_not_start,rabbitmq_management,

       {{shutdown,

            {failed_to_start_child,rabbit_mgmt_sup,

                {'EXIT',

                    {{shutdown,

                         [{{already_started,<6899.3247.0>},

                           {child,undefined,rabbit_mgmt_db,

                               {rabbit_mgmt_db,start_link,[]},

                               permanent,4294967295,worker,

                               [rabbit_mgmt_db]}}]},

                     {gen_server2,call,

                         [<0.695.0>,{init,<0.693.0>},infinity]}}}}},

        {rabbit_mgmt_app,start,[normal,[]]}}}

Log files (may contain more information):

   /var/log/rabbitmq/rab...@ip-10-11-40-37.log

   /var/log/rabbitmq/rab...@ip-10-11-40-37-sasl.log

{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,rabbitmq_management,{{shutdown,{failed_to_start_child,rabbit_mgmt_sup,{'EXIT',{{shutdown,[{{already_started,<6899.3247.0>},{child,undefined,rabbit_mgmt_db,{rabbit_mgmt_db,start_link,[]},permanent,4294967295,worker,[rabbit_mgmt_db]}}]},{gen_server2,call,[<0.695.0>,{init,<0.693.0>},infinity]}}}}},{rabbit_mgmt_app,start,[normal,[]]}}}}}

Thanks,

Shafeek

[Michael Klishin]

 On 23 December 2014 at 15:41:06, Shafeek VK (shaf...@gmail.com) wrote:
> I have a cluster setup of two nodes. Node 1 is started, but node
> 2 cannot be started if the management plugin is enabled.
> I can start the server if the plugin is disabled. (Then web API
> is not available).

Try 3.4.2 first.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Shafeek VK]

3.3.4 installed in my server, is there any work around to solve this in 3.3.4?

I was able to start and was working on it.

[Giri Chintala]

MK, 

     What could be the root cause of this error?. i got this error today, but was able to successfully start in second try.

Error description:

   {could_not_start,rabbitmq_management,

       {{shutdown,

            {failed_to_start_child,rabbit_mgmt_sup,

                {'EXIT',

                    {{shutdown,

                         [{{already_started,<8663.1021.0>},

                           {child,undefined,rabbit_mgmt_db,

                               {rabbit_mgmt_db,start_link,[]},

                               permanent,4294967295,worker,

                               [rabbit_mgmt_db]}}]},

                     {gen_server2,call,

                         [<0.403.0>,{init,<0.401.0>},infinity]}}}}},

        {rabbit_mgmt_app,start,[normal,[]]}}}

[Michael Klishin]

Management DB tried to start but found that another copy is running.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Giri Chintala]

i am using rabbit 3.5.6, it is the first time i encountered this issue in the last couple of months.  

is it a know issue in 3.5.6 or below? i am trying to re-produce it but not successful yet.

[Michael Klishin]

In some cases it is a natural race condition. Don't

worry too much about it, management plugin is getting a significant revamp internally quite soon.

[Giri Chintala]

do you recommend any temporary workaround? 

i am consistently getting this error on one node.

Error description:

   {could_not_start,rabbitmq_management,

       {{shutdown,

            {failed_to_start_child,rabbit_mgmt_sup,

                {'EXIT',

                    {{shutdown,

                         [{{already_started,<14730.1317.0>},

                           {child,undefined,rabbit_mgmt_db,

                               {rabbit_mgmt_db,start_link,[]},

                               permanent,4294967295,worker,

                               [rabbit_mgmt_db]}}]},

                     {gen_server2,call,

                         [<0.341.0>,{init,<0.339.0>},infinity]}}}}},

        {rabbit_mgmt_app,start,[normal,[]]}}}

{"init terminating in do_boot",{could_not_start,rabbitmq_management,{{shutdown,{failed_to_start_child,rabbit_mgmt_sup,{'EXIT',{{shutdown,[{{already_started,<14730.1317.0>},{child,undefined,rabbit_mgmt_db,{rabbit_mgmt_db,start_link,[]},permanent,4294967295,worker,[rabbit_mgmt_db]}}]},{gen_server2,call,[<0.341.0>,{init,<0.339.0>},infinity]}}}}},{rabbit_mgmt_app,start,[normal,[]]}}}}

[Michael Klishin]

See how to restart management db at the bottom of rabbitmq.com/management.html

[Giri Chintala]

MK, 

    rabbitmqctl doesn't work as the rabbit node it down.

    

$ rabbitmqctl eval 'exit(erlang:whereis(rabbit_mgmt_db), please_terminate).'

Error: unable to connect to node 'rabbit@xxxxxxx': nodedown

Segmentation fault

[Giri Chintala]

i resolved it by disable and re-enable mgmt_plugin. 

[Alexey Lebedeff]

Hi,

There is a patch for this issue, it's included in 3.5.7 and later versions:

https://github.com/rabbitmq/rabbitmq-management/commit/27fcfd890380f08ddab86daa55cf262a2151a09b

With this patch inability to start management DB is no longer considered
a fatal error that should prevent node startup.

Giri Chintala writes:

> do you recommend any temporary workaround?
>
> i am consistently getting this error on one node.
>
> Error description:
> {could_not_start,rabbitmq_management,
> {{shutdown,
> {failed_to_start_child,rabbit_mgmt_sup,
> {'EXIT',
> {{shutdown,
> [{{already_started,<14730.1317.0>},
> {child,undefined,rabbit_mgmt_db,
> {rabbit_mgmt_db,start_link,[]},
> permanent,4294967295,worker,
> [rabbit_mgmt_db]}}]},
> {gen_server2,call,
> [<0.341.0>,{init,<0.339.0>},infinity]}}}}},
> {rabbit_mgmt_app,start,[normal,[]]}}}
>
>
> {"init terminating in
> do_boot",{could_not_start,rabbitmq_management,{{shutdown,{failed_to_start_child,rabbit_mgmt_sup,{'EXIT',{{shutdown,[{{already_started,<14730.1317.0>},{child,undefined,rabbit_mgmt_db,{rabbit_mgmt_db,start_link,[]},permanent,4294967295,worker,[rabbit_mgmt_db]}}]},{gen_server2,call,[<0.341.0>,{init,<0.339.0>},infinity]}}}}},{rabbit_mgmt_app,start,[normal,[]]}}}}
>
>
>
> On Thursday, February 4, 2016 at 1:30:43 PM UTC-5, Michael Klishin wrote:
>>
>> In some cases it is a natural race condition. Don't
>> worry too much about it, management plugin is getting a significant revamp
>> internally quite soon.
>>
>> On 4 feb 2016, at 20:53, Giri Chintala <chintala....@gmail.com

>> email to rabbitmq-user...@googlegroups.com <javascript:>.

>> To post to this group, send email to rabbitm...@googlegroups.com

>> <javascript:>.

[Giri Chintala]

i encountered different error on mgmt , while i am trying to test this in 3.5.7 by reproduce similar scenario, but i never seen this before on 3.5.6

=INFO REPORT==== 8-Feb-2016::14:26:07 ===

autocluster: Registering node with consul

=INFO REPORT==== 8-Feb-2016::14:26:07 ===

Resetting Rabbit

=INFO REPORT==== 8-Feb-2016::14:26:07 ===

Clustering with ['rabbit@ip-xxx126','rabbit@ip-xxx-125',

                 'rabbit@ip-xxx124','rabbit@ip-xxx-115',

                 'rabbit@ip-xxx-108','rabbit@ip-xxx-107',

                 'rabbit@ip-xxx-106','rabbit@ip-xxx-104',

                 'rabbit@ip-xxx-100'] as disc node

=ERROR REPORT==== 8-Feb-2016::14:29:20 ===

Mnesia('rabbit@ip-xxx-120'): ** ERROR ** (core dumped to file: "/var/lib/rabbitmq/MnesiaCore.rabbit@ip-xxx-120_1454_959760_750877")

:$

=INFO REPORT==== 8-Feb-2016::14:29:03 ===

Setting user tags for user 'ism-manage-api' to [administrator]

=ERROR REPORT==== 8-Feb-2016::14:29:20 ===

Mnesia('rabbit@ip-xxx-120'): ** ERROR ** (core dumped to file: "/var/lib/rabbitmq/MnesiaCore.rabbit@ip-xxx-120_1454_959760_750877")

 ** FATAL ** Failed to merge schema: Incompatible schema storage types (remote). on 'rabbit@ip-xxx-120' cs [{name,schema},{type,set},{ram_copies,['rabbit@ip-xxx-123']},{disc_copies,['rabbit@ip-xxx-120','rabbit@ip-xxx-126','rabbit@ip-xxx-125','rabbit@ip-xxx-124','rabbit@ip-xxx-115','rabbit@ip-xxx-108','rabbit@ip-xxx-107','rabbit@ip-xxx-106','rabbit@ip-xxx-104','rabbit@ip-xxx-100']},{disc_only_copies,[]},{load_order,0},{access_mode,read_write},{majority,false},{index,[]},{snmp,[]},{local_content,false},{record_name,schema},{attributes,[table,cstruct]},{user_properties,[]},{frag_properties,[]},{storage_properties,[]},{cookie,{{1454956622666707308,-576460752303423475,1},'rabbit@ip-xxx-115'}},{version,{{16,1},{'rabbit@ip-xxx-120',{1454,959712,421956}}}}], on 'rabbit@ip-xxx-123' rcs [{name,schema},{type,set},{ram_copies,[]},{disc_copies,['rabbit@ip-xxx-123','rabbit@ip-xxx-120','rabbit@ip-xxx-126','rabbit@ip-xxx-125','rabbit@ip-xxx-124','rabbit@ip-xxx-115','rabbit@ip-xxx-108','rabbit@ip-xxx-107','rabbit@ip-xxx-106','rabbit@ip-xxx-104','rabbit@ip-xxx-100']},{disc_only_copies,[]},{load_order,0},{access_mode,read_write},{majority,false},{index,[]},{snmp,[]},{local_content,false},{record_name,schema},{attributes,[table,cstruct]},{user_properties,[]},{frag_properties,[]},{storage_properties,[]},{cookie,{{1454956622666707308,-576460752303423475,1},'rabbit@ip-xxx-115'}},{version,{{16,2},{'rabbit@ip-xxx-123',{1454,959759,926319}}}}]

=ERROR REPORT==== 8-Feb-2016::14:29:30 ===

** Generic server mnesia_monitor terminating

** Last message in was {'EXIT',<0.701.0>,killed}

** When Server state == {state,<0.701.0>,[],[],true,[],undefined,[],[]}

** Reason for termination ==

** killed

=ERROR REPORT==== 8-Feb-2016::14:29:30 ===

** Generic server mnesia_recover terminating

** Last message in was {'EXIT',<0.701.0>,killed}

** When Server state == {state,<0.701.0>,undefined,undefined,undefined,0,

                               false,true,[]}

** Reason for termination ==

** killed

=ERROR REPORT==== 8-Feb-2016::14:29:30 ===

** Generic server mnesia_snmp_sup terminating

** Last message in was {'EXIT',<0.701.0>,killed}

** When Server state == {state,

                            {local,mnesia_snmp_sup},

[Giri Chintala]

My environment looks. 

rabbitmq-server 3.5.7-1
erlang/18.2-1

rabbitmq-autocluster/0.4.1

I got below error while adding new( brand new EC2 instance) nodes to existing cluster.