RabbitMQ corrupt queue?

Mike Roest

unread,

Aug 16, 2016, 2:28:57 PM8/16/16

to rabbitmq-users

We have a RabbitMQ cluster with 2 nodes. We had a queue that I tried to delete (that queue contained a half a dozen bindings) . The delete seems to have failed in some way. The queue still shows in the web admin but not via rabbitmqctl list_queues.

I tried to delete the queue again via pika and that just hung with no response. The queue is only showing in the webui now as belonging on 1 node instead of existing on 2 which it did before I tried to delete the queue. I thought maybe it was just the one node that was in a bad state so I restarted the node to try to get it to clear out. But on startup I get this error in the startup_log

## ## Licensed under the MPL. See http://www.rabbitmq.com/

## ##

########## Logs: /var/log/rabbitmq/rab...@rabbitmq1.log

###### ## /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

##########

Starting broker...

BOOT FAILED

===========

Error description:

{error,{{badmatch,{error,[{<<149,242,33,30,66,247,121,151,21,73,228,180,181,

31,120,103>>,

"replicon"},

{<<139,12,224,72,32,237,89,121,247,232,1,240,160,

86,156,211>>,

"*"},

{<<135,223,75,10,43,90,124,155,128,187,197,170,87,

190,157,35>>,

"tenant"},

{root,none}],

["user","policy-set-assignment","modified"]}},

[{rabbit_exchange_type_topic,follow_down_get_path,2},

{rabbit_exchange_type_topic,'-remove_bindings/3-lc$^1/1-1-',1},

{rabbit_exchange_type_topic,remove_bindings,3},

{rabbit_binding,x_callback,4},

{rabbit_binding,'-process_deletions/1-fun-0-',2},

{dict,map_bucket,2},

{dict,map_bkt_list,2},

{dict,map_bkt_list,2}]}}

Log files (may contain more information):

/var/log/rabbitmq/rab...@rabbitmq1.log

/var/log/rabbitmq/rab...@rabbitmq1-sasl.log

Stack trace:

[{rabbit_misc,execute_mnesia_transaction,1},

{rabbit_misc,execute_mnesia_tx_with_tail,1},

{rabbit_amqqueue,recover,0},

{rabbit,recover,0},

{rabbit,'-run_boot_step/1-lc$^1/1-1-',1},

{rabbit,run_boot_step,1},

{rabbit,'-start/2-lc$^0/1-0-',1},

{rabbit,start,2}]

BOOT FAILED

===========

Error description:

{could_not_start,rabbit,

{bad_return,

{{rabbit,start,[normal,[]]},

{'EXIT',

{rabbit,failure_during_boot,

{error,

{{badmatch,

{error,

[{<<149,242,33,30,66,247,121,151,21,73,228,

180,181,31,120,103>>,

"replicon"},

{<<139,12,224,72,32,237,89,121,247,232,1,240,

160,86,156,211>>,

"*"},

{<<135,223,75,10,43,90,124,155,128,187,197,

170,87,190,157,35>>,

"tenant"},

{root,none}],

["user","policy-set-assignment","modified"]}},

[{rabbit_exchange_type_topic,follow_down_get_path,2},

{rabbit_exchange_type_topic,

'-remove_bindings/3-lc$^1/1-1-',1},

{rabbit_exchange_type_topic,remove_bindings,3},

{rabbit_binding,x_callback,4},

{rabbit_binding,'-process_deletions/1-fun-0-',2},

{dict,map_bucket,2},

{dict,map_bkt_list,2},

{dict,map_bkt_list,2}]}}}}}}}

Log files (may contain more information):

/var/log/rabbitmq/rab...@rabbitmq1.log

/var/log/rabbitmq/rab...@rabbitmq1-sasl.log

{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{error,{{badmatch,{error,[{<<16 bytes>>,"replicon"},{<<16 bytes>>,"*"},{<<16 bytes>>,"tenant"},{root,none}],["us

er","policy-set-assignment","modified"]}},[{rabbit_exchange_type_topic,follow_down_get_path,2},{rabbit_exchange_type_topic,'-remove_bindings/3-lc$^1/1-1-',1},{rabbit_exchange_type_topic,remove_bindings,3},{rabbit_binding,x_callback,4},{rabbit_binding,'-process_deletions/

1-fun-0-',2},{dict,map_bucket,2},{dict,map_bkt_list,2},{dict,map_bkt_list,2}]}}}}}}}}}

[FAILED]

I've tried clearing the mnesia DB for the node that is down but it's still not able to start and fails with the same error.

The UI shows that there are 9 messages in that queue (I've tried to fetch and purge the queue with no luck). Looking through the log I do see a few errors:

On the node that's up I do see some errors related to the queue

=ERROR REPORT==== 16-Aug-2016::17:45:05 ===

** Generic server <0.7689.125> terminating

** Last message in was {'$gen_cast',

{method,

{'queue.unbind',0,

<<"work-queue.saas-batch[3.0.0.286]">>,

<<"events">>,

<<"tenant.*.replicon.user.modification.modified">>,

[]},

none,noflow}}

** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.7686.125>,

<0.7686.125>,<0.7680.125>,

<<"<rab...@rabbitmq2.1.7680.125>">>,

{lstate,<0.7688.125>,false,false},

none,1,

{[],[]},

{user,<<"rabbitmq">>,

[administrator],

rabbit_auth_backend_internal,

{internal_user,<<"rabbitmq">>,

<<219,169,247,94,236,178,254,52,252,23,166,

75,88,132,138,62,138,189,132,17>>,

[administrator]}},

<<"/">>,<<>>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{state,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[],[]}}},

erlang},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{set,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{set,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

<0.7683.125>,

{state,fine,5000,#Ref<0.0.2035.52178>},

false,1,

{{0,nil},{0,nil}},

[],

[{<<"publisher_confirms">>,bool,true},

{<<"exchange_exchange_bindings">>,bool,true},

{<<"basic.nack">>,bool,true},

{<<"consumer_cancel_notify">>,bool,true},

{<<"connection.blocked">>,bool,true},

{<<"authentication_failure_close">>,bool,true}],

none}

** Reason for termination ==

** {{error,{{badmatch,{error,[{<<149,242,33,30,66,247,121,151,21,73,228,180,

181,31,120,103>>,

"replicon"},

{<<139,12,224,72,32,237,89,121,247,232,1,240,160,

86,156,211>>,

"*"},

{<<135,223,75,10,43,90,124,155,128,187,197,170,

87,190,157,35>>,

"tenant"},

{root,none}],

["user","modification","modified"]}},

[{rabbit_exchange_type_topic,follow_down_get_path,2},

{rabbit_exchange_type_topic,'-remove_bindings/3-lc$^1/1-1-',1},

{rabbit_exchange_type_topic,remove_bindings,3},

{rabbit_binding,x_callback,4},

{rabbit_binding,'-process_deletions/1-fun-0-',2},

{dict,map_bucket,2},

{dict,map_bkt_list,2},

{dict,map_bkt_list,2}]}},

[{rabbit_misc,execute_mnesia_transaction,1},

{rabbit_misc,execute_mnesia_tx_with_tail,1},

{rabbit_channel,binding_action,9},

{rabbit_channel,handle_cast,2},

{gen_server2,handle_msg,2},

{proc_lib,init_p_do_apply,3}]}

Anyone have any ideas on how I can recover this cluster? without having to flush everything.

RabbitMQ 3.2.3 on centos 6.5

Thanks

Michael Klishin

unread,

Aug 16, 2016, 2:49:35 PM8/16/16

to rabbitm...@googlegroups.com

This is bug 26014 in http://www.rabbitmq.com/release-notes/README-3.2.4.txt.

You should consider upgrading to 3.6.5 or at least 3.5.7.

On Tue, Aug 16, 2016 at 9:28 PM, Mike Roest <mike....@replicon.com> wrote:

We have a RabbitMQ cluster with 2 nodes. We had a queue that I tried to delete (that queue contained a half a dozen bindings) . The delete seems to have failed in some way. The queue still shows in the web admin but not via rabbitmqctl list_queues.
I tried to delete the queue again via pika and that just hung with no response. The queue is only showing in the webui now as belonging on 1 node instead of existing on 2 which it did before I tried to delete the queue. I thought maybe it was just the one node that was in a bad state so I restarted the node to try to get it to clear out. But on startup I get this error in the startup_log

RabbitMQ 3.2.3. Copyright (C) 2007-2013 GoPivotal, Inc.
## ## Licensed under the MPL. See http://www.rabbitmq.com/
## ##

########## Logs: /var/log/rabbitmq/rabbit@rabbitmq1.log
###### ## /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

/var/log/rabbitmq/rabbit@rabbitmq1.log
/var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

Michael Klishin

unread,

Aug 16, 2016, 2:55:43 PM8/16/16

to rabbitm...@googlegroups.com

You can try

`rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<“VHOST">>, queue, <<“QUEUE_NAME">>)), rabbit_amqqueue:delete_crashed(Q).’`

and then upgrade to 3.2.4 or 3.5.7 or 3.6.5 when you can.

I'm not 100% sure that the above functions are available in 3.2.x but they likely are.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Mike Roest

unread,

Aug 16, 2016, 3:04:35 PM8/16/16

to rabbitm...@googlegroups.com

I ran that command

sudo rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"/">>, queue, <<"work-queue.saas-batch[3.0.0.286]">>)), rabbit_amqqueue:delete_crashed(Q).'

and got

Error: {undef,

[{rabbit_amqqueue,delete_crashed,

[{amqqueue,

{resource,<<"/">>,queue,

<<"work-queue.saas-batch[3.0.0.286]">>},

true,false,none,[],<5746.3036.701>,[],[],

[{vhost,<<"/">>},

{name,<<"ha-all">>},

{pattern,<<".*">>},

{'apply-to',<<"all">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[],[]}]},

{erl_eval,do_apply,5},

{rpc,'-handle_call_call/6-fun-0-',5}]}

Based on the output it looks like something there didn't work.

Any ideas as to why the bindings in question would show up in the adminUI (from both the queue and exchange details pages) but cause this error when trying to unbind them. The bindings are getting created with the .Net RabbitMQ.Client v3.2.4. The issue we were having when trying to track this down is messages that appeared to not be getting into a queue even though there was a binding that appeared as if it should work (a similar binding created through the UI was picking up messages in a different queue created for debugging.

We'll get on the upgrade here as well but at this point I'm in disaster recovery mode as I have this one cluster degraded. And another cluster where we've done the unbind command and gotten the process crash and now I'm concerned that it will also have this same startup problem if it has to restart or our system restarts and tries to delete the queue that now has this invalid binding delete command pending.

--Mike

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Data's inconvienient when people have opinions.

Mike Roest

unread,

Aug 16, 2016, 3:23:50 PM8/16/16

to rabbitm...@googlegroups.com

I've tried recreating the binding from the UI (it returns a 201 created status code). But the down server still fails to startup (I removed the mnesia dir on that node).

I've also run the same on the other server where we sent a unbind command but have not tried deleting the queue or restarting any of the cluster nodes so although it still exhibits the behavior of the queue not receiving the message it is at least up and running with both cluster nodes (but I don't know what state it will get into if we deploy a new version of our component (which will try to delete that queue) or end up having to restart one of the cluster nodes).

--

Data's inconvienient when people have opinions.

Mike Roest

unread,

Aug 16, 2016, 3:34:36 PM8/16/16

to rabbitm...@googlegroups.com

Another interesting piece of data from the issue where the binding doesn't appear to be created correctly. We have 9 individual production environments and numerous staging environments all running the same 3.2.3 version of rabbit and only in 2 environments do we see the issue. The others the binding appears to be functioning as intended

So something seems to be creating these "invalid" bindings either intermittently or in response to something on particular clusters (we use puppet to create the clusters so they all should be basically identical).

Michael Klishin

unread,

Aug 16, 2016, 3:53:49 PM8/16/16

to rabbitm...@googlegroups.com

"undef" means that the function doesn't exist in your version of RabbitMQ.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Mike Roest

unread,

Aug 16, 2016, 3:58:57 PM8/16/16

to rabbitm...@googlegroups.com

ok at this point then is the only way to recover is to basically rebuild the cluster?

Michael Klishin

unread,

Aug 16, 2016, 4:09:13 PM8/16/16

to rabbitm...@googlegroups.com, Mike Roest

It's not clear whether you have tried upgrading to 3.2.4 first.

The issue comes down to edge cases handling in binding removal so I'd expect 3.2.4
to be able to start.

On 16 August 2016 at 22:58:58, Mike Roest (mike....@replicon.com) wrote:
> ok at this point then is the only way to recover is to basically rebuild
> the cluster?
>
> On Tue, Aug 16, 2016 at 1:53 PM, Michael Klishin

> wrote:
>
> > "undef" means that the function doesn't exist in your version of RabbitMQ.
> >
> >
> >
> > On Tue, Aug 16, 2016 at 10:34 PM, Mike Roest

> > wrote:
> >
> >> Another interesting piece of data from the issue where the binding
> >> doesn't appear to be created correctly. We have 9 individual production
> >> environments and numerous staging environments all running the same 3.2.3
> >> version of rabbit and only in 2 environments do we see the issue. The
> >> others the binding appears to be functioning as intended
> >>
> >> So something seems to be creating these "invalid" bindings either
> >> intermittently or in response to something on particular clusters (we use
> >> puppet to create the clusters so they all should be basically identical).
> >>
> >>
> >> On Tue, Aug 16, 2016 at 1:23 PM, Mike Roest

> >> wrote:
> >>
> >>> I've tried recreating the binding from the UI (it returns a 201 created
> >>> status code). But the down server still fails to startup (I removed the
> >>> mnesia dir on that node).
> >>>
> >>> I've also run the same on the other server where we sent a unbind
> >>> command but have not tried deleting the queue or restarting any of the
> >>> cluster nodes so although it still exhibits the behavior of the queue not
> >>> receiving the message it is at least up and running with both cluster nodes
> >>> (but I don't know what state it will get into if we deploy a new version of
> >>> our component (which will try to delete that queue) or end up having to
> >>> restart one of the cluster nodes).
> >>>
> >>>
> >>>
> >>> On Tue, Aug 16, 2016 at 1:04 PM, Mike Roest

> >>>> wrote:
> >>>>
> >>>>> You can try
> >>>>>
> >>>>> `rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<“VHOST">>,
> >>>>> queue, <<“QUEUE_NAME">>)), rabbit_amqqueue:delete_crashed(Q).’`
> >>>>>
> >>>>> and then upgrade to 3.2.4 or 3.5.7 or 3.6.5 when you can.
> >>>>>
> >>>>> I'm not 100% sure that the above functions are available in 3.2.x but
> >>>>> they likely are.
> >>>>>
> >>>>> On Tue, Aug 16, 2016 at 9:49 PM, Michael Klishin

> >>>>> wrote:
> >>>>>
> >>>>>> This is bug 26014 in http://www.rabbitmq.com/rel
> >>>>>> ease-notes/README-3.2.4.txt.
> >>>>>>
> >>>>>> You should consider upgrading to 3.6.5 or at least 3.5.7.
> >>>>>>
> >>>>>> On Tue, Aug 16, 2016 at 9:28 PM, Mike Roest

> >>>>>> wrote:
> >>>>>>
> >>>>>>> We have a RabbitMQ cluster with 2 nodes. We had a queue that I
> >>>>>>> tried to delete (that queue contained a half a dozen bindings) . The
> >>>>>>> delete seems to have failed in some way. The queue still shows in the web
> >>>>>>> admin but not via rabbitmqctl list_queues.
> >>>>>>> I tried to delete the queue again via pika and that just hung with
> >>>>>>> no response. The queue is only showing in the webui now as belonging on 1
> >>>>>>> node instead of existing on 2 which it did before I tried to delete the
> >>>>>>> queue. I thought maybe it was just the one node that was in a bad state so
> >>>>>>> I restarted the node to try to get it to clear out. But on startup I get
> >>>>>>> this error in the startup_log
> >>>>>>>
> >>>>>>> RabbitMQ 3.2.3. Copyright (C) 2007-2013 GoPivotal, Inc.
> >>>>>>> ## ## Licensed under the MPL. See http://www.rabbitmq.com/
> >>>>>>> ## ##

> >>>>>>> ########## Logs: /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>> ###### ## /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>> /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>> /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>> follow_down_get_path,2},
> >>>>>>> {rabbit_exchange_type_topic,
> >>>>>>> '-remove_bindings/3-lc$^1/1-1-',1},
> >>>>>>> {rabbit_exchange_type_topic,re
> >>>>>>> move_bindings,3},

> >>>>>>> {rabbit_binding,x_callback,4},
> >>>>>>> {rabbit_binding,'-process_dele
> >>>>>>> tions/1-fun-0-',2},

> >>>>>>> {dict,map_bucket,2},
> >>>>>>> {dict,map_bkt_list,2},
> >>>>>>> {dict,map_bkt_list,2}]}}}}}}}
> >>>>>>>
> >>>>>>> Log files (may contain more information):

> >>>>>>> /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>> /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>> send an email to rabbitmq-user...@googlegroups.com.

> >>>>>>> To post to this group, send email to rabbitm...@googlegroups.com
> >>>>>>> .
> >>>>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> MK
> >>>>>>
> >>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> MK
> >>>>>
> >>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>
> >>>>> --
> >>>>> You received this message because you are subscribed to the Google
> >>>>> Groups "rabbitmq-users" group.
> >>>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>>> an email to rabbitmq-user...@googlegroups.com.

> >>>>> To post to this group, send email to rabbitm...@googlegroups.com.
> >>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Data's inconvienient when people have opinions.
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Data's inconvienient when people have opinions.
> >>>
> >>
> >>
> >>
> >> --
> >> Data's inconvienient when people have opinions.
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "rabbitmq-users" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to rabbitmq-user...@googlegroups.com.

> >> To post to this group, send email to rabbitm...@googlegroups.com.
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> >
> >
> > --
> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "rabbitmq-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to rabbitmq-user...@googlegroups.com.

> > To post to this group, send email to rabbitm...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
>
>
>
> --
> Data's inconvienient when people have opinions.
>
> --

> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.

Michael Klishin

unread,

Aug 16, 2016, 4:09:32 PM8/16/16

to rabbitm...@googlegroups.com

Looking at the 3.2.3 code,

`rabbitmqctl eval '{ok, {amqqueue, _, _, _, _, _, Pid, _, _, _, _, _, _, _}} = rabbit_amqqueue:lookup(rabbit_misc:r(<<“VHOST">>, queue, <<“QUEUE_NAME">>)), rabbit_amqqueue:delete_immediately([Pid]).’`

might work.

Mike Roest

unread,

Aug 16, 2016, 4:18:31 PM8/16/16

to rabbitm...@googlegroups.com

That eval also failed.

Error: {{badmatch,

{ok,{amqqueue,

{resource,<<"/">>,queue,

<<"work-queue.saas-batch[3.0.0.286]">>},

true,false,none,[],<5746.3036.701>,[],[],

[{vhost,<<"/">>},

{name,<<"ha-all">>},

{pattern,<<".*">>},

{'apply-to',<<"all">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,0}],

[],[]}}},

[{erl_eval,expr,3}]}

I just updated to 3.2.4 on the down server and it still fails to start up

## ## Licensed under the MPL. See http://www.rabbitmq.com/

## ##

########## Logs: /var/log/rabbitmq/rab...@rabbitmq1.log

###### ## /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

##########

Starting broker...

BOOT FAILED

===========

Error description:

{error,{{badmatch,{error,[{<<188,169,169,0,103,98,111,64,60,75,49,4,225,243,

167,138>>,

"user"},

{<<149,242,33,30,66,247,121,151,21,73,228,180,181,

31,120,103>>,

"replicon"},

{<<139,12,224,72,32,237,89,121,247,232,1,240,160,

86,156,211>>,

"*"},

{<<135,223,75,10,43,90,124,155,128,187,197,170,87,

190,157,35>>,

"tenant"},

{root,none}],

["policy-set-assignment","modified"]}},

[{rabbit_exchange_type_topic,follow_down_get_path,2},

{rabbit_exchange_type_topic,'-remove_bindings/3-lc$^1/1-1-',1},

{rabbit_exchange_type_topic,remove_bindings,3},

{rabbit_binding,x_callback,4},

{rabbit_binding,'-process_deletions/1-fun-0-',2},

{dict,map_bucket,2},

{dict,map_bkt_list,2},

{dict,map_bkt_list,2}]}}

Log files (may contain more information):

/var/log/rabbitmq/rab...@rabbitmq1.log

/var/log/rabbitmq/rab...@rabbitmq1-sasl.log

Stack trace:

[{rabbit_misc,execute_mnesia_transaction,1},

{rabbit_misc,execute_mnesia_tx_with_tail,1},

{rabbit_amqqueue,recover,0},

{rabbit,recover,0},

{rabbit,'-run_boot_step/1-lc$^1/1-1-',1},

{rabbit,run_boot_step,1},

{rabbit,'-start/2-lc$^0/1-0-',1},

{rabbit,start,2}]

BOOT FAILED

===========

Error description:

{could_not_start,rabbit,

{bad_return,

{{rabbit,start,[normal,[]]},

{'EXIT',

{rabbit,failure_during_boot,

{error,

{{badmatch,

{error,

[{<<188,169,169,0,103,98,111,64,60,75,49,4,

225,243,167,138>>,

"user"},

{<<149,242,33,30,66,247,121,151,21,73,228,

180,181,31,120,103>>,

"replicon"},

{<<139,12,224,72,32,237,89,121,247,232,1,240,

160,86,156,211>>,

"*"},

{<<135,223,75,10,43,90,124,155,128,187,197,

170,87,190,157,35>>,

"tenant"},

{root,none}],

["policy-set-assignment","modified"]}},

[{rabbit_exchange_type_topic,follow_down_get_path,2},

{rabbit_exchange_type_topic,

'-remove_bindings/3-lc$^1/1-1-',1},

{rabbit_exchange_type_topic,remove_bindings,3},

{rabbit_binding,x_callback,4},

{rabbit_binding,'-process_deletions/1-fun-0-',2},

{dict,map_bucket,2},

{dict,map_bkt_list,2},

{dict,map_bkt_list,2}]}}}}}}}

Log files (may contain more information):

/var/log/rabbitmq/rab...@rabbitmq1.log

/var/log/rabbitmq/rab...@rabbitmq1-sasl.log

{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{error,{{badmatch,{error,[{<<16 bytes>>,"user"},{<<16 bytes>>,"replicon"},{<<16 bytes>>,"*"},{<<16 bytes>>,"tena

nt"},{root,none}],["policy-set-assignment","modified"]}},[{rabbit_exchange_type_topic,follow_down_get_path,2},{rabbit_exchange_type_topic,'-remove_bindings/3-lc$^1/1-1-',1},{rabbit_exchange_type_topic,remove_bindings,3},{rabbit_binding,x_callback,4},{rabbit_binding,'-pro

cess_deletions/1-fun-0-',2},{dict,map_bucket,2},{dict,map_bkt_list,2},{dict,map_bkt_list,2}]}}}}}}}}}

[FAILED]

Michael Klishin

unread,

Aug 16, 2016, 4:45:30 PM8/16/16

to rabbitm...@googlegroups.com, Mike Roest

`rabbitmqctl eval '{ok, {amqqueue, _, _ , _, _, _, Pid, _, _, _, _, _}} = rabbit_amqqueue:lookup(rabbit_misc:r(<<“VHOST">>, queue, <<“QUEUE_NAME">>)), rabbit_amqqueue:delete_immediately([Pid]).’`

> wrote:
>
> > Looking at the 3.2.3 code,
> >
> > `rabbitmqctl eval '{ok, {amqqueue, _, _, _, _, _, Pid, _, _, _, _, _, _,
> > _}} = rabbit_amqqueue:lookup(rabbit_misc:r(<<“VHOST">>, queue,
> > <<“QUEUE_NAME">>)), rabbit_amqqueue:delete_immediately([Pid]).’`
> >
> > might work.
> >
> >
> > On Tue, Aug 16, 2016 at 10:58 PM, Mike Roest

> > wrote:
> >
> >> ok at this point then is the only way to recover is to basically rebuild
> >> the cluster?
> >>
> >> On Tue, Aug 16, 2016 at 1:53 PM, Michael Klishin

> >> wrote:
> >>
> >>> "undef" means that the function doesn't exist in your version of
> >>> RabbitMQ.
> >>>
> >>>
> >>>
> >>> On Tue, Aug 16, 2016 at 10:34 PM, Mike Roest

> >>> wrote:
> >>>
> >>>> Another interesting piece of data from the issue where the binding
> >>>> doesn't appear to be created correctly. We have 9 individual production
> >>>> environments and numerous staging environments all running the same 3.2.3
> >>>> version of rabbit and only in 2 environments do we see the issue. The
> >>>> others the binding appears to be functioning as intended
> >>>>
> >>>> So something seems to be creating these "invalid" bindings either
> >>>> intermittently or in response to something on particular clusters (we use
> >>>> puppet to create the clusters so they all should be basically identical).
> >>>>
> >>>>
> >>>> On Tue, Aug 16, 2016 at 1:23 PM, Mike Roest

> >>>> wrote:
> >>>>
> >>>>> I've tried recreating the binding from the UI (it returns a 201
> >>>>> created status code). But the down server still fails to startup (I
> >>>>> removed the mnesia dir on that node).
> >>>>>
> >>>>> I've also run the same on the other server where we sent a unbind
> >>>>> command but have not tried deleting the queue or restarting any of the
> >>>>> cluster nodes so although it still exhibits the behavior of the queue not
> >>>>> receiving the message it is at least up and running with both cluster nodes
> >>>>> (but I don't know what state it will get into if we deploy a new version of
> >>>>> our component (which will try to delete that queue) or end up having to
> >>>>> restart one of the cluster nodes).
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Aug 16, 2016 at 1:04 PM, Mike Roest

> >>>>>>>>> ########## Logs: /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>>>> ###### ## /var/log/rabbitmq/rab...@rabbitmq1-sasl.log
> >>>>>>>>> ##########
> >>>>>>>>> Starting broker...
> >>>>>>>>>
> >>>>>>>>> BOOT FAILED
> >>>>>>>>> ===========
> >>>>>>>>>
> >>>>>>>>> Error description:

> >>>>>>>>> {error,{{badmatch,{error,[{<<149,242,33,30,66,247,121,151,2
> >>>>>>>>> 1,73,228,180,181,
> >>>>>>>>> 31,120,103>>,
> >>>>>>>>> "replicon"},
> >>>>>>>>> {<<139,12,224,72,32,237,89,12
> >>>>>>>>> 1,247,232,1,240,160,
> >>>>>>>>> 86,156,211>>,
> >>>>>>>>> "*"},
> >>>>>>>>> {<<135,223,75,10,43,90,124,15
> >>>>>>>>> 5,128,187,197,170,87,

> >>>>>>>>> 190,157,35>>,
> >>>>>>>>> "tenant"},
> >>>>>>>>> {root,none}],
> >>>>>>>>> ["user","policy-set-assignment
> >>>>>>>>> ","modified"]}},
> >>>>>>>>> [{rabbit_exchange_type_topic,follow_down_get_path,2},
> >>>>>>>>> {rabbit_exchange_type_topic,'-

> >>>>>>>>> [{<<149,242,33,30,66,247,121,
> >>>>>>>>> 151,21,73,228,
> >>>>>>>>> 180,181,31,120,103>>,
> >>>>>>>>> "replicon"},
> >>>>>>>>> {<<139,12,224,72,32,237,89,121
> >>>>>>>>> ,247,232,1,240,
> >>>>>>>>> 160,86,156,211>>,
> >>>>>>>>> "*"},
> >>>>>>>>> {<<135,223,75,10,43,90,124,155
> >>>>>>>>> ,128,187,197,

> >>>>>>>>> 170,87,190,157,35>>,
> >>>>>>>>> "tenant"},
> >>>>>>>>> {root,none}],

> >>>>>>>>> ["user","policy-set-assignmen
> >>>>>>>>> t","modified"]}},
> >>>>>>>>> [{rabbit_exchange_type_topic,

> >>>>>>>>> follow_down_get_path,2},
> >>>>>>>>> {rabbit_exchange_type_topic,
> >>>>>>>>> '-remove_bindings/3-lc$^1/1-1-',1},
> >>>>>>>>> {rabbit_exchange_type_topic,re
> >>>>>>>>> move_bindings,3},

> >>>>>>>>> {rabbit_binding,x_callback,4},
> >>>>>>>>> {rabbit_binding,'-process_dele
> >>>>>>>>> tions/1-fun-0-',2},

> >>>>>>>>> {dict,map_bucket,2},
> >>>>>>>>> {dict,map_bkt_list,2},
> >>>>>>>>> {dict,map_bkt_list,2}]}}}}}}}
> >>>>>>>>>
> >>>>>>>>> Log files (may contain more information):
> >>>>>>>>> /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>>>> /var/log/rabbitmq/rab...@rabbitmq1-sasl.log
> >>>>>>>>>

> >>>>>>>>> {"init terminating in do_boot",{rabbit,failure_durin
> >>>>>>>>> g_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[n

> >>>>>>>>> ormal,[]]},{'EXIT',{rabbit,failure_during_boot,{error,{{badmatch,{error,[{<<16
> >>>>>>>>> bytes>>,"replicon"},{<<16 bytes>>,"*"},{<<16 bytes>>,"tenant"},{root,none}]
> >>>>>>>>> ,["us

> >>>>>>>>> er","policy-set-assignment","modified"]}},[{rabbit_exchange_
> >>>>>>>>> type_topic,follow_down_get_path,2},{rabbit_exchange_type_top
> >>>>>>>>> ic,'-remove_bindings/3-lc$^1/1-1-',1},{rabbit_exchange_type_
> >>>>>>>>> topic,remove_bindings,3},{rabbit_binding,x_callback,4},{rabb

> >>>>>>>>> it_binding,'-process_deletions/
> >>>>>>>>> 1-fun-0-',2},{dict,map_bucket,2},{dict,map_bkt_list,2},{dict
> >>>>>>>>> ,map_bkt_list,2}]}}}}}}}}}
> >>>>>>>>> [FAILED]
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I've tried clearing the mnesia DB for the node that is down but
> >>>>>>>>> it's still not able to start and fails with the same error.
> >>>>>>>>>
> >>>>>>>>> The UI shows that there are 9 messages in that queue (I've tried
> >>>>>>>>> to fetch and purge the queue with no luck). Looking through the log I do
> >>>>>>>>> see a few errors:
> >>>>>>>>>
> >>>>>>>>> On the node that's up I do see some errors related to the queue
> >>>>>>>>>
> >>>>>>>>> =ERROR REPORT==== 16-Aug-2016::17:45:05 ===
> >>>>>>>>> ** Generic server <0.7689.125> terminating
> >>>>>>>>> ** Last message in was {'$gen_cast',
> >>>>>>>>> {method,
> >>>>>>>>> {'queue.unbind',0,

> >>>>>>>>> <<"work-queue.saas-batch[3.0.
> >>>>>>>>> 0.286]">>,
> >>>>>>>>> <<"events">>,
> >>>>>>>>> <<"tenant.*.replicon.user.mod

> >>>>>>>>> ification.modified">>,
> >>>>>>>>> []},
> >>>>>>>>> none,noflow}}
> >>>>>>>>> ** When Server state == {ch,running,rabbit_framing_amq
> >>>>>>>>> p_0_9_1,1,<0.7686.125>,
> >>>>>>>>> <0.7686.125>,<0.7680.125>,

> >>>>>>>>> ** {{error,{{badmatch,{error,[{<<149,242,33,30,66,247,121,151,2
> >>>>>>>>> 1,73,228,180,
> >>>>>>>>> 181,31,120,103>>,
> >>>>>>>>> "replicon"},
> >>>>>>>>> {<<139,12,224,72,32,237,89,121
> >>>>>>>>> ,247,232,1,240,160,
> >>>>>>>>> 86,156,211>>,
> >>>>>>>>> "*"},
> >>>>>>>>> {<<135,223,75,10,43,90,124,155
> >>>>>>>>> ,128,187,197,170,

> >>>>>>>>> send an email to rabbitmq-user...@googlegroups.com.

> >>>>>>>>> To post to this group, send email to
> >>>>>>>>> rabbitm...@googlegroups.com.
> >>>>>>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> MK
> >>>>>>>>
> >>>>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> MK
> >>>>>>>
> >>>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>
> >>>>>>> --
> >>>>>>> You received this message because you are subscribed to the Google
> >>>>>>> Groups "rabbitmq-users" group.
> >>>>>>> To unsubscribe from this group and stop receiving emails from it,

> >>>>>>> send an email to rabbitmq-user...@googlegroups.com.

> >>>>>>> To post to this group, send email to rabbitm...@googlegroups.com
> >>>>>>> .

> >>>>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Data's inconvienient when people have opinions.
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Data's inconvienient when people have opinions.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Data's inconvienient when people have opinions.
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "rabbitmq-users" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to rabbitmq-user...@googlegroups.com.

> >>>> To post to this group, send email to rabbitm...@googlegroups.com.
> >>>> For more options, visit https://groups.google.com/d/optout.
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> MK
> >>>
> >>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "rabbitmq-users" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send

> >>> an email to rabbitmq-user...@googlegroups.com.

> >>> To post to this group, send email to rabbitm...@googlegroups.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>
> >>
> >>
> >> --
> >> Data's inconvienient when people have opinions.
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "rabbitmq-users" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to rabbitmq-user...@googlegroups.com.

> >> To post to this group, send email to rabbitm...@googlegroups.com.
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> >
> >
> > --
> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "rabbitmq-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to rabbitmq-user...@googlegroups.com.

> > To post to this group, send email to rabbitm...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
>
>
>
> --
> Data's inconvienient when people have opinions.
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.

Mike Roest

unread,

Aug 16, 2016, 4:56:14 PM8/16/16

to Michael Klishin, rabbitm...@googlegroups.com

Well that eval worked (on the up server):

ok

...done.

But down server still fails to startup with or without removing the mnesia dir.

> ########## Logs: /var/log/rabbitmq/rabbit@rabbitmq1.log
> ###### ## /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> /var/log/rabbitmq/rabbit@rabbitmq1.log
> /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>>>> ########## Logs: /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>>>> ###### ## /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>>>> /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>>>> /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>>>> /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>>>> /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log
> >>>>>>>>>

> >>>>>>>>> send an email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>>>>>>>> To post to this group, send email to

> >>>>>>>>> rabbitmq-users@googlegroups.com.

> >>>>>>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> MK
> >>>>>>>>
> >>>>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> MK
> >>>>>>>
> >>>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>
> >>>>>>> --
> >>>>>>> You received this message because you are subscribed to the Google
> >>>>>>> Groups "rabbitmq-users" group.
> >>>>>>> To unsubscribe from this group and stop receiving emails from it,

> >>>>>>> send an email to rabbitmq-users+unsubscribe@googlegroups.com.
> >>>>>>> To post to this group, send email to rabbitmq-users@googlegroups.com

> >>>>>>> .
> >>>>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Data's inconvienient when people have opinions.
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Data's inconvienient when people have opinions.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Data's inconvienient when people have opinions.
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "rabbitmq-users" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to rabbitmq-users+unsubscribe@googlegroups.com.
> >>>> To post to this group, send email to rabbitmq-users@googlegroups.com.

> >>>> For more options, visit https://groups.google.com/d/optout.
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> MK
> >>>
> >>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "rabbitmq-users" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send

> >>> an email to rabbitmq-users+unsubscribe@googlegroups.com.
> >>> To post to this group, send email to rabbitmq-users@googlegroups.com.

> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>
> >>
> >>
> >> --
> >> Data's inconvienient when people have opinions.
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "rabbitmq-users" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to rabbitmq-users+unsubscribe@googlegroups.com.
> >> To post to this group, send email to rabbitmq-users@googlegroups.com.

> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> >
> >
> > --
> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "rabbitmq-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to rabbitmq-users+unsubscribe@googlegroups.com.
> > To post to this group, send email to rabbitmq-users@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
>
>
>
> --
> Data's inconvienient when people have opinions.
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

> To post to this group, send an email to rabbitmq-users@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

--

Michael Klishin

unread,

Aug 16, 2016, 5:03:56 PM8/16/16

to Mike Roest, rabbitm...@googlegroups.com

I find it very hard to believe that a node can fail to start with this very exception in the log with a blank

node data directory: there simply aren't any exchanges or bindings to load, or peers to sync from.

On Tue, Aug 16, 2016 at 11:56 PM, Mike Roest <mike....@replicon.com> wrote:

Well that eval worked (on the up server):

ok
...done.

But down server still fails to startup with or without removing the mnesia dir.

RabbitMQ 3.2.4. Copyright (C) 2007-2013 GoPivotal, Inc.
## ## Licensed under the MPL. See http://www.rabbitmq.com/
## ##

> >>>>>>>>> rabbitm...@googlegroups.com.

> >>>>>>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> MK
> >>>>>>>>
> >>>>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> MK
> >>>>>>>
> >>>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>
> >>>>>>> --
> >>>>>>> You received this message because you are subscribed to the Google
> >>>>>>> Groups "rabbitmq-users" group.
> >>>>>>> To unsubscribe from this group and stop receiving emails from it,
> >>>>>>> send an email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>>>>>> To post to this group, send email to rabbitm...@googlegroups.com

> >>>>>>> .
> >>>>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Data's inconvienient when people have opinions.
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Data's inconvienient when people have opinions.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Data's inconvienient when people have opinions.
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "rabbitmq-users" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send
> >>>> an email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>>> To post to this group, send email to rabbitm...@googlegroups.com.

> >>>> For more options, visit https://groups.google.com/d/optout.
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> MK
> >>>
> >>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "rabbitmq-users" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send
> >>> an email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>> To post to this group, send email to rabbitm...@googlegroups.com.

> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>
> >>
> >>
> >> --
> >> Data's inconvienient when people have opinions.
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "rabbitmq-users" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to rabbitmq-users+unsubscribe@googlegroups.com.

> >> To post to this group, send email to rabbitm...@googlegroups.com.

> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> >
> >
> > --
> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "rabbitmq-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to rabbitmq-users+unsubscribe@googlegroups.com.

> > To post to this group, send email to rabbitm...@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
>
>
>
> --
> Data's inconvienient when people have opinions.
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

> To post to this group, send an email to rabbitm...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

--
Data's inconvienient when people have opinions.

Michael Klishin

unread,

Aug 16, 2016, 5:04:36 PM8/16/16

to Mike Roest, rabbitm...@googlegroups.com

You can try deleting a single vhost the problematic exchange and bindings is/are in,

not everything.

Mike Roest

unread,

Aug 16, 2016, 5:10:58 PM8/16/16

to Michael Klishin, rabbitm...@googlegroups.com

Maybe I'm clearing the node data incorrectly. I'm doing

rm -rf /var/lib/rabbitmq/mnesia

on the down node.

How would I go about deleting the data for just that one vhost?

This is the contents of the mnesia directory on the down node after I attempt to start it

[msroest@rabbitmq1 mnesia]$ ls -l

total 8

drwxr-xr-x 2 rabbitmq rabbitmq 4096 Aug 16 21:07 rabbit@rabbitmq1

drwxr-xr-x 8 rabbitmq rabbitmq 4096 Aug 16 21:07 rabbit@rabbitmq1-plugins-expand

and then in the rabbit dir

[msroest@rabbitmq1 rabbit@rabbitmq1]$ ls -l

total 92

-rw-r--r-- 1 rabbitmq rabbitmq 75 Aug 16 21:07 cluster_nodes.config

-rw-r--r-- 1 rabbitmq rabbitmq 336 Aug 16 21:07 LATEST.LOG

-rw-r--r-- 1 rabbitmq rabbitmq 37 Aug 16 21:07 nodes_running_at_shutdown

-rw-r--r-- 1 rabbitmq rabbitmq 4602 Aug 16 21:07 rabbit_durable_exchange.DCD

-rw-r--r-- 1 rabbitmq rabbitmq 5222 Aug 16 21:07 rabbit_durable_queue.DCD

-rw-r--r-- 1 rabbitmq rabbitmq 12164 Aug 16 21:07 rabbit_durable_route.DCD

-rw-r--r-- 1 rabbitmq rabbitmq 297 Aug 16 21:07 rabbit_runtime_parameters.DCD

-rw-r--r-- 1 rabbitmq rabbitmq 3 Aug 16 21:07 rabbit_serial

-rw-r--r-- 1 rabbitmq rabbitmq 328 Aug 16 21:07 rabbit_user.DCD

-rw-r--r-- 1 rabbitmq rabbitmq 281 Aug 16 21:07 rabbit_user_permission.DCD

-rw-r--r-- 1 rabbitmq rabbitmq 127 Aug 16 21:07 rabbit_vhost.DCD

-rw-r--r-- 1 rabbitmq rabbitmq 27256 Aug 16 21:07 schema.DAT

-rw-r--r-- 1 rabbitmq rabbitmq 238 Aug 16 21:07 schema_version

We only have the default vhost.

Mike Roest

unread,

Aug 16, 2016, 5:13:21 PM8/16/16

to Michael Klishin, rabbitm...@googlegroups.com

Our rabbitmq.config is

[

{rabbit, [

{cluster_nodes,{['rabbit@rabbitmq1','rabbit@rabbitmq2'], disc}},

{frame_max,131072},

{collect_statistics_interval,5000},

{heartbeat,580}

]}

].

Michael Klishin

unread,

Aug 16, 2016, 5:33:11 PM8/16/16

to Mike Roest, rabbitm...@googlegroups.com

You are deleting node data correctly but then the node syncs from another

node.

Vhosts are deleted with rabbitmqctl, see its man page.

Mike Roest

unread,

Aug 16, 2016, 5:36:42 PM8/16/16

to Michael Klishin, rabbitm...@googlegroups.com

But deleting the vhost I would have to do on the running node correct? If I delete that vhost that's effectively the same as shutting down everything since it's the only vhost we have.

Mike Roest

unread,

Aug 17, 2016, 3:06:13 PM8/17/16

to Michael Klishin, rabbitm...@googlegroups.com

Any other possibility's you can think of for a non-destructive recovery? We're planning for a downtime on the weekend to rebuild the cluster but if I can get away without it that would be helpful.

Thanks again for all your help. I'm going to start an additional thread about the bindings that seem to not be working once I can reproduce it in a staging environment.

--Mike

Mike Roest

unread,

Aug 19, 2016, 9:20:33 AM8/19/16

to Michael Klishin, rabbitm...@googlegroups.com

So interesting piece of data. The corrupt queue seems to happen solely from attempting to delete the queue via the 3.2.3 admin UI. We've now had 2 other clusters get into this scenario with a broken queue when I try to work around the non working bindings by deleteing the queue through the UI and have our software rebuild the queue from a single process before the rest of processes come online.

I'm still working on reproducing the actual issue where we get queue bindings that show in the Admin but don't actually capture messages.

Michael Klishin

unread,

Aug 19, 2016, 9:26:39 AM8/19/16

to Mike Roest, rabbitm...@googlegroups.com

The issue is not with "corrupt queues" but with non-existent bindings that are still
left recorded in some tables.

On 19 August 2016 at 16:20:29, Mike Roest (mike....@replicon.com) wrote:
> So interesting piece of data. The corrupt queue seems to happen solely
> from attempting to delete the queue via the 3.2.3 admin UI. We've now had
> 2 other clusters get into this scenario with a broken queue when I try to
> work around the non working bindings by deleteing the queue through the UI
> and have our software rebuild the queue from a single process before the
> rest of processes come online.
>
> I'm still working on reproducing the actual issue where we get queue
> bindings that show in the Admin but don't actually capture messages.
>
>
>
> On Wed, Aug 17, 2016 at 1:06 PM, Mike Roest wrote:
>
> > Any other possibility's you can think of for a non-destructive recovery?
> > We're planning for a downtime on the weekend to rebuild the cluster but if
> > I can get away without it that would be helpful.
> >
> > Thanks again for all your help. I'm going to start an additional thread
> > about the bindings that seem to not be working once I can reproduce it in a
> > staging environment.
> >
> > --Mike
> >
> >
> > On Tue, Aug 16, 2016 at 3:36 PM, Mike Roest

> > wrote:
> >
> >> But deleting the vhost I would have to do on the running node correct?
> >> If I delete that vhost that's effectively the same as shutting down
> >> everything since it's the only vhost we have.
> >>
> >>
> >> On Tue, Aug 16, 2016 at 3:33 PM, Michael Klishin

> >> wrote:
> >>
> >>> You are deleting node data correctly but then the node syncs from another
> >>> node.
> >>>
> >>> Vhosts are deleted with rabbitmqctl, see its man page.
> >>>
> >>> On Wed, Aug 17, 2016 at 12:10 AM, Mike Roest

> >>>> wrote:
> >>>>
> >>>>> You can try deleting a single vhost the problematic exchange and
> >>>>> bindings is/are in,
> >>>>> not everything.
> >>>>>
> >>>>> On Wed, Aug 17, 2016 at 12:03 AM, Michael Klishin > >>>>> > wrote:
> >>>>>
> >>>>>> I find it very hard to believe that a node can fail to start with
> >>>>>> this very exception in the log with a blank
> >>>>>> node data directory: there simply aren't any exchanges or bindings to
> >>>>>> load, or peers to sync from.
> >>>>>>
> >>>>>> On Tue, Aug 16, 2016 at 11:56 PM, Mike Roest > >>>>>> > wrote:
> >>>>>>
> >>>>>>> Well that eval worked (on the up server):
> >>>>>>> ok
> >>>>>>> ...done.
> >>>>>>>
> >>>>>>> But down server still fails to startup with or without removing the
> >>>>>>> mnesia dir.
> >>>>>>>
> >>>>>>> RabbitMQ 3.2.4. Copyright (C) 2007-2013 GoPivotal, Inc.
> >>>>>>> ## ## Licensed under the MPL. See http://www.rabbitmq.com/
> >>>>>>> ## ##

> >>>>>>> ########## Logs: /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>> ###### ## /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>> ##########
> >>>>>>> Starting broker...
> >>>>>>>
> >>>>>>> BOOT FAILED
> >>>>>>> ===========
> >>>>>>>
> >>>>>>> Error description:
> >>>>>>> {error,{{badmatch,{error,[{<<188,169,169,0,103,98,111,64,60
> >>>>>>> ,75,49,4,225,243,
> >>>>>>> 167,138>>,
> >>>>>>> "user"},

> >>>>>>> {<<149,242,33,30,66,247,121,1
> >>>>>>> 51,21,73,228,180,181,
> >>>>>>> 31,120,103>>,
> >>>>>>> "replicon"},
> >>>>>>> {<<139,12,224,72,32,237,89,12
> >>>>>>> 1,247,232,1,240,160,
> >>>>>>> 86,156,211>>,
> >>>>>>> "*"},
> >>>>>>> {<<135,223,75,10,43,90,124,15
> >>>>>>> 5,128,187,197,170,87,
> >>>>>>> 190,157,35>>,
> >>>>>>> "tenant"},
> >>>>>>> {root,none}],

> >>>>>>> ["policy-set-assignment","modified"]}},

> >>>>>>> [{rabbit_exchange_type_topic,follow_down_get_path,2},
> >>>>>>> {rabbit_exchange_type_topic,'-
> >>>>>>> remove_bindings/3-lc$^1/1-1-',1},
> >>>>>>> {rabbit_exchange_type_topic,remove_bindings,3},
> >>>>>>> {rabbit_binding,x_callback,4},
> >>>>>>> {rabbit_binding,'-process_deletions/1-fun-0-',2},
> >>>>>>> {dict,map_bucket,2},
> >>>>>>> {dict,map_bkt_list,2},
> >>>>>>> {dict,map_bkt_list,2}]}}
> >>>>>>>
> >>>>>>> Log files (may contain more information):

> >>>>>>> /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>> /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>>
> >>>>>>> Stack trace:
> >>>>>>> [{rabbit_misc,execute_mnesia_transaction,1},
> >>>>>>> {rabbit_misc,execute_mnesia_tx_with_tail,1},
> >>>>>>> {rabbit_amqqueue,recover,0},
> >>>>>>> {rabbit,recover,0},
> >>>>>>> {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
> >>>>>>> {rabbit,run_boot_step,1},
> >>>>>>> {rabbit,'-start/2-lc$^0/1-0-',1},
> >>>>>>> {rabbit,start,2}]
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> BOOT FAILED
> >>>>>>> ===========
> >>>>>>>
> >>>>>>> Error description:
> >>>>>>> {could_not_start,rabbit,
> >>>>>>> {bad_return,
> >>>>>>> {{rabbit,start,[normal,[]]},
> >>>>>>> {'EXIT',
> >>>>>>> {rabbit,failure_during_boot,
> >>>>>>> {error,
> >>>>>>> {{badmatch,
> >>>>>>> {error,
> >>>>>>> [{<<188,169,169,0,103,98,111,
> >>>>>>> 64,60,75,49,4,
> >>>>>>> 225,243,167,138>>,
> >>>>>>> "user"},

> >>>>>>> {<<149,242,33,30,66,247,121,15
> >>>>>>> 1,21,73,228,
> >>>>>>> 180,181,31,120,103>>,

> >>>>>>> "replicon"},
> >>>>>>> {<<139,12,224,72,32,237,89,121
> >>>>>>> ,247,232,1,240,
> >>>>>>> 160,86,156,211>>,
> >>>>>>> "*"},
> >>>>>>> {<<135,223,75,10,43,90,124,155
> >>>>>>> ,128,187,197,
> >>>>>>> 170,87,190,157,35>>,
> >>>>>>> "tenant"},
> >>>>>>> {root,none}],

> >>>>>>> ["policy-set-assignment","mod

> >>>>>>> ified"]}},
> >>>>>>> [{rabbit_exchange_type_topic,
> >>>>>>> follow_down_get_path,2},
> >>>>>>> {rabbit_exchange_type_topic,
> >>>>>>> '-remove_bindings/3-lc$^1/1-1-',1},
> >>>>>>> {rabbit_exchange_type_topic,re
> >>>>>>> move_bindings,3},
> >>>>>>> {rabbit_binding,x_callback,4},
> >>>>>>> {rabbit_binding,'-process_dele
> >>>>>>> tions/1-fun-0-',2},
> >>>>>>> {dict,map_bucket,2},
> >>>>>>> {dict,map_bkt_list,2},
> >>>>>>> {dict,map_bkt_list,2}]}}}}}}}
> >>>>>>>
> >>>>>>> Log files (may contain more information):

> >>>>>>> /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>> /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>>
> >>>>>>> {"init terminating in do_boot",{rabbit,failure_durin
> >>>>>>> g_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[n
> >>>>>>> ormal,[]]},{'EXIT',{rabbit,failure_during_boot,{error,{{badmatch,{error,[{<<16

> >>>>>>> bytes>>,"user"},{<<16 bytes>>,"replicon"},{<<16 bytes>>,"*"},{<<16
> >>>>>>> bytes>>,"tena

> >>>>>>> nt"},{root,none}],["policy-set-assignment","modified"]}},[{r
> >>>>>>> abbit_exchange_type_topic,follow_down_get_path,2},{rabbit_ex
> >>>>>>> change_type_topic,'-remove_bindings/3-lc$^1/1-1-',1},{rabbit
> >>>>>>> _exchange_type_topic,remove_bindings,3},{rabbit_binding,x_ca

> >>>>>>>> > ########## Logs: /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>>> > ###### ## /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>>> > /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>>> > /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>>> > /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>>> > /var/log/rabbitmq/rab...@rabbitmq1-sasl.log
> >>>>>>>> >
> >>>>>>>> > {"init terminating in
> >>>>>>>> > do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit
> >>>>>>>> ,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,fai

> >>>>>>>> lure_during_boot,{error,{{badmatch,{error,[{<<16
> >>>>>>>> > bytes>>,"user"},{<<16 bytes>>,"replicon"},{<<16 bytes>>,"*"},{<<16
> >>>>>>>> > bytes>>,"tena

> >>>>>>>> > nt"},{root,none}],["policy-set-assignment","modified"]}},[{r
> >>>>>>>> abbit_exchange_type_topic,follow_down_get_path,2},{rabbit_ex
> >>>>>>>> change_type_topic,'-remove_bindings/3-lc$^1/1-1-',1},{rabbit
> >>>>>>>> _exchange_type_topic,remove_bindings,3},{rabbit_binding,x_ca
> >>>>>>>> llback,4},{rabbit_binding,'-pro
> >>>>>>>> > cess_deletions/1-fun-0-',2},{dict,map_bucket,2},{dict,map_bk
> >>>>>>>> t_list,2},{dict,map_bkt_list,2}]}}}}}}}}}
> >>>>>>>> > [FAILED]
> >>>>>>>> >
> >>>>>>>> >

> >>>>>>>> > >>>>>>>>> ########## Logs: /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>>> > >>>>>>>>> ###### ## /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>>> > >>>>>>>>> ##########
> >>>>>>>> > >>>>>>>>> Starting broker...
> >>>>>>>> > >>>>>>>>>
> >>>>>>>> > >>>>>>>>> BOOT FAILED
> >>>>>>>> > >>>>>>>>> ===========
> >>>>>>>> > >>>>>>>>>
> >>>>>>>> > >>>>>>>>> Error description:
> >>>>>>>> > >>>>>>>>> {error,{{badmatch,{error,[{<<1

> >>>>>>>> 49,242,33,30,66,247,121,151,2

> >>>>>>>> > >>>>>>>>> 1,73,228,180,181,
> >>>>>>>> > >>>>>>>>> 31,120,103>>,
> >>>>>>>> > >>>>>>>>> "replicon"},
> >>>>>>>> > >>>>>>>>> {<<139,12,224,72,32,237,89,12
> >>>>>>>> > >>>>>>>>> 1,247,232,1,240,160,
> >>>>>>>> > >>>>>>>>> 86,156,211>>,
> >>>>>>>> > >>>>>>>>> "*"},
> >>>>>>>> > >>>>>>>>> {<<135,223,75,10,43,90,124,15
> >>>>>>>> > >>>>>>>>> 5,128,187,197,170,87,
> >>>>>>>> > >>>>>>>>> 190,157,35>>,
> >>>>>>>> > >>>>>>>>> "tenant"},
> >>>>>>>> > >>>>>>>>> {root,none}],
> >>>>>>>> > >>>>>>>>> ["user","policy-set-assignment
> >>>>>>>> > >>>>>>>>> ","modified"]}},
> >>>>>>>> > >>>>>>>>> [{rabbit_exchange_type_topic,follow_down_get_path,2},
> >>>>>>>> > >>>>>>>>> {rabbit_exchange_type_topic,'-
> >>>>>>>> > >>>>>>>>> remove_bindings/3-lc$^1/1-1-',1},
> >>>>>>>> > >>>>>>>>> {rabbit_exchange_type_topic,remove_bindings,3},
> >>>>>>>> > >>>>>>>>> {rabbit_binding,x_callback,4},
> >>>>>>>> > >>>>>>>>> {rabbit_binding,'-process_deletions/1-fun-0-',2},
> >>>>>>>> > >>>>>>>>> {dict,map_bucket,2},
> >>>>>>>> > >>>>>>>>> {dict,map_bkt_list,2},
> >>>>>>>> > >>>>>>>>> {dict,map_bkt_list,2}]}}
> >>>>>>>> > >>>>>>>>>
> >>>>>>>> > >>>>>>>>> Log files (may contain more information):

> >>>>>>>> > >>>>>>>>> /var/log/rabbitmq/rab...@rabbitmq1.log
> >>>>>>>> > >>>>>>>>> /var/log/rabbitmq/rab...@rabbitmq1-sasl.log

> >>>>>>>> > >>>>>>>>>
> >>>>>>>> > >>>>>>>>> {"init terminating in do_boot",{rabbit,failure_durin
> >>>>>>>> > >>>>>>>>> g_boot,{could_not_start,rabbit
> >>>>>>>> ,{bad_return,{{rabbit,start,[n
> >>>>>>>> > >>>>>>>>> ormal,[]]},{'EXIT',{rabbit,fai
> >>>>>>>> lure_during_boot,{error,{{badmatch,{error,[{<<16
> >>>>>>>> > >>>>>>>>> bytes>>,"replicon"},{<<16 bytes>>,"*"},{<<16
> >>>>>>>> bytes>>,"tenant"},{root,none}]
> >>>>>>>> > >>>>>>>>> ,["us
> >>>>>>>> > >>>>>>>>> er","policy-set-assignment","m
> >>>>>>>> odified"]}},[{rabbit_exchange_
> >>>>>>>> > >>>>>>>>> type_topic,follow_down_get_pat
> >>>>>>>> h,2},{rabbit_exchange_type_top
> >>>>>>>> > >>>>>>>>> ic,'-remove_bindings/3-lc$^1/1

> >>>>>>>> -1-',1},{rabbit_exchange_type_
> >>>>>>>> > >>>>>>>>> topic,remove_bindings,3},{rabb

> >>>>>>>> > >>>> an email to rabbitmq-user...@googlegroups.com.

> >>>>>>>> > >>>> To post to this group, send email to
> >>>>>>>> rabbitm...@googlegroups.com.
> >>>>>>>> > >>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>> > >>>>
> >>>>>>>> > >>>
> >>>>>>>> > >>>
> >>>>>>>> > >>>
> >>>>>>>> > >>> --
> >>>>>>>> > >>> MK
> >>>>>>>> > >>>
> >>>>>>>> > >>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>> > >>>
> >>>>>>>> > >>> --
> >>>>>>>> > >>> You received this message because you are subscribed to the
> >>>>>>>> Google
> >>>>>>>> > >>> Groups "rabbitmq-users" group.
> >>>>>>>> > >>> To unsubscribe from this group and stop receiving emails from
> >>>>>>>> it, send

> >>>>>>>> > >>> an email to rabbitmq-user...@googlegroups.com.

> >>>>>>>> > >>> To post to this group, send email to
> >>>>>>>> rabbitm...@googlegroups.com.
> >>>>>>>> > >>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>> > >>>
> >>>>>>>> > >>
> >>>>>>>> > >>
> >>>>>>>> > >>
> >>>>>>>> > >> --
> >>>>>>>> > >> Data's inconvienient when people have opinions.
> >>>>>>>> > >>
> >>>>>>>> > >> --
> >>>>>>>> > >> You received this message because you are subscribed to the
> >>>>>>>> Google Groups
> >>>>>>>> > >> "rabbitmq-users" group.
> >>>>>>>> > >> To unsubscribe from this group and stop receiving emails from
> >>>>>>>> it, send an

> >>>>>>>> > >> email to rabbitmq-user...@googlegroups.com.

> >>>>>>>> > >> To post to this group, send email to
> >>>>>>>> rabbitm...@googlegroups.com.
> >>>>>>>> > >> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>> > >>
> >>>>>>>> > >
> >>>>>>>> > >
> >>>>>>>> > >
> >>>>>>>> > > --
> >>>>>>>> > > MK
> >>>>>>>> > >
> >>>>>>>> > > Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>> > >
> >>>>>>>> > > --
> >>>>>>>> > > You received this message because you are subscribed to the
> >>>>>>>> Google Groups
> >>>>>>>> > > "rabbitmq-users" group.
> >>>>>>>> > > To unsubscribe from this group and stop receiving emails from
> >>>>>>>> it, send an

> >>>>>>>> > > email to rabbitmq-user...@googlegroups.com.

> >>>>>>>> > > To post to this group, send email to
> >>>>>>>> rabbitm...@googlegroups.com.
> >>>>>>>> > > For more options, visit https://groups.google.com/d/optout.
> >>>>>>>> > >
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > --
> >>>>>>>> > Data's inconvienient when people have opinions.
> >>>>>>>> >
> >>>>>>>> > --
> >>>>>>>> > You received this message because you are subscribed to the
> >>>>>>>> Google Groups "rabbitmq-users"
> >>>>>>>> > group.
> >>>>>>>> > To unsubscribe from this group and stop receiving emails from it,

> >>>>>>>> send an email to rabbitmq-user...@googlegroups.com.

Mike Roest

unread,

Aug 19, 2016, 9:31:47 AM8/19/16

to Michael Klishin, rabbitm...@googlegroups.com

ahh that explains why list_queues from rabbitmqctl doesn't display the queue then.

So then we're just back to how the non-existant bindings get in there then. It really seems to have something to do with multiple processes trying to create the queues & bindings at the same time.

What we're doing is on each new version of our software they attempt to create a versioned queue with the corresponding bindings . When we deploy the software multiple servers all come up at the same time doing the Queue and binding declaration. When I start them one up first allow it to do all the initial creation. That seems then to create everything correctly, but we have 10-15 cases of when the multiple machines come up that the bindings don't actually get created correctly.

--Mike

> >>>>>>> ########## Logs: /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>> ###### ## /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>> /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>> /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>>> > ########## Logs: /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>>> > ###### ## /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>>> > /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>>> > /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>>> > /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>>> > /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log
> >>>>>>>> >
> >>>>>>>> > {"init terminating in

> >>>>>>>> > >>>>>>>>> ########## Logs: /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>>> > >>>>>>>>> ###### ## /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>>> > >>>>>>>>> /var/log/rabbitmq/rabbit@rabbitmq1.log
> >>>>>>>> > >>>>>>>>> /var/log/rabbitmq/rabbit@rabbitmq1-sasl.log

> >>>>>>>> > >>>>>>>>> rabbitmq-users@googlegroups.com.

> >>>>>>>> > >>>>>>>>> For more options, visit https://groups.google.com/d/op
> >>>>>>>> tout.
> >>>>>>>> > >>>>>>>>>
> >>>>>>>> > >>>>>>>>
> >>>>>>>> > >>>>>>>>
> >>>>>>>> > >>>>>>>>
> >>>>>>>> > >>>>>>>> --
> >>>>>>>> > >>>>>>>> MK
> >>>>>>>> > >>>>>>>>
> >>>>>>>> > >>>>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>> > >>>>>>>>
> >>>>>>>> > >>>>>>>
> >>>>>>>> > >>>>>>>
> >>>>>>>> > >>>>>>>
> >>>>>>>> > >>>>>>> --
> >>>>>>>> > >>>>>>> MK
> >>>>>>>> > >>>>>>>
> >>>>>>>> > >>>>>>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>> > >>>>>>>
> >>>>>>>> > >>>>>>> --
> >>>>>>>> > >>>>>>> You received this message because you are subscribed to
> >>>>>>>> the Google
> >>>>>>>> > >>>>>>> Groups "rabbitmq-users" group.
> >>>>>>>> > >>>>>>> To unsubscribe from this group and stop receiving emails
> >>>>>>>> from it,
> >>>>>>>> > >>>>>>> send an email to rabbitmq-users+unsubscribe@goo
> >>>>>>>> glegroups.com.
> >>>>>>>> > >>>>>>> To post to this group, send email to

> >>>>>>>> rabbitmq-users@googlegroups.com

> >>>>>>>> > >>>>>>> .
> >>>>>>>> > >>>>>>> For more options, visit https://groups.google.com/d/op
> >>>>>>>> tout.
> >>>>>>>> > >>>>>>>
> >>>>>>>> > >>>>>>
> >>>>>>>> > >>>>>>
> >>>>>>>> > >>>>>>
> >>>>>>>> > >>>>>> --
> >>>>>>>> > >>>>>> Data's inconvienient when people have opinions.
> >>>>>>>> > >>>>>>
> >>>>>>>> > >>>>>
> >>>>>>>> > >>>>>
> >>>>>>>> > >>>>>
> >>>>>>>> > >>>>> --
> >>>>>>>> > >>>>> Data's inconvienient when people have opinions.
> >>>>>>>> > >>>>>
> >>>>>>>> > >>>>
> >>>>>>>> > >>>>
> >>>>>>>> > >>>>
> >>>>>>>> > >>>> --
> >>>>>>>> > >>>> Data's inconvienient when people have opinions.
> >>>>>>>> > >>>>
> >>>>>>>> > >>>> --
> >>>>>>>> > >>>> You received this message because you are subscribed to the
> >>>>>>>> Google
> >>>>>>>> > >>>> Groups "rabbitmq-users" group.
> >>>>>>>> > >>>> To unsubscribe from this group and stop receiving emails
> >>>>>>>> from it, send

> >>>>>>>> > >>>> an email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>>>>>>> > >>>> To post to this group, send email to

> >>>>>>>> rabbitmq-users@googlegroups.com.

> >>>>>>>> > >>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>> > >>>>
> >>>>>>>> > >>>
> >>>>>>>> > >>>
> >>>>>>>> > >>>
> >>>>>>>> > >>> --
> >>>>>>>> > >>> MK
> >>>>>>>> > >>>
> >>>>>>>> > >>> Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>> > >>>
> >>>>>>>> > >>> --
> >>>>>>>> > >>> You received this message because you are subscribed to the
> >>>>>>>> Google
> >>>>>>>> > >>> Groups "rabbitmq-users" group.
> >>>>>>>> > >>> To unsubscribe from this group and stop receiving emails from
> >>>>>>>> it, send

> >>>>>>>> > >>> an email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>>>>>>> > >>> To post to this group, send email to

> >>>>>>>> rabbitmq-users@googlegroups.com.

> >>>>>>>> > >>> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>> > >>>
> >>>>>>>> > >>
> >>>>>>>> > >>
> >>>>>>>> > >>
> >>>>>>>> > >> --
> >>>>>>>> > >> Data's inconvienient when people have opinions.
> >>>>>>>> > >>
> >>>>>>>> > >> --
> >>>>>>>> > >> You received this message because you are subscribed to the
> >>>>>>>> Google Groups
> >>>>>>>> > >> "rabbitmq-users" group.
> >>>>>>>> > >> To unsubscribe from this group and stop receiving emails from
> >>>>>>>> it, send an

> >>>>>>>> > >> email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>>>>>>> > >> To post to this group, send email to

> >>>>>>>> rabbitmq-users@googlegroups.com.

> >>>>>>>> > >> For more options, visit https://groups.google.com/d/optout.
> >>>>>>>> > >>
> >>>>>>>> > >
> >>>>>>>> > >
> >>>>>>>> > >
> >>>>>>>> > > --
> >>>>>>>> > > MK
> >>>>>>>> > >
> >>>>>>>> > > Staff Software Engineer, Pivotal/RabbitMQ
> >>>>>>>> > >
> >>>>>>>> > > --
> >>>>>>>> > > You received this message because you are subscribed to the
> >>>>>>>> Google Groups
> >>>>>>>> > > "rabbitmq-users" group.
> >>>>>>>> > > To unsubscribe from this group and stop receiving emails from
> >>>>>>>> it, send an

> >>>>>>>> > > email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>>>>>>> > > To post to this group, send email to

> >>>>>>>> rabbitmq-users@googlegroups.com.

> >>>>>>>> > > For more options, visit https://groups.google.com/d/optout.
> >>>>>>>> > >
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> >
> >>>>>>>> > --
> >>>>>>>> > Data's inconvienient when people have opinions.
> >>>>>>>> >
> >>>>>>>> > --
> >>>>>>>> > You received this message because you are subscribed to the
> >>>>>>>> Google Groups "rabbitmq-users"
> >>>>>>>> > group.
> >>>>>>>> > To unsubscribe from this group and stop receiving emails from it,

> >>>>>>>> send an email to rabbitmq-users+unsubscribe@googlegroups.com.

> >>>>>>>> > To post to this group, send an email to

> >>>>>>>> rabbitmq-users@googlegroups.com.

Michael Klishin

unread,

Nov 17, 2016, 5:50:01 PM11/17/16

to rabbitmq-users

As a follow-up to this thread, there will be one scenario where `rabbit_amqqueue:delete_immediately/1` is not going to work:

it's when the node that hosts queue master is unavailable and can't or won't be brought back up.

See https://groups.google.com/forum/#!topic/rabbitmq-users/rpO3dFzm-yQ for how to deal with that case.

On Tuesday, August 16, 2016 at 11:45:30 PM UTC+3, Michael Klishin wrote:

`rabbitmqctl eval '{ok, {amqqueue, _, _ , _, _, _, Pid, _, _, _, _, _}} = rabbit_amqqueue:lookup(rabbit_misc:r(<<“VHOST">>, queue, <<“QUEUE_NAME">>)), rabbit_amqqueue:delete_immediately([Pid]).’`

Reply all

Reply to author

Forward