Cannot delete queue that used to exist on a node that has been deleted

[James Rybicki]

The title says it all.  The queue shows up in the Management UI, but we cannot delete it using any method we have tried.

Please tell us how we can get rid of it.

Thanks,

Jamie

[Michael Klishin]

Try searching for "delete_immediately" in list archives.

`rabbitmqctl eval` can do it.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Rybicki]

I did mate...I did.

None of those commands worked either...

I always get this back:

Error: {{badmatch,{error,not_found}},[{erl_eval,expr,3,[]}]}

[James Rybicki]

Also, We are on RabbitMQ 3.6.3 and Erlang 18.something.  We are going to move to 3.6.6 and Erland 19.1 when available and after some testing...

[James Rybicki]

Have tried the following commands, each ending with the same result:

rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"bar">>, queue, <<"foo">>)), Pid = rabbit_amqqueue:pid_of(Q), rabbit_amqqueue:delete_immediately([Pid]).'

rabbitmqctl eval '{ok, {amqqueue, _, _ , _, _, _, Pid, _, _, _, _, _}} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"bar">>, queue, <<"foo">>)), rabbit_amqqueue:delete_immediately([Pid]).'

rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"bar">>, queue, <<"foo">>)), rabbit_amqqueue:delete_crashed(Q).'

[Michael Klishin]

Are you specifying the correct vhost and queue name?

[James Rybicki]

I am supplying the names EXACTLY as listed in the management UI

Sent from my iPhone

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/rpO3dFzm-yQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

Queue and vhost names as passed to rabbit_amqqueue:lookup must be binaries, not strings.

<<"abc">>, not "abc"

[James Rybicki]

Yes I am using that format, <<"bar">>

Sent from my iPhone

[James Rybicki]

Did you see my earlier response? 

I have tried the following commands, each ending with the same result:  Error: {{badmatch,{error,not_found}},[{erl_eval,expr,3,[]}]}

rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"bar">>, queue, <<"foo">>)), Pid = rabbit_amqqueue:pid_of(Q), rabbit_amqqueue:delete_immediately([Pid]).'

rabbitmqctl eval '{ok, {amqqueue, _, _ , _, _, _, Pid, _, _, _, _, _}} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"bar">>, queue, <<"foo">>)), rabbit_amqqueue:delete_immediately([Pid]).'

rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"bar">>, queue, <<"foo">>)), rabbit_amqqueue:delete_crashed(Q).'

[James Rybicki]

I did not mention, the queue's master node no longer exists.  We got into this state as a result of one by one replacing all of the nodes in the cluster with new nodes.

Specifically, we performed the following steps:

0. We started with a 5 node cluster.

1. Added a new node to the cluster, bringing the number of nodes in the cluster to 6.

2. Waited for things to "settle down" by monitoring the rabbit log on the stats node.

3. Removed a node from the original 5 node cluster, bringing the number of nodes in the cluster back to 5.

4. Added a new node to the cluster, bringing the number of nodes in the cluster to 6.

5. Waited for things to "settle down" by monitoring the rabbit log on the stats node.

6. Removed another node from the original 5 node cluster, bringing the number of nodes in the cluster back to 5.

7. We repeated this process until all of the original nodes in the 5 node cluster had been replaced with new nodes.  We made sure to process the stats node last.

8. After all of this was done, several of our clients were having problems using the cluster.

9. It was then that we identified that there were queues showing as "Down" and those queues were shown to exist on the RabbitMQ nodes which no longer exist (as a result of the process we just went through).

10.  Our attempts to delete those queues so that the clients can recreate them have not been successful, as previously described, and hence our clients are continuing to fail.

[Michael Klishin]

if

> rabbit_amqqueue:lookup(rabbit_misc:r(<<"bar">>, queue, <<"foo">>))

returns a `not_found` error, then the queue cannot be found. It does not try to contact that

queue's process.

The queue should not appear in `rabbitmqctl list_queues`, for example. If it still appears

in the management plugin, resetting the stats DB should help with that.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[James Rybicki]

I reset the stats db, and the queue still shows up in the Management UI.

To reset the stats db, I performed:

rabbitmqctl eval 'supervisor2:terminate_child(rabbit_mgmt_sup_sup, rabbit_mgmt_sup),
                  rabbit_mgmt_sup_sup:start_child().'

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/rpO3dFzm-yQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

[Michael Klishin]

Can we first verify that the queue in question does not appear in `rabbitmqctl list_queues -p {vhost}`? 

On 17 November 2016 at 17:54:21, James Rybicki (rybick...@gmail.com) wrote:
> I reset the stats db, and the queue still shows up in the Management UI.
>
> To reset the stats db, I performed:
>
> rabbitmqctl eval 'supervisor2:terminate_child(rabbit_mgmt_sup_sup,
> rabbit_mgmt_sup),
> rabbit_mgmt_sup_sup:start_child().'
>
>
> On Thu, Nov 17, 2016 at 9:52 AM, Michael Klishin

> wrote:
>
> > if
> >
> > > rabbit_amqqueue:lookup(rabbit_misc:r(<<"bar">>, queue, <<"foo">>))
> >
> > returns a `not_found` error, then the queue cannot be found. It does not
> > try to contact that
> > queue's process.
> >
> > The queue should not appear in `rabbitmqctl list_queues`, for example. If
> > it still appears
> > in the management plugin, resetting the stats DB should help with that.
> >
> > On Thu, Nov 17, 2016 at 4:19 PM, James Rybicki

> > MK
> >
> > Staff Software Engineer, Pivotal/RabbitMQ
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "rabbitmq-users" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/
> > topic/rabbitmq-users/rpO3dFzm-yQ/unsubscribe.

> > To unsubscribe from this group and all its topics, send an email to
> > rabbitmq-user...@googlegroups.com.
> > To post to this group, send email to rabbitm...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send an email to rabbitm...@googlegroups.com.

[James Rybicki]

It does appear in the list returned by rabbitmqctl list_queues -p vhost

But when I try to delete it using the REST API or if I try and view the queue using the Management UI (by clicking on it) I am told it does not exist.

[Michael Klishin]

So here's how to delete such queue (hosted on a node that's no longer a cluster member):

```

rabbitmqctl eval 'Q = rabbit_misc:r(<<"VHOST NAME">>, queue, <<"QUEUE NAME">>), rabbit_amqqueue:internal_delete(Q).'

```

or

```

rabbitmqctl eval 'Q = {resource, <<"VHOST NAME">>, queue, <<"QUEUE NAME">>}, rabbit_amqqueue:internal_delete(Q).'

```

`rabbit_amqqueue:delete_immediately/1` and `rabbit_amqqueue:delete_crashed/1` will attempt to contact the hosting node,

which I incorrectly believed not to be the case.

I'd like to thank my Pivotal colleague Phil Pickett for assisting with finding the appropriate function

and setting up an environment where this could be reproduced and verified.

[James Rybicki]

Thanks will give a try

Sent from my iPhone

[James Rybicki]

I was able to delete the queue using the command:

rabbitmqctl eval 'Q = {resource, <<"VHOST NAME">>, queue, <<"QUEUE NAME">>}, rabbit_amqqueue:internal_delete(Q).'

Thank you to Michael and Phil!