How to get rid of phantom shovels?

[Brett Neumeier]

Hi all, I've got a strange problem and I wonder whether someone can help me with it.

I'm running rabbitmq 3.4.2 on erlang 17.3.

Recently we switched to a new cluster on a new set of servers. To do the switch without losing any in-process messages, we told all the producers and consumers to connect to the new cluster, then set up a bunch of shovels to move the messages from the old cluster to the new one. That all worked great!

However, now we can't get rid of the things. I'd really like to get rid of them!

In the admin web ui, under Admin / Shovel Status, I see rows like e.g.

Name: carapace.sms.mt.fast-shovel

Node: elkprmq01

Virtual Host: carapace-production

State: running

Source: amqp://elkpsplmys02.cloud.vibes.com/carapace-production

Destination: amqp://elkprmq01.cloud.vibes.com/carapace-production

Last changed: 2016-03-19 22:53:12

However, in the Shovel Management part of the UI, the "Shovels" section just shows: "... no shovels ...". So of course I can't destroy them from the web ui; there's no way to get to a shovel page for a shovel it doesn't admit to existing. Using the HTTP api fails to get rid of them as well. And "rabbitmqctl list_parameters" doesn't show any shovels either.

To make matters worse -- some of the shovels were not created properly (the password for the account has spaces in it, and initially we tried creating shovels without URL-encoding the spaces in the src-uri and dest-uri). Those shovels are not started properly -- still don't show up in the shovel management UI or rabbitmqctl -- and periodically try to start, resulting in error messages being written to the log files (snippets attached).

Is there any way to get rid of shovels in this weird and inconsistent state? If someone could suggest a snippet of erlang code I could throw in via "rabbitmqctl eval", or something, that would be awesome.

Thanks!

Brett

--

Brett Neumeier (bneu...@gmail.com)

[V Z]

Maybe the user id that you are using to view the shovels in the management UI does not have access to the virtual hosts containing the shovels?

[Brett Neumeier]

On Sun, Mar 20, 2016 at 12:20 PM, V Z <uvzu...@gmail.com> wrote:

Maybe the user id that you are using to view the shovels in the management UI does not have access to the virtual hosts containing the shovels?

Hi VZ,

Thanks for the response!

The user that's accessing the management UI is an admin account with full permissions on every vhost and the [administrator] tag.

What we're going to try next is, disable the shovel plugin, restart the brokers, and then re-enable the shovel plugin. Since there are no parameter entries that correspond to the phantom shovels, I'm hopeful that whatever lingering cruft there is in the rabbitmq database will get cleared out. (ProTip: at least with rabbitmq 3.4, restarting the brokers in between disabling and re-enabling plugins is a good idea because otherwise the broker crashes as soon as the plugin is re-enabled!)

--

Brett Neumeier (bneu...@gmail.com)

[Brett Neumeier]

On Sun, Mar 20, 2016 at 12:46 PM, Brett Neumeier <bneu...@gmail.com> wrote:

What we're going to try next is, disable the shovel plugin, restart the brokers, and then re-enable the shovel plugin. Since there are no parameter entries that correspond to the phantom shovels, I'm hopeful that whatever lingering cruft there is in the rabbitmq database will get cleared out. (ProTip: at least with rabbitmq 3.4, restarting the brokers in between disabling and re-enabling plugins is a good idea because otherwise the broker crashes as soon as the plugin is re-enabled!)

In case anyone is interested in this -- that worked fine. As soon as we disabled the shovel plugin, all the phantom shovels vanished (as expected). We didn't actually re-enable the plugin yet because we haven't needed to do any shoveling; I'll try to remember to post a follow-up here when we do that, especially if the phantom shovels reappear.

--

Brett Neumeier (bneu...@gmail.com)