Tracing AMQP commands on RabbitMQ

[refik turkeli]

We normally have around 300 queues in RabbitMQ. Today we noticed that it spiked up to 69,000. Majority of those queues are in the form amq.gen-***. I know that these are created when a queue is declared with an empty routing key. 

Our AMQP client is pika and we searched everywhere in our code but found no declare with an empty routing key. 

I've used the firehose tracer in order to find how these queues are being created but there was no log of AMQP commands such as "exchange.declare" there. After opening the trace, I've listened the exchange amq.rabbitmq.trace with # binding. 

My question is: how can I trace "exchange.declare" commands and find their origin?

If there is any other recommendation for solving this problem, that is also welcome.

We are using Ubuntu 12.04 server with RabbitMQ 3.3.5, Erlang R14B04 and pika 0.9.14

Also, to free up memory on that server we are trying to delete queues starting with amq.gen-. The way we are doing it is executing the command below for every single queue. Every time we execute, it takes about 2 seconds. If anyone knows a better way in terms of speed, I would appreciate it.

rabbitmqadmin --username=*** --password=***** delete queue name=

[Laing, Michael]

You could install rabbitmq_event_exchange to catch the queue.declare's etc.

Your erlang is old... 

I assume you have restarted your cluster and that didn't help.

We use pika extensively and have not seen this problem.

Other things that can create amq.gen queues might be shovels or federation - check policies and configuration?

Good luck,

Michael

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

On 11 October 2014 at 16:23:05, refik turkeli wrote:
> My question is: how can I trace "exchange.declare" commands
> and find their origin?

Wireshark or, if you can temporarily pass everything through a proxy,
Tracer [1].

1. http://www.rabbitmq.com/java-tools.html
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Simon MacMullen]

On 11/10/14 13:22, refik turkeli wrote:
> We normally have around 300 queues in RabbitMQ. Today we noticed that it
> spiked up to 69,000. Majority of those queues are in the form
> amq.gen-***. I know that these are created when a queue is declared with
> an empty routing key.
>
> Our AMQP client is pika and we searched everywhere in our code but found
> no declare with an empty routing key.
>
> I've used the firehose tracer in order to find how these queues are
> being created but there was no log of AMQP commands such as
> "exchange.declare" there.

Yes; the firehose just logs published and delivered messages, not AMQP
commands.

> My question is: how can I trace "exchange.declare" commands and find
> their origin?

Michael L's suggestion of the event exchange is probably easiest of you
can restart the broker. If not, I'd go with Michael K's suggestion of
wireshark.

> Also, to free up memory on that server we are trying to delete queues
> starting with amq.gen-. The way we are doing it is executing the command
> below for every single queue. Every time we execute, it takes about 2
> seconds. If anyone knows a better way in terms of speed, I would
> appreciate it.
>
> rabbitmqadmin --username=*** --password=***** delete queue name=

(I'm guessing that the performance problem here is in the client side by
the way.)

The big problem with using rabbitmqadmin in a loop is that it starts a
Python interpreter each time, and that's expensive. One day I'd like
rabbitmqadmin to get a REPL mode, but that's not going to be a short
term solution.

I suspect the best way to delete lots of queues quickly is to write your
own simple HTTP API client in your favourite programming language. If
that's too much work you could try replacing the rabbitmqadmin delete
queue invocation in whatever shell script you're using with something like:

curl -i -u username:password -H "content-type:application/json" \
-XDELETE http://server-name:15672/api/queues/%2f/queue-to-delete

since I think curl will start faster than Python.

Cheers, Simon

[Refik Turkeli]

Thanks a lot to everyone answered. I installed rabbitmq_event_exchange plugin and was able to see the queue.created events. However, there was no further information concerning the initiator of that event so I couldn't take any action.

Eventually it was Wireshark that saved the day. I was able to trace the problem to a single process working on a host. There is no problem with pika, it was only a programming error.

The tip for deleting queues with curl instead of rabbitmqadmin was immensely useful.