Feature Request: RabbitMQ Instrumentation for StatsD/Graphite and Graylog2

[chrishaine...@gmail.com]

Hi everyone,

At Betgenius we like to instrument everything we possibly can to give us as much visibility over our systems as possible. RabbitMQ is probably our least visible component in our stack. We are currently using StatsD in front of Graphite for performance metrics and Graylog2 for logs aggregation. To get stats out of the RabbitMQ broker we have to write custom scripts and poll the management ctl which can add quite a lot of stress under heavy load (management API often becomes unresponsive under load). Would it be possible to get the broker to push it's metrics to external sources instead of being polled for all metrics such as queue, exchange, timings, counts, etc? Also, shipping the logs to Graylog2 would be incredibly useful.

Cheers,

Chris

[Alvaro Videla]

Hi,

Have you seen this plugin? https://github.com/gmr/rabbitmq-pulse It pushes stats out of RabbitMQ.

Regards,

Alvaro

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[chrishaine...@gmail.com]

Hi Alvaro, 

The last check-in on that project was 3 years ago and it explicitly says do not use in Production. I would feel a lot more comfortable having something come from the official Rabbit team because this is something that could go easily wrong without full understanding of the architecture.

Cheers,

Chris

[Alvaro Videla]

Hi Chris,

Perhaps the author could use some feedback on why you need to use his project and such. Then he might update its status and so on.

Gavin usually replies here, he maintains the Python library for RabbitMQ and he's writing the book RabbitMQ in Depth, so I think he has an understanding of RabbitMQ ;-)

Regards,

Alvaro

[Chris Haines]

Hi Alvaro,

 

Thanks for that response. I should have done some research on Gavin, he sounds like just the man! :)

 

Hi Gavin,

 

Have you tried this plugin in Production? Your recommendation is not to use it. We run some extremely heavily loaded brokers in our Production environment pushing 20-40k messages/s and when we hit these levels the management API gets really sluggish and I have seen it become unusable before so polling that for data puts extra unnecessary load on the server and can leave us with gaps in our monitoring data.

 

Cheers,

Chris

[Gavin M. Roy]

Thanks for the vote of confidence Alvaro :)

I never got the plugin fully "done" because I wanted to figure out how mirrored_supervisor works. The plugin has been on my list of things to come back around and re-address since my knowledge of both erlang and RabbitMQ internals is a bit stronger than when I wrote it.

I'll bump it up towards the top of my list and circle back around when I get something viable for 3.5.

Gavin 

[Michael Klishin]

This is something a plugin can do, even with pluggable backends for storage.

--

[chrishaine...@gmail.com]

Hi Gavin,

Thank you so much, we would be eternally grateful! Happy to help test it on our Dev/UAT environments once you have a release if you want some good feedback.

Thanks,

Chris

[chrishaine...@gmail.com]

Looks like Gavin has the performance metrics covered. Is this possible through a plugin for the logs as well? I no absolutely noting about Erlang, so do you have a logging framework that supports something like appenders in log4net?

Thanks guys!

Chris

[Alvaro Videla]

At the moment the broker publishes logs to the "amq.rabbitmq.log" exchange which is a topic exchange. You could listen to those with any AMQP client and then republish them to these other solutions you are using.