tcp socket to rabbitmq to tcp socket - for Icinga2 to Graphite

[Chris Jefferies]

I saw only one inconclusive post about using Graphite (carbon-cache) with RabbitMQ so I'm posting in the hopes of learning a bit more.  I'm new to rabbitmq and queues in general, so apologies if this has been covered.

I use Icinga2 monitoring and it can output it's metric data directly to a TCP socket on port 2003.  Carbon-cache (the receiving component of Graphite) accepts the metric data on port 2003 and stores it in an RRD known as Whisper.

It seems  reasonable to insert rabbitmq into the icinga to carbon-cache tcp socket path to store and forward the data and to queue it up if the connection to carbon-cache is dropped.

I don't see anything that indicates a simple TCP socket input on rabbitmq so I assume I will need to write 2 scripts to:

1. receive the data coming in on tcp socket port 2003 and output to an AMQP "graphite" queue in to rabbitmq
   
2. consume the "graphite" queue out of rabbitmq and send it to the tcp socket port 2003 in carbon-cache
   

Pika seems to be the library of choice for python so I'm looking at that.  Now I need to learn how to manage the reconnection of tcp sockets and keep it running and reconnect when the connection returns.

Am I on the right track?

The other thread I found about graphite never seemed to be resolved but the poster indicated an issue connecting from rabbitmq to carbon-cache...

Thanks for any advice,

Chris.

[Michael Klishin]

Chris,

If intermediary queueing is required in your case then you are on the right track. Both apps will have to deal with connectivity issues

with both their inputs (Icinga2, RabbitMQ) and outputs (RabbitMQ, Carbon-cache).

Note that such "middleware" does add a fair share of complexity so I'd recommend weighting that against the benefit

you'd get.

There is a couple of RabbitMQ plugins that are basically "socket gateways" (e.g. the UDP exchange) to RabbitMQ exchanges

but I don't think I've seen anything that targets your specific needs and there are downsides to developing things as plugins

(from doing it in Erlang or Elixir to keeping track of RabbitMQ internal changes between releases).

Pika is the recommended library for Python.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Chris Jefferies]

Michael,

I appreciate your feedback.  The classic problem is that we have system of perhaps 20 hosts being monitored by Icinga2 and Icinga2 sends it's metric data to a separate graphite server over the tcp socket port 2003.  If that connection between Icinga2 and Graphite is dropped, we end up with a big gap in the data (which might be critical to diagnose the problem).

I'm looking for a reliable mechanism to store and forward later, when the connection is lost/regained.  I think the cost of the complexity is worth it...  but we'll see.  ;)

Part of what I'm not understanding is how to have a robust connection or a restart connection in pika whenever the connection gets broken between rabbitmq and graphite/carbon-cache.

And to explain a little further, I've installed rabbitmq on the Icinga2 server so it's a localhost:2003 connection from Icinga2 into a python/pika script.  On the graphite/carbon-cache side there's a network connection as it has always been but instead of a direct tcp socket port 2003, there's a python/pika script consuming the icinga queue and then sending to carbon-cache on port 2003.

Thanks,

Chris.

[Michael Klishin]

I don't know if Pika supports automatic connection recovery but several clients do.

You can take a look at what Bunny and Java client do:

http://rubybunny.info/articles/error_handling.html#automatic_recovery

http://www.rabbitmq.com/api-guide.html

this approach works quite well and has been provided by a few clients for 3-4 years now.

Then again, you can relatively easily recover your connection and the small topology your system

will need manually. RabbitMQ in Action covers this in one of the chapters, for example.

[vitaly numenta]

Pika doesn't auto-recover lost connections, but it's easy to orchestrate recovery in the app; just create a new connection instance. Pika's synchronous adapter, BlockingConnection, signals lost connection via the ConnectionClosed exception. Pika's asynchronous adapters (SelectConnection, TornadoConnection, etc.) signal lost connection via callback.