thoughts on collectd.

[Kevin Burton]

I spent a few hours tonight getting collectd working with kairosdb.  Here are my thoughts.

1.  collectd is kind of flakey.  The config file parser is prone to whitespace issues.

2.  the default kairosdb puts the hostname in the metric name.  this should be a tag.

3.  by default the hostname is mangled in the tag and the metric name to . -> _ 

4.  there's no way to tag a plugin.  the entire collectd system reports only a fixed set of tags to kairosdb. this means if you have daemons 'foo' and 'bar' running on your box you have no way to differentiate between them.

if collectd supported tags then we would be golden.  but it doesn't :-/

I'm going to investigate using multiple collectd daemons ... 

[Brian Hawkins]

Before you go to far Jeff has made some changes to the collectd plugin, making it very configurable.  https://github.com/jsabin/collectd-kairosdb

The goal is to have it pulled back into the owners branch.  Jeff added http support and the ability to pull parts of the metric and make them tags, and the . -> _ is configurable btw.

He would love input/help on it.  We used it to grab system/jmx metrics from our cassandra nodes.

Brian

[Kevin Burton]

Oh perfect.  I didn't spend much time on the previous one. I didn't see Jeff's plugin or I would have used it.

Could you share your collectd.conf ?  I will start migrating to it this early next week as this is a problem I need to get solved soon.

The JMX issue was the big motivator for me.

[jsabin]

Kevin,

collectd.conf - This is the main config file for collectd. The plugins not commented out are the ones we are currently using. We are currently watching load average, disk space, network, and cassandra JMX metrics.

Note the following lines adds the kairosdb and the cassndra configurations.

<Include "/opt/proofpoint/collectd/etc/collectd.d">
  Filter "*.conf"
</Include>

kairosdb.conf - configures the kairosdb collectd plugin. This is a collectd writer that sends the metrics to KairosDB. I modified this plugin to add the HTTP protocol. I also changed it to be more flexible in specifying the metric name.

cassandra.conf - This is the configuration for the FastJMX plugin that retrieves JMX metrics from Cassandra. 

cassndraFormatter.py - I wanted a way to take metrics for each column family and add them as a single metric with the column family name as a tag. So I changed the karios plugin to optionally call formatter code. This formatter pulls the column family name and adds it as a tag.

I also tried to document the plugin a little better. Glad to have feedback on it. 

Hope this is helpful.

Jeff

[Kevin Burton]

Nice!  Thanks. I'l deploy this on the weekend to how it works.

[Darshan Patil]

Hi Jeff, 

I am trying to configure collectd to insert data to Kairosdb. I came across this post https://code.google.com/p/kairosdb/wiki/Graphite which explains how you can enable graphite protocol in kairosdb, which will let you push data to Kairos from applications that normally push to Graphite. Will it work if I use the graphite module in collectd to send data to kairosdb after enabling the graphite protocol in kairosdb? or we have to use the collectd plugin( https://github.com/jsabin/collectd-kairosdb)? 

Darsh

[Brian Hawkins]

Use this plugin with collectd: https://github.com/kairosdb/collectd-kairosdb

The graphite protocol lacks ability to send tags and is a little harder to work with. The collectd plugin has a lot of nice features and provides logic to add formatters for your metrics.  We have added formatters for cpu that takes the cpu number and makes it a tag for example.

Brian

[Darshan Patil]

Thank you so much for your reply. I tried  https://github.com/kairosdb/collectd-kairosdb and it works perfectly fine.