node-exporter textfile collector samples...

[Kevin Lyda]

One of the nice features of node_exporter is the texfiles collector which allows some lesser used things to be monitored without bloating out the node_exporter binary itself. However there are no scripts with the repo that people could use out of the box or to use as examples or starting points for their own uses.

I was thinking that might be a good thing to change.

Not really sure what to call these - samples, contrib, something like that. But it might be nice to expose some of these. It might also help to do them up with some standard comments for how to install them.

I'm currently spinning up monitoring for myself and a few clients so I'm going to start putting the scripts I do up here:

https://github.com/lyda/node_exporter/tree/textfile-contrib

Ideas on how this could be improved and eventually accepted would be gratefully received. And if people have their own scripts they'd like to add to this that would be good too. I've started with a fail2ban metrics collector (which the fail2ban-client did not make easy) and am next going to do one for parsing apache logs. I also will do one to gather smartmon data.

Would this be of interest to people? Is contrib/ a good directory name for this? Other questions?

Kevin

[Brian Brazil]

That'd be great alright. In theory I can do lm-sensors from Go, but it'll be much easier from shell.

Would this be of interest to people? Is contrib/ a good directory name for this? Other questions?

textfile_scripts maybe? Contrib could be taken to mean normal node_exporter modules.

One thing we should have is a small shell wrapper that'll take care of atomicity and aborting on error.

Brian

 

Kevin

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Kevin Lyda]

On Wednesday, October 21, 2015 at 6:36:21 PM UTC+1, Brian Brazil wrote:

textfile_scripts maybe? Contrib could be taken to mean normal node_exporter modules.

 

Sure. Like that. Done.

One thing we should have is a small shell wrapper that'll take care of atomicity and aborting on error.

Yes, that's clearly needed.  Something like:

*/5 * * * * root /path/to/textfile_runner SCRIPT

It would derive the name of the script and the output file from SCRIPT and add SCRIPT_exitcode to the output.

Kevin

[Brian Brazil]

I was thinking more that if it failed that it wouldn't write the output file, so at least you can get the old samples still.

Brian

 

Kevin

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Björn Rabenstein]

Another idea for an example: SMART data from `smartctl`. With that,
you can alert on disks giving SMART warnings.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Martín Ferrari]

Hi all,

On 21/10/15 20:45, Kevin Lyda wrote:

> One thing we should have is a small shell wrapper that'll take care
> of atomicity and aborting on error.

> Yes, that's clearly needed. Something like:
>
> */5 * * * * root /path/to/textfile_runner SCRIPT
>
> It would derive the name of the script and the output file from SCRIPT
> and add SCRIPT_exitcode to the output.

Thinking about this, I got this idea. Might not be worth the trouble,
but I would maybe implement it anyway from my own use:

* Store all textfile scripts in a directory, i.e.
/etc/prometheus-node-exporter/scripts.d
* Have a script that then runs them with the same semantics as
run-parts, handles errors and atomicity and creates some extra metrics
for each script.
* Maybe have node-exporter trigger the run instead of cron? this way,
there is no need to duplicate the target file configuration, and there
are less moving parts.

What do you think?

--
Martín Ferrari (Tincho)

[Kevin Lyda]

Björn comments:

> Another idea for an example: SMART data from `smartctl`. With that,

> you can alert on disks giving SMART warnings.

Yes. That's what I'm working on now. I had a disk die again so this is why I'm doing monitoring at this point in the first place.

Martín comments:

> Thinking about this, I got this idea. Might not be worth the trouble,

> but I would maybe implement it anyway from my own use:

>

> * Store all textfile scripts in a directory, i.e.

>   /etc/prometheus-node-exporter/scripts.d

I like this idea when packaging it. Though I think by FHS rules maybe it goes in /usr/libexec/prometheus or something like that. However when it comes to running it...

> * Have a script that then runs them with the same semantics as

>   run-parts, handles errors and atomicity and creates some extra metrics

>   for each script.

I don't think this works for a few reasons. Primarily permissions and frequency.

I run node_exporter as the node-exporter user (I use bbrazil's debs at the moment) so scripts can reasonably be run as the node-exporter or root users. The former is preferable but some scripts will need root access (the fail2ban one does for instance; I suspect smartctl will as well).

Scripts won't all need to be run with the same frequency. Some scripts might put load on a machine so you might not want it to run as often. Others will be lightweight and can be run more often. Eventually you'd just re-implement cron and since cron is almost always there...

It would be nice; it would make installation easier. But I think things like /etc/cron.d make cron pretty easy to script for configuration.

Kevin

[Kevin Lyda]

On Wednesday, October 21, 2015 at 10:54:16 PM UTC+1, Björn Rabenstein wrote:

Another idea for an example: SMART data from `smartctl`. With that,
you can alert on disks giving SMART warnings.

There's a first pass in there but needs work.

Kevin 

[Brian Brazil]

That's pretty much been my thinking on this question, and why the textfile collector works the way it does. I don't think we gain much by reinventing cron, versus just using cron.

Brian

 

Kevin

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[René Treffer]

Hi,

Sorry for reviving an old discussion, but I just started monitoring my home box with prometheus. As this is a home-grown disk box (>8 2.5'' disk, btrfs raid10) I wanted to monitor disk activity (which node-exporter) as well as smart attributes.

I didn't want to parse smartctl output so I linked against libatasmart. I've dumped the (ugly+initial) code into a gist (including sample output of my laptops SSD) (feel free to do whatever you want with the code)
https://gist.github.com/rtreffer/4ca899ed926955078099b8f623ff3c59
gcc main.c -latasmart && sudo ./a.out /dev/sd?
Each metric is exported as
- The smart "value" (usually 100-0, 200-0 or 253-0)
- The raw value
- The pretty value (e.g. degree kelvin, with a suffix if possible, e.g. _ms, _bytes, temp_k, _percent....) (provided by libatasmart)
- The remaining health (value - threshold) (negative values or zero means predicted disk failure - note that the scale is vendor dependent and thus pointless except for sign)
Each data point is labeled with device path (e.g. /dev/sdX) and smart identity values (model, serial, firmware revision). This should make it easy to monitor all disks of a certain kind for common failures.
libatasmart was written because smartctl is not a lib and thus doesn't offer good integration with monitoring tools or the desktop.... The only downside I've found is it won't handle smart attributes of disks behind a megaraid controller in RAID mode (I just read the code but it seems like it's missing an equivalent to smartctl -d megaraid,X /dev/sda).

Side note: smartctl and libata smart use a command pass through mode which is only available for root and binaries with special capabilities. So the assertion that a smart util requires root will hold. Simply strace what smartctl is doing, or use smartctl -r ioctl to see what's going on.

Temperature works pretty well: https://drive.google.com/file/d/0Bxx_x6DuLA2hdnZxdl9sMXNwalk/view (cron currently running every 1h for testing, so not a smooth chart)

Now I'd like to build a disk error rate chart to see if any disk starts spilling out errors. For this I'd like to sum up all error related smart attributes. I'd like to keep it in one chart as most of the time it will hopefully be a zero rate and increases should just be one disk.
This means I'd have to add smart attribute raw values, but each disk has a different set of metrics, basically a sparse instant vector, and the intersect will be empty / all disks will be dropped from the graph (e.g. the SSD has wear counters whereas the HDD reports seek errors - adding both will drop all disks).

Is there a way to add sparse instant vectors? Is this planned for the future?

Regards,
  Rene Treffer

PS: I once had a RAID of disks known to loose data during SMART inqueries. I never bought a pack of the same disk for a RAID again. Just in case anyone is wondering.

[Richard Hartmann]

On Tue, May 17, 2016 at 8:51 AM, René Treffer <rtre...@gmail.com> wrote:
> This means I'd have to add smart attribute raw values, but each disk has a
> different set of metrics, basically a sparse instant vector, and the
> intersect will be empty / all disks will be dropped from the graph (e.g. the
> SSD has wear counters whereas the HDD reports seek errors - adding both will
> drop all disks).

Just to make sure: Are you planning to ingest as one timeseries per
disk or one time series per metric?

If the latter, you just need to sum & select the relevant ones. If the
former, that's an anti-pattern.


I will most likely use your code once it's settled down :)


Richard

[Ben Kochie]

We wrote a shell script to parse smartctl output, but I haven't had time to clean it up for committing to the node_exporter repo.  Having something that uses the smart library is much better.

We generally separate the model stuff into an INFO metric like this:

smartmon_device_info{disk="/dev/bus/0",type="sat+megaraid,0",model_family="Intel 730 and DC S35x0/3610/3700 Series SSDs",device_model="INTEL SSDSC2BB300G4",serial_number="XXXX",firmware_version="XXXX"} 1

Then the actual metrics have only labels we would use to aggregate over:

smartmon_current_pending_sector_raw_value{disk="/dev/bus/0",type="sat+megaraid,0",smart_id="197"} 0

On Tue, May 17, 2016 at 8:51 AM, René Treffer <rtre...@gmail.com> wrote:

--

[Richard Hartmann]

On Tue, May 17, 2016 at 10:06 AM, Ben Kochie <sup...@gmail.com> wrote:
> We wrote a shell script to parse smartctl output, but I haven't had time to
> clean it up for committing to the node_exporter repo. Having something that
> uses the smart library is much better.

I do wonder if an incubator of sorts for "we use this, it works, and
it's ugly. please play and improve" might make sense. The obvious risk
is that this creates de-facto standards, but that's more a problem on
the user's than on the project's side.


Richard

[René Treffer]

Am 17.05.2016 9:29 vorm. schrieb "Richard Hartmann" <richih.ma...@gmail.com>:

On Tue, May 17, 2016 at 8:51 AM, René Treffer <rtre...@gmail.com> wrote:
> This means I'd have to add smart attribute raw values, but each disk has a
> different set of metrics, basically a sparse instant vector, and the
> intersect will be empty / all disks will be dropped from the graph (e.g. the
> SSD has wear counters whereas the HDD reports seek errors - adding both will
> drop all disks).

Just to make sure: Are you planning to ingest as one timeseries per
disk or one time series per metric?

If the latter, you just need to sum & select the relevant ones. If the
former, that's an anti-pattern.

One timeseries per disk, in one chart.

I should perhaps explain it a bit more verbose....

Each disk has a set of error counters, e.g. for my laptop ssd I'd say Erase_Fail_Count_Chip,Program_Fail_Cnt_Total,Erase_Fail_Count_Total,Reported_Uncorrect,Hardware_ECC_Recovered for an HDD it might include Seek_Error_Rate. Reallocated sector metrics might indicate errors, too.

The raw value is pointless, but the rate at which errors happen is quite relevant. If a disk throws more than a handful of errors over 24h for a few days it's usually broken, or will be broken soonish.

So I'd like to say "rate(error_type_a + error_type_b + .... by (model, serial)". How much information I'm getting per disk will vary, but it should be a 0 rate so I don't care if an increase on Disk A is comparable to Disk B - I only care if errors happen at all and over some time. error_type_a might include only SSDs, whereas error_type_b might include only HDD, meaning the set of labels between all values has an empty intersect, resulting in an empty result.

As this is usually a 0 rate the graph would be quite boring, usually everything would be a flat line unless some disks begin to fail.

Now I could go ahead and write this down as a metric for each model (or even model + firmware) and split those by serial, but I don't like writing that much :-) For monitoring a datacenter (few disk vendors vs. many disks) it might be worth it, but not for my home setup.

From my understanding such a query is currently not supported by prometheus. It would _usually_ result in garbage, however in this specific case I think it would be useful....

 

Regards,

  Rene Treffer

[Matthias Rampke]

I wouldn't sum these errors at all – they are not the same dimension
so summing them makes no sense. It may seem "useful" to aggregate
before alerting, but in Prometheus it is much easier to just define a
single alert that covers all disks and all types of errors separately:

ALERT DiskErrors
IF increase({__name__=~"error_type_.*"}[24h]) > 0

This will generate alerts for all disks and type of error separately,
and Alertmanager lets you handle these alerts in aggregate. A certain
disk not having a certain metrics just means that that specific label
combination doesn't happen and thus will never be >0. There is no need
to handle all the combinations.

If you absolutely insist on summing over different error types, move
it into a label dimension:

disk_errors{error_type="Erase_Fail_Count_Chip",model="Laptop
SSD",serial="dead"} 1
disk_errors{error_type="Seek_Error_Rate",model="Desktop HDD",serial="1234"} 10


This way,

sum by (model,serial) (rate(disk_errors[24h]))

does what you want. Or you still use the alert above, and the
`__name__=~` hack is no longer necessary:

ALERT DiskErrors
IF increase(disk_errors[24h]) > 0

/MR

> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-devel...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Matthias Rampke
Engineer

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany | +49
173 6395215

[René Treffer]

For alerting I'm convinced :-)

But I'd still like to _graph_ them as "disk errors reported by disk per time unit". I think the labels already explains the limitation (it's what the disk errors reports and thus by definition vendor/firmware dependent) and what to expect.
It would be the sum of raw values of all attributes that represent error counters (which in the worst case has to take vendor specific attributes into account) with a rate/increase applied. Not all disks report the same set of errors, true. But 100 reported errors / hour should always translate into 100+ latency issues on a busy disk and is thus s.th. useful to see in an overview IMHO.

For alerting or debugging I'd always use the split metrics.

Merging metrics into one and using labels sounds interesting although I dislike doing magic while reading the smart attributes, but I guess I have to think again what should go into labels and metric names. Ben Kochie posted a different schema, too.

Regards,

  Rene Treffer

[Richard Hartmann]

On Tue, May 17, 2016 at 11:58 AM, René Treffer <rtre...@gmail.com> wrote:
> For alerting I'm convinced :-)
>
> But I'd still like to _graph_ them as "disk errors reported by disk per time
> unit".

The query for graphing and alerting is usually the same (with the
exception of rate vs irate)

> For alerting or debugging I'd always use the split metrics.

So you need to keep them anyway ;)

> Merging metrics into one and using labels sounds interesting although I
> dislike doing magic while reading the smart attributes, but I guess I have
> to think again what should go into labels and metric names. Ben Kochie
> posted a different schema, too.

It's not magic, you are simply slicing and dicing your data the way you need it.



Richard