FYI: Sysctl exporter for FreeBSD

[Ed Schouten]

Hi there,

A couple of days ago I started working on a small project that I
thought I'd announce on this list as well, namely a generic metrics
exporter for sysctls on FreeBSD.

One of the goals of the Prometheus node exporter is to create
structure where there is a lack of it. It fetches data from /proc, a
handful of sysctls, various system calls, maybe even ioctl, and
exports it in a unified format. On systems that provide sysctl, there
is already access to a large number of metrics with some structure.
Unfortunately, it's not trivial to map all of these straight into
Prometheus, as the semantics differ. Consider this sysctl object:

$ sysctl hw.acpi.thermal.tz0.temperature
hw.acpi.thermal.tz0.temperature: 27.8C

You could use some simple substitution to map this to:

sysctl_hw_acpi_thermal_tz0_temperature 27.8

While ideally you'd want to obtain something like this:

sysctl_hw_acpi_thermal_temperature_celcius{thermal_zone="tz1"} 27.8

To make this possible, I've made two changes to FreeBSD.

1. Label annotations for in-kernel sysctl objects.

I've extended the FreeBSD in-kernel sysctl API, so that objects in the
sysctl tree can be annotated to have an aggregation label. In the
example I gave above, it allows us to attach a "thermal_zone" label to
the "tz0" object, which applications can use as a hint that it makes
sense to aggregate underlying objects with the ones from its siblings.

https://svnweb.freebsd.org/base/head/sys/dev/acpica/acpi_thermal.c?r1=301562&r2=310054

All of these changes are already present on FreeBSD SVN HEAD.

2. A simple exporter for the FreeBSD base system.

I've written a tool that acts similar to the sysctl(8) command line
tool, except that it prints all sysctls in Prometheus' metrics format.
By running it through inetd, its output can be scraped by Prometheus
directly. What is annoying is that it has to make use of a fair number
of private APIs, so that it can do the following:

- walk the sysctl tree,
- do object-to-name translations,
- extract descriptions, which the exporter uses for '# HELP' annotations,
- extract the aggregation labels,
- extract type information, so temperatures/timestamps can have
_celcius/_seconds suffixes, etc.

For this reason I think it makes a lot of sense to have it included in
the FreeBSD source tree, as opposed to being independent or under the
Prometheus project's umbrella. The code for this exporter is currently
in review at FreeBSD:

https://reviews.freebsd.org/D8792

What I like about this approach so far is that it really lowers the
barrier for exporting even more metrics. We can now add a sysctl to
the kernel and start scraping it immediately, without needing to write
yet another collector. It also means that the node exporter doesn't
need to gain collectors for things that only a handful of people may
find interesting. The exporter executable is also tiny (~20 kB) and
has a small CPU/memory footprint, so it may also be useful for
monitoring embedded FreeBSD-based systems.

My goal is that this exporter will be part of FreeBSD 12 (due late
2017, early 2018, I guess). I can't merge it back to the 11.x/10.x
stable branches, for the reason that the label annotations feature
breaks the in-kernel binary interface.

Best regards,
--
Ed Schouten <e...@nuxi.nl>
Nuxi, 's-Hertogenbosch, the Netherlands
KvK-nr.: 62051717

[Ben Kochie]

Having native instrumentation like this is great.  It simplifies things on both the Prometheus side, and the kernel development side.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CABh_MKk0F_Jawvg81DJR293Vm9dyam2SvE%2BaivuGPo%2BWgzaOHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.