Beefing up FreeBSD support in node_exporter

708 views
Skip to first unread message

Derek M

unread,
Feb 2, 2017, 9:13:46 AM2/2/17
to Prometheus Developers
Hello!

Been playing around with node_exporter on FreeBSD.   Would like some feedback on some thoughts I've had about implementing functionally before I sit down, code it, and throw it over the wall. :D

I anticipate these to be some series of smaller projects, assuming that we are going ahead with a roll-out internally, but here's the big picture:

Build capability:

  • export missing relevant values (stat):

    • node_procs_running

    • node_procs_blocked

    • node_forks (sysctl)

    • node_context_switches (sysctl)

    • node_intr (sysctl)

  • export missing relevant values (disk):

    • node_disk_io_time_ms (devstat call)

    • node_disk_sectors_read (devstat call)

    • node_disk_sectors_written (devstat call)

  • Export ZFS values to align with linux_zfs when work there settles down a bit

Why not overlap linux labels where it makes sense to do so? (so the prometheus consoles mostly work out of the box)

  • move node_cpu_seconds_total from cpu to stat node_cpu to align with linux

  • rename relevant memory counters to match /proc/mem, add missing:

  • node_memory_Buffers (sysctl)
    node_memory_SwapTotal (sysctl)
    node_memory_SwapFree (kmv_ call)


Thoughts, feedback, discussion welcome and appreciated.
Derek

Brian Brazil

unread,
Feb 2, 2017, 9:24:26 AM2/2/17
to Derek M, Prometheus Developers
On 2 February 2017 at 14:13, Derek M <derek.m...@experiencepoint.com> wrote:
Hello!

Been playing around with node_exporter on FreeBSD.   Would like some feedback on some thoughts I've had about implementing functionally before I sit down, code it, and throw it over the wall. :D

I anticipate these to be some series of smaller projects, assuming that we are going ahead with a roll-out internally, but here's the big picture:

Build capability:

  • export missing relevant values (stat):

    • node_procs_running

    • node_procs_blocked

    • node_forks (sysctl)

    • node_context_switches (sysctl)

    • node_intr (sysctl)

  • export missing relevant values (disk):

    • node_disk_io_time_ms (devstat call)

    • node_disk_sectors_read (devstat call)

    • node_disk_sectors_written (devstat call)

  • Export ZFS values to align with linux_zfs when work there settles down a bit

More metrics are good. Beware some of the Linux ones aren't the best named.
 

Why not overlap linux labels where it makes sense to do so? (so the prometheus consoles mostly work out of the box)

  • move node_cpu_seconds_total from cpu to stat node_cpu to align with linux


The BSD name is better practice, we'll be changing the Linux metric name to it at some point.
 

  • rename relevant memory counters to match /proc/mem, add missing:

  • node_memory_Buffers (sysctl)
    node_memory_SwapTotal (sysctl)
    node_memory_SwapFree (kmv_ call)

I'd not worry too much about getting these identical, it's very unlikely the semantics are exactly identical across kernels.

Brian 


Thoughts, feedback, discussion welcome and appreciated.
Derek

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/61b280b6-9c13-43e1-b269-c25485599bb4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Derek M

unread,
Feb 24, 2017, 7:10:58 AM2/24/17
to Prometheus Developers
As mentioned (and have received good feedback on), I've submitted some PRs with regard to FreeBSD support.

I've got a new branch that builds on the memory branch, using the kvm_ calls to determine swap usage:


I have some architectural concerns with respect to the way that things get split out.   It would seem to me that the modules are organized loosely by "where" as opposed to separating the "where" and the "what".

Put differently, it would make sense to me to have a model+view separation, where there are drivers that know how to read from say, sysctl + kvm, and the view drivers consume and transform the data from the models, and present them in a related fashion.   Specifically, it would seem that swap used should live under the memory utilization section, and that it's an implementation detail that FreeBSD exposes this through kvm_ calls.

Looking forward to feedback/thoughts
Derek

Matthias Rampke

unread,
Feb 27, 2017, 5:35:00 AM2/27/17
to Derek M, Prometheus Developers
For Linux, there is a start made factoring out the "model" into https://github.com/prometheus/procfs – would something similar make sense for *BSD?

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6c7b7100-e8a6-44fb-b32e-2385cc1ad49a%40googlegroups.com.

Johannes Ziemke

unread,
Feb 27, 2017, 5:42:22 AM2/27/17
to Matthias Rampke, Derek M, Prometheus Developers
Agreed, extracting more stuff into 'model' packages like procfs probably makes sense and is easier to coordinate than a big refactoring of the node-exporter. But I'm open either way.

On Mon, Feb 27, 2017 at 11:34 AM, Matthias Rampke <m...@soundcloud.com> wrote:
For Linux, there is a start made factoring out the "model" into https://github.com/prometheus/procfs – would something similar make sense for *BSD?
On Fri, Feb 24, 2017 at 12:10 PM Derek M <derek.marcotte@experiencepoint.com> wrote:
As mentioned (and have received good feedback on), I've submitted some PRs with regard to FreeBSD support.

I've got a new branch that builds on the memory branch, using the kvm_ calls to determine swap usage:


I have some architectural concerns with respect to the way that things get split out.   It would seem to me that the modules are organized loosely by "where" as opposed to separating the "where" and the "what".

Put differently, it would make sense to me to have a model+view separation, where there are drivers that know how to read from say, sysctl + kvm, and the view drivers consume and transform the data from the models, and present them in a related fashion.   Specifically, it would seem that swap used should live under the memory utilization section, and that it's an implementation detail that FreeBSD exposes this through kvm_ calls.

Looking forward to feedback/thoughts
Derek

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAFU3N5VAMV%2Bh4T19w6oF2W2OVH4z45tADgO7hRozp0pU2FgR0Q%40mail.gmail.com.

Tobias Schmidt

unread,
Mar 1, 2017, 10:16:51 AM3/1/17
to Johannes Ziemke, Matthias Rampke, Derek M, Prometheus Developers
I agree that the naming of some of the collectors is not ideal right now and your proposal to group them around "what they expose" makes sense to me. Similar to #150 concerned with consolidating metric names, consolidating collector names seems sound.


On Mon, Feb 27, 2017 at 6:42 AM Johannes Ziemke <fi...@freigeist.org> wrote:
Agreed, extracting more stuff into 'model' packages like procfs probably makes sense and is easier to coordinate than a big refactoring of the node-exporter. But I'm open either way.
On Mon, Feb 27, 2017 at 11:34 AM, Matthias Rampke <m...@soundcloud.com> wrote:
For Linux, there is a start made factoring out the "model" into https://github.com/prometheus/procfs – would something similar make sense for *BSD?
On Fri, Feb 24, 2017 at 12:10 PM Derek M <derek.m...@experiencepoint.com> wrote:
As mentioned (and have received good feedback on), I've submitted some PRs with regard to FreeBSD support.

I've got a new branch that builds on the memory branch, using the kvm_ calls to determine swap usage:


I have some architectural concerns with respect to the way that things get split out.   It would seem to me that the modules are organized loosely by "where" as opposed to separating the "where" and the "what".

Put differently, it would make sense to me to have a model+view separation, where there are drivers that know how to read from say, sysctl + kvm, and the view drivers consume and transform the data from the models, and present them in a related fashion.   Specifically, it would seem that swap used should live under the memory utilization section, and that it's an implementation detail that FreeBSD exposes this through kvm_ calls.

Looking forward to feedback/thoughts
Derek

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CANi%3DR%3D1nF9FkRb6%2BM51ZTwn3u4JWJHFE8JNRdkey_W5baNdJGQ%40mail.gmail.com.

Derek M

unread,
Mar 1, 2017, 11:27:21 AM3/1/17
to Prometheus Developers, fi...@freigeist.org, m...@soundcloud.com, derek.m...@experiencepoint.com
Thanks.  Kind of related to that thread, while it has been established that it doesn't make sense for all the metric names to exactly align between platforms, I think it'd make sense for the Subsystem names to be fairly common as we move up the level of abstraction, eg:

physical

cpu
memory
disk
network

kernel
exec/scheduler
zfs/volume

process
filesystem
sockets

Additionally I'll take a look at the procfs work and see what I can make of it.

Thanks for the feedback so far.

Derek

On Wednesday, 1 March 2017 10:16:51 UTC-5, Tobias Schmidt wrote:
I agree that the naming of some of the collectors is not ideal right now and your proposal to group them around "what they expose" makes sense to me. Similar to #150 concerned with consolidating metric names, consolidating collector names seems sound.


On Mon, Feb 27, 2017 at 6:42 AM Johannes Ziemke <fi...@freigeist.org> wrote:
Agreed, extracting more stuff into 'model' packages like procfs probably makes sense and is easier to coordinate than a big refactoring of the node-exporter. But I'm open either way.
On Mon, Feb 27, 2017 at 11:34 AM, Matthias Rampke <m...@soundcloud.com> wrote:
For Linux, there is a start made factoring out the "model" into https://github.com/prometheus/procfs – would something similar make sense for *BSD?
On Fri, Feb 24, 2017 at 12:10 PM Derek M <derek.m...@experiencepoint.com> wrote:
As mentioned (and have received good feedback on), I've submitted some PRs with regard to FreeBSD support.

I've got a new branch that builds on the memory branch, using the kvm_ calls to determine swap usage:


I have some architectural concerns with respect to the way that things get split out.   It would seem to me that the modules are organized loosely by "where" as opposed to separating the "where" and the "what".

Put differently, it would make sense to me to have a model+view separation, where there are drivers that know how to read from say, sysctl + kvm, and the view drivers consume and transform the data from the models, and present them in a related fashion.   Specifically, it would seem that swap used should live under the memory utilization section, and that it's an implementation detail that FreeBSD exposes this through kvm_ calls.

Looking forward to feedback/thoughts
Derek

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

Derek M

unread,
Mar 27, 2017, 4:33:44 PM3/27/17
to Prometheus Developers, fi...@freigeist.org, m...@soundcloud.com, derek.m...@experiencepoint.com
Been mulling over this for a bit.  Here's what I came up with:

Namespaces

Takes a set of collectors and transforms them into a unified namespace.

1 Node Exporter

has at least one 

Namespace

has at least one, and is a

Collector

Detail

Namespaces are responsible for the composition of lower level collectors, e.g. sysctl vs kvm vs procfs, into a single set of:

- end-user facing names
- system dependant wiring/translation details
- - Generic Namespace struct is very sparse, may only have one value, two methods (say init, and collect)
- including arbitration of shared state between child collectors (e.g. resource handles, etc)

Namespaces could be composed of other namespaces, but it seems like this is too much, in terms of wiring details, for example.

This will allow us to compose a set of wheres into a what that makes sense.

For example, meminfo, has got a very strange coupling of system-dependent code.  What this could look like is:


node_exporter -> namespace("meminfo")(BSD) -> sysctl
                                           -> kvm
              -> namespace("meminfo")(Darwin) -> sysctl
                                              -> host_statistics
              -> namespace("meminfo")(Linux) -> procfs


So we decouple the sysctl code, the kvm code and the procfs code into self-contained libraries.

Is there any interest in a PoC for something like this?

Thanks
Derek

Derek M

unread,
Apr 6, 2017, 5:04:09 AM4/6/17
to Prometheus Developers, fi...@freigeist.org, m...@soundcloud.com, derek.m...@experiencepoint.com
Hi!

I've skipped the namespace business, but I now have a pull request in that moves memory reporting on freebsd + dragonfly out from under meminfo, and adds support for checking swap.



The CI fails on cross-build with different permutations of:

"C source files not allowed when not using cgo or SWIG: kvm_bsd.c"

Okay - that's fine.  Kind of expected.  What I don't get is why we don't get the same with devstat_freebsd.c?  Is it as simple as the filename?  Any pointers would be appreciated here.

Thanks
Derek

Johannes Ziemke

unread,
Apr 6, 2017, 5:16:19 AM4/6/17
to Derek M, Prometheus Developers, Matthias Rampke
Hi Derek,

the design you proposed sounds reasonable to me, but would like to hear some other opinion before I tell you to go for it and then people end up complaining on the PR later.

Regarding your PR: What is the reason for moving it out from meminfo? Already as part for further redesign? If so, we should decide for that first, otherwise the change is confusing.

Derek
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/83840971-7ffe-4509-b1d5-8310083c1e50%40googlegroups.com.

Derek M

unread,
Apr 6, 2017, 6:19:40 AM4/6/17
to Prometheus Developers, derek.m...@experiencepoint.com, m...@soundcloud.com
Thanks for your thoughts.  I've had this kvm/stuff stuff hanging around for about a month in my tree, so I thought I'd get it committed (good) before a larger decision is made regarding the structure of things (perfect).

In this case, I feel hamstrung by the constraints imposed in meminfo.go, for example:

- I'd like to extend meminfoCollector in a system-dependent way
- Memory stats aren't always Gauges
- Might be nice to add better descriptions (although I haven't yet)

This is what precipitated the move - it felt like an unnecessary abstraction that wasn't adding much value.  (Rather than an attempt to re-organize things WRT naming/structure.)

Hope that helps!
Derek
Derek
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CANi%3DR%3D1nF9FkRb6%2BM51ZTwn3u4JWJHFE8JNRdkey_W5baNdJGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

Brian Brazil

unread,
Apr 6, 2017, 6:23:18 AM4/6/17
to Derek M, Prometheus Developers, Matthias Rampke
On 6 April 2017 at 11:19, Derek M <derek.m...@experiencepoint.com> wrote:
Thanks for your thoughts.  I've had this kvm/stuff stuff hanging around for about a month in my tree, so I thought I'd get it committed (good) before a larger decision is made regarding the structure of things (perfect).

In this case, I feel hamstrung by the constraints imposed in meminfo.go, for example:

- I'd like to extend meminfoCollector in a system-dependent way
- Memory stats aren't always Gauges

They should be listed as untyped.

Brian
 
Derek
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages