metrics.sh meets Prometheus

[patrick...@gmail.com]

Hey folks

Check out metrics.sh, a metrics collector and forwarding daemon written in shell scripts[1].

Prometheus support is available in form of an HTTP endpoint which serves data compatible with the exposition format v0.4.0[2]. The HTTP endpoint is provided using a simple `nc` multi-liner.

Let me know what you think and if there's anything that could be done to ensure compatibility.


Best,
Patrick


[1] https://github.com/pstadler/metrics.sh
[2] https://github.com/pstadler/metrics.sh/blob/master/reporters/prometheus.sh

[Julius Volz]

Hey Patrick,

Thanks for the Prometheus integration. While I personally prefer running statically built binaries like the node exporter for host metrics, I can imagine this coming in handy for some people.

I tried running metrics.sh locally, but noticed some issues:

1) whenever requesting http://localhost:9100/metrics, it seems to clear any metrics state for subsequent requests, so the next scrape will show an empty or reduced metrics state, until more metrics "accumulate" again. In Prometheus, a scrape should have no effect on the data (two parallel scrapes at the same time should get the same data). For example, counters should simply be in-memory values that are continuously incremented and a scrape would simply export their current state. Similar for the other metric types.

2) The metrics.sh metrics are of the form:

  metrics_sh{metric="network_io.out"}

This is not following Prometheus's metrics data model (http://prometheus.io/docs/concepts/data_model/). The metric name should be the first word before the brackets, and the labels should differentiate dimensional aspects of the same metric. For example, in Prometheus's node exporter, the same metric is exported as:

node_network_receive_bytes{device="eth0"}

node_network_receive_bytes{device="bond0"}

[...]

A job="<exporter-job-name>" label will be automatically added by Prometheus, telling you where the metric came from.

3) you set client-side timestamps for the samples, which should only be done in very rare power-user circumstances. Prometheus will automatically attach scrape-time timestamps.

Hope this helps!

Cheers,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julius Volz]

Also, Adrian's Bash Prometheus client library might be interesting: https://github.com/aecolley/client_bash

[patrick...@gmail.com]

Julius, thanks for your reply.

I'm trying to keep this short.

1) The idea is that when Prometheus scrapes data every n seconds, it will only get the current state per metric. To avoid this one could push to a gateway. Is this correct? (I'm aware of the fact that this makes timestamps superfluous)

2) Are gauges the right data type in my case?

3) Can I safely skip the curly braces for some metrics? Example:

cpu 10.3
memory 50.4
swap: 17.2
network_io{label="in"} 1.2
network_io{label="out"} 2.7


Cheers,
Patrick

[Brian Brazil]

On 29 March 2015 at 22:38, <patrick...@gmail.com> wrote:

Julius, thanks for your reply.

I'm trying to keep this short.

1) The idea is that when Prometheus scrapes data every n seconds, it will only get the current state per metric. To avoid this one could push to a gateway. Is this correct? (I'm aware of the fact that this makes timestamps superfluous)

Hmm, these all appear to be machine-level stats. You want https://github.com/prometheus/node_exporter#textfile-collector rather than push gateway in that case, which will also give you things like memory, cpu and disk for free.

2) Are gauges the right data type in my case?

Yes, though you should try to export counters where possible rather than doing math on the client side.

 

3) Can I safely skip the curly braces for some metrics? Example:

  cpu 10.3
  memory 50.4
  swap: 17.2
  network_io{label="in"} 1.2
  network_io{label="out"} 2.7

Yes, if there's no label there's no need for braces.

Brian