On 07/22/2018 01:13 PM, Sergey Arlashin wrote:
> I'm using node_exporter to export basic servers' metrices. So I'd
> like to have nodes' hostnames to be visible in the alerts that I send
> using Alertmanager. The hostname is visible in the following metric
> on all my nodes that have node_exporter installed -
> node_uname_info{nodename="my-node-name"}. But I don't know how to put
> nodename value in alert's body.
>
> As I understand I need to relabel $labels.instance somehow.
Basically, I see several options:
(1) As you said, you can decide to use the hostname as the instance
value. This can be implemented using relabeling [1] (e.g. based on
__address__) or setting target-specific labels [2]. In any case, you
would need a source for the hostname (which cannot come from another
metric such as node_uname_info, as far as I know).
(2) You can also decide to keep the prometheus-side data as is and
instead enrich the data which is generated from your alerts. This way,
alertmanager would have access to those extra labels. An example can be
found at the RobustPerception blog as well [3] (although that
description is not tied to alertmanager in any way). You would have to
make sure that all your alert expressions contain this pattern.
(3) If all your alerts already have some label which contains the
necessary data but not in the required format (e.g.
instance=
foo.example.org:9100), then you could re-format this in your
templates (e.g. using reReplaceAll [4]).
It depends on your actual environment, what option fits best. For
example, we do service discovery using file_sd_config anyway, so this is
where we explicitly set the instance label to a value which makes sense
to us (a unique id from a CMDB). Therefore, we just use $labels.instance
in our alert templates (option #1). On the other hand, we pull in
further labels (e.g. for alert routing) using a custom metric (option #2).
Let us know if you need further guidance.
Kind regards,
Christian
[1]
https://www.robustperception.io/controlling-the-instance-label
[2]
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cstatic_config%3E
[3]
https://www.robustperception.io/exposing-the-software-version-to-prometheus
[4]
https://prometheus.io/docs/prometheus/latest/configuration/template_reference/