Alertmanager: display hostnames in alerts

1,578 views
Skip to first unread message

Sergey Arlashin

unread,
Jul 22, 2018, 7:13:28 AM7/22/18
to Prometheus Users
Hi!

I'm new to Prometheus Alertmanager. Searched the Web for some time but unfortunately haven't found an answer.

I'm using node_exporter to export basic servers' metrices. So I'd like to have nodes' hostnames to be visible in the alerts that I send using Alertmanager. The hostname is visible in the following metric on all my nodes that have node_exporter installed - node_uname_info{nodename="my-node-name"}. But I don't know how to put nodename value in alert's body.

As I understand I need to relabel $labels.instance somehow.

Can anybody help me?

Thanks!

Regards,
Sergey


Christian Hoffmann

unread,
Jul 22, 2018, 7:32:17 AM7/22/18
to Sergey Arlashin, Prometheus Users
On 07/22/2018 01:13 PM, Sergey Arlashin wrote:
> I'm using node_exporter to export basic servers' metrices. So I'd
> like to have nodes' hostnames to be visible in the alerts that I send
> using Alertmanager. The hostname is visible in the following metric
> on all my nodes that have node_exporter installed -
> node_uname_info{nodename="my-node-name"}. But I don't know how to put
> nodename value in alert's body.
>
> As I understand I need to relabel $labels.instance somehow.

Basically, I see several options:

(1) As you said, you can decide to use the hostname as the instance
value. This can be implemented using relabeling [1] (e.g. based on
__address__) or setting target-specific labels [2]. In any case, you
would need a source for the hostname (which cannot come from another
metric such as node_uname_info, as far as I know).

(2) You can also decide to keep the prometheus-side data as is and
instead enrich the data which is generated from your alerts. This way,
alertmanager would have access to those extra labels. An example can be
found at the RobustPerception blog as well [3] (although that
description is not tied to alertmanager in any way). You would have to
make sure that all your alert expressions contain this pattern.

(3) If all your alerts already have some label which contains the
necessary data but not in the required format (e.g.
instance=foo.example.org:9100), then you could re-format this in your
templates (e.g. using reReplaceAll [4]).

It depends on your actual environment, what option fits best. For
example, we do service discovery using file_sd_config anyway, so this is
where we explicitly set the instance label to a value which makes sense
to us (a unique id from a CMDB). Therefore, we just use $labels.instance
in our alert templates (option #1). On the other hand, we pull in
further labels (e.g. for alert routing) using a custom metric (option #2).

Let us know if you need further guidance.

Kind regards,
Christian

[1] https://www.robustperception.io/controlling-the-instance-label
[2]
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cstatic_config%3E
[3]
https://www.robustperception.io/exposing-the-software-version-to-prometheus
[4]
https://prometheus.io/docs/prometheus/latest/configuration/template_reference/

Sergey Arlashin

unread,
Jul 22, 2018, 3:55:40 PM7/22/18
to Christian Hoffmann, Prometheus Users
Christian,

Thank you so much for your answer.

Since in my environment service discovery is implemented with Consul, I ended up using Consul meta for this

relabel_configs:
- source_labels: [__meta_consul_node]
regex: (.+)
target_label: instance
replacement: '${1}'


Regards,
Sergey
Reply all
Reply to author
Forward
0 new messages