Getting hostnames along with IPs in Alerts.

[Yagyansh S. Kumar]

I am providing IPs as my targets and have configured many alerts rules based on different conditions. Since, node_uname_info gives the nodename for the provided instance, is there any way to get that nodename to be printed in the alerts along with the instance's IP?

I have more than 2500+ servers in my config file hence, I don't want to reconfigure my whole config file and add DNS names  as targets instead of IPs.

Thanks!

[Brian Candler]

Yes. There are two basic approaches.

The first (and more complex) is to join node_uname_info to pick extra labels to add to your alert.  Example (untested):

  - alert: DiskFull

    expr: |

      (node_filesystem_avail_bytes < 100000000 and node_filesystem_size_bytes > 100000000)

      * on (instance) group_left(nodename) node_uname_info

More info:

* https://www.robustperception.io/how-to-have-labels-for-machine-roles

* https://www.robustperception.io/exposing-the-software-version-to-prometheus

* https://www.robustperception.io/left-joins-in-promql

The downside is that it makes all your alerting rules more complex.

The second approach, specifically for your problem of having the hostnames in alerts, is to put a meaningful name in the "instance" label, rather than the IP address.  This approach is described here:

* https://www.robustperception.io/target-labels-are-for-life-not-just-for-christmas/

* https://www.slideshare.net/roidelapluie/taking-advantage-of-prometheus-relabeling-109483749

This makes not only alerts but also dashboards much more usable.

Taking this to the limit, I create my targets files with entries of the form "foo/1.2.3.4" and use relabelling so that the name "foo" ends up in the instance label, and the IP address is the target which is actually scraped.  Details:

https://groups.google.com/d/msg/prometheus-users/VafEFQQKdWc/YrChiAePBQAJ

[Brian Candler]

On Thursday, 5 March 2020 16:13:33 UTC, Brian Candler wrote:

The second approach, specifically for your problem of having the hostnames in alerts, is to put a meaningful name in the "instance" label, rather than the IP address.  This approach is described here:

Sorry, I meant to link to this:

* https://www.robustperception.io/controlling-the-instance-label

[Yagyansh S. Kumar]

Cool, thanks a ton.

Just one doubt. I have tried and have been using the join method that you mentioned, in grafana for getting the hostnames as legends but how do I extract the hostname from the expression in alertmanager? I mean we have the expression but I am stuck at getting the nodename out of it.

expr: |

      (node_filesystem_avail_bytes < 100000000 and node_filesystem_size_bytes > 100000000)

      * on (instance) group_left(nodename) node_uname_info

Now, how do I extract the nodename? $labels.nodename ?

[Yagyansh S. Kumar]

Found my stupid mistake. It works now. Thanks!

I have another query though. This maybe just a ridiculous question, but is there any way I can get the hostnames for the targets I configured as ICMP ping targets for Blackbox exporter? I know Blackbox exporter itself does not expose any hostname but still is there something that can be done?

[Brian Candler]

On Thursday, 5 March 2020 19:23:51 UTC, Yagyansh S. Kumar wrote:

Found my stupid mistake. It works now. Thanks!

I have another query though. This maybe just a ridiculous question, but is there any way I can get the hostnames for the targets I configured as ICMP ping targets for Blackbox exporter? I know Blackbox exporter itself does not expose any hostname but still is there something that can be done?

Not a ridiculous question.  You can do exactly the same with blackbox: you need to set the __param_target rather than the __address__ (because you will set __address__ to point to blackbox_exporter itself)

  - job_name: blackbox

    file_sd_configs:

      - files:

        - /etc/prometheus/targets.d/blackbox_targets.yml

    metrics_path: /probe

    relabel_configs:

      - source_labels: [__address__]

        regex: '([^/]+)'  # name or address only

        target_label: instance

      - source_labels: [__address__]

        regex: '([^/]+)'  # name or address only

        target_label: __param_target

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: instance

        replacement: '${1}'

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: __param_target

        replacement: '${2}'

      - source_labels: [module]

        target_label: __param_module

      - target_label: __address__

        replacement: 127.0.0.1:9115  # Blackbox exporter

Then you create /etc/prometheus/targets.d/blackbox.yml like this:

- labels:

    module: icmp

  targets:

    - foo/1.2.3.4

    - bar/5.6.7.8

- labels:

    module: http_2xx

  targets:

    - baz/9.10.11.12

[Yagyansh S. Kumar]

Cool. Thank you.

Just started getting comfortable with Prometheus, so getting a lot of questions in mind. Thanks for the quick responses! :)