How to configure nodename label for alert

Skip to first unread message

rajendar t

Jan 20, 2022, 12:21:52 PMJan 20
Hi All,

K8s: 1.24
Helm 3.0

I have Prometheus alert , I need node Name in the labels

- alert: example-alert
        description: Memory on node currently at % 
          is under pressure
        summary: Memory usage is under pressure, system may become unstable.
      expr: |
        100 - ((node_memory_MemAvailable_bytes{job="node-exporter"} * 100) / node_memory_MemTotal_bytes{job="node-exporter"}) > 50
      for: 2m
        severity: warning

{endpoint="metrics",instance="",job="node-exporter",namespace="monitoring",pod="mypromoperator-prometheus-node-exporter-gg5nl",service="mypromoperator-prometheus-node-exporter"} 67.09431138997289
{endpoint="metrics",instance="",job="node-exporter",namespace="monitoring",pod="mypromoperator-prometheus-node-exporter-9mfn2",service="mypromoperator-prometheus-node-exporter"} 52.7483247365166e

but want to see node name n the query , how to configure alert so that I will get node name

Brian Candler

Jan 20, 2022, 1:03:44 PMJan 20
to Prometheus Users
The best way in my opinion is to change to meaningful instance labels.  That is: arrange at scrape time that your metric has {instance="foo"} instead of {instance=""}.  Then the label you want is right there already.

To do this requires a bit of relabelling in the scrape job.  Here's the config I use:

  - job_name: node
    scrape_interval: 1m
      - files:
          - /etc/prometheus/targets.d/node_targets.yml
    metrics_path: /metrics
      - source_labels: [__address__]
        regex: '([^ ]+)'    # name or address only
        target_label: instance
      - source_labels: [__address__]
        regex: '(.+) (.+)'  # name address
        target_label: instance
        replacement: '${1}'
      - source_labels: [__address__]
        regex: '(.+) (.+)'  # name address
        target_label: __address__
        replacement: '${2}'
      - source_labels: [__address__]
        target_label: __address__
        replacement: '${1}:9100'

Now in your targets file you can put a plain DNS name or IP address (without the :9100 suffix), and this will become the instance label.  Or: you can put a name followed by a space and a DNS name or IP address, like this:

- labels:
    netbox_type: device
    - foo
    - bar

The target to be scraped will be "", but the instance label will be "foo"

Other approaches are significantly more difficult.  You can do a PromQL many-to-one "join" between your alerting expression and node_uname_info, matching on the "instance" label, to add other labels from node_uname_info to your alert.  But this means that every alerting expression becomes significantly more complex.  For the technique, see:

rajendar t

Jan 21, 2022, 6:20:32 AMJan 21
to Brian Candler, Prometheus Users
Thanks for reply .
I have attached my repo , I have created this from the helm chart repo .
this   - job_name: node, where should I include  ?
Can you please elaborate?

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To view this discussion on the web visit

Brian Candler

Jan 21, 2022, 6:47:55 AMJan 21
to Prometheus Users
This mailing list is for prometheus (just the software).

Helm charts which deploy prometheus are separate, so if you have problems with that you should ask at the group or mailing list for the particular helm chart you're using.
Reply all
Reply to author
0 new messages