How to track an availability of hosts and include hostname

[Skyl...@mail.ru]

Hello,

I faced difficulty when I configured alerts for virtual machines about their availability.

I want to see a hostname in an alert message. So for RAM, CPU, etc I use the query:

metric_name  * on(instance) group_left(hostname) windows_cs_hostname

the result:

{hostname="SRV-HOST01",instance="192.168.1.12",job="Hyper-v nodes",version="go1.13.3"}  

That's what I need. But when I've tried this query:

up{job="Hyper-v nodes"} * on(instance) group_left(hostname) windows_cs_hostname

I've got nothing.

Is it possible to add the label "hostname" to metric "UP"? Or maybe someone knows a workaround on how to track the availability of hosts with the opportunity to add the label "hostname"?

[Brian Candler]

On Thursday, 15 October 2020 12:44:05 UTC+1, Skyl...@mail.ru wrote:

{hostname="SRV-HOST01",instance="192.168.1.12",job="Hyper-v nodes",version="go1.13.3"}  

That's what I need. But when I've tried this query:

up{job="Hyper-v nodes"} * on(instance) group_left(hostname) windows_cs_hostname

I've got nothing.

Can you show the results separately of these two queries? (a few lines of each):

up{job="Hyper-v nodes"}

windows_cs_hostname

My guess is that your "instance" labels don't match up, maybe because one or the other includes port numbers.  To solve this, change your instance labels so they don't include the port number:

https://www.robustperception.io/controlling-the-instance-label

With some rewriting tricks, you can use an arbitrary label like SRV-HOST01 directly for the "instance" label while still using 192.168.1.12 for the scrape __address__.  This example is for Linux node_exporter but you can adapt it for Windows targets:

  - job_name: node

    file_sd_configs:

      - files:

        - /etc/prometheus/targets.d/node_targets.yml

    metrics_path: /metrics

    relabel_configs:

      - source_labels: [__address__]

        regex: '([^/]+)'  # name or address only

        target_label: instance

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: instance

        replacement: '${1}'

      - source_labels: [__address__]

        regex: '(.+)/(.+)'  # name/address

        target_label: __address__

        replacement: '${2}'

      - source_labels: [__address__]

        target_label: __address__

        replacement: '${1}:9100'

This lets you create a targets file like this:

- targets:

    - SRV-HOST01/192.168.1.12

and then you'll get metric{instance="SRV-HOST01",...} but the scrape target will be 192.168.1.12:9100.  The advantage here is that the instance label is meaningful by itself, and you don't need to do the join to pick up the hostname.

[Skyl...@mail.ru]

up{job="Hyper-v nodes"}:

up{instance="192.168.1.12:9182",job="Hyper-v nodes"} 

windows_cs_hostname:

windows_cs_hostname{domain="xxx.local",fqdn=" SRV-HOST01  .xxx.local",hostname=" SRV-HOST01  ",instance="192.168.1.12",job="Hyper-v nodes"}  

 

My job:

  - job_name: 'Hyper-v nodes'

    scrape_interval: 30s

    static_configs:

    - targets:

       - 192.168.1.12:9182

    metric_relabel_configs:

      # Delete port from instance

      - source_labels: [instance]

        regex: '(.*):.*'

        replacement: '$1'

        target_label: instance

I deleted "metric_relabel_configs", then I ran "up{job="Hyper-v nodes"} * on(instance) group_left(hostname) windows_cs_hostname" and I got the desired result! Thanks a lot, man!!!

четверг, 15 октября 2020 г. в 15:31:44 UTC+3, b.ca...@pobox.com:

[Brian Candler]

On Thursday, 15 October 2020 14:55:40 UTC+1, Skyl...@mail.ru wrote:

up{job="Hyper-v nodes"}:

up{instance="192.168.1.12:9182",job="Hyper-v nodes"} 

windows_cs_hostname:

windows_cs_hostname{domain="xxx.local",fqdn=" SRV-HOST01  .xxx.local",hostname=" SRV-HOST01  ",instance="192.168.1.12",job="Hyper-v nodes"}  

Thank you.  You should be able to see the problem now: the up metric has instance="192.168.1.12:9182", while the windows_cs_hostname has instance="192.168.1.12"

Note that you should not change the instance label in metric relabelling - you should change it in the initial target relabelling (i.e. "relabel_configs", not "metric_relabel_configs", which is done *before* metric collection).  Changing this should fix the instance label on the "up" metric.

The 'instance' label won't be set at the start of relabel_configs, only __address__.  If the instance label is still unset after relabel_configs, then prometheus will set it to be the same as __address__.  But if you set it explicitly, then that value will be kept.