How to define exact hostname or IP in prometheus.yml file

[Sadhana Kumari]

Hi Team,

Below is the configuration done to monitor few targets of APIGEE customer.

The IP with 10.2.0.* is the target IP, whereas 10.2.3.17 is the node exporter IP from where its scraping the metrics. Whenever we are receiving the alerts its coming with node exporter IP , we want the exact host name or IP to be displayed in the alert. Any suggestions

  - job_name: 'apigeenode'

    static_configs:

- targets:

- http://10.2.3.17:8080/10.2.0.85

- http://10.2.3.17:8080/10.2.0.86

- http://10.2.3.17:8080/10.2.0.87

- http://10.2.3.17:8080/10.2.0.101

- http://10.2.3.17:8080/10.2.0.102

- http://10.2.3.17:8080/10.2.0.103

- http://10.2.3.17:8080/10.2.0.117

- http://10.2.3.17:8080/10.2.0.118

- http://10.2.3.17:8080/10.2.0.119

    relabel_configs:

- source_labels: [__address__]

target_label: __param_target

- source_labels: [__param_target]

target_label: instance

- target_label: __address__

replacement: 10.2.3.17:8080

[Brian Candler]

See  https://www.robustperception.io/controlling-the-instance-label

- you can set the instance label separately from the __address__ that is scraped

- job_name: 'apigeenode'

  static_configs:

    - targets:

      - 10.2.0.85

      - 10.2.0.86

      - ...

  relabel_configs:

    - source_labels: [__address__]

      target_label: instance

    - source_labels: [__address__]

      target_label: __param_target

      replacement: http://10.2.3.17:8080/$1

    - target_label: __address__

      replacement: 10.2.3.17:8080

With the above, the instance label will be "10.2.0.85" for example.

I don't really understand why both the address and param_target are being set to 10.2.3.17:8080.  It looks like maybe you want to set __metric_path__ instead.  But if it works, it works.

[Sadhana Kumari]

Thanks Brian,I will try the below suggested one and update you.

[Sadhana Kumari]

I have done the changes as suggested and it worked. Thanks a lot for your help. Below is the format we are receiving the alert now.

Labels
alertname = InstanceDown
customer = APIGEE
instance = 10.2.0.85
job = apigeenode
severity = critical
Annotations
description = 10.2.0.85 of job apigeenode has been down for more than 1 minutes.
summary = Endpoint 10.2.0.85 down

[Sadhana Kumari]

Hi Brian,

Can you give example how to do set__metric__path since the format of alert which we wanted was achieved , but the end point is still showing down in prometheus., as its not able to scrape the metrics due to metrics path not correct.

http://10.2.3.17:8080/metrics     in this instead of /metrics it should be the http://10.2.3.17:8080/10.2.0.85

On Sunday, June 14, 2020 at 2:03:11 AM UTC+5:30, Brian Candler wrote:

[Brian Candler]

My mistake, it's __metrics_path__ not __metric_path__.

it's just another internal label like __address__, so you set it in relabelling:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

You'd use it if the URL path you want to scrape varies per target, rather than using the same path (e.g. "/metrics") for every target.

Therefore, if you want to scrape http://10.2.3.17:8080/10.2.0.85 (where the target is 10.2.0.85) I think it would be something like this:

- job_name: 'apigeenode'

  static_configs:

    - targets:

      - 10.2.0.85

      - 10.2.0.86

      - ...

  relabel_configs:

    - source_labels: [__address__]

      target_label: instance

    - source_labels: [__address__]

      target_label: __metrics_path__

[Sadhana Kumari]

Thanks Brian... In the same relabel config  can we do for both metrics and target?

[Brian Candler]

Yes if you need to - but that's why I was querying why you put the target in two places.  You could generate

http://10.2.3.17:8080/10.2.0.85?target=10.2.0.85
                      ^^^^^^^^^        ^^^^^^^^^
                   __metrics_path__  __param_target

but it seems rather redundant.  Surely the exporter you're talking to will be looking at only one or the other?