Help Required for Prometheus config || multiple Labels Segregation on the basis of multiple targets in single single job name

[vinay jaiswal]

Hi , 

Can anyone help me to define labels on the basis of targets in a single job name so that receptive targets  can pick the label name accordingly in prometheus config . i don't want to define multiple job name for labels wise segregation . 

i tried to define this but it doesn't work appropriate .  My dashboard pick the filter on the basis  of labels . 

Thanks in Advance . 

- job_name: postgres_exporter_lr-1m0s

  scrape_interval: 10s

  scrape_timeout: 10s

  metrics_path: /metrics

  static_configs:

  - targets:

    - x.x.x.x:9187

    - y.y.y.y:9187

    labels:

      agent_type: postgres_exporter

      node_name: ip-x.x.x.x

      node_name: ip-y.y.y.y

      node_type: generic

      region: eastus1

      service_type: postgresql

      service_name: ip-x.x.x.x

      service_name: ip-y.y.y.y

[vinay jaiswal]

tried this way it's working fine  please suggest  if there is any alternative way 

- job_name: postgres_exporter_lr-1m0s

  scrape_interval: 10s

  scrape_timeout: 10s

  metrics_path: /metrics

  static_configs:

  - targets:

    - x.x.x.x:9187

    labels:

      agent_type: postgres_exporter

      node_name: ip-x.x.x.x

      node_type: generic

      region: eastus1

      service_type: postgresql

      service_name: ip-x.x.x.x

  - targets:

    -  y.y.y.y:9187

    labels:

      agent_type: postgres_exporter

      node_name: ip-y.y.y.y

      node_type: generic

      region: eastus1

      service_type: postgresql

      service_name: ip-y.y.y.y

[Brian Candler]

Yes that's correct, but you'll find it easier to use file_sd_configs.  The YAML is basically the same:

- targets:

    - ...

    - ...
  labels:
    ...: ...

but by putting it in a separate file, you don't need to touch prometheus.yml.  Also, prometheus automatically picks up when file_sd files change (you don't need to HUP prometheus)

You might also want to look at this:

https://www.robustperception.io/controlling-the-instance-label

It's very helpful to keep the exporter port out of the instance label, so you end up with

foo{instance="x.x.x.x"}

instead of

foo{instance="x.x.x.x:9187"}

Indeed, it may be a good idea to keep the IP address out of the instance label, and use a meaningful name like

foo{instance="db1"}

but set the __address__ to be x.x.x.x:9187 so this is what gets scraped.

[vinay jaiswal]

Hi , 

Thanks for your valuable suggestion which help me to use config file in proper way .  i tired to use this config to achieve the same and everything is working fine except node_name as label  which is unique for each target . i want to replace or append node_name as label in target_label with _address_ value  . But it doesn't work with relabel_config . Can you please suggest for the same . 

  file_sd_configs:

    - files:

      - /etc/prometheus/targets.yaml

  relabel_configs:

    - source_labels: [_address_]

      regex: '172*'

      replacement: '$1'

      target_label: node_name

On Saturday, 18 April 2020 16:19:58 UTC+5:30, vinay jaiswal wrote:

[Brian Candler]

$1 is a capture group - a parenthesised expression with the regexp - so you need to include the parentheses to capture the value you want:

regex: '(172*)'

However, in a regexp, "*" means "zero or more instances of the preceeding character".  So what you've written matches "17" followed by zero or more instances of "2", and that won't work.  For example it would match 1722222 but not 172.1.2.3

What you want is "172" followed by zero or more of any character, which is:

regex: '(172.*)'

Dot in a regexp has a special meaning of "any character".

If you want to match "172." followed by anything, then to match a literal dot without backslash hell I'd do:

regex: '(172[.].*)'