How to add labels to some metrics according to metric names?

3,851 views
Skip to first unread message

Qiheng Li

unread,
Jun 6, 2017, 8:31:39 AM6/6/17
to Prometheus Users, liqi...@huawei.com
I have many metrics,but only a part of them should be added a label

how can I do that?


thanks

Brian Brazil

unread,
Jun 6, 2017, 8:40:34 AM6/6/17
to Qiheng Li, Prometheus Users, liqi...@huawei.com
On 6 June 2017 at 13:31, Qiheng Li <emp...@gmail.com> wrote:
I have many metrics,but only a part of them should be added a label

how can I do that?

 


thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5e5119ed-74d2-4f59-a9b0-0ca010cd2d17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Qiheng Li

unread,
Jun 7, 2017, 4:03:00 AM6/7/17
to Prometheus Users, emp...@gmail.com, liqi...@huawei.com
I used following to add a label to a metric whose name contains string 'custom':

  - source_labels: [__name__]
    regex: .*custom.*
    replacement: 'cs'
    target_label: rwlabel

the label named rwlabel and the value of the label is always cs

but it can't work,what's wrong with me?


thanks

在 2017年6月6日星期二 UTC+8下午8:40:34，Brian Brazil写道：
On 6 June 2017 at 13:31, Qiheng Li <emp...@gmail.com> wrote:
I have many metrics,but only a part of them should be added a label

how can I do that?

 


thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--

Brian Brazil

unread,
Jun 7, 2017, 4:09:42 AM6/7/17
to Qiheng Li, Prometheus Users, liqi...@huawei.com
On 7 June 2017 at 09:03, Qiheng Li <emp...@gmail.com> wrote:
I used following to add a label to a metric whose name contains string 'custom':

  - source_labels: [__name__]
    regex: .*custom.*
    replacement: 'cs'
    target_label: rwlabel

the label named rwlabel and the value of the label is always cs

That should work. What behaviour are you seeing?

Brian
 

but it can't work,what's wrong with me?


thanks

在 2017年6月6日星期二 UTC+8下午8:40:34，Brian Brazil写道：
On 6 June 2017 at 13:31, Qiheng Li <emp...@gmail.com> wrote:
I have many metrics,but only a part of them should be added a label

how can I do that?

 


thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
 To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5e5119ed-74d2-4f59-a9b0-0ca010cd2d17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5430a048-1769-432c-bcfc-535a9a1181e2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Qiheng Li

unread,
Jun 7, 2017, 4:13:01 AM6/7/17
to Prometheus Users, emp...@gmail.com, liqi...@huawei.com
I queried a metric whose name contains string 'custom' ,but couldn't find any labels named rwlabel

The version of my prometheus server is 1.6.1

thanks

在 2017年6月7日星期三 UTC+8下午4:09:42，Brian Brazil写道：



--

Brian Brazil

unread,
Jun 7, 2017, 4:32:41 AM6/7/17
to Qiheng Li, Prometheus Users, liqi...@huawei.com
On 7 June 2017 at 09:13, Qiheng Li <emp...@gmail.com> wrote:
I queried a metric whose name contains string 'custom' ,but couldn't find any labels named rwlabel

The version of my prometheus server is 1.6.1

Did you put the relabel action in metric_relabel_configs?

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/83ef7d57-6a99-4070-ba7b-2b8a068e739a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Qiheng Li

unread,
Jun 7, 2017, 4:37:25 AM6/7/17
to Prometheus Users, emp...@gmail.com, liqi...@huawei.com
Yes,here is the actual parameter file:

global:
  evaluation_interval: 1m
  external_labels: {monitor: P1}
  scrape_interval: 60s
  scrape_timeout: 10s
rule_files: [alert_rules/*.rules]
remote_write:
  #- url: 'http://10.41.31.58:9990/PromReceiver'
  #- url: 'http://10.60.77.80:50020'
scrape_configs:
- job_name: P1
  metric_relabel_configs:
  - action: drop
    regex: go_.*
    source_labels: [__name__]
  scrape_interval: 60s
  scrape_timeout: 9s
  static_configs:
    - targets: ['10.60.77.168:8999']
   # - targets: ['10.41.31.58:9118']
   # - targets: ['10.41.31.58:9116']
- consul_sd_configs:
  - {datacenter: zkzone-001, server: '10.60.77.168:8500', services: ['P1_6000_3000_MySQLOS','P1_6000_3000_MySQL']}
  job_name: overwritten-from-consul
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'mysql_up'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_global_status_uptime'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_global_status_threads_connected'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_global_status_threads_running'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_slave_status_slave_io_running'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_slave_status_slave_sql_running'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_slave_status_seconds_behind_master'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_slave_status_last_errno'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_global_variables_read_only'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'mysql_global_variables_gtid_mode'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'node_filesystem_size'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: 'node_filesystem_avail'
    replacement: 'tokafka'
    target_label: remoteWrite
  - source_labels: [__name__]
    regex: .*custom.*
    replacement: 'tokafka'
    target_label: remoteWrite
  - action: drop
    regex: go_.*
    source_labels: [__name__]
  relabel_configs:
  - action: replace
    regex: (.*);(.*)
    replacement: ${1}:${2}
    separator: ;
    source_labels: [__meta_consul_service_address, __meta_consul_service_port]
    target_label: __address__
  - action: keep
    regex: ^P1_.*
    source_labels: [__meta_consul_service_id]
  - action: replace
    regex: (?i).*,hostname=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: hostname
  - action: replace
    regex: (?i).*,ipAddress=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: ipAddress
  - action: replace
    regex: (?i).*,userId=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: userId
  - action: replace
    regex: (?i).*,dbName=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: dbName
  - action: replace
    regex: (?i).*,instanceName=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: instanceName
  - action: replace
    regex: (?i).*,monType=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: monType
  - action: replace
    regex: (?i).*,promName=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: promName
  - action: replace
    regex: (?i).*,tenantName=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: tenantName
  - action: replace
    regex: (?i).*,eMail=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: eMail
  - action: replace
    regex: (?i).*,instanceId=([^,]+),.*
    replacement: ${1}
    source_labels: [__meta_consul_tags]
    target_label: instanceId
  scrape_interval: 60s
  scrape_timeout: 10s

thanks

在 2017年6月7日星期三 UTC+8下午4:32:41，Brian Brazil写道：



--

Brian Brazil

unread,
Jun 7, 2017, 4:45:36 AM6/7/17
to Qiheng Li, Prometheus Users, liqi...@huawei.com
On 7 June 2017 at 09:37, Qiheng Li <emp...@gmail.com> wrote:
Yes,here is the actual parameter file:

Are you sure the metrics aren't coming in from the P1 job?

You should also really be using relabelling on remote_write to select what you want to send out, not messing with all your metrics.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/31bc66e0-75a8-4849-a763-9368a69ee48a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Qiheng Li

unread,
Jun 7, 2017, 4:57:07 AM6/7/17
to Prometheus Users, emp...@gmail.com, liqi...@huawei.com
in fact I can get the metric without label remoteWrite to filter but can't if using it to filter.

what I really want to do is to add a label remoteWrite to some metrics so that I can send the related metric to kafka(another team already wrote a go program to send metrics to kafka)

I also used a federate to filter the metrics,but seems don't work either,the configuration file of the federate is as following:

remote_write:
  - url: 'http://10.41.31.58:9990/PromReceiver'
scrape_configs:
- job_name: 'federate'
  scrape_interval: 60s
  honor_labels: true
  #metrics_path: '/federate'
  params:
    'match[]':
      - '{remoteWrite="tokafka"}'
      - '{__name__=~".*custom.*"}'
  static_configs:
    - targets:
      - '10.60.77.168:8999'
  metric_relabel_configs:
  - action: drop
    regex: go_.*
    source_labels: [__name__]
  - action: drop
    regex: prometheus_.*
    source_labels: [__name__]
  - action: drop
    regex: http_.*
    source_labels: [__name__]
  - action: drop
    regex: process_.*
    source_labels: [__name__]

it seems that match[] don't work and action: drop don't work either.

thanks

在 2017年6月7日星期三 UTC+8下午4:45:36，Brian Brazil写道：



--

Qiheng Li

unread,
Jun 7, 2017, 7:00:34 AM6/7/17
to Prometheus Users, emp...@gmail.com, liqi...@huawei.com
it seems for the metrics generated by rules can't be relabeled.


all the metrics whose name contain string 'custom'  are defined in rule files which are generated after they are be evaluated.


thanks

在 2017年6月7日星期三 UTC+8下午4:45:36，Brian Brazil写道：



--

Brian Brazil

unread,
Jun 7, 2017, 7:01:49 AM6/7/17
to Qiheng Li, Prometheus Users, liqi...@huawei.com
On 7 June 2017 at 12:00, Qiheng Li <emp...@gmail.com> wrote:
it seems for the metrics generated by rules can't be relabeled.

Yes, it's expected you'll do whatever logic you need in the rule itself.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5e762a09-1da7-48d7-a38a-043f7bf4f71a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Qiheng Li

unread,
Jun 7, 2017, 8:04:12 AM6/7/17
to Prometheus Users, emp...@gmail.com, liqi...@huawei.com
yes,thanks a lot for your help

在 2017年6月7日星期三 UTC+8下午7:01:49，Brian Brazil写道：



--
Reply all
Reply to author
Forward
0 new messages