Remove port number from instance value

[kiran]

Hello all,

I am getting metrics correctly from netdata into prometheus with promethues.yml file below(part of the file):

What do I do to not have the port number associated with the IP address in the instance label?

  - job_name: 'netdata'

    metrics_path: '/api/v1/allmetrics'

    params:

      format: [prometheus]

    honor_labels: true

    file_sd_configs:

    - files:

      - 'nodes.yaml'

    relabel_configs:

      - source_labels: [__address__]

        regex: (.*):(9100)

        target_label: __param_target

        replacement:   '${1}:19999'

      - source_labels: [__address__]

        regex: (.*):(9100)

        target_label: instance

        replacement:   '${1}:19999'

      - source_labels: [__param_target]

        regex: (.*)

        target_label: __address__

        replacement:   '${1}'

[Brian Candler]

Simply set the "instance" label to what you want.  In your example you can change

      - source_labels: [__address__]

        regex: (.*):(9100)

        target_label: instance

        replacement:   '${1}:19999'

to

      - source_labels: [__address__]

        regex: (.*):(9100)

        target_label: instance

        replacement:   '${1}'

I'm not sure why you're setting __param_target.  This sets "?target=x.x.x.x" in the scrape URL; it is normally used when scraping blackbox_exporter or snmp_exporter, where the exporter is on one host and the target is on a different, remote host.  As far as I'm aware, with netdata you just scrape the host directly and there's no need to set 'target'.

You can improve this further by not putting the :9100 suffix in the targets file at all.  Then when you're scraping node_exporter, add :9100 to the address as part of your rewriting.

Therefore it might simplify to:

  - job_name: 'node'

    file_sd_configs:

    - files:

      - 'nodes.yaml'

    relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __address__

        replacement:   '${1}:9100'

  - job_name: 'netdata'

    metrics_path: '/api/v1/allmetrics'

    params:

      format: [prometheus]

    file_sd_configs:

    - files:

      - 'nodes.yaml'

    relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __address__

        replacement:   '${1}:19999'

[kiran]

Thank you, Brian. I have a question on the relabel_configs section you suggested. We are mapping the same source_label([__address__]) twice to different things. Could you please help me understand the logic and the sequence of the mappings?

relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __address__

        replacement:   '${1}:19999'

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2bf6c48d-1ea7-4f45-8d41-3b5802f891deo%40googlegroups.com.

[Brian Candler]

On Thursday, 17 September 2020 03:47:42 UTC+1, kiran wrote:

I have a question on the relabel_configs section you suggested. We are mapping the same source_label([__address__]) twice to different things. Could you please help me understand the logic and the sequence of the mappings?

relabel_configs:

      - source_labels: [__address__]

        target_label: instance

      - source_labels: [__address__]

        target_label: __address__

        replacement:   '${1}:19999'

__address__ is what's set initially from the contents of the targets file; and __address__ is also what's used as the actual target after relabelling.

instance is just a normal label, but if you don't set it explicitly, prometheus automatically sets it to the content of __address__ used for scraping.

All labels which start __ are excluded from the label set in the resulting metric.

So in the above example, say that your targets file contains an entry "1.2.3.4"

* Before relabelling, you have __address__ = 1.2.3.4

* The first relabel rule copies 1.2.3.4 to the instance label.

* The second relabel rule copies 1.2.3.4:19999 to the __address__ label, replacing the previous value of __address__.  This is the actual target+port used for scraping.

You have to do it in that order, because the value you want for the instance label is the plain "1.2.3.4", not the one with the port appended.

Since the instance label has already been set to something, prometheus does *not* then replace the instance label with 1.2.3.4:19999

HTH,

Brian.

[kiran]

Thank you Brian for such a great detailed explanation. Truly appreciate your time.

So I am assuming similar relabelling can be used for any exporter(where exporter is on same host unlike blackbox exporter etc.).

I currently have this relabelling for aws service discovery. I am not sure what to modify here to get just the instance and not the port. I tried to remove '19999', but somehow it did not work.

  - job_name: 'aws_netdata'

    params:

    honor_labels: true

    ec2_sd_configs:

      - access_key: "aaaaaa"

        secret_key: "bbbbbbbbbbb"

        port: 19999

    relabel_configs:

      - source_labels: [__meta_ec2_instance_state]

        regex: running

        action: keep

      - source_labels: [__meta_ec2_public_ip]

        regex: (.+)

        target_label: __address__

        replacement:   '${1}:19999'

      - source_labels: [__metrics_path__]

        regex: (.+)

        target_label: __metrics_path__

        replacement:   '/api/v1/allmetrics'

      - source_labels: [__meta_ec2_tag_Name]

        target_label: name

      - source_labels: [__meta_ec2_instance_id]

        target_label: id

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9ca73c1a-1389-4b72-989e-8127295e783fo%40googlegroups.com.

[Brian Candler]

On Thursday, 17 September 2020 15:35:12 UTC+1, kiran wrote:

  - job_name: 'aws_netdata'

    params:

    honor_labels: true

    ec2_sd_configs:

      - access_key: "aaaaaa"

        secret_key: "bbbbbbbbbbb"

        port: 19999

Aside: I believe that ec2 service discovery initializes __address__ to <private-ip>:<port>.  However, since you're replacing the entire __address__ label anyway, you don't need to set the port here.

You also probably should not set honor_labels.  This is only needed if the scraped data contains a 'job' or 'instance' label, and you want the value from the scrape to override the value set by prometheus.  This is usually dangerous, except in certain special cases like federation.

 

    relabel_configs:

      - source_labels: [__meta_ec2_instance_state]

        regex: running

        action: keep

      - source_labels: [__meta_ec2_public_ip]

        regex: (.+)

        target_label: __address__

        replacement:   '${1}:19999'

As you know, that sets the __address__ label to <public_ip>:19999.

Therefore, if you want to set the instance label to the public IP (without port), then add:

      - source_labels: [__meta_ec2_public_ip]

        regex: (.+)

        target_label: instance

        replacement:   '${1}'

The only reason you got instance label with the port before was because you hadn't set any instance label, so it defaults to __address__.

The defaults are regex: '(.*)' and replacement: '${1}', so this rule can be simplified down to:

      - source_labels: [__meta_ec2_public_ip]

        target_label: instance

You don't have to use the public_ip for the instance label; you might prefer __meta_ec2_instance_id for example.  It just has to be something that's unique to this instance.  __meta_ec2_instance_id has the benefit that if the instance is rebooted and changes public IP, you can still track it as part of the same timeseries.

 

      - source_labels: [__metrics_path__]

        regex: (.+)

        target_label: __metrics_path__

        replacement:   '/api/v1/allmetrics'

Just as a side note, this could be done more simply by setting

    metrics_path: /api/v1/allmetrics

at the job level.  But using relabelling works too.

HTH,

Brian.

[kiran]

Hi Brian,

Thank you for suggesting to use __meta_ec2_instance_id instead of public IP. That really helps.

Another question: Lets say I have netdata(or node_exporter) and also cadvisor on every host and I am using Ec2 service discovery. I now have 2 separate jobs, one for cadvisor and one for netdata. So in prometheus.yml file for each job instead of putting targets file, we pass AWS keys/role in every job and the re-label section tells which port to scrape? Is my understanding correct?

[Brian Candler]

Yes that's correct.  And I believe prometheus SD is smart enough not to run the service discovery query twice, but to re-use the previous answers, when seeing the same SD with the same parameters.

[kiran]

Thank you, Brian.

I wish there was more like a global variables section for cloud provider access information so it does not have to be repeated for every job level, but at the same time have override flexibility at job level helps.

On Saturday, September 19, 2020, Brian Candler <b.ca...@pobox.com> wrote:

Yes that's correct.  And I believe prometheus SD is smart enough not to run the service discovery query twice, but to re-use the previous answers, when seeing the same SD with the same parameters.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/aa4d6eeb-875f-49f9-818f-2df4645d9a8ao%40googlegroups.com.