Changing "instance" label in relabel_configs doesn't work but works in metric_relabel_configs

[Igor Sverkos]

Hi,

I'd like to have a "instance" label value without ":<port>". This seems to be a common task according to

- https://www.robustperception.io/controlling-the-instance-label/

- https://groups.google.com/g/prometheus-users/c/I4dVtwI5YrM

- https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2#8559

But none of this is working for me.

However, doing it in "metric_relabel_configs" instead works.

Config looks like

scrape_configs:
  - job_name: node-external
    scheme: "https"
    metrics_path: "/exporters/node/metrics"
    static_configs:
      - targets:
          - host1.example.org
          - host2.example.org
    relabel_configs:
    - source_labels: [__address__]
      regex: "([^:]+):.+"
      target_label: "instance"

I even tried without the regex just in case, i.e.

    relabel_configs:
    - target_label: "instance"
      replacement: "foo"

but "instance" value will stick to "host1.example.org:443" for example.

If I'll do the relabeling in metric_relabel_configs instead, the shown config will work.

I am using prometheus v2.37.8.

Was this changed in the recent past? I cannot believe that all the documentations and blog postings are wrong. Am I missing something?

--

Igor

[Brian Candler]

Your "targets" under "static_configs" don't include a port number, so they won't match your regex.  Therefore, you can either add :443 to the targets, or you could do

    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

Your second test config will likely create scraping errors as you'll end up with duplicate metrics all sharing the same label sets with {instance="foo"}.  But if you make your scrape job have a single target, it ought to work, and I don't know why it doesn't.

Are you using regular prometheus directly, or via some docker / helm chart?  The problem might be with the latter.