Metric relabeling replacing char

58 views
Skip to first unread message

Aleksey Sysoev

unread,
Jul 4, 2022, 7:25:53 AMJul 4
to Prometheus Users

Hi.
Is it possible to replace certain char in label?

I have a metric like this:
my_metric{label1=“foo-foo-foo”, label2=“foo”}

label1 may have values:
foo
foo-foo
foo-foo-foo
and so on

It isn’t known how many ‘-’ it has.

I want to replace all dashes with underscore in label1 value:
my_metric{label1=“foo_foo_foo”, label2=“foo”}

Again, i don’t know how many dashes it may has.

I tried using “metric_relabel_configs” with replace mechanism, but it seems not possible to do with that.

Brian Candler

unread,
Jul 4, 2022, 8:52:08 AMJul 4
to Prometheus Users
> I tried using “metric_relabel_configs” with replace mechanism, but it seems not possible to do with that.

If the number is bounded, say no more than 4 dashes, then you can just have 4 rewriting rules, each of which rewrites one.  Just repeat the rule N times:

      - source_labels: [label1]
        regex: '(.*)-(.*)'
        replacement: '${1}_${2}'
        target_label: label1
      - source_labels: [label1]
        regex: '(.*)-(.*)'
        replacement: '${1}_${2}'
        target_label: label1
      - source_labels: [label1]
        regex: '(.*)-(.*)'
        replacement: '${1}_${2}'
        target_label: label1
      - source_labels: [label1]
        regex: '(.*)-(.*)'
        replacement: '${1}_${2}'
        target_label: label1

But I ask the question, why?

If it's your own exporter - then fix the exporter.
If it's someone else's exporter - then use the labels as they are.
If you still need it - then write a scraping proxy (where you scrape the proxy, the proxy scrapes the target, and the proxy modifies the labels in whatever way you like)

Aleksey Sysoev

unread,
Jul 4, 2022, 9:10:33 AMJul 4
to Prometheus Users
Thank you for answer.
I will try this as a workaround solution. I think there couldn't be more than 5 '-'.


>If it's your own exporter - then fix the exporter.
No it's a cloud exporter


>If it's someone else's exporter - then use the labels as they are.
it isn't possible because they use label as part of the metric name with '_'


>If you still need it - then write a scraping proxy
It's a good case to investigate further.


понедельник, 4 июля 2022 г. в 15:52:08 UTC+3, Brian Candler:

Aleksey Sysoev

unread,
Jul 4, 2022, 9:57:58 AMJul 4
to Prometheus Users
Thank you, it works, but with minor fix with your code

          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)'

            replacement: "${1}_${2}"
          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)'
            replacement: "${1}_${2}_${3}"
          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)'
            replacement: "${1}_${2}_${3}_${4}"
          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)'
            replacement: "${1}_${2}_${3}_${4}_{5}"

It's well enough as a workaround for now, but with agreement that wouldn't be more then four '-' or '_'
P.S. their might be '_' or '-', didn't write it first to make task easier

понедельник, 4 июля 2022 г. в 16:10:33 UTC+3, Aleksey Sysoev:

Brian Candler

unread,
Jul 4, 2022, 9:58:49 AMJul 4
to Prometheus Users
On Monday, 4 July 2022 at 14:10:33 UTC+1 a...@fevlake.com wrote:
>If it's someone else's exporter - then use the labels as they are.
it isn't possible because they use label as part of the metric name with '_'

I don't understand what you're saying.  You gave the following example:

my_metric{label1=“foo-foo-foo”, label2=“foo”}

Here the metric name is "my_metric" and label1 is "foo-foo-foo".  I don't see the relationship.  (Of course there is a hidden label, __name__="my_metric")

In any case, you can aggregate over labels independently of the metric name.  Can you give an example of an actual metric from this cloud exporter, and what problem it causes?

Brian Candler

unread,
Jul 4, 2022, 10:00:50 AMJul 4
to Prometheus Users
On Monday, 4 July 2022 at 14:57:58 UTC+1 a...@fevlake.com wrote:
Thank you, it works, but with minor fix with your code

          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)'
            replacement: "${1}_${2}"

Hmm, I can't see what the problem with the original version was.  If there was already an underscore, it would have left it alone.

Aleksey Sysoev

unread,
Jul 4, 2022, 10:33:03 AMJul 4
to Prometheus Users
>I don't understand what you're saying.  You gave the following example:
my_metric{label1=“foo-foo-foo”, label2=“foo”}

Ok.
My real example. I have postgres "my-db-name" in a cloud and need to construct grafana dashboard, for example "query count" for it.
1. I get variable (database) in grafana from metric which has label with database name (_pg_database_size{dbname="my-db-name", dbunderscore="my_db_name"<other labels>}): 
label_values(_pg_database_size{resource_id="$instance"}, dbunderscore)
where 'dbunderscore' it's a label i extracted from label "dbname" replacing '-' to '_' using “metric_relabel_configs” method.

2. then i need metric query count:
From exporter:
pooler_my_db_name_query_count{<many labels>} it doesn't have label for databasename, the databasename is in metric name but all '-' replaced with '_'

So, in grafana promql i make query like this: pooler_${database}_query_count


понедельник, 4 июля 2022 г. в 16:58:49 UTC+3, Brian Candler:

Brian Candler

unread,
Jul 4, 2022, 3:55:42 PMJul 4
to Prometheus Users
On Monday, 4 July 2022 at 15:33:03 UTC+1 a...@fevlake.com wrote:
From exporter:
pooler_my_db_name_query_count{<many labels>} it doesn't have label for databasename, the databasename is in metric name but all '-' replaced with '_'

Ugh, that's terrible design. It should have been:

pooler_query_count{database="my-db-name",<many labels>}
 
I think you could use metric relabelling the other way round, to turn those metrics into this format.  (I believe you can change the __name__ label to rename a metric; haven't tested it though)

However, you won't know whether an underscore in the metric name was originally an underscore or a dash, unless one of <many labels> carries the correct database name.

Aleksey Sysoev

unread,
Jul 6, 2022, 4:29:23 AMJul 6
to Prometheus Users
>Ugh, that's terrible design. It should have been:
pooler_query_count{database="my-db-name",<many labels>}

I'm totally agreed.
But we have what we have, unfortunately

понедельник, 4 июля 2022 г. в 22:55:42 UTC+3, Brian Candler:

Brian Candler

unread,
Jul 6, 2022, 6:59:25 AMJul 6
to Prometheus Users
So which cloud exporter is it? (So I can avoid that cloud :-)

Aleksey Sysoev

unread,
Jul 8, 2022, 1:44:06 AMJul 8
to Prometheus Users
Yandex

среда, 6 июля 2022 г. в 13:59:25 UTC+3, Brian Candler:

Aleksey Sysoev

unread,
Jul 8, 2022, 2:10:28 AMJul 8
to Prometheus Users
>If you still need it - then write a scraping proxy (where you scrape the proxy, the proxy scrapes the target, and the proxy modifies the labels in whatever way you like)

May be you know any scraping proxy solutions? Could you please give a link?

пятница, 8 июля 2022 г. в 08:44:06 UTC+3, Aleksey Sysoev:

Brian Candler

unread,
Jul 8, 2022, 8:30:31 AMJul 8
to Prometheus Users
> May be you know any scraping proxy solutions? Could you please give a link?

Nothing in particular: just a few lines of code in python, go, or your favourite language.

You could look at an existing exporter which makes API calls behind the scenes, e.g.
Reply all
Reply to author
Forward
0 new messages