Metric relabeling replacing char

[Aleksey Sysoev]

Hi.
Is it possible to replace certain char in label?

I have a metric like this:
my_metric{label1=“foo-foo-foo”, label2=“foo”}

label1 may have values:
foo
foo-foo
foo-foo-foo
and so on

It isn’t known how many ‘-’ it has.

I want to replace all dashes with underscore in label1 value:
my_metric{label1=“foo_foo_foo”, label2=“foo”}

Again, i don’t know how many dashes it may has.

I tried using “metric_relabel_configs” with replace mechanism, but it seems not possible to do with that.

[Brian Candler]

> I tried using “metric_relabel_configs” with replace mechanism, but it seems not possible to do with that.

If the number is bounded, say no more than 4 dashes, then you can just have 4 rewriting rules, each of which rewrites one.  Just repeat the rule N times:

      - source_labels: [label1]
        regex: '(.*)-(.*)'
        replacement: '${1}_${2}'
        target_label: label1
      - source_labels: [label1]
        regex: '(.*)-(.*)'
        replacement: '${1}_${2}'
        target_label: label1

      - source_labels: [label1]

        regex: '(.*)-(.*)'
        replacement: '${1}_${2}'
        target_label: label1

      - source_labels: [label1]
        regex: '(.*)-(.*)'
        replacement: '${1}_${2}'
        target_label: label1

But I ask the question, why?

If it's your own exporter - then fix the exporter.

If it's someone else's exporter - then use the labels as they are.

If you still need it - then write a scraping proxy (where you scrape the proxy, the proxy scrapes the target, and the proxy modifies the labels in whatever way you like)

[Aleksey Sysoev]

Thank you for answer.
I will try this as a workaround solution. I think there couldn't be more than 5 '-'.

>If it's your own exporter - then fix the exporter.

No it's a cloud exporter

>If it's someone else's exporter - then use the labels as they are.

it isn't possible because they use label as part of the metric name with '_'

>If you still need it - then write a scraping proxy

It's a good case to investigate further.

понедельник, 4 июля 2022 г. в 15:52:08 UTC+3, Brian Candler:

[Aleksey Sysoev]

Thank you, it works, but with minor fix with your code

          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)'

            replacement: "${1}_${2}"

          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)'
            replacement: "${1}_${2}_${3}"
          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)'
            replacement: "${1}_${2}_${3}_${4}"
          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)[-_]+([[:alnum:]]+)'
            replacement: "${1}_${2}_${3}_${4}_{5}"

It's well enough as a workaround for now, but with agreement that wouldn't be more then four '-' or '_'
P.S. their might be '_' or '-', didn't write it first to make task easier

понедельник, 4 июля 2022 г. в 16:10:33 UTC+3, Aleksey Sysoev:

[Brian Candler]

On Monday, 4 July 2022 at 14:10:33 UTC+1 a...@fevlake.com wrote:

>If it's someone else's exporter - then use the labels as they are.
it isn't possible because they use label as part of the metric name with '_'

I don't understand what you're saying.  You gave the following example:

my_metric{label1=“foo-foo-foo”, label2=“foo”}

Here the metric name is "my_metric" and label1 is "foo-foo-foo".  I don't see the relationship.  (Of course there is a hidden label, __name__="my_metric")

In any case, you can aggregate over labels independently of the metric name.  Can you give an example of an actual metric from this cloud exporter, and what problem it causes?

[Brian Candler]

On Monday, 4 July 2022 at 14:57:58 UTC+1 a...@fevlake.com wrote:

Thank you, it works, but with minor fix with your code

          - source_labels: [ label1 ]
            target_label: label1
            regex: '([[:alnum:]]+)[-_]+([[:alnum:]]+)'
            replacement: "${1}_${2}"

Hmm, I can't see what the problem with the original version was.  If there was already an underscore, it would have left it alone.

[Aleksey Sysoev]

>I don't understand what you're saying.  You gave the following example:

my_metric{label1=“foo-foo-foo”, label2=“foo”}

Ok.
My real example. I have postgres "my-db-name" in a cloud and need to construct grafana dashboard, for example "query count" for it.
1. I get variable (database) in grafana from metric which has label with database name (_pg_database_size{dbname="my-db-name", dbunderscore="my_db_name"<other labels>}): 
label_values(_pg_database_size{resource_id="$instance"}, dbunderscore)
where 'dbunderscore' it's a label i extracted from label "dbname" replacing '-' to '_' using “metric_relabel_configs” method.

2. then i need metric query count:
From exporter:
pooler_my_db_name_query_count{<many labels>} it doesn't have label for databasename, the databasename is in metric name but all '-' replaced with '_'

So, in grafana promql i make query like this: pooler_${database}_query_count

понедельник, 4 июля 2022 г. в 16:58:49 UTC+3, Brian Candler:

[Brian Candler]

On Monday, 4 July 2022 at 15:33:03 UTC+1 a...@fevlake.com wrote:

From exporter:
pooler_my_db_name_query_count{<many labels>} it doesn't have label for databasename, the databasename is in metric name but all '-' replaced with '_'

Ugh, that's terrible design. It should have been:

pooler_query_count{database="my-db-name",<many labels>}

 

I think you could use metric relabelling the other way round, to turn those metrics into this format.  (I believe you can change the __name__ label to rename a metric; haven't tested it though)

However, you won't know whether an underscore in the metric name was originally an underscore or a dash, unless one of <many labels> carries the correct database name.

[Aleksey Sysoev]

>Ugh, that's terrible design. It should have been:

pooler_query_count{database="my-db-name",<many labels>}

I'm totally agreed.
But we have what we have, unfortunately

понедельник, 4 июля 2022 г. в 22:55:42 UTC+3, Brian Candler:

[Brian Candler]

So which cloud exporter is it? (So I can avoid that cloud :-)

[Aleksey Sysoev]

Yandex

среда, 6 июля 2022 г. в 13:59:25 UTC+3, Brian Candler:

[Aleksey Sysoev]

>If you still need it - then write a scraping proxy (where you scrape the proxy, the proxy scrapes the target, and the proxy modifies the labels in whatever way you like)

May be you know any scraping proxy solutions? Could you please give a link?

пятница, 8 июля 2022 г. в 08:44:06 UTC+3, Aleksey Sysoev:

[Brian Candler]

> May be you know any scraping proxy solutions? Could you please give a link?

Nothing in particular: just a few lines of code in python, go, or your favourite language.

You could look at an existing exporter which makes API calls behind the scenes, e.g.

https://github.com/prometheus-pve/prometheus-pve-exporter