Need urgent help!!! Want to modify tags "keys" to lowercase scraping from Cloudwatch-Exporter in Prometheus before sending to Mimir #13912

[Vaibhav Ingulkar]

We are using **Cloudwatch-exporter** to scrape cloudwatch metrics and adding exporter job in **Prometheus** to scrape those metrics from exporter and then forwarding those metrics from prometheus to mimir **to enable alerting on metrics.**

There is no consistency in **keys** of tags at aws side, due to which we are facing issue of getting **blank values of tags in alert email** as many of them **are not matching keys in alert template code** as we can add **only one pattern (Uppercase or lowercase)** in template code.

So we are trying to **standardise** the incoming tags from all aws services **to one fix pattern as lowercase** and then forward them to mimir so this issue will be overcome and will able to print all tags in alert email.

Please suggest how we can achieve this with prometheus job scrapinng config that we have added as below -
- job_name: 'cloudwatch-exporter'    
  kubernetes_sd_configs:
  - role: service
        scrape_interval: 1m
        metrics_path: /metrics
        relabel_configs:
        - target_label: __address__
           replacement: XXX-prom-cloudwatch-exp-XXX.XXX.svc.cluster.local:9106

tags we are getting through below metrics -
**aws_resource_info**{job="aws_ec2",instance="",**tag_Name**="XXX",**tag_Owner**="XXX",tag_businessunit="XXX",**tag_environment**="dev"} 1.0
So, here we want to modify **tag_Name** to *tag_name*, **tag_Owner** to *tag_owner*

Please suggest.

[Vaibhav Ingulkar]

We are using Cloudwatch-exporter to scrape cloudwatch metrics and adding exporter job in Prometheus to scrape those metrics from exporter and then forwarding those metrics from prometheus to mimir to enable alerting on metrics.

There is no consistency in keys of tags at aws side, due to which we are facing issue of getting blank values of tags in alert email as many of them are not matching keys in alert template code as we can add only one pattern (Uppercase or lowercase) in template code.

So we are trying to standardise the incoming tags from all aws services to one fix pattern as lowercase and then forward them to mimir so this issue will be overcome and will able to print all tags in alert email.

Please suggest how we can achieve this with prometheus job scrapinng config that we have added as below -

• job_name: 'cloudwatch-exporter'
  kubernetes_sd_configs:

• • role: service
    scrape_interval: 1m
    metrics_path: /metrics
    relabel_configs:
    

• • - target_label: address
    replacement: XXX-prom-cloudwatch-exp-XXX.XXX.svc.cluster.local:9106

tags we are getting through below metrics -

aws_resource_info{job="aws_ec2",instance="",tag_Name="XXX",tag_Owner="XXX",tag_businessunit="XXX",tag_environment="dev"} 1.0
So, here we want to modify tag_Name to tag_name, tag_Owner to tag_owner

Please suggest.

[Brian Candler]

> Need urgent help!!!

See https://www.catb.org/~esr/faqs/smart-questions.html#urgent

> we can add only one pattern (Uppercase or lowercase) in template code.

At worst you can match like this: tag_Name=~"[fF][oO][oO][bB][aA][rR]"

I don't know of any way internally to prometheus to lowercase labels. What you could do though is to write a HTTP proxy: you scrape the proxy from prometheus, the proxy scrapes the upstream source, and modifies the labels before returning the results to prometheus.

Or: since you're using an external package anyway (cloudwatch_exporter), you could modify and recompile it yourself.

IMO it would better if you fix the data at source, i.e. make your tags be consistent in AWS. Prometheus faithfully reproduces the data you give it. Garbage in, garbage out.

[Ben Kochie]

Prometheus can lower/upper in relabeling.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c6d8f693-8238-4929-9dbc-d96e64b57180n%40googlegroups.com.

[Brian Candler]

On Thursday 18 April 2024 at 09:42:41 UTC+1 Ben Kochie wrote:

Prometheus can lower/upper in relabeling.

Thanks! That was added in v2.36.0, and I missed it.

[Vaibhav Ingulkar]

Thanks @Brian Kochie

Correct me if I am wrong but I think  lower/upper action in relabeling works to make "values" of labels to lower/upper and not "keys" i.e. label name itself wont get convert to lowercase. Right?

Because I an using v2.41.0 and  have tried it and it is converting all values of labels to lowercase.

Here my requirement is to convert labels i.e. keys to lowercase for ex. tag_Budget_Code to tag_budget_code or tag_Name to tag_name

[Vaibhav Ingulkar]

Additionally , I have prepare below config under metric_relable_configs

- action: labelmap

  regex: 'tag_(.*)'

  replacement: $1

It is giving one me new set of all label starting with word 'tag_' as added in regex but not converting them to lowercase and removing "tag_" from label name, for ex. tag_Name is converted only "Name"

Also existing label tag_Name is also remaining as it is .i.e. old label tag_Name and new label Name

So Firstly I want that "tag_" should remain as it it in new label and it should get converted to lower case i.e. for ex. tag_Budget_Code to tag_budget_code or tag_Name to tag_name

Secondly need to remove old label for ex. tag_Budget_Code , tag_Name , etc

[Brian Candler]

You mean you're seeing tag_owner, tag_Owner, tag_OWNER from different instances? Because the tags weren't entered consistently?

I don't see a lowercasing version of the "labelmap" action. So I think you're back to either:

1. fixing the data at source (e.g. using the EC2 API to read the tags and reset them to the desired values; and then make policies and procedures so that new instances have consistent tag names); or

2. proxying / modifying the exporter

> I think  lower/upper action in relabeling works to make "values" of labels to lower/upper 

I believe so. The way I interpret it, "lowercase" action is the same as "replace", but the concatenated values from source_labels are lowercased first. Hence the fixed target_label that you specify will get the lowercased value, after any regex matching/capturing.

The test case here agrees:

https://github.com/prometheus/prometheus/blob/v2.45.4/model/relabel/relabel_test.go#L461-L482

[Vaibhav Ingulkar]

Thanks @Brian Candler

Actually not possible fixing the data at source due to multiple variations in diff aws services and huge data modification. So looking to make it dynamically by capturing labels starting with "tag_".

As mentioned here https://github.com/prometheus/prometheus/blob/v2.45.4/model/relabel/relabel_test.go#L461-L482 can you please give me one example of config to achieve it dynamically for all labels starting with "tag_"

It will be great help if that works for me. :)

[Brian Candler]

No. That test case demonstrates that it is the label *values* that are downcased, not the label names, exactly as you said.

[Vaibhav Ingulkar]

Ok

Thanks @ Brian Candler for your valuable time on this.

Do you have any idea or way to achieve this in relabel_configs or metric_relabel_configs

[Vaibhav Ingulkar]

Hello Guys,

Does anyone has any idea/solution over this?

Your help will be appreciated!