How to work with scrape jobs introducing label name changes?

28 views

Skip to first unread message

tejaswini vadlamudi

unread,

Feb 23, 2023, 10:16:56 AM2/23/23

to Prometheus Users

Hi,

I have the two sets of metrics in my TSDB based on my scrape job configuration:

Old scrape job
my_client_request_duration_seconds_sum{instance=“35.226.217.238:8002”, job=“my-old-scrape-job”, method=“POST”, status_code=“200”, k8s_namespace=“staging”} 10.62420876675
my_client_request_duration_seconds_sum{instance=“44.110.218.239:8002”, job=“my-old-scrape-job”, method=“POST”, status_code=“200”, k8s_namespace=“staging”} 10.62420876675

New scrape job

my_client_request_duration_seconds_sum{instance=“36.226.217.238:8002”, job=“my-new-scrape-job”, method=“POST”, status_code=“200”, namespace=“staging”} 20.62420876675
my_client_request_duration_seconds_sum{instance=“45.110.218.239:8002”, job=“my-new-scrape-job”, method=“POST”, status_code=“200”, namespace=“staging”} 20.62420876675

I wonder how I can aggregate and update my existing queries in PromQL client's, recording rules, alert rules, HPA scaling queries, and dashboards to support & enable these label name changes from k8s_namespace to namespace. Any ideas or suggestions on this migration process?