We currently add a label to the target scrape job to give a bit of context to the service the host provides and route alerts based on service. After reviewing some targets, I noticed the service label is incorrect in a few cases so have attempted to update the configuration.
Following the configuration update, I can see the new metric data now includes the update service label (as I intended) however all the old metric data still uses the old label value. While this is not a major issue, it has caused some issues with our dashboards when displaying historical data as essential its see’s two different scraped targets.
Is there anyway to update/edit all historical metrics which match a certain label and value? From what I can tell the Prometheus HTTP API does not seem to provide this functionality?
Any input would be appreciated. Thank you