Hello,
From my prometheus, i am monitoring a storage system.I have an issue with the alerts coming out of this storage monitoring.
I have my alert manager grouping alerts by instance. the storage system is a single instance but it has directories for each application and alerts are in place on the total space used by these apps wrt to allocated space. The alerts that are triggered when the usage reaches threshold is being group and i am only receiving one instance but I want to receive multiple alerts for each directory (referred as 'name' in the label).
I don't want to add this(name) to the group_by in alert manager as this would distort another infra monitoring alerts in place.
My idea is to set intance as combination of instance and directoryname but doesn't like this approach much. While recording might help, but i am unable to relabel labels in recording rules.
- job_name: 'pure_flashblade'
scrape_interval: 60s
scrape_timeout: 30s
metrics_path: /metrics/flashblade
relabel_configs:
- source_labels: [__address__]
target_label: __param_endpoint
- source_labels: [__pure_apitoken]
target_label: __param_apitoken
- source_labels: [__address__]
target_label: instance
replacement: ${1}_${name}
- target_label: __address__
replacement: pure-exporter:9491
static_configs:
labels:
__pure_apitoken: "${PURE_API_TOKEN}"
env: "prod"
Any help is much appreciated.
Thanks
Eswar