Reducing the content in prometheus alerts

[Guna Kambalimath]

Hi,

0

I have kube state metrics, prometheus and alertmanager deployed an VM. I have configured prometheus and alertmanager to get alerts , when number of restarts increases certain value in a certain amount of time. It is working fine. But, a lot of unnecessary data is coming as a part of the alerts. Basically, I do not want all the labels seen in prometheus to be a part of alert.

Currently what I am receving:

alertname = RestartsAlerts

container = kube-state-metrics

endpoint = http

exported_container = kube-scheduler

exported_namespace = kube-system

....

alertname = RestartsAlerts

container = kube-state-metrics

endpoint = http

exported_container = kube-scheduler

exported_namespace = kube-system

Alert configuration: 

- name: Pod-Restarts    

  rules:      - alert: RestartsAlerts        

  expr: max_over_time(kube_pod_container_status_restarts_total[3m]) - min_over_time(kube_pod_container_status_restarts_total[3m]) > 1        

   labels:         

       severity: critical       

   annotations:          

       summary: "More than 1 restart in pod {{ $labels.exported_pod }}"          

       description: "{{ $labels.exported_container }} container has restarted {{ $value }} times.\n Instance: {{ $labels.instance }}"

 

In the above alert, I don't want namespace, container, etc to be a part of the alert. I only want the summary and description. 

Please help with the same.

Thank you,

Guna

[Brian Candler]

On Thursday, 5 November 2020 05:58:05 UTC, Guna Kambalimath wrote:

But, a lot of unnecessary data is coming as a part of the alerts. Basically, I do not want all the labels seen in prometheus to be a part of alert.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alert_relabel_configs

However, I'd advise you to keep all the labels.  Information about things like namespace and container could be really helpful to someone trying to identify exactly where the problem is coming from.

[Guna Kambalimath]

Could you please provide some example ? As documentation is not quite explanatory. 

[Brian Candler]

"Alert relabeling ... has the same configuration format and actions as target relabeling."

That configuration is shown immediately above: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

You would need the "labeldrop" action.

    alert_relabel_configs:
      - regex: 'exported_namespace|exported_container'
        action: labeldrop

There is also "labelkeep" which drops all labels *except* the given pattern.  However, you said "I only want the summary and description", but those are not labels - they are annotations.  You REALLY REALLY don't want to drop all labels.  For one thing, alertmanager uses the labels to distinguish between one alert and another.  You really don't want alert B going away to resolve alert A.

It sounds to me that what you really want is to change the alertmanager template to simplify the E-mails it sends out - NOT to remove the labels sent from prometheus to alertmanager.

Google "prometheus alertmanager templates" for lots of examples how to do that.  Here's one hit:

https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/

[Guna Kambalimath]

Thank you. This was really helpful.