Issues with Prometheus AlertManager Configuration

[Baburao Ganpatrao]

Dear Support Team,

I am currently experiencing some discrepancies and unexpected behavior with the configuration of Prometheus AlertManager. Despite configuring alerts to fire every 4 hours, I am encountering issues with the actual number of alerts sent and received by AlertManager. There seems to be a significant disparity between the expected and observed counts.

Furthermore, even when no conditions are satisfied for triggering alerts, Prometheus continues to send alerts every 4 to 5 hours, and AlertManager subsequently forwards the payload to the configured webhook. I have noticed this behavior persisting even after a prolonged period of no data changes – in one instance, alerts were still being received 20 hours after data had ceased to change.

Below are the relevant configuration files: prometheus.yml: Scrape Interval: 2m

rules.yml: groups:  
  - name: recording-rule
    interval: 1h
    rules:  
      - record: myRecord
        expr: [expression calculating ratio by dividing two metrics, greater than a specified value]
  - name: alerting-rule
    interval: 4h
    rules:
      - alert: myAlert
        expr: max_over_time(myRecord[4h])
        labels:
          severity: warning
        annotations:
          summary: "Summary"
alertmanager.yml: group_by: ['alertname']
group_wait: 30s
group_interval: 1h
repeat_interval: 4h
I would appreciate your assistance in resolving the following issues:

1. Identifying and rectifying the discrepancies in the number of alerts between Prometheus and AlertManager.
2. Investigating and addressing the persistence of alerts being fired by Prometheus and forwarded by AlertManager, even in the absence of new data or changes.
3. Resolving the issue of receiving alerts at regular intervals, despite a prolonged period of no new data.

It's worth noting that a substantial amount of data is being processed for alerts, more than 25,000 unique metrics.

Your prompt attention to these matters would be greatly appreciated.

Thank you for your assistance.

[Ben Kochie]

This is not a support team, this is a users discussion list.

If you require professional support, please see our support page: https://prometheus.io/support-training/

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com.

[Baburao Ganpatrao]

Actually, I initially thought of this as support, then now I know its discussion forum. And I am seeking for help to resolve this strange behavior of prometheus alertmanager.

[Nash Saman]

Please remove me from this email list 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2ff8af79-ebd0-4cab-a058-366ce160820fn%40googlegroups.com.