deactive alert after hook

[Milad Devops]

hi all 

I use Prometheus to create alert rules and hook alerts using alertmanager.
My scenario is as follows:
- The log publishing service sends logs to Prometheus Exporter
- Prometheus takes the logs every second and matches them with our rules
- If the log applies to our rules, the alertmanager sends an alert to the frontend application. It also saves the alert in the elastic

My problem is that when sending each alert, all the previous alerts are also stored in Elastic in the form of a single log and sent to my front service as a notification (web hook).

Is there a way I can change the alert status to resolved after the hook so that it won't be sent again on subsequent hooks?
Or delete the previous logs completely after the hook from Prometheus
Or any other suggested way you have
Thank you in advance

[Stuart Clark]

I'm not sure I really understand what you are asking due to your
mentioning of logs.

Are you saying that you are using an exporter (for example mtail) which
is consuming logs and then generating metrics?

When you create an alerting rule in Prometheus it performs the PromQL
query given, and if there are any results an alert is fired. Once the
PromQL query stops returning results (or has a different set of time
series being returned) the alert is resolved.

So for example if you had a simple query that said "alert if the number
of error logs [stored in a counter metric] increases by 5 or more in the
last 5 minutes" as soon as the metric returned an increase of at least 5
over the last 5 minutes it would fire. It would then continue to fire
until that is no longer true - so if the counter kept recording error
log lines such that the increase was still over 5 per 5 minutes it would
keep firing. It would only resolve once there were no more than 5 new
long lines recorded over the past 5 minutes.

Alertmanager just routes alerts that are generated within Prometheus to
other notification/processing systems, such as email or webhooks. It
would normally fire the webhook once the alert starts firing, and then
periodically (if it keeps firing, at a configurable interval) and then
finally (optionally) once it resolves. This is a one-way process -
nothing about the notification has any impact on the alert firing or
not. Only the PromQL query controls the alert.

I'm not sure if that helps.

--
Stuart Clark

[Milad Devops]

Hello Stuart
I'm sorry I couldn't ask my question properly
Actually, I use prometheus/alertmanager as an event pipeline to alert every event that occurs.

For example, I show two of the rules that are exported from the log exporter service:
- name: plate
    rules:
      - alert: "plate"
        expr: 'plate_log {plate_number="123456877Lay"}'
        for: 1s
        annotations:
          title: "plate detection {{ $labels.model_camera_id }}"
          description: "plate detection with confidence : {{ $labels.confidence }} "
        labels:
          severity: "critical"
          type: "plate"

  - name: human
    rules:
      - alert: "human"
        expr: 'number_of_Human > 15'
        for: 1s
        annotations:
          title: "human detection {{ $labels.model_camera_id }}"
          description: "human detection with confidence : {{ $labels.confidence }} "
        labels:
          severity: "critical"
          type: "human"
 

Also, the alertmanager configuration is as follows:

global:

route:
  receiver: webhook
  group_by: ["alertname"]
  group_wait: 1s
  group_interval: 1s
  # repeat_interval: 6d
  routes:
  - receiver: webhook
    continue: true
Receivers:
- name: webhook
  webhook_configs:
  - send_resolved: false
    http_config: {}
    url: "http://192.168.10.20:7000/visual"
    max_alerts: 0
  - url: "http://192.168.10.20:9200/alerts/_doc"
    send_resolved: false

My problem is exactly that if 5 alerts are hooked at different times, for the sixth log, all the previous 5 logs are also hooked.
I felt that because the logs are still in firing mode after the hook, they are sent again to Front and Elastic with new logs.

Stuart Clark در تاریخ یکشنبه ۲۴ ژوئیهٔ ۲۰۲۲ ساعت ۱۹:۵۹:۳۵ (UTC+4:30) نوشت:

[Brian Candler]

Events are not metrics.

For the behaviour you want, you should be looking at an event-based system like Loki.