New in this world

[Jorge Fernández Muñoz]

Hello, I am a Spanish guy trying to understand this tool! (Sorry for my English) 

My question is... why I can’t see in the api (of alert manager or Prometheus) the alerts that I receive in the alert mail? I just want create an script that can read the api and print the alerts of the day, like just a summary or something like that.

Thanks!

[sayf.eddi...@gmail.com]

Hello,

Can you explain what you tried and how did you setup your monitoring system?

[Jorge Fernández Muñoz]

I think the problem is the time, I don’t know what label says the up of the alert. Because my alerts is grouping by technology and it sends the mail with the same startsAt value, I think because wait an regroup all same technology alerts in pending and then send the email.

[Brian Candler]

On Thursday, 29 October 2020 08:06:01 UTC, Jorge Fernández Muñoz wrote:

My question is... why I can’t see in the api (of alert manager or Prometheus) the alerts that I receive in the alert mail? I just want create an script that can read the api and print the alerts of the day, like just a summary or something like that.

Prometheus generates timeseries ALERTS and ALERTS_FOR_STATE that you can query to get a history of triggering alerts. e.g.

ALERTS{alertname="ProbeFail", alertstate="firing", instance="gw1-he", job="blackbox", module="icmp", severity="critical"} => 1 @[1603968515.41]

ALERTS_FOR_STATE{alertname="ProbeFail", instance="gw1-he", job="blackbox", module="icmp", severity="critical"} => 1603378977 @[1603968515.41]

You can get also prometheus to scrape alertmanager:

  - job_name: alertmanager

    scrape_interval: 1m

    static_configs:

      - targets: ['localhost:9093']

This gives you additional statistics around notifications, e.g. failed and successful notification deliveries.

If you just want to show currently firing alerts and/or silences, then use the alertmanager API.  This is what alertmanager front-ends like karma do.