AlertManager Alerts API requires clients to 'continuously re-send'

32 views
Skip to first unread message

Alex Seaton

unread,
Jun 18, 2020, 5:00:32 AM6/18/20
to Prometheus Users
Hi,


"Clients are expected to continuously re-send alerts as long as they are still active (usually on the order of 30 seconds to 3 minutes)"

- Is this accurate? Using the POST /v1/alerts endpoint I tried creating an alert with a single request. It was created successfully and did not close.
- If it is accurate, what does it mean? How frequently is 'continuously'?
- How was the 30 seconds to 3 minutes interval arrived at? How is it configured?
- How does this statement in the docs interact with the "endsAt" field in the payload?

Thanks,

Alex Seaton

Brian Brazil

unread,
Jun 18, 2020, 5:06:19 AM6/18/20
to Alex Seaton, Prometheus Users
On Thu, 18 Jun 2020 at 10:00, Alex Seaton <alex....@man.com> wrote:
Hi,


"Clients are expected to continuously re-send alerts as long as they are still active (usually on the order of 30 seconds to 3 minutes)"

- Is this accurate? Using the POST /v1/alerts endpoint I tried creating an alert with a single request. It was created successfully and did not close.

Yes.
 
- If it is accurate, what does it mean? How frequently is 'continuously'?
- How was the 30 seconds to 3 minutes interval arrived at? How is it configured?

That's a guideline, I'd not go any longer than about a minute as otherwise you'll have issues when the Alertmanager restarts.
 
- How does this statement in the docs interact with the "endsAt" field in the payload?

You should ensure that the next send happens before endsAt.

Brian


Thanks,

Alex Seaton

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4d6760ae-12b5-48fe-9f2a-cfd5065f1458o%40googlegroups.com.


--

Alex Seaton

unread,
Jun 18, 2020, 5:50:01 AM6/18/20
to Prometheus Users
Thanks very much for the reply Brian! What would the consequences of only sending alerts a single time be?


On Thursday, 18 June 2020 10:06:19 UTC+1, Brian Brazil wrote:
On Thu, 18 Jun 2020 at 10:00, Alex Seaton <alex....@man.com> wrote:
Hi,


"Clients are expected to continuously re-send alerts as long as they are still active (usually on the order of 30 seconds to 3 minutes)"

- Is this accurate? Using the POST /v1/alerts endpoint I tried creating an alert with a single request. It was created successfully and did not close.

Yes.
 
- If it is accurate, what does it mean? How frequently is 'continuously'?
- How was the 30 seconds to 3 minutes interval arrived at? How is it configured?

That's a guideline, I'd not go any longer than about a minute as otherwise you'll have issues when the Alertmanager restarts.
 
- How does this statement in the docs interact with the "endsAt" field in the payload?

You should ensure that the next send happens before endsAt.

Brian


Thanks,

Alex Seaton

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

Brian Brazil

unread,
Jun 18, 2020, 6:05:38 AM6/18/20
to Alex Seaton, Prometheus Users
On Thu, 18 Jun 2020 at 10:50, Alex Seaton <alex....@man.com> wrote:
Thanks very much for the reply Brian! What would the consequences of only sending alerts a single time be?

If everything doesn't align perfectly time wise or there's any sort of transient failure, your alert can be lost. You need to keep on sending an alert once a minute or so for as long as it is still active.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/372ef6a1-aee9-4610-934d-c04f6767f0fdo%40googlegroups.com.


--
Reply all
Reply to author
Forward
0 new messages