startsAt and endsAt clarification.

Yagyansh S. Kumar

unread,

Apr 21, 2020, 6:08:00 AM4/21/20

to Prometheus Users

Hi. What and how are the timestamps gathered for startsAt and endsAt? Is there any documentation for this?

From what I have observed startsAt seems to be giving the correct time of when the alert entered the "firing" state. But endsAt time seem to be ambiguous to me, because one of my Filesystem Alert started firing on 13th April 2:30pm, now it is still active and I can see endsAt time at 21st April, 3:26pm. Is this based on my repeat_interval?

Can someone explain what time is taken for endsAt?

Thanks!

Yagyansh S. Kumar

unread,

Apr 21, 2020, 6:08:36 AM4/21/20

to Prometheus Users

I am sorry I forgot to mention that I am using v1 Alertmanager API to capture this data. /api/v1/alerts

Julius Volz

unread,

Apr 21, 2020, 7:30:13 AM4/21/20

to Yagyansh S. Kumar, Prometheus Users

As long as the alert stays active on the Prometheus side, Prometheus will send it repeatedly with an ongoingly updated "endsAt" timestamp every time Prometheus evaluates and sends the alert to Alertmanager. The endsAt timestamp will be chosen to be a couple of minutes into the future in such a way that it stays current until after the next expected resend by Prometheus, so that Alertmanager doesn't expire it in the meantime.

So you should set it far enough into the future from when you are sending the alert such that your next re-send will happen before "endsAt". But also not too far into the future, if you want Alertmanager to automatically time out and remove alerts that aren't being sent anymore after a while.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/907b9ba3-4f19-4335-9f5b-62cfc4aab355%40googlegroups.com.

Julius Volz

unread,

Apr 21, 2020, 7:31:45 AM4/21/20

to Yagyansh S. Kumar, Prometheus Users

(I described what you should do if you were to implement a client sending alerts to AM, but your question was more about interpreting the timestamps sent by Prometheus. But I guess it still holds and does that as well :))

Yagyansh S. Kumar

unread,

Apr 21, 2020, 7:58:13 AM4/21/20

to Prometheus Users

Thanks a lot for such a clear explanation.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

Reply all

Reply to author

Forward