Some questions about `stale` series data

[zichen chuh]

Hi, here is what I read from https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ :

> For pending and firing alerts, Prometheus also stores synthetic time series of the form ALERTS{alertname="<alert name>", alertstate="pending|firing", <additional alert labels>}. The sample value is set to 1 as long as the alert is in the indicated active (pending or firing) state, and the series is marked stale when this is no longer the case.

Can I ask what does stale mean?

Does it mean the series data is deleted from the database? 

If not, can I query these stale data, and how can I mark a serie stale by myself?

Is there more documents about `stale`?

[Christian Hoffmann]

Hi,

On 5/18/20 12:38 PM, zichen chuh wrote:
> https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ :
>
>> For pending and firing alerts, Prometheus also stores synthetic time
> series of the form ALERTS{alertname="<alert name>",
> alertstate="pending|firing", <additional alert labels>}. The sample
> value is set to 1 as long as the alert is in the indicated active
> (pending or firing) state, and the series is marked stale when this is
> no longer the case.
>
> Can I ask what does stale mean?
> Does it mean the series data is deleted from the database?

No, the old values stay and it's more like a marker which denotes: this
used to be there but is now gone.

> If not, can I query these stale data

I don't think there are standard APIs for doing that. You can probably
find out when directly analyzing the data on disk.

> and how can I mark a serie stale
> by myself?

I don't think so. It's supposed to be an internal implementation detail.
Can you elaborate what you are trying to accomplish?

> Is there more documents about `stale`?

Maybe this blog post, the slides and the video provides more insight:
https://www.robustperception.io/staleness-and-promql
https://de.slideshare.net/brianbrazil/staleness-and-isolation-in-prometheus-20-promcon-2017
https://www.youtube.com/watch?v=GcTzd2CLH7I

Kind regards,
Christian

[zichen chuh]

Hi Christian,

Got it. Thanks for your prompt response.

在 2020年5月18日星期一 UTC+8下午7:17:56，Christian Hoffmann写道：