Last Scrape time

[BHARATH KUMAR]

Hello,

Can we know the last time when the Prometheus scrape the metrics. Is there any metric to find the time when the Prometheus did the scrape?

Thanks & regards,

Bharath Kumar.

[Ben Kochie]

timestamp(up)

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e28a24ea-921a-4710-9540-0f0c66556ab8n%40googlegroups.com.

[BHARATH KUMAR]

Yes... It will give the timestamp... It is giving the time even though the specific instance is down. It should not give the timestamp if it didn't scrape as the instance/server was down. 

Can we check whether the scrape was successful or not?

The main intention is to find the timestamp because when a server goes down we can know at what time the last scrape happen.

[Brian Candler]

> Can we check whether the scrape was successful or not?

up == 1

Unfortunately,

timestamp(up == 1)

doesn't work as you might hope; it returns the current time (the instant that the query was made for), not the time of the last successful scrape.

You can do:

timestamp(up) unless up == 0

but then you'll only get the timestamp if the server is currently up, and nothing if it is down.

You can use a subquery, which scans the expression over a time range - but I already explained that to you a few weeks ago:

https://groups.google.com/g/prometheus-users/c/1zSp7XTbChY/m/3my54aefBwAJ

max_over_time((time() * up)[24h:5m])

Or I think this will work too:

max_over_time((timestamp(up == 1))[24h:5m])

Both of these give an approximation. The subquery as shown evaluates the expression at 5 minute intervals over the last 24 hours; so the timestamp you get is rounded up to the next 5 minute step.

[Brian Candler]

> Unfortunately,

> timestamp(up == 1)

> doesn't work as you might hope; it returns the current time (the instant that the query was made for), not the time of the last successful scrape.

I should add: this is not a bug, this is by design, and it wouldn't do what you want anyway.

The query "up == 1" means "show me all values of the timeseries 'up' whose value is 1 *at the current point in time*".  It will look backwards in time to find the most recent value of the 'up' metric, and will only look back 5 minutes by default (query.lookback-delta).   Once it has found the most recent value, that's the one it will use.  If the most recent value of "up" seen is 0, the filter "== 1" will eliminate it from the answer; and if no scrape has been attempted for more than 5 minutes, it also won't appear in the result set.

Hence, for what you're asking, you need to look back over a defined time range - using a subquery, as I showed.

[BHARATH KUMAR]

I remember that you explained me last week, I thought like that is related to the servers that were down. Here I want to query such that it should give the last time when the scrape was successful for both Up and Down Servers.

The subquery you mentioned above :

max_over_time((time() * up)[24h:5m]) -----> In This query we are getting values for both up and down, but for the servers which were down I am getting the wrong time(i.e the year was 1970) and the servers which were up, the timestamp was correct.

Or 

max_over_time((timestamp(up == 1))[24h:5m]) ------> This query we are getting output which were up, I am not getting the value which were down

Is there any other metric or method to know the timestamp of the last successful scrape for both up and down servers?

Thanks & regards,

Bharath Kumar.

[Brian Candler]

On Monday, 19 September 2022 at 04:14:14 UTC+1 chembakay...@gmail.com wrote:

The subquery you mentioned above :

max_over_time((time() * up)[24h:5m]) -----> In This query we are getting values for both up and down, but for the servers which were down I am getting the wrong time(i.e the year was 1970) and the servers which were up, the timestamp was correct.

This will give zero if the servers have been down for more than 24 hours, i.e. the value of the "up" metric is zero at every point during the last 24 hours.  Essentially, a result of zero here means "server has been down for more than 24 hours".

You can increase the size of the time window from 24h to whatever is needed - but the query will become more expensive as it has to look over a wider range of data.

As far as I know, there is no existing metric you can query which says "the time a scrape was last successful". Given that the prometheus server could be restarted at any time, this would involve maintaining state when the server is shutdown and restarted.

You could perhaps synthesise one using recording rules. That is: I believe it's possible to write a recording rule which depends on its own previous value.  Without careful rule writing, it might still fail if the prometheus server itself is shut down for more than 5 minutes.

 

Or 

max_over_time((timestamp(up == 1))[24h:5m]) ------> This query we are getting output which were up, I am not getting the value which were down

Again for the same reason: the servers which are down for more than 24 hours have 'up' value of 0, so the filter "up == 1" excludes them at all points in the 24 hour period that you are querying.