should exporters include sample timestamps

[Christoph Anton Mitterer]

Hey.

I'm writing an exporter[0] and wondered whether one should set the timestamps on the metrics themselves, i.e.

  foo{...} value timestamp

respectively the add_metric(timestamp=...) parameter from https://prometheus.github.io/client_python/collector/custom/ .

I couldn't really find a lot of documentation on them merely some comments in some github issues that using them would be usually discouraged.

Also there's https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling

which says "Accordingly, you should not set timestamps on the metrics you expose, let Prometheus take care of that.".

In [0] my data comes actually from varying sources... which I first collect (concurrently) on every collection. So by that already my source data may be from various times.

The service where I get the data from (dCache) even caches some of the data for e.g. up to a minute (or even more). In the future, dCache may however give me the information when exactly it was collected (within dCache)... so I could use this on the samples.

My naive guess would have been that the timestamp should be from the time at which the actual underlying data was collected.

In particular I will also have metrics like:

   last_heartbeat_timestamp_seconds

[1]

which would then be the epoch seconds of when the (at the time of the scrape) last heartbeat from something was received.

In that case, the value would probably be the same as the sample timestamp?

So questions are:

1) What are they actually?

2) When / for what should one set them? Should one make their inclusion e.g. configurable?

3) What should the timestamp actually be? Like the start of when the data was collected? When that finished? The middle? Something else?

4) What should one do in more complex cases? Above I write "time at which the actual underlying data was collected"... now I do have cases, where e.g. the value of a metric may be from timestamp A, which some of it's labels might be from other timestamps B, C, etc.?

Thanks,

Chris.

[0] https://gitlab.com/calestyo/prometheus-dcache-exporter

[1] btw: Everyone seems to recommend Gauges for these... why not Counters? Shouldn't the epoch only increase?

[David Leadbeater]

On Tue, Jul 14, 2026 at 06:20:48PM -0700, Christoph Anton Mitterer wrote:
[...]

> I couldn't really find a lot of documentation on them merely some comments
> in some github issues that using them would be usually discouraged.
> Also there's
> https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling
> which says "Accordingly, you should not set timestamps on the metrics you
> expose, let Prometheus take care of that.".

Is that not enough documentation? It says not to do it.

> In particular I will also have metrics like:
> last_heartbeat_timestamp_seconds
> [1]
> which would then be the epoch seconds of when the (at the time of the
> scrape) last heartbeat from something was received.
> In that case, the value would probably be the same as the sample timestamp?

This is a fine metric, but it makes no sense to set a timestamp on it.

If the heartbeat has a timestamp on it, then it might go stale when the
exporter is still returning it, depending how your exporter works. This
could potentially break an alert on a condition like the heartbeat being
older than a threshold. (It's obviously possible to still alert on such
a condition, but consider how to make the metrics nice for your users,
setting a timestamp does not make for reliable querying.)

> So questions are:
> 1) What are they actually?
> 2) When / for what should one set them? Should one make their inclusion
> e.g. configurable?

They shouldn't be configurable. Pretty much the only use for them is
when moving samples from Prometheus to another Prometheus (e.g.
federation). It just happens the metric format allows them as the same
format is used for federation, but unless you're doing that you can
safely pretend they don't exist.

Setting them outside of the monitoring system has problems such as
breaking staleness as mentioned above, making it very hard to debug if
another system has a clock out of sync, etc.

> 3) What should the timestamp actually be? Like the start of when the data
> was collected? When that finished? The middle? Something else?
> 4) What should one do in more complex cases? Above I write "time at which
> the actual underlying data was collected"... now I do have cases, where
> e.g. the value of a metric may be from timestamp A, which some of it's
> labels might be from other timestamps B, C, etc.?

These two questions show more of the reasons why this isn't a good idea.
If every timestamp is set by Prometheus you don't have to worry about
the edge cases here and in fact there's no reason to.

[Christoph Anton Mitterer]

Hey David.

On Wed, 2026-07-15 at 12:44 +1000, David Leadbeater wrote:
> > Also there's
> > https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling
> > which says "Accordingly, you should not set timestamps on the
> > metrics you
> > expose, let Prometheus take care of that.".
>
> Is that not enough documentation? It says not to do it.

Well, I simply would like to understand why, especially since it seems
pretty counter-intuitive.
Also it's the only real documentation that indicates it... e.g. the
Python client doesn't mentioned anything about that one shouldn't use
timestamp=.


I mean as I've said... the underlying data, returned by my exporter may
be already somewhat old when it gets it, because the monitored software
allows it to be cached (which the user may basically freely configure).

The default of that software would be to cache it for 1min, so if my
exporter would be scraped every 10s, it would return roughly 6 times
the very same data, with prometheus having no real chance to notice
this (my assumption was, it would with a timestamp).

Even if'd always get fresh data, my exporter might e.g. run for a while
(this a big one, monitoring a clustered software, so on a production
site it may easily 50-100k samples per scrape.

Which currently takes it already about 2-3s to process.

Not much, but still, when prometheus gets it, the data is already at
least that old.

> > In particular I will also have metrics like:
> >    last_heartbeat_timestamp_seconds
> > [1]
> > which would then be the epoch seconds of when the (at the time of
> > the
> > scrape) last heartbeat from something was received.
> > In that case, the value would probably be the same as the sample
> > timestamp?
>
> This is a fine metric, but it makes no sense to set a timestamp on
> it.
>
> If the heartbeat has a timestamp on it, then it might go stale when
> the
> exporter is still returning it, depending how your exporter works.

Could possibly you explain why? Or do you mean, if e.g. the attached
timestamp would be like older than the 5mins?

But then again, wouldn't one want it to go stale, if the actual data is
from that time and hasn't been updated since?

Take again my above case, where the raw data I collect is cached... it
seems wrong to present it as fresh if it actually isn't.


I'm also still not sure what the attached timestamps actually do? Is it
simply that without a timestamp, prometheus takes it's own time from
when the data was received... while with a timestamp it simply takes
whatever is attached on the sample?

 
>
> They shouldn't be configurable. Pretty much the only use for them is
> when moving samples from Prometheus to another Prometheus (e.g.
> federation). It just happens the metric format allows them as the
> same
> format is used for federation, but unless you're doing that you can
> safely pretend they don't exist.

I see (albeit not fully understand) ^^
Shouldn't one then perhaps add this do the documentation of the Python
client?

> > 3) What should the timestamp actually be? Like the start of when
> > the data
> > was collected? When that finished? The middle? Something else?
> > 4) What should one do in more complex cases? Above I write "time at
> > which
> > the actual underlying data was collected"... now I do have cases,
> > where
> > e.g. the value of a metric may be from timestamp A, which some of
> > it's
> > labels might be from other timestamps B, C, etc.?
>
> These two questions show more of the reasons why this isn't a good
> idea.

I can understand this from the PoV, that when Prometheus itself sets
the timestamp - they're the same on all the samples of a given scrape
(I presume)... which is probably nice.

But it also means that Prometheus has to make such choice, doesn't it?
Does it take the time from when the scrap started... or when it
finished.

Plus it would simply "ignore" the complex cases I've mentioned... i.e.
pretending as if all data was basically atomically collected and fresh
at the timestamp it sets.


And don't get me wrong... I'm not saying that this may not be the best
thing one should do in practise... I'm just trying to understand it :-)



Thanks,
Chris.