prometheus not scrapping targets when timestamp field is present

Venkata Bhagavatula

unread,

Jul 2, 2020, 5:33:20 AM7/2/20

to Prometheus Users

Hi,

We are using prometheus version 2.11.1, In our application, the scrape target has timestamp field. when timestamp field is present, then prometheus is not scrapping any metrics.

Following is the output of the curl request for scrape target:

cmd: curl http://<readacted>:24231/metrics

meas_gauge{id="Filtered",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0 1593680400000
meas_gauge{id="Rejected",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0 1593680400000
meas_gauge{id="ReprocessedIn",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0 1593680400000
meas_gauge{id="Created",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0 1593680400000
meas_gauge{id="Duplicated",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0 1593680400000
meas_gauge{id="Stored",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 336.0 1593680400000
meas_gauge{id="Retrieved",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 354.0 1593680400000
meas_gauge{id="ReducedInMerging",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0 1593680400000

I checked that time is in sync between the prometheus node and the target node.
Following is the epoch time on the prometheus node:

cmd: date +'%s%3N'
1593681793979

Epoch difference between the prometheus node and the time stamp present in the sample is more than an hour.

difference = ( 1593681793979 - 1593680400000) / 1000 = 1393sec = 23min

Scrape_interval is configured as 300s

honor_timestamps is set to true.

Can you let us know why prometheus is not able to scrape the targets? Is it due to the timestamp difference between prometheus and target?

How much difference will prometheus tolerate?

Thanks n Regards,

Chalapathi

Venkata Bhagavatula

unread,

Jul 6, 2020, 3:31:40 AM7/6/20

to Prometheus Users

Hi All,

Can any one respond to my queries? Also we observed the following:

1. If for eg timestamp(epoch) in the scrape is 12:00:00, then prometheus is not scrapping the targets

2. If for eg timestamp(epoch) in the scrape is 12:00:01, then prometheus is scrapping the targets.

Thanks & regards,

Chalapathi

Aliaksandr Valialkin

unread,

Jul 6, 2020, 4:23:24 AM7/6/20

to Venkata Bhagavatula, Prometheus Users

Prometheus doesn't support storing historical data and samples with out-of-order timestamps. If you need writing such data, then take a look at other Prometheus-inspired solutions. See, for example, https://victoriametrics.github.io/#backfilling .

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABXnQPuB5iWDhDw06OLOepmz5_XgC2a%3DC9uuVaDKcczR9B-%2BAA%40mail.gmail.com.