msg="Error on ingesting samples that are too old or are too far into the future" ?

[petar...@gmail.com]

Hi folks,

I'm scraping matrics from several instances (6 linux instances).

5 instances was in timezone where is prometheus server.

1 instance was in different time zone.

On prometheus server instance in log I can see many of this log for this instance which is in another time zone:

msg="Error on ingesting samples that are too old or are too far into the future"

In grafana also for this 5 instances my graph works ok, but for this one not.

In prometheus web interface I can regular open /metrics for this problematic instance and I can see metrics.

Can someone explain me what this error mean, and how can I resolve this error.

Thanks in advanced.

[Brian Brazil]

This error isn't related to timezones, as Prometheus doesn't use time zones. It sounds like you're scraping something that is exposing timestamps which are not for now - which you should stop doing. 

--

Brian Brazil

www.robustperception.io

[Petar Kozic]

How can I figure what exposing timestamp ?

I’m using collectd for export and collectd write_prometheus plugin.

Thank’s Brian !

[Brian Brazil]

On 20 March 2018 at 13:56, Petar Kozic <petar...@gmail.com> wrote:

How can I figure what exposing timestamp ?

I’m using collectd for export and collectd write_prometheus plugin.

It'd be collectd then, which does expose timestamps. It's taking over an hour for data to get to Prometheus, you should scrape more directly.

Brian

 

Thank’s Brian !

On March 20, 2018 at 2:55:12 PM, Brian Brazil (brian.brazil@robustperception.io) wrote:

On 20 March 2018 at 13:52, <petar...@gmail.com> wrote:

Hi folks,

I'm scraping matrics from several instances (6 linux instances).

5 instances was in timezone where is prometheus server.

1 instance was in different time zone.

On prometheus server instance in log I can see many of this log for this instance which is in another time zone:

msg="Error on ingesting samples that are too old or are too far into the future"

In grafana also for this 5 instances my graph works ok, but for this one not.

In prometheus web interface I can regular open /metrics for this problematic instance and I can see metrics.

Can someone explain me what this error mean, and how can I resolve this error.

This error isn't related to timezones, as Prometheus doesn't use time zones. It sounds like you're scraping something that is exposing timestamps which are not for now - which you should stop doing. 

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io

[uvlig...@gmail.com]

Saying that prometheus is not using timezones cannot be correct as any timestamp has a timezone of some sorts. So its always the local system timezone. 
I worked with a lot of systems which stored their timedata always in GMT after converting timestamps with the timezone info.

Which is already the answer to the questions I have looked for.

If prometheus is using the local timezone  and my timestamps come from another system, the timezones have to be synced.

Also it sounds that the limit is one hour. which is the original question about how big is the range of timestamps currently accepted.

Thank you!

[Ben Kochie]

https://en.wikipedia.org/wiki/Unix_time

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/078459c5-347a-4519-b18d-f96f9bfaed1f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.