Error on ingesting samples that are too old or are too far into the future

[imra...@gmail.com]

I am trying to expose metrics from a file (using a simple http server on my dev box). I have a local instance of prometheus running, fresh without any data. When I launch it with target pointing to my http server, I get the following warning and data is not ingested.

level=warn ts=2017-11-17T01:31:04.68807107Z caller=scrape.go:908 component="target manager" scrape_pool=zodiac target=http://localhost:9999/data/metrics.txt msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

Any idea how to get rid of this message? I tried with multiple samples, different timestamps, but data doesn't get ingested. The timestamp is GMT: Thursday, November 16, 2017 8:09:44 PM

http://localhost:9999/data/metrics.txt

------------------

# HELP zodiac_metric_hv_count Count of hvs.

# TYPE zodiac_metric_hv_count gauge

zodiac_metric_hv_count{dim="total",domain="xxx2",region="test",state="ok",instance="prod",job="zodiac"} 2 1510862984

Config:

# my global config

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with

  # external systems (federation, remote storage, Alertmanager).

  external_labels:

      monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'zodiac'

    honor_labels: true

    scrape_interval: 5s

    metrics_path: '/data/metrics.txt'

    static_configs:

      - targets:

         - 'localhost:9999'

        labels:

[Ben Kochie]

It's recommended to omit the timestamp.  The timestamp in the spec is there to allow for federation or importing of data from external TDSBs.  For normal application metrics, you want to assign the timestamp of the scrape.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9aad417e-98ad-44fe-a247-56417dba50d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Kochie]

Also, you should omit the labels like instance, job, and probably domain and region.  These look like target identifying labels that are added at scrape time by the server.  Metrics exposed by applications should only expose labels that relate to the metric, not the target host.

[Imran]

Thank you for the reply, Ben. 

Couple of questions:

1. How old of a timestamp can prometheus accept? I tried ingesting values that are an hour old, and it worked. But when tried ingesting a day old values, it throws me the same error.

2. I am trying for ways to feed in output from a prometheus server API to another prometheus instance.  If I use federation, would it retain timestamps? I want to run a prometheus server running on my laptop to federate to one behind a gateway. Whenever I start the server on my laptop, I expect it to catch up. Is that possible?

thanks,
Imran

[Ben Kochie]

On Nov 17, 2017 09:04, "Imran" <imra...@gmail.com> wrote:

Thank you for the reply, Ben. 

Couple of questions:

1. How old of a timestamp can prometheus accept? I tried ingesting values that are an hour old, and it worked. But when tried ingesting a day old values, it throws me the same error.

Sorry, but this is not how Prometheus is designed to be used.  It's a monitoring system, not a generic time-series database.  Simply remove the timestamps.

2. I am trying for ways to feed in output from a prometheus server API to another prometheus instance.  If I use federation, would it retain timestamps? I want to run a prometheus server running on my laptop to federate to one behind a gateway. Whenever I start the server on my laptop, I expect it to catch up. Is that possible?

Federation is not for mirroring / replication. It will not work this way.

I would look into using a remote write destination.

Here are two SaaS options:

https://kausal.co

https://www.weave.works/features/prometheus-monitoring/

[Imran]

Hi Ben,

Thank you for the SaaS references and guidance. 

-Imran

thanks,
Imran

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9aad417e-98ad-44fe-a247-56417dba50d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

thanks,
Imran

[uvlig...@gmail.com]

I have a similar issue. I am reading data from a nagios system which contains valid timestamps.
Replacing them with random values depending on the duration of a https query feels terribly wrong!

So I think the question regarding the actual limits that are causing  this error is totally  valid and I think refering to the deprecation is not very constructive.
This is our current use case nevertheless. 

Also the threat with the deprecation can always be resolved by not updating prometheus :)

So which are those limits please?

thanks,
Imran

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.