Cannot save metrics with timestamp

[shimpeko]

Hello,

Metrics are not saved when I put a timestamps with metrics.

I've got through github issue and some of prometheus codes but not yet getting the root cause.

I would be happy if I can get why I'm getting below errors.

Errors

Error 1

I got below error when executing prometheus with prometheus.yml (v1). Please see blow for the prometheus.yml (v1).

prometheus_1  | level=warn ts=2020-05-25T12:29:17.216Z caller=scrape.go:1240 component="scrape manager" scrape_pool=my-app target=https://<internal server address>:443/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=600

Error2

I got massive amount of below log when running prometheus with prometheus.yml (v2). Please see below for prometheus.yml (v2)

prometheus_1  | level=info ts=2020-05-25T15:57:25.857Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=1164 last=1165 duration=2.1818ms

prometheus_1  | level=info ts=2020-05-25T15:57:25.867Z caller=head.go:662 component=tsdb msg="head GC completed" duration=509.9µs

prometheus_1  | level=info ts=2020-05-25T15:57:25.879Z caller=head.go:662 component=tsdb msg="head GC completed" duration=549.7µs

prometheus_1  | level=info ts=2020-05-25T15:57:25.882Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=1166 last=1167 duration=2.7017ms

prometheus_1  | level=info ts=2020-05-25T15:57:25.892Z caller=head.go:662 component=tsdb msg="head GC completed" duration=564.7µs

prometheus_1  | level=info ts=2020-05-25T15:57:25.903Z caller=head.go:662 component=tsdb msg="head GC completed" duration=651.9µs

prometheus_1  | level=info ts=2020-05-25T15:57:25.906Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=1168 last=1169 duration=2.8861ms

prometheus_1  | level=info ts=2020-05-25T15:57:25.916Z caller=head.go:662 component=tsdb msg="head GC completed" duration=562.8µs

prometheus_1  | level=info ts=2020-05-25T15:57:25.925Z caller=head.go:662 component=tsdb msg="head GC completed" duration=633µs

Background

I'm testing prometheus for custom app.

I've created endpoint which expose metrics in text-based format for the app.

Since metrics are calculated with batch job, I've put a timestamps to metrics to show when the metrics was calculated.

The batch job is executed in every 5 min, so the timestamp will not be older than 5 min compared to the current timestamp.

Here is an example of exposed data. There are about 600 lines in real data.

#HELP processed_rows_total the total number of processed rows
#TYPE processed_rows_total counter
processed_rows_total{stream_name="a", result="success"} 93163 1590422899
processed_rows_total{stream_name="a", result="error"} 0 1590422899
processed_rows_total{stream_name="b", result="success"} 45695851 1590422899
processed_rows_total{stream_name="b", result="error"} 0 1590422899

Environment

Running with docker.

docker-compose.yml

prometheus:

  image: prom/prometheus:v2.17.2

  entrypoint:

    - "/bin/prometheus"

    - "--log.level=debug"

    - "--config.file=/etc/prometheus/prometheus.yml"

    - "--storage.tsdb.path=/prometheus"

    - "--web.console.libraries=/usr/share/prometheus/console_libraries"

    - "--web.console.templates=/usr/share/prometheus/consoles"

  volumes:

    - ./prometheus.yml:/etc/prometheus/prometheus.yml

  ports:

    - "127.0.0.1:9090:9090"

prometheus.yml (v1)

# my global config

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'prometheus'

    static_configs:

    - targets: ['localhost:9090']

  - job_name: 'my-app'

    metrics_path: '/metrics'

    scrape_interval: 1m

    scheme: 'https'

    static_configs:

    - targets: ['<internal server address>']

prometheus.yml (v2)

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'my-app'
    metrics_path: '/metrics'
    scrape_interval: 1m
    scheme: 'https'
    static_configs:
    - targets: ['<internal server address>']

Other notes

With honor-timestamps: false configuration in prometheus.yml, metrics are saved correctly.

[shimpeko]

Resolved.

Precision of timestamp must be millisecond but I've had put second.

[Brian Candler]

Aside: if your metrics are only updated every 5 minutes from a batch job, then node_exporter's textfile collector is the simplest way to do this.

The recommended way to expose the time when a metric was updated is to have a separate metric with the timestamp. The textfile collector does this automatically:

$ curl -Ss localhost:9100/metrics | grep node_textfile

# HELP node_textfile_mtime_seconds Unixtime mtime of textfiles successfully read.

# TYPE node_textfile_mtime_seconds gauge

node_textfile_mtime_seconds{file="role.prom"} 1.505382562e+09

# HELP node_textfile_scrape_error 1 if there was an error opening or reading a file, 0 otherwise

# TYPE node_textfile_scrape_error gauge

node_textfile_scrape_error 0