prometheus notis “sample timestamp out of order”

[Jason Tom]

This is my prometheus.yml config info:

$ cat /etc/prometheus/prometheus.yml 
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

rule_files:
  - "/etc/prometheus/alert.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    metrics_path: "/metrics"

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['172.30.30.6:9090']
        labels:
          group: 'monitor'

  - job_name: 'cadvisor'
    metrics_path: "/metrics"
    scrape_interval: 5s
    static_configs:
      - targets: ['172.30.30.5:8080', '172.30.30.6:8080', '172.30.30.11:8080']
        labels:
          group: 'monitor'

  - job_name: 'node-exporter'
    metrics_path: "/metrics"
    scrape_interval: 5s
    static_configs:
      - targets: ['172.30.30.5:9100', '172.30.30.6:9100', '172.30.30.11:9100']
        labels:
          group: 'monitor'

  - job_name: 'alertmanager'
    metrics_path: "/metrics"
    scrape_interval: 5s
    static_configs:
      - targets: ['172.30.30.6:9093']
        labels:
          group: 'monitor'

In the Prometheus ui, all target states are up.In the graph of the UI, no datapoints found.

Here is the information for the Prometheus log.

Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{group="monitor", instance="172.30.30.11:8080", job="cadvisor"} => 1 @[1492947990.969] source="scrape.go:586" 
Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{group="monitor", instance="172.30.30.11:8080", job="cadvisor"} => 0.122665982 @[1492947990.969] source="scrape.go:589" 
Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Scrape sample count sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{group="monitor", instance="172.30.30.11:8080", job="cadvisor"} => 0.122665982 @[1492947990.969] source="scrape.go:592" 
Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Scrape sample count post-relabeling sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{group="monitor", instance="172.30.30.11:8080", job="cadvisor"} => 0.122665982 @[1492947990.969] source="scrape.go:595" 
Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Error on ingesting out-of-order samples" numDropped=1332 source="scrape.go:533" 
Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{group="monitor", instance="172.30.30.5:9100", job="node-exporter"} => 1 @[1492947991.37] source="scrape.go:586" 
Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{group="monitor", instance="172.30.30.5:9100", job="node-exporter"} => 0.067132553 @[1492947991.37] source="scrape.go:589" 
Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Scrape sample count sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{group="monitor", instance="172.30.30.5:9100", job="node-exporter"} => 0.067132553 @[1492947991.37] source="scrape.go:592" 
Apr 23 19:46:31 localhost prometheus[31806]: time="2017-04-23T11:46:31Z" level=warning msg="Scrape sample count post-relabeling sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{group="monitor", instance="172.30.30.5:9100", job="node-exporter"} => 0.067132553 @[1492947991.37] source="scrape.go:595"

[Brian Brazil]

There's likely something up with time on the machine. Try running an ntp client.

Brian

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/83e91437-81cc-49e7-afe7-e4795cb7a868%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Ben Kochie]

Yes, I was just going to ask about time on this system.  It seems like time is being stepped in a very bad way.  Is this a VM of some kind?

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLoDvyUA3C1QSVQOhjpdw%3DigEVAX1dnUc9kwE%3D%2BfoNDKaQ%40mail.gmail.com.

[Jason Tom]

Yes, This is running in the VM. but the time is UTC, containers and VM are the same time.

在 2017年4月25日星期二 UTC+8下午3:47:10，Ben Kochie写道：

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/83e91437-81cc-49e7-afe7-e4795cb7a868%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

Additional questions:

* Which hypervisor/cloud provider is this?

* Are you running an NTP client? (Which one?)

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/da8201d6-c294-46dc-81a3-2e8767b35dd5%40googlegroups.com.

[Jason Tom]

Run on the internal EXI, not run ntp, I will try it.

Thanks

在 2017年4月25日星期二 UTC+8下午4:30:11，Ben Kochie写道：