msg="Error on ingesting out-of-order samples" num_dropped=8 in my promethues

[anton...@gmail.com]

Hi all

     When i checked the log in my prometheus server, i found some warning messages, it just like following :

     level=warn ts=2018-03-08T06:45:14.900579756Z caller=scrape.go:915 component="scrape manager" scrape_pool=federate target="http://ph02v-pms01.ae007.com:9090/federate?match%5B%5D=%7Bjob%3D~%22icmp.%2A%22%7D&match%5B%5D=%7Bjob%3D~%22probe_pbcom.%2A%22%7D&match%5B%5D=%7Bjob%3D~%22probe_rcbc.%2A%22%7D&match%5B%5D=%7Bjob%3D~%22sw_snmp.%2A%22%7D&match%5B%5D=%7Bjob%3D~%22snmp.%2A%22%7D&match%5B%5D=%7Bjob%3D%22node_linux%22%7D&match%5B%5D=%7Bjob%3D%22node_win%22%7D" msg="Error on ingesting out-of-order samples" num_dropped=8

     i have checked the time status :

      [root@ph71v-pms01 lis]# timedatectl

           Local time: Thu 2018-03-08 14:48:24 +08

           Universal time: Thu 2018-03-08 06:48:24 UTC

           RTC time: Thu 2018-03-08 06:48:24

          Time zone: Asia/Manila (+08, +0800)

          NTP enabled: yes

          NTP synchronized: yes

          RTC in local TZ: no

          DST active: n/a

And the ntp status is just like:

[root@ph71v-pms01 lis]# systemctl status chronyd 

鈼 chronyd.service - NTP client/server

   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)

   Active: active (running) since Sat 2017-11-18 08:25:18 +08; 3 months 18 days ago

 Main PID: 678 (chronyd)

   CGroup: /system.slice/chronyd.service

           鈹斺攢678 /usr/sbin/chronyd

Feb 13 10:49:40 ph71v-pms01 chronyd[678]: Selected source 10.71.40.1

Feb 13 15:35:16 ph71v-pms01 chronyd[678]: Selected source 10.71.40.1

What should i do to solve the issue 

[Brian Brazil]

You most likely have duplicate time series coming from different targets. Make sure that all your targets have unique target/external labels.

 

--

Brian Brazil

www.robustperception.io

[anton...@gmail.com]

I have checked all my targets in my server,but i did not find the targets with the same label

在 2018年3月8日星期四 UTC+8下午3:10:05，Brian Brazil写道：

[anton...@gmail.com]

I have found the reason, in my environment i have the global prometheus with federation, In addition,i have configured the same rule as in branch promethues,After i remove the rule and  Adding  '{__name__=~"probe_packetloss:.*"}' under the job to scrape the packet loss metric from all the server,then it does work

在 2018年3月8日星期四 UTC+8下午2:49:55，anton...@gmail.com写道：