Prometheus - Rule evaluation result discarded err="duplicate sample for timestamp"
1,234 views
Skip to first unread message
Umut Cokbilir
Dec 13, 2022, 12:54:06 AM12/13/22
to Prometheus Users
Hi All,
I couldn't understand the root cause of the problem below. How to solve it?
Prometheus version: prometheus-2.40.3.linux-amd64
snmp_exporter version: snmp_exporter-0.20.0
Rules:
- record: fix:min_in
expr: min_over_time(access:ifHCInOctets:FIX_rate[1d])
- record: fix:min_out
expr: min_over_time(access:ifHCOutOctets:FIX_rate[1d])
- record: fix:max_in
expr: max_over_time(access:ifHCInOctets:FIX_rate[1d])
- record: fix:max_out
expr: max_over_time(access:ifHCOutOctets:FIX_rate[1d])
- record: fix:avg_in
expr: avg_over_time(access:ifHCInOctets:FIX_rate[1d])
- record: fix:avg_out
expr: avg_over_time(access:ifHCOutOctets:FIX_rate[1d])
- record: fix:avg_in:12h
expr: avg_over_time(access:ifHCInOctets:FIX_rate[12h])
- record: fix:avg_out:12h
expr: avg_over_time(access:ifHCOutOctets:FIX_rate[12h])
04:41:23 --ts=2022-12-13T01:41:23.771Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:min_in index=16 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:min_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 22676.373333333333 @[1670895366553]"
04:41:25 --ts=2022-12-13T01:41:25.099Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 45519.09881820317 @[1670895366553]"
04:41:25 --ts=2022-12-13T01:41:25.495Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571800.1416907905 @[1670895366553]"
04:41:26 --ts=2022-12-13T01:41:25.864Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 37126.089073190036 @[1670895366553]"
04:41:26 --ts=2022-12-13T01:41:26.208Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 568421.9923866978 @[1670895366553]"
05:21:23 --ts=2022-12-13T02:21:23.715Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 45353.12701316737 @[1670897766553]"
05:21:24 --ts=2022-12-13T02:21:24.126Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571748.0096685543 @[1670897766553]"
05:21:24 --ts=2022-12-13T02:21:24.453Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 31094.769935320466 @[1670897766553]"
05:21:25 --ts=2022-12-13T02:21:24.794Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 567518.4603637857 @[1670897766553]"
06:06:26 --ts=2022-12-13T03:06:25.984Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 45189.166572681956 @[1670900466553]"
06:06:26 --ts=2022-12-13T03:06:26.353Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571701.6636125795 @[1670900466553]"
06:06:26 --ts=2022-12-13T03:06:26.679Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 29880.13207330078 @[1670900466553]"
06:06:27 --ts=2022-12-13T03:06:27.010Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 567379.9435644317 @[1670900466553]"
06:21:27 --ts=2022-12-13T03:21:27.089Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 45110.958791205536 @[1670901366553]"
06:21:27 --ts=2022-12-13T03:21:27.496Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571723.652811564 @[1670901366553]"
06:21:27 --ts=2022-12-13T03:21:27.832Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 29745.59755724392 @[1670901366553]"
06:21:28 --ts=2022-12-13T03:21:28.173Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 567638.6622438562 @[1670901366553]"
06:56:28 --ts=2022-12-13T03:56:28.058Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 44958.979497887565 @[1670903466553]"
06:56:28 --ts=2022-12-13T03:56:28.650Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571641.2677371124 @[1670903466553]"
06:56:29 --ts=2022-12-13T03:56:29.089Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 29316.18369902085 @[1670903466553]"
06:56:29 --ts=2022-12-13T03:56:29.713Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 567614.2432218753 @[1670903466553]"
07:06:08 --ts=2022-12-13T04:06:07.854Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCInOctets:rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 26498.213333333337 @[1670904066553]"
07:06:10 --ts=2022-12-13T04:06:09.708Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCOutOctets:rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 569222.2666666667 @[1670904066553]"
07:06:12 --ts=2022-12-13T04:06:11.181Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCInOctets:FIX_rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 26498.213333333337 @[1670904066553]"
07:06:13 --ts=2022-12-13T04:06:11.676Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 569222.2666666667 @[1670904066553]"
07:51:27 --ts=2022-12-13T04:51:27.065Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 44893.41348050907 @[1670906766553]"
07:51:27 --ts=2022-12-13T04:51:27.564Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571623.9997716311 @[1670906766553]"
07:51:28 --ts=2022-12-13T04:51:27.907Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 28273.031953867194 @[1670906766553]"
07:51:28 --ts=2022-12-13T04:51:28.245Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 568057.166386401 @[1670906766553]"
08:01:08 --ts=2022-12-13T05:01:08.140Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCInOctets:rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 26605.573333333334 @[1670907366553]"
08:01:10 --ts=2022-12-13T05:01:09.841Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCOutOctets:rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 566597.4400000001 @[1670907366553]"
08:01:12 --ts=2022-12-13T05:01:11.039Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCInOctets:FIX_rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 26605.573333333334 @[1670907366553]"
08:01:13 --ts=2022-12-13T05:01:11.295Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 566597.4400000001 @[1670907366553]"
08:16:26 --ts=2022-12-13T05:16:26.233Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 44828.267673529954 @[1670908266553]"
08:16:26 --ts=2022-12-13T05:16:26.661Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571600.3475860697 @[1670908266553]"
08:16:27 --ts=2022-12-13T05:16:27.003Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 27422.084942333982 @[1670908266553]"
08:16:27 --ts=2022-12-13T05:16:27.338Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 568112.2042330011 @[1670908266553]"
04:41:25 --ts=2022-12-13T01:41:25.099Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 45519.09881820317 @[1670895366553]"
04:41:25 --ts=2022-12-13T01:41:25.495Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571800.1416907905 @[1670895366553]"
04:41:26 --ts=2022-12-13T01:41:25.864Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 37126.089073190036 @[1670895366553]"
04:41:26 --ts=2022-12-13T01:41:26.208Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 568421.9923866978 @[1670895366553]"
05:21:23 --ts=2022-12-13T02:21:23.715Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 45353.12701316737 @[1670897766553]"
05:21:24 --ts=2022-12-13T02:21:24.126Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571748.0096685543 @[1670897766553]"
05:21:24 --ts=2022-12-13T02:21:24.453Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 31094.769935320466 @[1670897766553]"
05:21:25 --ts=2022-12-13T02:21:24.794Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 567518.4603637857 @[1670897766553]"
06:06:26 --ts=2022-12-13T03:06:25.984Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 45189.166572681956 @[1670900466553]"
06:06:26 --ts=2022-12-13T03:06:26.353Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571701.6636125795 @[1670900466553]"
06:06:26 --ts=2022-12-13T03:06:26.679Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 29880.13207330078 @[1670900466553]"
06:06:27 --ts=2022-12-13T03:06:27.010Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 567379.9435644317 @[1670900466553]"
06:21:27 --ts=2022-12-13T03:21:27.089Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 45110.958791205536 @[1670901366553]"
06:21:27 --ts=2022-12-13T03:21:27.496Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571723.652811564 @[1670901366553]"
06:21:27 --ts=2022-12-13T03:21:27.832Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 29745.59755724392 @[1670901366553]"
06:21:28 --ts=2022-12-13T03:21:28.173Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 567638.6622438562 @[1670901366553]"
06:56:28 --ts=2022-12-13T03:56:28.058Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 44958.979497887565 @[1670903466553]"
06:56:28 --ts=2022-12-13T03:56:28.650Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571641.2677371124 @[1670903466553]"
06:56:29 --ts=2022-12-13T03:56:29.089Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 29316.18369902085 @[1670903466553]"
06:56:29 --ts=2022-12-13T03:56:29.713Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 567614.2432218753 @[1670903466553]"
07:06:08 --ts=2022-12-13T04:06:07.854Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCInOctets:rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 26498.213333333337 @[1670904066553]"
07:06:10 --ts=2022-12-13T04:06:09.708Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCOutOctets:rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 569222.2666666667 @[1670904066553]"
07:06:12 --ts=2022-12-13T04:06:11.181Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCInOctets:FIX_rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 26498.213333333337 @[1670904066553]"
07:06:13 --ts=2022-12-13T04:06:11.676Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 569222.2666666667 @[1670904066553]"
07:51:27 --ts=2022-12-13T04:51:27.065Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 44893.41348050907 @[1670906766553]"
07:51:27 --ts=2022-12-13T04:51:27.564Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571623.9997716311 @[1670906766553]"
07:51:28 --ts=2022-12-13T04:51:27.907Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 28273.031953867194 @[1670906766553]"
07:51:28 --ts=2022-12-13T04:51:28.245Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 568057.166386401 @[1670906766553]"
08:01:08 --ts=2022-12-13T05:01:08.140Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCInOctets:rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 26605.573333333334 @[1670907366553]"
08:01:10 --ts=2022-12-13T05:01:09.841Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCOutOctets:rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 566597.4400000001 @[1670907366553]"
08:01:12 --ts=2022-12-13T05:01:11.039Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCInOctets:FIX_rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 26605.573333333334 @[1670907366553]"
08:01:13 --ts=2022-12-13T05:01:11.295Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 566597.4400000001 @[1670907366553]"
08:16:26 --ts=2022-12-13T05:16:26.233Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in index=20 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 44828.267673529954 @[1670908266553]"
08:16:26 --ts=2022-12-13T05:16:26.661Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out index=21 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 571600.3475860697 @[1670908266553]"
08:16:27 --ts=2022-12-13T05:16:27.003Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in:12h index=22 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_in:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 27422.084942333982 @[1670908266553]"
08:16:27 --ts=2022-12-13T05:16:27.338Z caller=manager.go:684 level=warn component="rule manager" file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out:12h index=23 msg="Rule evaluation result discarded" err="duplicate sample for timestamp" sample="{__name__=\"fix:avg_out:12h\", ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", instance=\"10.85.4.3\", job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => 568112.2042330011 @[1670908266553]"
Thanks
Julius Volz
Dec 13, 2022, 4:07:31 AM12/13/22
to Umut Cokbilir, Prometheus Users
Hi Umut,
This error occurs when something in Prometheus tries to append a sample to a time series with the same timestamp as the last already stored sample in the series, but with a different sample value. Usually this is because of some accidental collision between time series identities (metric name + label set). Is there any chance that anything else in your Prometheus setup also produces metrics with the same names as the ones in your recording rules? Maybe the recording rules are duplicated, but with different values (probably in the same rule group, since it's the same timestamp)?
Regards,
Julius
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com.
Julius Volz
PromLabs - promlabs.com
Umut Cokbilir
Dec 13, 2022, 5:55:19 AM12/13/22
to Prometheus Users
Hi
Julius,
Please find my recording rules file attached. There isn't any rules using same names.
Please also check it? Where i am doing wrong?
Thanks
13 Aralık 2022 Salı tarihinde saat 12:07:31 UTC+3 itibarıyla juliu...@promlabs.com şunları yazdı:
Brian Candler
Dec 13, 2022, 6:44:31 AM12/13/22
to Prometheus Users
The first error explicitly mentions a particular metric "fix:min_in", and the input to the recording rule which generates that is the metric "access:ifHCInOctets:FIX_rate" (rule 16, counting from zero)
But it also mentions an error for the metric "access:ifHCInOctets:FIX_rate", which takes its input from "ifHCInOctets" (rule 4)
So my guess is that you have duplicate data somewhere, posisbly in ifHCInOctets. Try these queries:
ifHCInOctets{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||"}[1h]
access:ifHCInOctets:FIX_rate{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||"}[1h]
Do you see multiple data points with the same timestamp? In which timeseries? if not, try extending the time range to [1d]
This would however require you to have multiple values of ifHCInOctets with the same value of all labels (same ifAlias, same ifDescr, same ifIndex).
If that's what you're seeing then you have to work backwards: are you scraping the same target twice? Or is there a bug in your snmp_exporter config? Try scraping the device manually using curl:
curl -g 'localhost:9115/snmp?module=XXXX&target=10.85.4.3'
(Having said that, if there were duplicates at this point, I'd be expecting them to be rejected at scrape time).
Are you doing any sort of high-availability or federation? If so, you need different external labels for each data source.
Umut Cokbilir
Dec 13, 2022, 7:21:15 AM12/13/22
to Prometheus Users
Thanks for your detail explanation. You can see my answers below.
I am not doing federation or any sort of availability.
I've checked [1h] then [1d] and haven't seen any multiple data points :(
I am not scraping the same targets twice but i am updating target.yml every hour. Does it affect?
i am not sure there is any bug in my exporter config but it scrapes data from mydevices every 5 minutes.
Please find curl output attached.
13 Aralık 2022 Salı tarihinde saat 14:44:31 UTC+3 itibarıyla Brian Candler şunları yazdı:
Julius Volz
Dec 13, 2022, 8:34:10 AM12/13/22
to Umut Cokbilir, Prometheus Users
Ok, that's strange. Yeah, the TSDB should also reject duplicate samples and out-of-order samples for the same series if they came directly from targets and not a recording rule. So I'm not sure what's happening here. Maybe you could share your prometheus.yml and a sample target.yml as well for good measure?
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6d5070eb-b057-4b6f-9fe6-447cbcfe2c14n%40googlegroups.com.
Julien Pivotto
Dec 13, 2022, 8:42:12 AM12/13/22
to Julius Volz, Umut Cokbilir, Prometheus Users
Is this the same behaviour with Prometheus 2.37?
Otherwise could it be a bug, duplicate of
https://github.com/prometheus/prometheus/issues/11484 ?
> >>>>> .
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Julius Volz
> >>>> PromLabs - promlabs.com
> >>>>
> >>> --
> > You received this message because you are subscribed to the Google Groups
> > "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to prometheus-use...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-users/6d5070eb-b057-4b6f-9fe6-447cbcfe2c14n%40googlegroups.com
> > <https://groups.google.com/d/msgid/prometheus-users/6d5070eb-b057-4b6f-9fe6-447cbcfe2c14n%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>>>
> >>>>
> >>>> --
> >>>> Julius Volz
> >>>> PromLabs - promlabs.com
> >>>>
> >>> --
> > You received this message because you are subscribed to the Google Groups
> > "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to prometheus-use...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-users/6d5070eb-b057-4b6f-9fe6-447cbcfe2c14n%40googlegroups.com
> > .
> >
>
>
> --
> Julius Volz
> PromLabs - promlabs.com
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5yC1LDY1eZKixKN9LB65TcyErb%2B8yfEyRSEqLkrO-1dZQ%40mail.gmail.com.
>
>
> --
> Julius Volz
> PromLabs - promlabs.com
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
--
Julien Pivotto
@roidelapluie
Brian Candler
Dec 13, 2022, 9:37:43 AM12/13/22
to Prometheus Users
I ran your curl output through "promtool check metrics", but it only warns about label and metric naming conventions (which SNMP violates to be consistent with standard MIB usage). There are no duplicate timeseries.
# /opt/prometheus/promtool check metrics <ert3.txt 2>&1 | sort | uniq -c
11247 ifAlias label names should be written in 'snake_case' not 'camelCase'
1 ifAlias metric names should be written in 'snake_case' not 'camelCase'
13767 ifDescr label names should be written in 'snake_case' not 'camelCase'
1 ifDescr metric names should be written in 'snake_case' not 'camelCase'
1 ifHCInOctets counter metrics should have "_total" suffix
11247 ifHCInOctets label names should be written in 'snake_case' not 'camelCase'
1 ifHCInOctets metric names should be written in 'snake_case' not 'camelCase'
1 ifHCOutOctets counter metrics should have "_total" suffix
11247 ifHCOutOctets label names should be written in 'snake_case' not 'camelCase'
1 ifHCOutOctets metric names should be written in 'snake_case' not 'camelCase'
1 sysName label names should be written in 'snake_case' not 'camelCase'
1 sysName metric names should be written in 'snake_case' not 'camelCase'
1 sysUpTime metric names should be written in 'snake_case' not 'camelCase'
1 ifAlias metric names should be written in 'snake_case' not 'camelCase'
13767 ifDescr label names should be written in 'snake_case' not 'camelCase'
1 ifDescr metric names should be written in 'snake_case' not 'camelCase'
1 ifHCInOctets counter metrics should have "_total" suffix
11247 ifHCInOctets label names should be written in 'snake_case' not 'camelCase'
1 ifHCInOctets metric names should be written in 'snake_case' not 'camelCase'
1 ifHCOutOctets counter metrics should have "_total" suffix
11247 ifHCOutOctets label names should be written in 'snake_case' not 'camelCase'
1 ifHCOutOctets metric names should be written in 'snake_case' not 'camelCase'
1 sysName label names should be written in 'snake_case' not 'camelCase'
1 sysName metric names should be written in 'snake_case' not 'camelCase'
1 sysUpTime metric names should be written in 'snake_case' not 'camelCase'
Changing the target.yml file periodically won't make a difference; Prometheus will just pick up the new one for the next scrape. So this is odd.
Umut Cokbilir
Dec 13, 2022, 10:28:37 AM12/13/22
to Prometheus Users
Hi All,
I added my all config files attached.
Thank you so much for this deep diving.
13 Aralık 2022 Salı tarihinde saat 17:37:43 UTC+3 itibarıyla Brian Candler şunları yazdı:
Brian Candler
Dec 13, 2022, 10:42:40 AM12/13/22
to Prometheus Users
Thank you for the full files (although you probably should have scrubbed your snmp community)
honor_timestamps: true
which generally shouldn't be needed except in very special circumstances. I don't *think* it will make any difference here, since snmp_exporter doesn't give timestamps with its metrics anyway, but can you try commenting it out?
Otherwise, follow Julien's advice of trying the same configuration with v2.37.5 (which is an LTS release) and see if the problem recurs.
Umut Cokbilir
Dec 13, 2022, 3:59:55 PM12/13/22
to Prometheus Users
Hi All,
I tried commenting it out but it didn't solve the problem that I faced. I'll try 2.37.5 LTS version, if it solves then i'll update you.
Thanks
13 Aralık 2022 Salı tarihinde saat 18:42:40 UTC+3 itibarıyla Brian Candler şunları yazdı:
Julius Volz
Dec 15, 2022, 8:01:25 AM12/15/22
to Umut Cokbilir, Prometheus Users
Btw., even though we haven't found your specific problem yet, this thread motivated me to write a blog post about these errors: https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/05123d4b-11de-41b8-bbf0-d8332b6b76ebn%40googlegroups.com.
Brian Candler
Dec 15, 2022, 8:58:40 AM12/15/22
to Prometheus Users
Nice, thank you!
Regards buggy client side timestamps, I didn't realise until now that "honor_timestamps: true" was the default. So for the OP, I should have said to try setting "honor_timestamps: false" instead of removing it. But in any case, snmp_exporter doesn't export timestamps, so it shouldn't make any difference in this case.
Umut Cokbilir
Dec 15, 2022, 1:06:04 PM12/15/22
to Prometheus Users
Hi Brian and Julian,
I would like to update you that I've tried 2.37.5 LTS version which is not solved my problem. :(
If you want, i can share my log again.
Thank you
15 Aralık 2022 Perşembe tarihinde saat 16:58:40 UTC+3 itibarıyla Brian Candler şunları yazdı:
Reply all
Reply to author
Forward
0 new messages