Prometheus Scraping Abnormally

57 views
Skip to first unread message

Umut Cokbilir

unread,
Apr 6, 2021, 1:42:50 AM4/6/21
to Prometheus Users
Hi All,

Do you know the root cause of the problem at attachment? Some devices display the interface data abnormally.

Device type: huawei router
prometheus version: prometheus-2.25.1.linux-amd64
scrape_interval: 2m scrape_timeout: 2m
snmp_exporter version: snmp_exporter-0.20.0.linux-amd64
snmp timeout:2m

Thanks
Prometheus Scraping Problem.JPG

Ben Kochie

unread,
Apr 6, 2021, 3:11:51 AM4/6/21
to Umut Cokbilir, Prometheus Users
What is the query and query parameters for that graph?

It could either be due to a lost scrape, or a caching bug in the device's SNMP stack. I've seen that kind of thing when SNMP result caching is used on the device.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fcfd370b-289a-4d2e-b52b-f8c075edcb82n%40googlegroups.com.

Umut Cokbilir

unread,
Apr 6, 2021, 3:28:44 AM4/6/21
to Prometheus Users
I haven't seen this problem in a previous version of snmp_exporter and prometheus.

Queries:
((rate(ifHCInOctets{ifAlias=~".*_NNI.*",ifDescr=~"[[int]]"}[10m]) * 8) and on (instance) sysName{sysName =~ ".*$name.*"})
((rate(ifHCOutOctets{ifAlias=~".*_NNI.*",ifDescr=~"[[int]]"}[10m]) * 8) and on (instance) sysName{sysName =~ ".*$name.*"})

6 Nisan 2021 Salı tarihinde saat 10:11:51 UTC+3 itibarıyla sup...@gmail.com şunları yazdı:

Ben Kochie

unread,
Apr 6, 2021, 3:39:10 AM4/6/21
to Umut Cokbilir, Prometheus Users
The version of Prometheus or snmp_exporter don't have anything to do with this. It's a data or scrape problem with the device.

If you do a console query for ifHCInOctets{ifAlias="X",ifDescr="Y",instance="Z"}[2h] at 22:00 for the series that you're seeing, it will return the raw data in the TSDB. That should help you debug what happened.

Reply all
Reply to author
Forward
0 new messages