snmp alerts
803 views
Skip to first unread message
Andrew Meyer
Oct 16, 2017, 4:23:22 PM10/16/17
to Prometheus Users
Hello,
I am trying to setup SNMP alerts via Alertmanager.
I have been following the directions on https://blog.svedr.in/posts/prometheus-quick-start.html
I just wanted to make sure that I am going down the correct path.
Most of the stuff that I plan to monitor with the snmp_exporter will be hardware. Will the examples given by this person work for hardware?
Ben Kochie
Oct 16, 2017, 4:40:06 PM10/16/17
to Andrew Meyer, Prometheus Users
Looks somewhat reasonable, except it uses direct building from go source, rather than using officially released binaries. This will likely give you un-released and unexpected bugs. I would suggest using official binaries when possible.
What kind of hardware are you looking to monitor? There are lots of modules for different types.
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b3d5aacd-1d3f-44b9-adaf-e7a089c839de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Andrew Meyer
Oct 16, 2017, 4:42:29 PM10/16/17
to Ben Kochie, Prometheus Users
So I am using the binaries. My installation is CentOS 7.
I am trying to monitoring Juniper, HP, Aruba, & Cisco.
If you have any examples I am trying to write my own documentation.
Thank you!
Please remember to reply to all if there are multiple recipients.
Ben Kochie
Oct 16, 2017, 4:47:07 PM10/16/17
to Andrew Meyer, Prometheus Users
All of those network devices should be possible, make sure to read the documentation on the snmp_exporter generator[0] for customizing the configuration for various devices and authentication. It will save a lot of time creating valid snmp_exporter configurations with custom walks.
The code is a bit out of date, but we use ansible to deploy Prometheus to deploy SNMP monitoring for FOSDEM[1].
With FOSDEM 2018 coming up in February we will start working on this again soon.
Andrew Meyer
Oct 16, 2017, 4:49:26 PM10/16/17
to Ben Kochie, Prometheus Users
So i've already got the snmp.yml generated. Now I need to get my alerts setup. Will the alerts that were on the website I sent initially work?
Do you have any examples?
Please remember to reply to all if there are multiple recipients.
Ben Kochie
Oct 16, 2017, 4:55:28 PM10/16/17
to Andrew Meyer, Prometheus Users
All of the alerts seem valid, but may not be appropriate for your data. Most of them I would not use in production as they don't fit symptom based alerting[0] philosophy.
You will need to think about what you want to alert on and why in order to generate alerts that are appropriate for your situation. There is no one-size-fits-all here.
Andrew Meyer
Oct 16, 2017, 4:59:52 PM10/16/17
to Ben Kochie, Prometheus Users
Agreed that there is no 1 size fits all.
Ok. I'm going to take what he has a hack away at it.
If I need more help I'll reach out through here.
Thank you.
Please remember to reply to all if there are multiple recipients.
Andrew Meyer
Oct 16, 2017, 7:17:18 PM10/16/17
to Ben Kochie, Prometheus Users
So i've configured the following alerts:
ALERT snmp_down
IF up == 0 AND job="snmp"
FOR 1m
LABELS { severity="critical" }
ANNOTATIONS {
summary = "System is down",
description = "System has been unreachable for more than 1 minute.",
}
ALERT ifLinkUpDownTrapEnable
IF ifName{ifIndex="4",ifName="BV1"}
FOR 1m
LABELS { severity="critical" }
ANNOTATIONS {
summary = "The port is down",
description = "AP Port is down",
}
However i'm not seeing any alerts getting generated. I'm not seeing much in the logs as to why.
Please remember to reply to all if there are multiple recipients.
Ben Kochie
Oct 17, 2017, 2:50:40 AM10/17/17
to Andrew Meyer, Prometheus Users
Alerts are only generated when the query returns data. The best way to find out if something will alert is to put the 'IF' query into the /graph URL. (not including the IF keyword of course)
The first alert query is probably invalid. I'm guessing what you really want is this:
ALERT SNMPDown
IF up{job="snmp"} == 0
FOR 1m
LABELS { severity="critical" }
ANNOTATIONS {
summary = "SNMP is down",
description = "SNMP exporter has been unreachable for more than 1 minute.",
}
The second query doesn't make much sense either as it's trying to alert on an OID that is a constant, and is filtering for a single interface port. Are you trying to alert on something like this:
ALERT SNMPLinkDown
IF ifOperStatus != 1 AND ifLinkUpDownTrapEnable == 1
FOR 1m
LABELS { severity="critical" }
ANNOTATIONS {
summary = "The port is down",
description = "Monitored port is down (state={{$value}})",
}
That will alert if a port is in configured for link status traps any state is other than up[0]. Note that I put the ifOperStatus first, as the left most metric value is passed as $value to the template.
[0]:
ifOperStatus OBJECT-TYPE
SYNTAX INTEGER {
up(1), -- ready to pass packets
down(2),
testing(3), -- in some test mode
unknown(4), -- status can not be determined
-- for some reason.
dormant(5),
notPresent(6), -- some component is missing
lowerLayerDown(7) -- down due to state or
-- lower-layer interface(s)
}
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAEAR6cNKKZhdZNBEg_7z72fpJ3RdWdQjg%2BquUkZhOu6itNhW%2Bw%40mail.gmail.com.
roger.l...@basalt.se
Oct 17, 2017, 11:30:21 AM10/17/17
to Prometheus Users
If you are also looking into sending SNMP traps from alert manager when alarms are raised you might want to try https://github.com/chrusty/prometheus_webhook_snmptrapper
Best regards
Roger
Reply all
Reply to author
Forward
0 new messages