Re: [prometheus-users] snmp_exporter and the generator
1,565 views
Skip to first unread message
Brian Brazil
Jul 31, 2018, 7:59:05 AM7/31/18
to Michael Cooper, Prometheus Users
On 31 July 2018 at 12:53, Michael Cooper <nexus...@gmail.com> wrote:
Hello Everyone,Very new to prometheus and grafana but I am learning everyday. We have a need to use the snmp_exporter as we have some non standard devices to monitor. One being Raritan PDUs the have MIBs and such I can communicate with them. However i have no idea how to add the modue and OIDs I need for this to scrape. We also need to monitor the APC Temp and Humidity sensors on out cages. However I think Arista and PaloAlto already have some Modules and OIDs in the standard snmp.yml file.I have created a dir below my username called /.snmp/mibs and I copied the relevant MIB files in to it and another location i believe is /usr/local/share/snmp/mibs as well. So at this point I misunderstood how the generator works. I was thinking that it read the MIB files and put the relevant Modules and OIDs after reading an article by Consul Labs De, I believe my understanding was incorrect.So now I need something to teach me how to read the MIB files and get the OIDs I need in order to scrape the proper metrics.
Putting the MIBs in the right directory makes them available to the generator, but doesn't change its output. Walking every possible object would be too slow for monitoring.
What you want to do is add the OIDs you care about to the walk of a module in the generator.yml, and then regenerate the snmp.yml.
Brian
--Any help would be greatly apprecited and Thanks for your time in Advance.P.S. I apologize if this is a stupid request.Thanks again,
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cec987f2-3c10-4bb0-856e-495d1bb21e06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Brian Brazil
Michael Cooper
Jul 31, 2018, 8:04:38 AM7/31/18
to Prometheus Users
Thank you Brian, I appreciate the quick response. Oh I got you okay that makes sense. So how do I add new device such as Raritan PDUs? Sorry for the stupid question I am seriously new to this and I am learning as I go.
Thanks again Brian
On Tuesday, July 31, 2018 at 7:59:05 AM UTC-4, Brian Brazil wrote:
On 31 July 2018 at 12:53, Michael Cooper <nexus...@gmail.com> wrote:Hello Everyone,Very new to prometheus and grafana but I am learning everyday. We have a need to use the snmp_exporter as we have some non standard devices to monitor. One being Raritan PDUs the have MIBs and such I can communicate with them. However i have no idea how to add the modue and OIDs I need for this to scrape. We also need to monitor the APC Temp and Humidity sensors on out cages. However I think Arista and PaloAlto already have some Modules and OIDs in the standard snmp.yml file.I have created a dir below my username called /.snmp/mibs and I copied the relevant MIB files in to it and another location i believe is /usr/local/share/snmp/mibs as well. So at this point I misunderstood how the generator works. I was thinking that it read the MIB files and put the relevant Modules and OIDs after reading an article by Consul Labs De, I believe my understanding was incorrect.So now I need something to teach me how to read the MIB files and get the OIDs I need in order to scrape the proper metrics.Putting the MIBs in the right directory makes them available to the generator, but doesn't change its output. Walking every possible object would be too slow for monitoring.What you want to do is add the OIDs you care about to the walk of a module in the generator.yml, and then regenerate the snmp.yml.Brian
Any help would be greatly apprecited and Thanks for your time in Advance.P.S. I apologize if this is a stupid request.Thanks again,
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cec987f2-3c10-4bb0-856e-495d1bb21e06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Brian Brazil
Jul 31, 2018, 8:06:59 AM7/31/18
to Michael Cooper, Prometheus Users
On 31 July 2018 at 13:04, Michael Cooper <nexus...@gmail.com> wrote:
Thank you Brian, I appreciate the quick response. Oh I got you okay that makes sense. So how do I add new device such as Raritan PDUs? Sorry for the stupid question I am seriously new to this and I am learning as I go.
Once the MIBs are in place, generally you'd add a new module (and clear out the ones you don't need - getting all the MIBs for the default config is a little annoying), and set the walk to whatever OIDs you care about.
Brian
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9b77c726-a278-4893-824d-9daf88312220%40googlegroups.com.
Michael Cooper
Jul 31, 2018, 8:09:43 AM7/31/18
to Prometheus Users
Alrighty, I will give it shot and see what I come up with Thank you very much.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9b77c726-a278-4893-824d-9daf88312220%40googlegroups.com.
Nicolai von Neudeck
Jul 31, 2018, 10:48:13 AM7/31/18
to Prometheus Users
Hi Michael,
SNMP is an old metrics, config and alert protocol that was primarily designed for network appliances in the 1980ies and 1990ies. Tons of devices speak some version of it. Network gear, servers, printers, copiers, even some larger office coffee machines.
## GET, SET, TRAP
There are three important modes of communication:
1. One where your agent (in our case the snmp_exporter) asks the SNMP-device for information and gets a response.
SNMP has a tree structure. To get a certain metric you need to request an OID, which is a long line of decimal numbers divided by dots. Each number gives means you choose a certain branch in the tree. E.g.: 1.3.6.1.4.1.9.9.719.1.1.1.1.16 You could just start at the beginning and ask the device for the next point (as they are numbered that is pretty easy to do) until you have gotten everything. That is called a "complete walk". But as Brian mentioned that takes forever (can easily take half an hour on some devices) and is nothing you want to do all the time.
Furthermore these numbers are uninformative for humans and also you don’t know what the return value will be, e.g. a table, a string or a number or a hex-value.
So there are MIBs, those are databases (text files in a somewhat human decipherable format) where each OID in a certain subbranch is documented with a name, an explanation and what kind of value is returned.
Each MIB file only documents its subbranch of the tree, so you need some for the basic values and another one for your device. And for some vendors even more than one.
# The snmp.yml and the generator.yml
The exporter uses the snmp.yml file to know which OID it should walk, how they are and how they should be combined (for example to match certain values with certain names, etc.) and correct some mistakes in the MIB (wrong types and spelling mistakes are common).
You can write the snmp.yml by hand but it is more comfortable to let the generator make one you need based on the generator.yml. Sometimes you have to correct the snmp.yml a little bit by hand because the generator cannot yet generate all the magic the snmp_exporter is capable to execute.
I was in a similar situation to you a while ago and there are a few different concepts to understand. Please forgive me, if I mention something you know already, as I might not know that you know and also this is hopefully helpful for others that search for SNMP later. I’ll likely forget things and might make mistakes. Please correct me if you see them.
# A few basics about SNMP
# A few basics about SNMP
SNMP is an old metrics, config and alert protocol that was primarily designed for network appliances in the 1980ies and 1990ies. Tons of devices speak some version of it. Network gear, servers, printers, copiers, even some larger office coffee machines.
## GET, SET, TRAP
There are three important modes of communication:
1. One where your agent (in our case the snmp_exporter) asks the SNMP-device for information and gets a response.
2. One where you configure a setting on the device and get a response.
3. One were the SNMP-device sends a message without prior request to your agent.
Number 3 is called a TRAP and is used for alerts (that have the unfortunate tendency to get silently lost in a broken network), number 2 is called a SET and is not used much because of security issues and because there are more comfortable config management mechanisms nowadays and number 1 is a GET (but not the HTTP kind) and used for classical metrics.
The Prometheus snmp_exporter only handles the number 1 kind of communication.
## OID and MIBs
3. One were the SNMP-device sends a message without prior request to your agent.
Number 3 is called a TRAP and is used for alerts (that have the unfortunate tendency to get silently lost in a broken network), number 2 is called a SET and is not used much because of security issues and because there are more comfortable config management mechanisms nowadays and number 1 is a GET (but not the HTTP kind) and used for classical metrics.
The Prometheus snmp_exporter only handles the number 1 kind of communication.
## OID and MIBs
SNMP has a tree structure. To get a certain metric you need to request an OID, which is a long line of decimal numbers divided by dots. Each number gives means you choose a certain branch in the tree. E.g.: 1.3.6.1.4.1.9.9.719.1.1.1.1.16 You could just start at the beginning and ask the device for the next point (as they are numbered that is pretty easy to do) until you have gotten everything. That is called a "complete walk". But as Brian mentioned that takes forever (can easily take half an hour on some devices) and is nothing you want to do all the time.
Furthermore these numbers are uninformative for humans and also you don’t know what the return value will be, e.g. a table, a string or a number or a hex-value.
So there are MIBs, those are databases (text files in a somewhat human decipherable format) where each OID in a certain subbranch is documented with a name, an explanation and what kind of value is returned.
Each MIB file only documents its subbranch of the tree, so you need some for the basic values and another one for your device. And for some vendors even more than one.
# The snmp.yml and the generator.yml
The exporter uses the snmp.yml file to know which OID it should walk, how they are and how they should be combined (for example to match certain values with certain names, etc.) and correct some mistakes in the MIB (wrong types and spelling mistakes are common).
You can write the snmp.yml by hand but it is more comfortable to let the generator make one you need based on the generator.yml. Sometimes you have to correct the snmp.yml a little bit by hand because the generator cannot yet generate all the magic the snmp_exporter is capable to execute.
Here is a small example from a generator.yml that makes use of some of the most common functions:
So what does this do?
1. We walk the cucsFaultOccur to get a counter for errors. Each will be labeled with an index that is given by the device.
2. Because that index is rather useless we then look up cucsFaultDn which contains a long string with useful informative values (like what is broken on which machine with wich error code?) and match and replace the aforementioned cucsFaultIndex with it. So we get a nice label with useful information and have good timeseries.
module: cisco-ucs: walk: - 1.3.6.1.4.1.9.9.719.1.1.1.1.16 #cucsFaultOccur lookups: - old_index: cucsFaultIndex new_index: cucsFaultDn overrides: cucsFaultDn: type: DisplayString |
1. We walk the cucsFaultOccur to get a counter for errors. Each will be labeled with an index that is given by the device.
2. Because that index is rather useless we then look up cucsFaultDn which contains a long string with useful informative values (like what is broken on which machine with wich error code?) and match and replace the aforementioned cucsFaultIndex with it. So we get a nice label with useful information and have good timeseries.
3. In a later relabeling step in the config.yml we split that one long string label into single labels to make it more useful when querying and allow queries like "count all errors of hard drives by blade".
4. And because unfortunately the cucsFaultDn is slightly faulty described in the MIB, hence we override its type to what it actually is, a DisplayString.
Of course your values will be different as these are highly vendor and somewhat device specific and of course you need to get the metrics that are of interest for you.
4. And because unfortunately the cucsFaultDn is slightly faulty described in the MIB, hence we override its type to what it actually is, a DisplayString.
Of course your values will be different as these are highly vendor and somewhat device specific and of course you need to get the metrics that are of interest for you.
I hope this gives you some more insight. I have to attend a meeting now, but feel free to ask more questions.
I wish you fun and perseverance!
Nicolai
I wish you fun and perseverance!
Nicolai
Michael Cooper
Jul 31, 2018, 1:50:24 PM7/31/18
to Prometheus Users
Thank Nicolai. Very Well explained.
Ben Kochie
Jul 31, 2018, 2:27:06 PM7/31/18
to Nicolai von Neudeck, Prometheus Users
This is excellent, I would like to format this and add it to the Prometheus official blog. Maybe even a whole guide.
Understanding SNMP is pretty difficult for newcomers. And translating SNMP to be Prometheus is a whole new difficulty. This covers things very well.
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f5bfd1bc-1e71-4993-aeaa-10f946cbb57e%40googlegroups.com.
Nicolai von Neudeck
Oct 8, 2018, 11:27:47 AM10/8/18
to Prometheus Users
Thank you Ben, hey everyone!
I want to work soon on polishing this to be an actual useful guide/blog post.
The idea is to explain how it works, less of a how to but more of a guide to empower others to get enough knowledge and insight to set up their own snmp-exporters to their (and hopefully the larger community’s) needs.
For that I think it is critical to make another post/guide about the "proxy pattern" for exporters (blackbox, snmp, ipmi, are there others?). Is there already a text I could build upon? What is the official™ name of the "proxy pattern"?
If there is no opposition, I would just write something using the blackbox exporter as an example because of its simplicity and larger popularity.
If there is no opposition, I would just write something using the blackbox exporter as an example because of its simplicity and larger popularity.
Nicolai
Brian Brazil
Oct 8, 2018, 11:34:43 AM10/8/18
to von Neudeck, Nicolai, promethe...@googlegroups.com
On Mon, 8 Oct 2018 at 16:27, Nicolai von Neudeck <nicolai.v...@sap.com> wrote:
Thank you Ben, hey everyone!I want to work soon on polishing this to be an actual useful guide/blog post.The idea is to explain how it works, less of a how to but more of a guide to empower others to get enough knowledge and insight to set up their own snmp-exporters to their (and hopefully the larger community’s) needs.For that I think it is critical to make another post/guide about the "proxy pattern" for exporters (blackbox, snmp, ipmi, are there others?). Is there already a text I could build upon? What is the official™ name of the "proxy pattern"?
All exporters are proxies really. The typical name of that type of exporter is blackbox-style or SNMP-style after the relevant exporters, see also https://prometheus.io/docs/instrumenting/writing_exporters/#deployment
Brian
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/923a653f-4037-4494-8ce8-4f9e43581e53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Ben Kochie
Oct 8, 2018, 11:43:55 AM10/8/18
to Nicolai von Neudeck, Prometheus Users
I've been calling them "proxy exporters", since they are not 1:1 with their targets.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/923a653f-4037-4494-8ce8-4f9e43581e53%40googlegroups.com.
Nicolai von Neudeck
Oct 9, 2018, 5:43:35 AM10/9/18
to Prometheus Users
Thanks for your explanations.
This will be nice and make the entry into this field of metrics much kinder and friendlier.
I’ll make two issues in the docs repo and will start to work on PRs.
I’ll make two issues in the docs repo and will start to work on PRs.
Reply all
Reply to author
Forward
0 new messages