skip some network interfaces with snmp exporter

[californi...@gmail.com]

Hi,

I scraped several network devices with snmp exporter and some devices took 5min to respond. That's too long. In general, some network devices generates logical interfaces (vlan interfaces) internally. The number of logical interfaces depends on the configuration and it's over 1000 in our environment. We do not need to scrape those logical interfaces. Current monitoring tool (Munin) can set a filter to skip them.

Please let me know if snmp exporter can skip part of interfaces. This is a huge problem to choose prometheus as our next monitoring system.

[californi...@gmail.com]

In addition, ifType(1.3.6.1.2.1.2.2.1.3) is expected as a key to filter.

[Brian Brazil]

This is not something the snmp exporter can currently do. All I can suggest is ensuring the snmp exporter is right beside the switch to cut down on network latency. 

--

Brian Brazil

www.robustperception.io

[californi...@gmail.com]

thank you for your reply. the problem is high cpu usage of network device that continues during scrape. high cpu usage requires reducing the number of target MIB objects that can be scraped by prometheus. I also have to put long interval of scrape enough to ensure that network devices do not fail because of scrape. these are totally opposite to what I want to achieve with prometheus.

[Ben Kochie]

I would be interested to see the walk that Munin is doing in order to reduce the metrics returned.

If you can give us more specific details about the devices, walks, and your network configuration, it would help.

On Aug 31, 2017 12:48 PM, <californi...@gmail.com> wrote:

thank you for your reply. the problem is high cpu usage of network device that continues during scrape. high cpu usage requires reducing the number of target MIB objects that can be scraped by prometheus. I also have to put long interval of scrape enough to ensure that network devices do not fail because of scrape. these are totally opposite to what I want to achieve with prometheus.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/04236e7e-c71a-4e6b-ad0d-31982e9c9a16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[californi...@gmail.com]

We are IaaS provider, maintaining large network resource pool for our customers. The core L2 switch of the network has thousands of VLANs defined in it because each of our customer consumes VLANs of the pool. This L2 switch creates VLAN I/F internally and automatically for each VLAN to manage it. The VLAN I/Fs are not used to transmit traffic but required for VLAN management. we do not need to scrape VLAN I/Fs but they have their own OID and can be scraped.

Munin plugin, which is a perl script, do the followings:

1. We defined specific values of ifType that we want to scrape for ifEntry(1.3.6.1.2.1.2.2.1) and ifXEntry(1.3.6.1.31.1.1.1). For example, 6 for ethernetCsmacd (regular ethernet I/F) and 161 for ieee8023adLag (bunch of regular ethernet I/Fs). Please refer to the definition of ifType value at https://www.iana.org/assignments/ianaiftype-mib/ianaiftype-mib.

2. walk ifType (1.3.6.1.2.1.2.2.1.3) first with Munin plugin script and stores it in hash (interface id and its ifEntry value)

3. Munin plugin script checks the ifEntry value for each entry of hash. The script discards the entry if ifType value is not 6 nor 161 (refer to #1)

4. walk ifEntry and ifXEntry of interfaces that remain in the hash. Munin plugin script retrieves each value one by one with snmp get request for single MIB object.

Multiple requests for single MIB object in #4 might consume more resources of snmp exporter. But they are mandatory for large network I think.

[Brian Brazil]

Do you know the range indexes in advance, or must you check ifType to find out which you need?

Brian

 

Multiple requests for single MIB object in #4 might consume more resources of snmp exporter. But they are mandatory for large network I think.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b04466a7-2d80-4aff-9476-5d5519c61d5d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[californi...@gmail.com]

Do you mean how to identify which value of ifType I need? I refer to https://www.iana.org/assignments/ianaiftype-mib/ianaiftype-mib in advance. Sorry if I misunderstand your question.

[Brian Brazil]

On 31 August 2017 at 13:51, <californi...@gmail.com> wrote:

Do you mean how to identify which value of ifType I need? I refer to https://www.iana.org/assignments/ianaiftype-mib/ianaiftype-mib in advance. Sorry if I misunderstand your question.

 

My question is if you have to look at ifType to figure out which indexes you care about, or if you could hardcode it.

--

Brian Brazil

www.robustperception.io

[californi...@gmail.com]

I have to look at ifType because indexes vary depends on the model or configuration.

[Ben Kochie]

What Brian is referring to is https://github.com/prometheus/snmp_exporter/issues/131 and the related issue https://github.com/prometheus/snmp_exporter/issues/176

Instead of doing it by ifType dynamically, some devices group virtual interfaces into ranges that could be ignored statically.

But it sounds like this isn't always the case.

Making a walk/get filtered by ifType sounds like a great feature, but I don't think we have time to implement it as it's a fairly edge case for most SNMP polling that I'm familiar with.

If you would be willing to sponsor the work, it's something users would appreciate.

I'm also curious what SNMP version you are using in this polling.  We have noticed some users using SNMP v1, which tends to be very inefficient due to the lack of Bulk walk requests.

On Thu, Aug 31, 2017 at 3:07 PM, <californi...@gmail.com> wrote:

I have to look at ifType because indexes vary depends on the model or configuration.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e7095ddd-7f00-4f82-a183-1ec9fea860d1%40googlegroups.com.

[californi...@gmail.com]

We are using version 2c. I also check our L2 switch again to figure out any specific range of indexes exists. If so, issue 131 will help us.

[Brian Brazil]

On 31 August 2017 at 14:14, Ben Kochie <sup...@gmail.com> wrote:

What Brian is referring to is https://github.com/prometheus/snmp_exporter/issues/131 and the related issue https://github.com/prometheus/snmp_exporter/issues/176

Instead of doing it by ifType dynamically, some devices group virtual interfaces into ranges that could be ignored statically.

But it sounds like this isn't always the case.

Making a walk/get filtered by ifType sounds like a great feature, but I don't think we have time to implement it as it's a fairly edge case for most SNMP polling that I'm familiar with.

There's also a question of how much complexity it'd add to the snmp exporter. Such a feature would require pretty fundamental changes to the exporter, for what is a fairly niche thing. Doing funky stuff to control what is walked based on user-provided logic from walk results is possibly out of scope for the snmp exporter. As a general rule, if you need to parse English (as distinct from things specified directly in the MIB) to do something with SNMP then that logic belongs in Prometheus rather than in the exporter.

Brian

 

If you would be willing to sponsor the work, it's something users would appreciate.

I'm also curious what SNMP version you are using in this polling.  We have noticed some users using SNMP v1, which tends to be very inefficient due to the lack of Bulk walk requests.

On Thu, Aug 31, 2017 at 3:07 PM, <californi...@gmail.com> wrote:

I have to look at ifType because indexes vary depends on the model or configuration.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e7095ddd-7f00-4f82-a183-1ec9fea860d1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrWmReZZi8UGeK1KQv0%2B9n%2BBRJE%2BsnZfBTqSScCPALXcA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[californi...@gmail.com]

I have found that our L2 switch has fixed ranges for each kind of interfaces. For example:

ifIndex            type of interface

0 to 999          physical ethernet port

1000 to 1999  link aggregation port

2000 to 9999  vlan interface

This means solution for issue #131 will help us to skip snmpwalk for vlan interfaces (I can instruct snmp_exporter to walk physical ethernet port and link aggregation port only. The feature that I posted above is not necessary.