Can not collect metric on Cisco Device by snmp-exporter

[Hieu Nguyen]

Dear Admin,

I have a problem when I collect a metric on Cisco devices.

An error has occurred during metrics gathering:
error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {}, variableLabels: []}: error walking target 10.10.10.101: Request timeout (after 3 retries)

Although I can check with snmpwalk. It's very fast.

Please help me fix a bug.

[Brian Candler]

"Timeout" simply means it failed.  If you communicate with a device but give the wrong community string, or you send from an IP address which is blocked by the device's ACL, you will get no response.

Note that your snmpbulkwalk uses a different community string than the default of "public".  I suspect the problem is that you have not set the community properly in snmp.yml.

Therefore:

1. Show us your snmp.yml

2. Use curl to scrape snmp_exporter directly, and show us the exact command you're using: e.g.

curl 'http://127.0.0.1:9116/snmp?target=10.10.10.101&module=FOO'

(we don't know what you're using for "FOO")

Once you have curl scraping directly working, then you can update your prometheus configuration to match.

[Daniel Swarbrick]

If I remember correctly, snmpwalk uses a lower max-repetitions than snmp_exporter by default. I experienced bogus "request timeouts" when walking certain large Juniper devices, where it would work flawlessly with snmpwalk (or snmpbulkwalk), but fail with snmp_exporter.

Try setting max_repetitions to 20 or less in your exporter config.

[Ben Kochie]

The default max_repetitions is 25, which matches the screenshot posted.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7aafd2e3-ffa2-4ed0-8fef-9bd8d633e4b1%40googlegroups.com.

[Brian Candler]

That's helpful, but the OP has already ruled this out: they provided -Cr25 to snmpbulkwalk (increasing the max_repetitions from 10 to 25), and 25 is also the default from snmp_exporter.

[Hieu Nguyen]

Hi Brian Candler.

1. Here is my snmp.yml config. 

cisco:

  walk:

  - 1.3.6.1.2.1.2.2.1.1

  metrics:

  - name: ifIndex

    oid: 1.3.6.1.2.1.2.2.1.1

    type: gauge

    help: A unique value, greater than zero, for each interface - 1.3.6.1.2.1.2.2.1.1

    indexes:

    - labelname: ifIndex

      type: gauge

  version: 2

  max_repetitions: 25

  retries: 3

  timeout: 10s

  auth:

    community: xxxpuclic

2. I tested with the comment: snmpwalk -v2c -c xxxpublic 10.10.10.101 1.3.6.1.2.1.2.2.1.1

Thanks for your support.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/172e70d7-06ae-4866-b65a-2862a3aa837b%40googlegroups.com.

[Hieu Nguyen]

I tried to change max_repetitions, request timeouts but it not work.

[Brian Candler]

In snmp.yml you are using community "xxxpuclic", but at the snmpwalk command line you're using "xxxpublic".  Both of these are different to what you posted before (but deleted).

Anyway, you can most likely debug it yourself using tcpdump: something like

tcpdump -i eth0 -nn -s0 -v udp port 161

(replace "eth0" with your interface name)

Run the scrape using curl; look at the packet it sends.  You should see it retry several times.

Then run the snmpbulkwalk that works.  Compare the first packet of that with the packets generated by the scrape.  Something will be different: e.g. the source IP address, the community string, the SNMP version, the OID.

[Hieu Nguyen]

I am sure I using 1 community string for snmp.yml and snmpwalk. "xxx" because I want to hidden community value. So sorry went  I was delivery information not cleanly.  

  When I run scrape using curl. I checked with tcpdump bit I only see... 

It's not work. :(

[Hieu Nguyen]

But when I checked with snmpwalk i saw packet send to Prometheus server.

[Brian Candler]

Once again, you are using a different community string in both those screenshots ("ldgpublic" versus "ldgpuclic")

Also, try to capture the *very first* packet that is sent by snmpbulkwalk, and compare it with the packets being sent by snmp_exporter.

[Hieu Nguyen]

Now, It is work.

Thanks for your support.