snmp_exporter fails with 'Error scraping target'

[John Fox]

Hello all!

I've been pulling my hair out trying to get snmp_exporter to crawl my Unifi devices. The error that it comes back with is:

An error has occurred while serving metrics: error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {}, variableLabels: []}: error getting target 192.168.8.1: request timeout (after 3 retries)

Yet I can snmpwalk without issue - for example:  snmpwalk -v 2c -c unifi -On 192.168.8.17 1.3.6.1.2.1.1.3.0 - returning:

.1.3.6.1.2.1.1.3.0 = Timeticks: (9931546) 1 day, 3:35:15.46

I'm not really sure what to make of this issue or how to solve it. When I see a timeout message, I thought to increase the scrape and timeout parameters to 5m, however that just results in an Error 500 message in Prometheus after about 2 minutes.

Wondering what I'm doing wrong here.

[Ben Kochie]

Your target IPs don't match.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9d0dc146-cd37-4a7c-8051-e92ba3d47530n%40googlegroups.com.

[Brian Candler]

And if that's not it, it's probably that the community string is wrong.  The supplied snmp.yml uses "public" as the community string; if you need something else, you need to modify snmp.yml or generate a completely new one from generator.yml

[John Fox]

This was my error insofar as pasting the wrong thing above. I can guarantee that the walk is correct for the given IP. See below:

pi@network:~ $ snmpwalk -v 2c -c unifi -On 192.168.8.1 1.3.6.1.2.1.1.3.0
.1.3.6.1.2.1.1.3.0 = Timeticks: (31280439) 3 days, 14:53:24.39

[John Fox]

I think you might be on to something, Brian.  I looked in my snmp.yml file and found no reference at all to the community string. I tried adding 

community: 

  - unifi 

to the top of the file immediately following ubiquiti_unifi: but when restarting the snmp service, it fails stating msg="Error parsing config file" err="yaml: unmarshal errors:\n  line 2: field community not found in type config.plain".

[Brian Candler]

Here's the approach I recommend:

1. modify the header line of the ubiquiti_unifi section so it looks like this:

ubiquiti_unifi: &ubiquiti_unifi

(i.e. add "&ubiquiti_unifi" to the end of that line)

2. at the very end of the file, put:

ubiquiti_unifi_1:
  <<: *ubiquiti_unifi
  version: 2
  auth:
    community: unifi

3. Restart snmp_exporter, and then invoke it with with module=ubiquiti_unifi_1 instead of ubiquiti_unifi.

(This uses a YAML trick to make a new entry which copies an existing entry, and then overrides some settings. It's especially useful if there are multiple community strings in use)

[John Fox]

My man! That did the trick. Made the edits you suggest and a quick restart of both snmp_exporter and prometheus later, the targets are polling! Now to add them to my Grafana dashboards.

Thank you so much, Brian!!

[karthik b.v.n.s]

Hi John,

Can you please help me understand this, even I am facing the same error. I have not using generator.yml instead using default snmp.yml file and modified the community string accordingly but still the same issue.

can you please guide me and tell me the changes you did that will be much helpful

[Brian Candler]

The problem is with your setup, not John's, so I think it's better if you provide the following:

1. Demonstrate that SNMP is working on the target device, by showing an snmpwalk command line which succeeds

2. Show what changes you've made to snmp.yml

3. Show how you're testing snmp_exporter (e.g. the curl command line you're using)

4. Show what error message you're getting back

There are lots of reasons why SNMP won't respond:

- target device is offline, traffic blocked by a firewall/ACL, or not running an SNMP agent

- client is using wrong SNMP version and/or wrong SNMP credentials

- target device has an SNMP ACL that does not include the client's IP address