prometheus-snmp-generator error (Debian 10)

[Mario Polario]

Hi,

I am trying to set-up prometheus-snmp-exporter to collect data from my Synology NAS.

I downloaded mibs (Synology mibs included) to /usr/share/snmp/mibs.

When running prometheus-snmp-generator I get an error:

root@monitoring:/etc/prometheus# prometheus-snmp-generator generate

INFO[0000] Loading MIBs from $HOME/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/ietf:/usr/share/mibs/netsnmp  source="net_snmp.go:136"

WARN[0000] NetSNMP reported 2 parse errors               source="main.go:103"

FATA[0000] Error parsing yml config: Invalid metric type override 'EnumAsInfo'  source="main.go:44"

I tried removing all from generator.yml except Synology module, but no change.

I was following instructions noted in /usr/share/doc/prometheus-snmp-exporter/README.Debian but I can't seem to generate snmp.yml file.

Any help in understanding what I did wrong is more than welcomed.

Thanks in advance!

Best regards,

Mario.

[Ben Kochie]

What version is this, it's likely out of date. I would recommend building from source.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b8f2f195-efc1-4c48-b85b-88a8687964bfn%40googlegroups.com.

[Mario Pranjic]

Hi Ben,

I use versions available in Debian repository:
ii  prometheus                              2.7.1+ds-3+b11                amd64        Monitoring system and time series database
ii  prometheus-alertmanager                 0.15.3+ds-3+b1                amd64        Handle and deliver alerts created by Prometheus
ii  prometheus-node-exporter                0.17.0+ds-3+b11               amd64        Prometheus exporter for machine metrics
ii  prometheus-snmp-exporter                0.14.0+ds-1+b10               amd64        Prometheus exporter for SNMP-enabled devices

I try building as little as possible since all provisioning is handled by Ansible at the end.

If there is prometheus external repository for Debian, that could be worth checking, but I didn't find one.

[Ben Kochie]

I highly recommend against using the Debian packages, as they are extremely out of date.

If you're using Ansible, you can use these high quality roles.

https://github.com/cloudalchemy

For example, https://github.com/cloudalchemy/ansible-prometheus

We are planning an official Deb repo, but it's not ready yet.

For the generator, it's going to be one thing you do need to build, because it needs to be compiled with a distro specific net-snmp dev package.

[Mario Pranjic]

Hm.

I wrote the roles myself for Prometheus and Grafana but I used debian packages for Prometheus and external repository for Grafana.

I am fully aware Debian often has older versions of packages (sometimes pretty old). To be able to use prometheus repository (as Grafana provides one), would make our lives a lot easier. :)  

I will look into these and see if things are better with newer versions.

Thanks for the tip!

Best regards,

Mario.

[Mario Pranjic]

Hi Ben,

I ran another VM and used roles:
-  https://github.com/cloudalchemy/ansible-snmp-exporter  
-  https://github.com/cloudalchemy/ansible-prometheus  

along with Grafana. Prometheus, node.exporter and Grafana work as intended (as before).

However, when I try to enable snmp_exporter:
  - job_name: snmp
    metrics_path: /snmp
    params:
      module:
      - if_mib
    relabel_configs:
    - source_labels:
      - __address__
      target_label: __param_target
    - source_labels:
      - __param_target
      target_label: instance
    - replacement: 127.0.0.1:9116
      target_label: __address__
    static_configs:
    - targets:
      - 192.168.10.187

I get error in syslog:

Aug 21 14:43:23 monnew snmp_exporter[363]: level=info ts=2020-08-21T12:43:23.189Z caller=collector.go:224 module=if_mib target=192.168.10.187 msg="Error scraping target" err="scrape canceled (possible timeout) getting target 192.168.10.187"

snmp is enabled on device on 192.168.10.187. I tried snmp version 2c and 3 but the error message is the same. I can connect on that IP:port from prometheus/snmp_exporter node:
root@monnew:~# telnet 192.168.10.187 161
Trying 192.168.10.187...
Connected to 192.168.10.187.
Escape character is '^]'.


Port 9116 is opened on localhost on prometheus/snmp_exporter node:

root@monnew:/etc/snmp_exporter# netstat -na|grep 9116
tcp        0      0 127.0.0.1:35274         127.0.0.1:9116          ESTABLISHED
tcp6       0      0 :::9116                 :::*                    LISTEN
tcp6       0      0 127.0.0.1:9116          127.0.0.1:35274         ESTABLISHED
root@monnew:/etc/snmp_exporter# telnet 127.0.0.1 9116
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

What I am missing here is that generator thing. I didn't install any  mibs in this case, and, from what I see in Ansible role, no mibs or generator has been set/run.

Instead, snmp.yml file is being pulled from remote location:
"https://raw.githubusercontent.com/prometheus/snmp_exporter/v{{ snmp_exporter_version }}/snmp.yml"

Could you shed some light here, because I am obviously missing some step along the way.

Thanks in advance!

Best regards,

Mario.

[Ben Kochie]

The SNMP exporter uses UDP for transport, so telnet to 161 isn't going to help much. I've seen very few devices implement RFC 3430.

There are usually two problems people run into.

* The community string is wrong (defaults to "public")

* The device has a management IP control and won't allow traffic.

SNMP implementations generally don't reject incorrect authentication, rather they simply drop the packets. So it can be hard to diagnose.

[Mario Pranjic]

Hi Ben,

Community string was wrong. I fixed that. Now I can tell snmp_exporter can communicate properly.

I can get in Prometheus: sysUpTime{instance="192.168.10.187",job="snmp"}

And value is read from the device (incrementing on refresh too).

But as for mibs, I still can't use mibs as described in the Synology snmp guide.

https://global.download.synology.com/download/Document/Software/DeveloperGuide/Firmware/DSM/All/enu/Synology_DiskStation_MIB_Guide.pdf

There is still a step missing in the snmp_exporter role in regards to mibs and/or proper snmp.yml generation.

Best regards,

Mario.

[Ben Kochie]

You will probably need to have a good understanding of SNMP and MIBs in order create new generator configs. Unfortunately, the Synology MIBs have a large number of mistakes, making it even harder to work with.

I created a simple example of how to use the generator based on what I found in the SYNOLOGY-RAID-MIB.

https://github.com/SuperQ/tools/tree/master/snmp_exporter/Synology

I corrected as many of the errors as I could, but Synology didn't format the raidStatus as an ENUM.

[Mario Pranjic]

Hi Ben,

Thank you very much for your time and effort.

Indeed, I am not an expert in snmp (that was always somehow left on TODO list). Maybe this is a good opportunity to fill in the gaps in knowledge. :)

The thing is... With snmp.yml which comes as a result of that snmp_exporter role, I do get some communication with Synology from Prometheus, but the list is pretty awkward.

By pointing browser to: http://127.0.0.1:9116/snmp?target=192.168.10.187
 i get output as noted in list.txt file.

I see no info about disks and raid.

However, if I run snmpget from Prometheus/snmp_exporter node against Synology, I can read values documented by Synology.

For example, disk temperature (disk0 and disk1):

root@monnew:/# snmpget -c public -v 2c -O qv 192.168.10.187 1.3.6.1.4.1.6574.2.1.1.6.0
29
root@monnew:/# snmpget -c public -v 2c -O qv 192.168.10.187 1.3.6.1.4.1.6574.2.1.1.6.1
31

Or some  info:

root@monnew:/# snmpget -c public -v 2c -O qv 192.168.10.187 1.3.6.1.4.1.6574.1.5.1.0
"DS216j"

I tried to use snmp.yml you provided in example or building one based on your example generator.yml, but in both cases  http://127.0.0.1:9116/snmp?target=192.168.10.187 will return error:
Unkown_module 'if_mib'

The question why snmp_exporter is returning only a small portion of values data from Synology is, I presume, related to the snmp.yml file.

How to generate the right one is an enigma. :/

Attaching snmp.yml provisioned by snmp_exporter role for the reference.

Best regards,

Mario.

[Mario Pranjic]

Hi,

A small update: indeed there are errors in provided MIB files by Synology. I ran some of them into  http://www.muonics.com/Tools/smicheck.php and reported some examples to Synology.

A case is opened on the Synology side and they will send it to their dev team. We will see if/when they come up with something useful.

Best regards,

Mario.

[Ben Kochie]

If we're going to talk to Synology, maybe we can get them to support Prometheus directly. I'd sell my QNAP if they did that. ;-)

For example, here's a patch to the MIB file that helps with the types:

https://gitlab.com/superq/mibs/-/commit/40ddc8af074961abfa89d2d94da9fd290e8e3f71

[Mario Pranjic]

Hi Ben,

I will gladly help. Just let me know how to approach the matter.

Initially I opened a request on Synology asking if they support Prometheus node exporter, since it is debian-based Linux on DSM.

I could attempt to install it on my own, but then Synology support is off and they don't guarantee anything (understandable, but not optimal).

So they mentioned snmp instead. And there we are, now. :)

[Mario Pranjic]

Hi,

To inform about Synology support case opened and MIB status.

They are now aware of some issues in MIB definitions. I did mention a patch to fix one of those MIB files, but, frankly, I didn't get the impression that will be a drive for them to get more involved into the matter.

From their response: 
"I've already forwarded the issue to our developer and they acknowledge that there're some problem related to the Couter64 in Synology RAID MIB file.

Most of the 64bit unit is only supported SNMPv2 but not v1 which causing this issue.
They had already made a record on this issue and review Synology's MIBs and fix the MIB files if necessary.
However, they confirm currently, the problem won't affect the MIB files to work normally through SNMP."

And, of course, the fact that this MIB file can cause a snmp generator to produce an unusable yml file for prometheus snmp has been ignored.

Best regards,

Mario.