SNMP exporter for Prometheus does not collect data

[maks...@balticom.eu]

Hello, everyone!

Can you help me in finding where is the issue? My SNMP exporter cannot collect SNMP data.

This shows after running http://172.27.15.206:9116/snmp?target=172.27.15.206 :

An error has occurred during metrics gathering:

error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {}, variableLabels: []}: Error walking target 172.27.15.206: Request timeout (after 3 retries)

And in console:

ERRO[0694] Error scraping target 172.27.15.206: Error walking target 172.27.15.206: Request timeout (after 3 retries)  source="collector.go:126"
ERRO[0704] Error scraping target 172.27.15.206: Error walking target 172.27.15.206: Request timeout (after 3 retries)  source="collector.go:126"
ERRO[0714] Error scraping target 172.27.15.206: Error walking target 172.27.15.206: Request timeout (after 3 retries)  source="collector.go:126"
ERRO[0724] Error scraping target 172.27.15.206: Error walking target 172.27.15.206: Request timeout (after 3 retries)  source="collector.go:126"
ERRO[0734] Error scraping target 172.27.15.206: Error walking target 172.27.15.206: Request timeout (after 3 retries)  source="collector.go:126"
ERRO[0744] Error scraping target 172.27.15.206: Error walking target 172.27.15.206: Request timeout (after 3 retries)  source="collector.go:126"

prometheus.yml:

global:

  scrape_interval: 10s

  evaluation_interval: 10s

scrape_configs:

  - job_name: 'snmp'

    metrics_path: /snmp

    params:

      module: [default]

    static_configs:

      - targets:

        - 172.27.15.206  # SNMP device - add your IPs here

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [__param_target]

        target_label: instance

      - target_label: __address__

        replacement: 127.0.0.1:9116  # SNMP exporter.

Head of snmp.yml:

default:

  version: 2

  auth:

    community: bcomsnmpadmin

apcups:

  walk:

  - 1.3.6.1.2.1.1.3

  - 1.3.6.1.2.1.2

  - 1.3.6.1.4.1.318.1.1.1.12

  - 1.3.6.1.4.1.318.1.1.1.2

[Ben Kochie]

The setting in snmp.yml are a series of individual modules.

The "default" module does not imply that settings to other modules are inherited.

You probably want something like this:

In snmp.yml:

apcups:

  version: 2
  auth:
    community: bcomsnmpadmin

  walk:
  - 1.3.6.1.2.1.1.3
  - 1.3.6.1.2.1.2
  - 1.3.6.1.4.1.318.1.1.1.12
  - 1.3.6.1.4.1.318.1.1.1.2

Then:

curl "http://172.27.15.206:9116/snmp?target=172.27.15.206&module=apcups"

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2692b2fe-6be6-4552-8a14-c7d7c0d44a73%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maksims Edelmans]

Thank you very much, Ben! That helped me.

Ar cieņu,
Maksims Edeļmans
Telekomunikāciju inženieris
A/S "BALTICOM"  Internet Service Provider
Adrese: Straupes 5k3, Riga LV-1073
E-pasts: ad...@balticom.lv

[Ben Kochie]

One thing we recommend is to put your community auth strings into a local copy of the generator.yml, and use that to generate your snmp.yml.  This way you can adjust the configuration to meet your needs, and make sanitized snmp.yml configs.

[Ben Kochie]

I'm guessing you don't want delta, but rate.

That would be:

rate(IfInOctets[5m])

But of course that's going to be bytes, so you will need to multiply by 8 to get bps.

rate(IfInOctets[5m]) * 8

Of course you can do other math as well.

(rate(IfInOctets[5m]) + rate(IfOutOctets[5m])) * 8

You leave this value in the base unit (bytes/sec, bits/sec), and then you configure Grafana to automatically display the correct order of magnitude. (Mbps) This is the Unit option under the Axes tab in the graph editor.

On Wed, Jul 5, 2017 at 7:06 PM, Maksims Edelmans <maks...@balticom.eu> wrote:

Hello, Ben!

Can you suggest me how can I get delta from two different values? For example I have two values of IfInOctets and I want to put them on a graph in Grafana but not as-values, but as a delta (Mbps)? Thank you!

[Ben Kochie]

On Wed, Jul 5, 2017 at 8:36 PM, Maksims Edelmans <maks...@balticom.eu> wrote:

Thanks for Your advice!

But my graph has disappeared after 18:15. Before 18:15 there is a chart and after 18:15 it is empty. If I query simply IfIndex then i get all indexes up to current time, so there is no problem with SNMP queries. Could you tell me what might be wrong or where should I look for a solution?

I don't have any good idea about this, there are a number of possibilities.  I would examine your logs.

You may also want to check the system clock on your Prometheus and exporter servers compared to the clock on your browser.  Prometheus uses standard unix epoch time to perform queries.  We highly recommend using NTP to sync all clocks.  Sorry if that sounds basic, but even I mess it up sometimes. :-)

 

By the way,

We have thousands of switches that I want to query for interface statistics, CPU, memory and maybe temperature. I think it is going to be up to 100k of time series.

100k is well within a normal Prometheus server limit.  A modern dual-socket xeon can handle a couple million series.

 

I image it to work this way:

1. I get info about a new switch that had been installed. For example, querying external DB. At this point I know IP and model.

2. Then I put IP address in an appropriate device_list.yml file that is bind to a job of prometheus.yml. I can to it manually now and it is working.

The recommended way would be to write a script that reads from your inventory database and writes out a series of device yaml files like you say, and include them using `file_sd_configs` per required snmp module.

 

 

3. Grafana/Prometheus sees a new device and automatically links it to a graph template.

Does this sound real? Any suggestions on how to accomplish this?

There's not much of a need to link prometheus to grafana, you just need to tell grafana about the Prometheus datasource.

 

What if I generate hundreds or even thousands of dashboards in Grafana? E.g. dashboard per switch where are interface statistics graphs. Up to 30 graphs per dashboard.

There's no need to generate many dashboards.  Grafana supports templating, which will allow you to have one dashboard to represent many devices.  You tell Grafana to query and automatically populate a variable based on the instance label, and then you will get a drop down menu that lists all SNMP devices.  You can also have Grafana perform a sub query that based on the device will let you select individual ports.  One dashboard can cover all switches and routers.

 

Thank you!

Ar cieņu,
Maksims Edeļmans
Telekomunikāciju inženieris
A/S "BALTICOM"  Internet Service Provider
Adrese: Straupes 5k3, Riga LV-1073
E-pasts: ad...@balticom.lv

[Maksims Edelmans]

Hello, Ben!

Your advice helped me a lot. Now I am struggling with sorting out targets/instances. 

I have around 2000 switches' IP addresses. They are targets/instances. It looks like this:

There are 4 models allover - GS220024, GS22008, GS221024 and GS22108. Model = job = module.

Now I have created a template that lets me to choose a model:

The problem is that instance/IP drop-down list doesn't depend on model and I cannot get this dependency.

I want to choose a model, then IP address from the list according to the model and then, port, for example. How can I do this? By the way, it is Grafana + Prometheus + SNMP exporter.

Thank you!

Ar cieņu,
Maksims Edeļmans
Telekomunikāciju inženieris
A/S "BALTICOM"  Internet Service Provider
Adrese: Straupes 5k3, Riga LV-1073
E-pasts: ad...@balticom.lv

[Brian Brazil]

On 10 July 2017 at 17:36, Maksims Edelmans <maks...@balticom.eu> wrote:

Hello, Ben!

Your advice helped me a lot. Now I am struggling with sorting out targets/instances. 

I have around 2000 switches' IP addresses. They are targets/instances. It looks like this:

There are 4 models allover - GS220024, GS22008, GS221024 and GS22108. Model = job = module.

Now I have created a template that lets me to choose a model:

The problem is that instance/IP drop-down list doesn't depend on model and I cannot get this dependency.

I want to choose a model, then IP address from the list according to the model and then, port, for example. How can I do this?

You need to use query_result and a regex to extract that out, look at the $Interface variable in https://grafana.com/dashboards/1124 for an example.

Brian

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CANEE6sjLsSSU0rQyaSsmn5Xwa_7YO5KN4ygBfWOiejv3uxKf_A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Maksims Edelmans]

Hello, Brian!

I cannot find any example at https://grafana.com/dashboards/1124. Could please send direct link? Thank you.

Ar cieņu,
Maksims Edeļmans
Telekomunikāciju inženieris
A/S "BALTICOM"  Internet Service Provider
Adrese: Straupes 5k3, Riga LV-1073
E-pasts: ad...@balticom.lv