Re: [prometheus-users] snmp_exporter can't scrape

[Ben Kochie]

If you `curl` the snmp_exporter, how long does it take to respond? I'm guessing you're scrape_timeout in Prometheus is the default of 10 seconds.

On Thu, Sep 3, 2020 at 2:20 AM Andrew Meyer <andre...@gmail.com> wrote:

Hello,

I have been working on getting some devices working with my snmp_exporter today.  I was successful in making this work but I am no longer able to scrape any snmp device.  I am running CentOS 8.

snmp_exporter-0.19.0-1.el8.x86_64

prometheus2-2.20.1-2.el8.x86_64

Sep  2 16:51:09 mon001 snmp_exporter[1238]: level=info ts=2020-09-02T21:51:09.014Z caller=collector.go:224 module=synology target=10.150.1.18 msg="Error scraping target" err="scrape canceled (possible timeout) getting target 10.150.1.18"
Sep  2 16:51:09 mon001 snmp_exporter[1238]: level=info ts=2020-09-02T21:51:09.563Z caller=collector.go:224 module=nortel target=10.150.1.2 msg="Error scraping target" err="scrape canceled (possible timeout) getting target 10.150.1.2"
Sep  2 16:51:13 mon001 snmp_exporter[1238]: level=info ts=2020-09-02T21:51:13.333Z caller=collector.go:224 module=synology target=10.150.1.2 msg="Error scraping target" err="scrape canceled (possible timeout) getting target 10.150.1.2"
Sep  2 16:51:13 mon001 snmp_exporter[1238]: level=info ts=2020-09-02T21:51:13.678Z caller=collector.go:224 module=fortigate_snmp target=10.150.1.1 msg="Error scraping target" err="scrape canceled (possible timeout) walking target 10.150.1.1"
Sep  2 16:51:21 mon001 snmp_exporter[1238]: level=info ts=2020-09-02T21:51:21.713Z caller=collector.go:224 module=fortinet target=10.150.1.1 msg="Error scraping target" err="scrape canceled (possible timeout) walking target 10.150.1.1"
Sep  2 16:51:23 mon001 snmp_exporter[1238]: level=info ts=2020-09-02T21:51:23.953Z caller=collector.go:224 module=synology target=10.150.1.19 msg="Error scraping target" err="scrape canceled (possible timeout) getting target 10.150.1.19"
Sep  2 16:51:24 mon001 snmp_exporter[1238]: level=info ts=2020-09-02T21:51:24.014Z caller=collector.go:224 module=synology target=10.150.1.18 msg="Error scraping target" err="scrape canceled (possible timeout) getting target 10.150.1.18"

I saw an issue on github where someone had a similar issue it was the default timeout that was not set.  Could this be what my issue is? 

Also I can go to the snmp_exporter web page and scrape stuff from there.  I can snmpwalk to the devices just fine.

Thanks! 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d32debe9-faf5-48fb-a1bc-2d82ee43edd1n%40googlegroups.com.

[Andrew Meyer]

I figured it out.

[Mario Pranjic]

Hi

Was it related to scrape_timeout?

Best regards,

Mario.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAEAR6cNHytA51UicPXmcA3ybHjOKTdjP%3DJ3OSiA8sOOm6x_35Q%40mail.gmail.com.

[Andrew Meyer]

I THINK that might have been part of the issue.  This my home environment so it was left in a not so wonderful state.

I also found that I had issues with the modules specifically for fortinet/fortigate devices.  I am still having trouble with my nortel switch.  But that is on the agenda to fix and reintroduce the specific snmp modules.

Please remember to reply to all if there are multiple recipients.

Andrew Meyer
andre...@gmail.com

314-277-3307 (Cell)

314-266-4837 (Google Voice)

[Andrew Meyer]

Ok,

So I spent the last hour re-enabling things and testing.  Specifically still having issues with fortinet devices.

If I comment out the params and modules for the Fortinet devices and let snmp scrape it, it works fine.  If I specify the module I want to use, it won't snmpwalk to the device.

I am doing this twice because I wasn't sure what the difference was between the two.  So if someone can help clarify that would be great.  

Here is the fortinet portions of my config:

  - job_name: 'snmp_fortinet'

    static_configs:

      - targets: [ 10.150.1.1 ]

    scrape_interval: 15s

    metrics_path: /snmp

    params:

      module: [fortinet]

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [__param_target]

        target_label: instance

      - target_label: __address__

        replacement: 10.150.10.26:9116  # SNMP exporter.

  - job_name: 'snmp_fortigate'

    static_configs:

      - targets: [ 10.150.1.1 ]

    scrape_interval: 15s

    metrics_path: /snmp

    params:

      module: [fortigate_snmp]

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [__param_target]

        target_label: instance

      - target_label: __address__

        replacement: 10.150.10.26:9116  # SNMP exporter

Also, the device shows DOWN in the prometheus targets page but when I click on the link for the module it will query just fine.  Which makes no sense.

[Andrew Meyer]

So this is what is showing up in the targets page

Get "http://10.150.10.26:9116/snmp?module=fortigate_snmp&target=10.150.1.1": context deadline exceeded

Please remember to reply to all if there are multiple recipients.

Andrew Meyer

[Ben Kochie]

I would run the exporter with `--log-leve=debug` to see what individual walks are timing out.

[Andrew Meyer]

So once again I commented out all of the snmp devices I have setup.  Restarted prom and snmp_exporter and then started to re-enable the snmp devices starting with the fortinet scrapers.  It worked.  I haven't re-enabled the rest.  But I would be curious to see if it breaks again.  Do you still think I should go down the log-level=debug route?  This seems like it is something else.

Thank you!

[Andrew Meyer]

Trying the --log.level=debug.  Will start adding devices back into the mix and report back.