snmp_exporter. server returned HTTP status 500 Internal Server Error

[Hemedi Salim]

Hi all,

I am babie to this world(prometheus, snmp_exporter).  I recently configure Prometheus/snmp_exporter to monitors swithes. I want to monitor 5 switches  but Prometheus scrap metrics form 2 switches and the other three shows an error ( server returned HTTP status 500 Internal Server Error).

am using the default smnp.yml which I generate form snmp generator. and my Prometheus.yml setting look like this:

 job_name: snmp

metrics_path: /snmp params: module: [if_mib] static_configs: - targets: - 127.0.0.1 relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9116 # URL as shown on the UI

I tried to do snmp walk to troubleshout with command (snmpwalk -v2c -c public x.x.x.x) the x.x.x.x represent switch ip. but it gives me time out error.

then I figure out the switches which gives error has snmp community string set up. so I tried to run snmpwalk again replacing default public communit string with the community string set on this switches and I was able to snmpwalk on these swathes.

my question is, how can I make the Prometheus scrape from these remaining switches, is there any setting am missing to facilitate that?

please help.

[Brian Candler]

Aside: curl is useful for debugging:

curl -g `http://127.0.0.1:9116/snmp?module=if_mib&target=x.x.x.x&debug=true`

However in your case, you've found the problem: you need to set a community string.

Unfortunately, snmp_exporter makes it hard to set this, because the MIB and the device authentication are bundled together in the "module".  Here's the easiest way I've found to do this:

1. Edit snmp.yml (that was supplied with snmp_exporter, or that you generated)

2. Find the MIB that you're interested in using: in this case if_mib.  Modify the first line like this:

if_mib:

becomes

if_mib: &if_mib

(this is called an "anchor" in YAML)

3. Add to the very *end* of the file:

if_mib_secret:
  <<: *if_mib
  version: 2
  auth:
    community: BlahBlah123

What you're doing is taking a copy of the "if_mib" section, and then overriding some of the parameters (the 'auth' and 'version' keys) without changing anything else.

[Aside: you should be able to do steps 1-3 using the snmp generator instead, but you'll have to duplicate the if_mib section in generator.yml instead, and then run the generator]

4. Restart snmp_exporter, and check this works using curl:

curl -g `http://127.0.0.1:9116/snmp?module=if_mib_secret&target=x.x.x.x&debug=true`

5. Change your scrape job for these devices to use module if_mib_secret instead of if_mib.

Oddly, your scrape job shows a static target of only 127.0.0.1, i.e. it looks like you're sending queries to the snmpd running on prometheus itself, not to any remote devices.  I would suggest that you use the file_sd mechanism, and also that you pick up the module from this, so that a single scrape job can handle any combination of targets and SNMP modules.

  - job_name: snmp
    metrics_path: /snmp
    file_sd_configs:
      - files:
          - /etc/prometheus/targets.d/snmp_targets.yml
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

      - source_labels: [__address__]
        target_label: __param_target

      - source_labels: [module]
        target_label: __param_module
      - target_label: __address__
        replacement: 127.0.0.1:9116  # SNMP exporter

then in snmp_targets.yml you can put:

- labels:
    module: if_mib
  targets:
    - foo
    - bar

- labels:
    module: if_mib_secret
  targets:
    - baz
    - qux

[Hemedi Salim]

Hi 

Thanks for the reply.

I follow the steps as you showed me, but problem still exist. I run the command to debug (curl -g `http://127.0.0.1:9116/snmp?module=if_mib&target=x.x.x.x&debug=true`) and it give me request timeout error:

error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {}, variableLabels: []}: error getting target x.x.x.x: request timeout (after 3 retries)

and the on the prometheus target shows:

just few questions:

do those changes have to be done in snmp generator ? and then generate a snmp.yml? or it can be added straight away in snmp.yml?

What should I need to add, if for example the switch snmp version is V3?

Is it possible to have more than one string for snmp authentication in snmp.yml?

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/aa9f124e-e927-4ad6-91c5-e261d3c52312n%40googlegroups.com.

[Hemedi Salim]

also, if I edit generator.yml  please which part exactly on generator.yml need to be duplicated?  

generator.yml

modules:

  # Default IF-MIB interfaces table with ifIndex.

  if_mib:

    walk: [sysUpTime, interfaces, ifXTable]

    version: 2  # SNMP version to use. Defaults to 2.

    auth:

      # Community string is used with SNMP v1 and v2. Defaults to "public".

      community: aspire-lan

    lookups:

      - source_indexes: [ifIndex]

        lookup: ifAlias

      - source_indexes: [ifIndex]

        # Uis OID to avoid conflict with PaloAlto PAN-COMMON-MIB.

        lookup: 1.3.6.1.2.1.2.2.1.2 # ifDescr

      - source_indexes: [ifIndex]

        # Use OID to avoid conflict with Netscaler NS-ROOT-MIB.

        lookup: 1.3.6.1.2.1.31.1.1.1.1 # ifName

    overrides:

      ifAlias:

        ignore: true # Lookup metric

      ifDescr:

        ignore: true # Lookup metric

      ifName:

        ignore: true # Lookup metric

      ifType:

        type: EnumAsInfo

# Cisco Wireless LAN Controller

  cisco_wlc:

    walk:

      - interfaces

      - ifXTable

      - 1.3.6.1.4.1.14179.2.1.1.1.38   # bsnDot11EssNumberofMobileStations

      - 1.3.6.1.4.1.14179.2.2.2.1.2    # bsnAPIfType

      - 1.3.6.1.4.1.14179.2.2.2.1.4    # bsnAPIfPhyChannelNumber

      - 1.3.6.1.4.1.14179.2.2.2.1.15   # bsnApIfNoOfUsers

      - 1.3.6.1.4.1.14179.2.2.6.1      # bsnAPIfDot11CountersTable

      - 1.3.6.1.4.1.14179.2.2.13.1.3   # bsnAPIfLoadChannelUtilization

      - 1.3.6.1.4.1.14179.2.2.15.1.21  # bsnAPIfDBNoisePower

    lookups:

      - source_indexes: [bsnDot11EssIndex]

        lookup: bsnDot11EssSsid

        drop_source_indexes: true

      - source_indexes: [bsnAPDot3MacAddress]

        lookup: bsnAPName

        drop_source_indexes: true

    overrides:

      ifType:

        type: EnumAsInfo

# APC/Schneider UPS Network Management Cards

....................

.................

..............

[Brian Candler]

On Tuesday, 1 March 2022 at 15:58:48 UTC hemj...@gmail.com wrote:

just few questions:

do those changes have to be done in snmp generator ? and then generate a snmp.yml? or it can be added straight away in snmp.yml?

I apply them straight away to snmp.yml

 

What should I need to add, if for example the switch snmp version is V3?

Example:

if_mib3:
  <<: *if_mib
  version: 3
  auth:
    security_level: authPriv
    username: bla
    auth_protocol: SHA
    password: blabla
    priv_protocol: AES
    priv_password: blablabla

 

Is it possible to have more than one string for snmp authentication in snmp.yml?

Yes, but you need to instantiate a separate module for each one:

if_mib_foo:
  <<: *if_mib
  version: 2
  auth:
    community: tcpip123

if_mib_bar:
  <<: *if_mib
  version: 2
  auth:
    community: xyzzy

...etc

[Brian Candler]

On Tuesday, 1 March 2022 at 16:14:16 UTC hemj...@gmail.com wrote:

also, if I edit generator.yml  please which part exactly on generator.yml need to be duplicated?  

Every module that you want to apply a different community string to.  e.g.

There you've put already changed the community to "aspire-lan".  If you want to run with another community as well, then either duplicate the *whole* section as if_mib_foo, with everything identical except the auth section; or use the YAML trick I showed before, like this.

modules:

  # Default IF-MIB interfaces table with ifIndex.

  if_mib: &if_mib

    walk: [sysUpTime, interfaces, ifXTable]

    version: 2  # SNMP version to use. Defaults to 2.

    auth:

      # Community string is used with SNMP v1 and v2. Defaults to "public".

      community: aspire-lan

    lookups:

      - source_indexes: [ifIndex]

        lookup: ifAlias

      - source_indexes: [ifIndex]

        # Uis OID to avoid conflict with PaloAlto PAN-COMMON-MIB.

        lookup: 1.3.6.1.2.1.2.2.1.2 # ifDescr

      - source_indexes: [ifIndex]

        # Use OID to avoid conflict with Netscaler NS-ROOT-MIB.

        lookup: 1.3.6.1.2.1.31.1.1.1.1 # ifName

    overrides:

      ifAlias:

        ignore: true # Lookup metric

      ifDescr:

        ignore: true # Lookup metric

      ifName:

        ignore: true # Lookup metric

      ifType:

        type: EnumAsInfo

...

if_mib_foo:

  <<: *if_mib
  version: 2
  auth:
    community: BlahBlah123

You'll have to repeat this as many times as you have different community strings.  And if you want multiple auths for for cisco_wlc, then you'll need to do the same for that as well.

Re-run the generator, of course, with this new input.

[Hemedi Salim]

Thanks very much Brian. got it working 👍

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4a795a1c-6172-4532-99ad-458f62d88458n%40googlegroups.com.