Individual OIDs scraping but not together.

[sabarish narayanan]

Hi, I'm trying to monitor a Fortigate firewall using snmp_exporter.

In my snmp_exporter generator.yml file when I use one oid, the exporter scrapes properly but when I use 2 individually working OIDs together it gives the following error.

snmp_exporter.service - SNMP Exporter
   Loaded: loaded (/etc/systemd/system/snmp_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-09-29 12:58:41 IST; 1min 13s ago
 Main PID: 482901 (snmp_exporter)
    Tasks: 8 (limit: 23823)
   Memory: 13.2M
   CGroup: /system.slice/snmp_exporter.service
           ââ482901 /usr/local/bin/snmp_exporter/snmp_exporter --config.file /etc/prometheus/snmp.yml

Sep 29 12:59:32 ipmadmin snmp_exporter[482901]: level=info ts=2020-09-29T07:29:32.659Z caller=collector.go:224 module=fortigate_snmp target=xx.xx.xx.xx msg="Error scraping target" err="scrape canceled (possible timeout) walking target xx.xx.xx.xx"
Sep 29 12:59:34 ipmadmin snmp_exporter[482901]: level=info ts=2020-09-29T07:29:34.619Z caller=collector.go:224 module=fortigate_snmp target=xx.xx.xx.xx msg="Error scraping target" err="scrape canceled (possible timeout) walking target xx.xx.xx.xx"
Sep 29 12:59:36 ipmadmin snmp_exporter[482901]: level=info ts=2020-09-29T07:29:36.624Z caller=collector.go:224 module=fortigate_snmp target=xx.xx.xx.xx msg="Error scraping target" err="scrape canceled (possible timeout) walking target xx.xx.xx.xx"
Sep 29 12:59:39 ipmadmin snmp_exporter[482901]: level=info ts=2020-09-29T07:29:39.783Z caller=collector.go:224 module=fortigate_snmp target=xx.xx.xx.xx msg="Error scraping target" err="scrape canceled (possible timeout) walking target xx.xx.xx.xx"

Here's my generator.yml :-

modules:
  XPPC-MIB:
    walk:
      - upsThreePhaseBatteryTimeRemain
      - upsThreePhaseBatteryTemperature
      - upsThreePhaseOutputFrequency
      - upsThreePhaseOutputVoltageR
      - upsThreePhaseOutputVoltageS
      - upsThreePhaseOutputVoltageT
      - upsThreePhaseOutputLoadPercentageR
      - upsThreePhaseOutputLoadPercentageS
      - upsThreePhaseOutputLoadPercentageT
      - upsConfigOutputVA
    version: 3
    max_repetitions: 25
    retries: 3
    timeout: 10s
    auth:
      username: *********
      security_level: authNoPriv
      password: *********
      auth_protocol: MD5
    lookups:
      - source_indexes: [upsThreePhaseBatteryTimeRemain]
        lookup: TimeRemain
      - source_indexes: [upsThreePhaseBatteryTemperature]
        lookup: BatteryTemperature
      - source_indexes: [upsThreePhaseOutputFrequency]
        lookup: OutputFrequency
      - source_indexes: [upsThreePhaseOutputVoltageR]
        lookup: OutputVoltageR
      - source_indexes: [upsThreePhaseOutputVoltageS]
        lookup: OutputVoltageS
      - source_indexes: [upsThreePhaseOutputVoltageT]
        lookup: OutputVoltageT
      - source_indexes: [upsThreePhaseOutputLoadPercentageR]
        lookup: OutputLoadPercentageR
      - source_indexes: [upsThreePhaseOutputLoadPercentageS]
        lookup: OutputLoadPercentageS
      - source_indexes: [upsThreePhaseOutputLoadPercentageT]
        lookup: OutputLoadPercentageT
      - source_indexes: [upsConfigOutputVA]
        lookup: ApparentPowerVA
       
  fortigate_snmp:
    walk:
      # - ifXTable
      - fgSystem
      - fgVpn
      # - fgIntf
      # - fgInetProto
    version: 3
    max_repetitions: 25
    timeout: 180s
    auth:
      username: *********  # Required, no default. -u option to NetSNMP.
      security_level: authPriv  # Defaults to noAuthNoPriv. -l option to NetSNMP.
                                    # Can be noAuthNoPriv, authNoPriv or authPriv.
      password: ********  # Has no default. Also known as authKey, -A option to NetSNMP.
                      # Required if security_level is authNoPriv or authPriv.
      auth_protocol: MD5  # MD5 or SHA, defaults to MD5. -a option to NetSNMP.
                          # Used if security_level is authNoPriv or authPriv.
      priv_protocol: AES  # DES or AES, defaults to DES. -x option to NetSNMP.
                          # Used if security_level is authPriv.
      priv_password: ********* # Has no default. Also known as privKey, -X option to NetSNMP.
                                # Required if security_level is authPriv.

If I comment out either fgSystem or fgVpn, it scrapes properly but if both are uncommented then I get the error. Both these OIDs are in the same MIB file.





Here's the prometheus.yml file in case it's required:-




# Global config
global:
  scrape_interval:     1s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 1s # Evaluate rules every 15 seconds. The default is every 1 minute.
#   scrape_timeout: 15s  # scrape_timeout is set to the global default (10s).
# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.

rule_files:
  - prometheus_rules.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  # - job_name: 'prometheus'
  #   # metrics_path defaults to '/metrics'
  #   # scheme defaults to 'http'.
  #   static_configs:
  #   - targets: ['localhost:9090']

  - job_name: 'apc3p-snmp'
    static_configs:
      - targets:
        - xx.xx.xx.xx  # SNMP device.
    metrics_path: /snmp
    params:
      module: [XPPC-MIB]
      # module: [APC3P-MIB]
      # module: [XPPC-MIB]
      # module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.
        # replacement: 192.168.0.75:9116  # The SNMP exporter's real hostname:port.
 
  - job_name: 'fortigate-snmp'
    static_configs:
      - targets:
        - xx.xx.xx.xx # fortigate device.
    metrics_path: /snmp
    params:
      module: [fortigate_snmp]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # SNMP exporter.

Any help is much appriciated.

Thanks in advance.


[Brian Candler]

You have set scrape_interval: 1s.  This automatically sets the deadline to slightly less than 1 second (0.5 seconds I think).

Therefore my guess is that the total scrape time is slightly greater than 0.5 seconds when all OIDs are scraped, and slightly less when you comment one out.

This can easily be confirmed by increasing the scrape_interval to say 10 seconds, then look at the metrics "snmp_scrape_duration_seconds" (which comes from the exporter) and "scrape_duration_seconds" (which comes from prometheus itself).

[sabarish narayanan]

Thank you so much. You hit the nail on the head. That was the exact problem. When I increased the scrape_interval it scraped properly.