Prometheus + SNMP-Exporter, Relabeling if_mib:ifAlias

[Luke O'Donnell]

Hi All,

I'm new to the stack so apologies if this is a silly question.

I'm intending to use Prometheus + SNMP-Exporter to collect stats from a large number of network devices (switches, routers, etc) using if_mib.

An issue i'm having difficulty with is extracting a subset of the "ifAlias" label returned by SNMP-Exporter. These represent the "description" of network interfaces on networking gear.

A subset of network interfaces are used for Services, which have a well-known regex pattern. Unfortunately there is some degree of inconsistency in the exact format of Interface descriptions, so regex-like matching is required to pick out just the ServiceID. For example sake:
  Pattern: /Z[0-9]{5}A/
  Example ifAlias values:
     Z12345A
     Svc Z12345A

     Service Z12345A

I'm trying to create a new label called "Service" that for all of the above 3 examples would just contain "Z12345A".

Example data from snmp-collector:

ifHCInOctets{ifAlias="Svc Z12345A",ifDescr="TenGigabitEthernet0/0/4.ServiceInstance.1111",ifIndex="9",ifName="Te0/0/4.SI.11111"} 408

I'm intending to build various dashboards (grafana), alerting etc that requires the matching of the ServiceID. It therefore seems to make sense to me to store these as a new label whenever there is a match. It could be done at query-time, however there are many use-cases for using this (hypothetical) label so it'd reduce query complexity to just take the hit on storage and store it as a label.

I'm unsure of how to go about this. My interpretation of relabel_configs is that it can't be used for relabeling labels from scraped data, rather only labels provided at SD time. 

Is there an approach using either SNMP-Exporter config or Prometheus config that can achieve this? Or do i need to solve it either with some sort of custom intermediate service, or accept that there will be additional complexity (and performance hit) at query-time?

Thanks in advance,

Luke

[Ben Kochie]

"relabel_configs" are for target labels, which are applied before the scrape. You're looking for "metric_relabel_configs", which happen after the scrape.

Base example from the snmp_exporter + relabel to match your example regexp.

scrape_configs:
  - job_name: 'snmp'
    static_configs:
      - targets:
        - 192.168.1.2  # SNMP device.
        - switch.local # SNMP device.
        - tcp://192.168.1.3:1161  # SNMP device using TCP transport and custom port.
    metrics_path: /snmp
    params:
      auth: [public_v2]
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.
    metric_relabel_configs:
      - source_labels: [ifAlias]
        regex: ".*(Z[0-9]{5}A)"
        target_label: service_id

This is a common/standard thing to do.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ae94d201-e913-41b8-915c-69e09d5bcb6an%40googlegroups.com.

[Luke O'Donnell]

Hi Ben

That's done it, thanks for the assist. I must have been blind missing that in the doco.

Luke