Extracting Group Data String for Alert Grouping

[Brian Bowen]

Hi all,

We are attempting to set up alerting with Prometheus and Alertmanager using some SNMP data. The basic use case is that we would like to group by a substring of label data rather than an entire label. Let's say our interfaces have the ifAlias label in the following format:

ifAlias="<brief description> - device 1 port 5 to device 2 port 7 - <cable ID>" and I want to group alerts only by "device 1 port 5 to device 2 port 7" (assuming this description is consistent across  both devices), leaving the rest of the description and cableID out.

Is there a way to do this? We have not had success extracting this as a separate label through snmp_exporter. I thought potentially we could do some regex matching under the group_by rules with Alertmanager, but I haven't seen any documentation/examples showing how to do this either.

Let me know if there are any files I should attach.

[Ben Kochie]

If you have your ifAlias well standardized you can use metric_relabel_configs to extract data.

metric_relabel_configs:
- source_labels: [ifAlias]
  regex: "(.+) - (.+) - (.+)"
  replacement: "$1"
  target_label: port_description
- source_labels: [ifAlias]
  regex: "(.+) - (.+) - (.+)"
  replacement: "$2"
  target_label: port_location
- source_labels: [ifAlias]
  regex: "(.+) - (.+) - (.+)"
  replacement: "$3"
  target_label: cable_id

This will separate out your ifAlias into the component label parts.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/31b5a66a-0aa5-432b-b527-764ac392e1d4n%40googlegroups.com.

[Brian Candler]

Alternatively, I'm not sure about this, but I think you could just add these extra labels in your alerting rules.

Labels added there are templated, and there are various template functions available, including reReplaceAll:

https://prometheus.io/docs/prometheus/latest/configuration/template_reference/

However it would have to be repeated on every alerting rule where you wanted to do this sort of grouping.

[Brian Bowen]

Thanks to both of you. The regex matching with metric_relabel_configs worked for my scenario since ifAlias is well standardized. I had previously tried this but misunderstood how to get the regex matching. The example above pushed us in the right direction.

[jaouad zarrabi]

BullionStar is Singapore's Premier Bullion Dealer For Sell  : GOLD / SILVER / BARS / COINS
- Over 1,000 Different Products
-  Cash & Bullion Account
- Attractive Prices
- Quick & Easy
-Tax Free Bullion
- Financial Strength
- Global Reach
- Multi-Jurisdiction
https://www.bullionstar.com/?r=27869