Specif alerts do not work when using instance with special characters

53 views
Skip to first unread message

Frédéric BIERRY

unread,
Jul 21, 2021, 8:20:38 AM7/21/21
to Prometheus Users
Hello all !

I am using prometheus and alert manager for more than one year now and totally appreciated it.
I'm trying to override default alert email by creating specific routes like this :

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'email'
  routes:
  - match_re:
      alertname: WEBLOGIC-wls_threadpool_hogging_thread_count
      instance: (ljade)*
    receiver: mailJADE
  - match_re:
      alertname: LINUX-node_filesystem_nfs_used_percent
      instance: ljade*
    receiver: mailJADE

receivers:
- name: 'email'
  email_configs:
  - to: mymail
- name: 'mailJADE'
  email_configs:
  - to: 'mymail, othersemail'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

The alerts arrive only on the main mail 'email' and never on 'mailJADE' while alerts with an instance name beginning with ljade are generated
I tried different syntax with parenthesis and also old version match deprecated without success.

Thanks in advance for your help

Regards
Frederic

Brian Candler

unread,
Jul 21, 2021, 12:53:24 PM7/21/21
to Prometheus Users
  - match_re:
      alertname: WEBLOGIC-wls_threadpool_hogging_thread_count
      instance: (ljade)*

That regular expession matches only:

""
"ljade"
"ljadeljade"
"ljadeljadeljade"  ... etc

 - match_re:
      alertname: LINUX-node_filesystem_nfs_used_percent
      instance: ljade*


That regular expression matches only:

"ljad"
"ljade"
"ljadee"
"ljadeee"   ... etc

To match any string beginning ljade, you need:

      instance: 'ljade.*'

(The quotes may not always be necessary but are a good idea, since YAML can do some odd things with special symbols in certain situations, particularly colons and asterisks)

Ian Billett

unread,
Jul 22, 2021, 5:54:49 AM7/22/21
to Brian Candler, Prometheus Users
Hey Fredeic,

When folks raise questions about unexpected routing in their alertmanager routing trees, I always recommend that you try amtool, specifically the amtool config routes test command.

Best,

Ian

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/acdbfca0-00b2-4232-8cae-5f6fc98b2fbdn%40googlegroups.com.

Frédéric BIERRY

unread,
Jul 22, 2021, 11:36:54 AM7/22/21
to Prometheus Users
Big thanks !
It works 👍

Frédéric BIERRY

unread,
Jul 22, 2021, 11:38:04 AM7/22/21
to Prometheus Users
Hi Ian
yes i know this tool but it didn't notice anything wrong in my syntax

Ian Billett

unread,
Jul 22, 2021, 12:25:54 PM7/22/21
to Frédéric BIERRY, Prometheus Users
Frédéric,

My advice was not to use the tool to see if you configuration was valid, though that is a very useful feature.

The suggestion was to use the 'amtool config route test' command - and provide a set of labels that represent your alert, then the tool will show you which receivers it matches. This is a powerful tool in identifying logical bugs in your configuration like you experienced above.

Best,

Ian 

Reply all
Reply to author
Forward
0 new messages