Prometheus AlertManager filter.
839 views
Skip to first unread message
Zhang Zhao
Jul 14, 2020, 1:22:08 PM7/14/20
to Prometheus Users
I added a filter in the alertmanager config so that only alerts that contain "inc:servicenow" label are able to be fed to ServiceNow. However it didn't work as expected. I still saw events that do not contain this label getting fed to ServiceNow. Below was my config. Please advice where was wrong. Appreciate it.
global:
resolve_timeout: 15m
receivers:
- name: prometheus-snow
webhook_configs:
- url: "https://servicenow.com"
http_config:
basic_auth:
username: "xxx"
password: "xxx"
route:
group_by: ['...']
group_interval: 15m
group_wait: 15m
repeat_interval: 1h
receiver: prometheus-snow
routes:
- receiver: "prometheus-snow"
match:
inc: servicenow
Zhang Zhao
Jul 14, 2020, 1:24:20 PM7/14/20
to Prometheus Users
Sorry about the background color in the code section.. Repost the code.
global:
resolve_timeout: 15m
receivers:
- name: prometheus-snow
webhook_configs:
- url: "https://servicenow.com"
http_config:
basic_auth:
username: "xxx"
password: "xxx"
route:
group_by: ['...']
group_interval: 15m
group_wait: 15m
repeat_interval: 1h
receiver: prometheus-snow
routes:
- receiver: "prometheus-snow"
match:
inc: servicenow
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7eaaca80-ddf6-4637-8217-99404763c2eeo%40googlegroups.com.
Christian Hoffmann
Jul 14, 2020, 1:36:17 PM7/14/20
to Zhang Zhao, Prometheus Users
Hi,
On 7/14/20 7:24 PM, Zhang Zhao wrote:
>> I added a filter in the alertmanager config so that only alerts that
>> contain "inc:servicenow" label are able to be fed to ServiceNow.
>> However it didn't work as expected. I still saw events that do not
>> contain this label getting fed to ServiceNow. Below was my config.
>> Please advice where was wrong. Appreciate it.
>>
[...]
>> receiver: prometheus-snow
^^^^^^^^^^
want to null-route everything which doesn't match, you can define a
receiver without any details (such as "devnull") and use that as a default.
Another option would be dropping all irrelevant alerts from ever
reaching alertmanager by using alert relabelling.
Kind regards,
Christian
On 7/14/20 7:24 PM, Zhang Zhao wrote:
>> I added a filter in the alertmanager config so that only alerts that
>> contain "inc:servicenow" label are able to be fed to ServiceNow.
>> However it didn't work as expected. I still saw events that do not
>> contain this label getting fed to ServiceNow. Below was my config.
>> Please advice where was wrong. Appreciate it.
>>
>> receiver: prometheus-snow
^^^^^^^^^^
>> routes:
>> - receiver: "prometheus-snow"
>> match:
>> inc: servicenow
You are still setting prometheus-snow as your default receiver. If you
>> - receiver: "prometheus-snow"
>> match:
>> inc: servicenow
want to null-route everything which doesn't match, you can define a
receiver without any details (such as "devnull") and use that as a default.
Another option would be dropping all irrelevant alerts from ever
reaching alertmanager by using alert relabelling.
Kind regards,
Christian
Zhang Zhao
Jul 14, 2020, 1:41:31 PM7/14/20
to Christian Hoffmann, Prometheus Users
Hi Christian,
Thank you for your quick reply. I thought the match stanza does the filter, it didn’t work that way? And how do I set a receiver as default?
Zhang
Thank you for your quick reply. I thought the match stanza does the filter, it didn’t work that way? And how do I set a receiver as default?
Zhang
Zhang Zhao
Jul 14, 2020, 1:50:08 PM7/14/20
to Christian Hoffmann, Prometheus Users
Hi Christian,
Based on your advice, I added a “devnull” receiver and set it as default as below. Does it look correct?
global:
resolve_timeout: 15m
receivers:
- name: devnull
- name: prometheus-snow
webhook_configs:
- url: “https://
http_config:
basic_auth:
username: “xx”
password: “xx”
route:
group_by: ['...']
group_interval: 15m
group_wait: 15m
repeat_interval: 1h
receiver: devnull
routes:
- receiver: "prometheus-snow"
match:
inc: servicenow
Zhang
On Jul 14, 2020, at 10:36 AM, Christian Hoffmann <ma...@hoffmann-christian.info> wrote:
Zhang Zhao
Jul 14, 2020, 2:45:59 PM7/14/20
to Christian Hoffmann, Prometheus Users
Hi Christian,
After I updated the config below, seems everything stopped feeding to ServiceNow even the ones with “inc:servicenow” label.. Any idea?
global:
resolve_timeout: 15m
receivers:
- name: devnull
- name: prometheus-snow
webhook_configs:
- url: “https://
http_config:
basic_auth:
username: “xx”
password: “xx”
route:
group_by: ['...']
group_interval: 15m
group_wait: 15m
repeat_interval: 1h
receiver: devnull
routes:
- receiver: "prometheus-snow"
match:
inc: servicenow
Zhang
Christian Hoffmann
Jul 14, 2020, 4:48:14 PM7/14/20
to Zhang Zhao, Prometheus Users
On 7/14/20 8:45 PM, Zhang Zhao wrote:
> Hi Christian,
> After I updated the config below, seems everything stopped feeding to
> ServiceNow even the ones with “inc:servicenow” label.. Any idea?
Hrm, your config looks fine to me. Can you show us an example alert
> Hi Christian,
> After I updated the config below, seems everything stopped feeding to
> ServiceNow even the ones with “inc:servicenow” label.. Any idea?
definition which is not routed correctly?
Kind regards,
Christian
Reply all
Reply to author
Forward
0 new messages