Alert Manager Configuration Issue

[Azhar Iqbal]

Hi 

I am having an issue with Alert manager in particular the yml config. When i use the following alert manager yml configuration I have no issues

global:

 resolve_timeout: 1m

 slack_api_url: 'Removed Temp'

route:

 receiver: 'slack-notifications'

receivers:

- name: 'slack-notifications'

  slack_configs:

   - channel: '#critical-alerts'

     send_resolved: true

I have then tried to enhance the alert manager configuration with the expected outcomes

• Black hole for the alert stating out of hours
• If the alert is critical send to critical alerts receiver otherwise use the default receiver
  
• the inhibitions are intended to silence alerts that occur during the out of hours defined timing and to suppress warning alerts if a critical with the same name has been fired

global:

route:

  receiver: 'default_reciever'

  group_by: [alertname]

  routes:

  - receiver: blackhole

    match_re:

      alertname: "OutsideBusinessHours$"

  - receiver: 'critical_alert'

    group_by: [alertname]

    match:

      severity: critical

 

receivers:

- name: blackhole

- name: 'critical_alerts'

  slack_configs:

  - channel: '#critical-alerts'

    send_resolved: true

    api_url: 'REMOVED TEMP'

    text: '{{ template "slack.default.text" . }}'

    title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"

- name: 'default_reciever'

  slack_configs:

  - channel: '#standard-alerts'

    send_resolved: true

    api_url: 'REMOVED TEMP'

    title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"

    text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

inhibit_rules:

- source_match:

    alertname: OutsideBusinessHours

  target_match_re:

    alertname: .*

- source_match:

    severity: 'critical'

  target_match:

    severity: 'warning'

  equal: ['alertname', 'instance']

I am unable to run alert manager when using the enhanced config and receive an error (see attached). I have checked the yml parsing and the routing tree editor on the Prometheus website and both seem fine. I am not sure what is wrong with the enhanced alert manager configuration.

Can someone please advise?

[Brian Candler]

Run "journalctl -eu alertmanager", it should show the logs from alertmanager as to why it's not starting.

[azha...@googlemail.com]

Hi 

I have resolved this. Just for anyone who may want to refer to this I ran the following;

./alertmanager --config.file="path to alert manager yml"

which returned the following error. I had a typo in one of my receiver declarations....

level=error ts=2020-08-19T08:10:47.636Z caller=coordinator.go:124 component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/alertmanager.yml err="undefined receiver \"critical_alert\" used in route"

level=info ts=2020-08-19T08:10:47.637Z caller=cluster.go:632 component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=58.872855ms