Alerting in Prometheus federated environment.

[Dennis Murtic]

Hi all,

Currently we have different environments where Prometheus and Alertmanager are used. These environments are also protected by certain network settings, so we need to use PushProx. To collect data from all environments, we use a "global" Prometheus server that federates with the environments Prometheus server.

Alarms are emailed directly from the environments. This means that the alarm rules are also configured per environment. Now we would like to send the alerts from all environments to one of our Slack channels as well. Since the environments are protected by certain network settings, it is not possible to send the alarms from the environments directly to our Slack channel. One of our ideas is to configure an alert in the "global" alert manager that looks like this:

- alert: AlertName

expr: ALERTS{alertname!="AlertName",alertstate="firing",severity="critical"}

labels:

severity: critical

annotations:

alertname: '{{ $labels.alertname }}'

summary: Alert {{ $labels.alertname }} fired.

This seems to work, but unfortunately this is not the best solution as alarms often switch between firing/resolved state and also the summary is not available here. 

Does anyone have any ideas on how we can improve this? 

Thanks! 

[Julius Volz]

Hi,

Indeed that's not ideal and annotations and such will only be available in the source Prometheus servers (and then pushed to the Alertmanagers). I'm not sure what exactly your network constraints are, but they seem to prohibit Alertmanager in each location talking directly to Slack. Would they also prohibit each location's Alertmanager pushing additionally to a global Alertmanager? Because if that's possible, you could point each location's Prometheus to both a local and a global Alertmanager, and both would receive all alerting info, and would be able to route alerts differently.

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAFERo3qk1zjB3krqNePeomzgvBXdVzTyTVd%3DkD9SbCJuM3WWhA%40mail.gmail.com.

--

Julius Volz

PromLabs - promlabs.com

[Julius Volz]

s/each location's Alertmanager/each location's Prometheus/

[Julius Volz]

Another option: if you cannot reach Slack from the individual environments, but you can provision some other global endpoint that would be reachable from them, you could have the local Alertmanagers send all of their alerts to a webhook receiver (https://prometheus.io/docs/alerting/latest/configuration/#webhook_config), which you implement on the other side, and which can then do anything. Or alternatively, for the Slack receiver, you can actually configure the "api_url", and maybe you can point that to a reachable proxy address of yours that can then simply proxy the request to Slack?