Service Discovery of Alertmanagers?

[Dave Cadwallader]

I'm using EC2 service discovery to find nodes and it's working great! 

Wondering if it's also possible to use service discovery to find alertmanager instances for my alerting config, so that I could replace the hardcoded hosts below with discovered hosts.  If it is possible, is it a good/bad idea?  I'm assuming that hardcoded hostnames are always good candidates for being replaced with service discovery.  But if that's not the case I'd love to hear more about the philosophy.  Thanks!

alerting:

alert_relabel_configs:

- source_labels: [dc]

regex: (.+)\d+

target_label: dc

alertmanagers:

- static_configs:

- targets: ['alertmanager1.company.com:9093', 'alertmanager2.company.com:9093']

[Tom Wilkie]

Yes it is possible - in fact, in Prometheus 2 it'll be the only way!

The config is literally identical to service discovery, for instance on Kubernetes I do:

alerting:
  alertmanagers:
  - path_prefix: /admin/alertmanager
    kubernetes_sd_configs:
      - role: pod
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_name]
      regex: alertmanager
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      regex: default
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      regex:
      action: drop

Docs are here: https://prometheus.io/docs/operating/configuration/#<alertmanager_config>

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8f090647-d9df-412e-823b-2eff64a76331%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sundeep kumar]

Hi Tom,

I used the config for alertmanager shared by you here, however prometheus is not showing the alertmanager.

When i click on Status-> Runtime and build information from the prometheus it shows none in the alertmanager.

Alertmanagers

Endpoint

Can you guide me as where i need to look for the trouble .

I am using kubernetes engine provided by Google Cloud . K8 version is 1.8 .

[Tom Wilkie]

My example explicitly looks for alertmanager pods in the default namespace with a "name" _label_ set to "alertmanager".  Unless you run your pods like that, it won't work.  Whats the output of "kubectl get pod -n default -l name=alertmanager"?

Thanks

Tom

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e94217e4-66af-42ba-8c42-85c79982a9f4%40googlegroups.com.

[sundeep kumar]

Hi Tom,

I am running the prometheus stack in a different namespace called "monitoring" . Please advice me on what need to be changed to the service discovery part .

Please find the output below . 

kubectl get pod -n default -l name=alertmanager

No resources found.

kubectl get pod -n monitoring -l name=alertmanager

No resources found.

kubectl get pods -n monitoring

NAME                                    READY     STATUS             RESTARTS   AGE

alertmanager-4247043424-mf4hl           1/1       Running            0          1m

node-exporter-mgvtf                     1/1       Running            0          1m

node-exporter-vclnd                     1/1       Running            0          1m

node-exporter-vtkk5                     1/1       Running            0          1m

prometheus-deployment-430511021-jr9j7   0/1       CrashLoopBackOff   3          1m

[Tom Wilkie]

Hi Sundeep

You need to change the line `regex: default` to `regex: monitoring`.  You may find it easier to go with static service discovery, an example of which is in the Dave's email that began this thread.

Thanks

Tom

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0104fb25-524f-4b57-ab68-38497d71bf09%40googlegroups.com.

[kold...@gmail.com]

Wonder if it's possible to make Prometheus send alerts to different Alertmanagers based on Alert labels.

On Friday, December 15, 2017 at 3:28:25 AM UTC-8, Tom Wilkie wrote:

Hi Sundeep

You need to change the line `regex: default` to `regex: monitoring`.  You may find it easier to go with static service discovery, an example of which is in the Dave's email that began this thread.

Thanks

Tom

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Simon Pasquier]

On Wed, May 15, 2019 at 4:46 PM <kold...@gmail.com> wrote:
>
> Wonder if it's possible to make Prometheus send alerts to different Alertmanagers based on Alert labels.

You can use the keep and drop actions in alert_relabel_configs for this.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f571886c-f289-4918-aca9-a320ea0a47a9%40googlegroups.com.