Prometheus Operator and auto discovery of AlertManagers

[Richard Moe]

Hi!

I am struggling a bit to figure out how to correctly configure autodiscovery of AlertManagers in a Kubernetes cluster.

Anyone have a fully working example Prometheus custom resource of this?

Just putting example config from Prometheus doc doesn't seem to work as the resource
expects a list of alertmanager endpoints (https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#alertmanagerendpoints)

Example config not working:

alerting:

  alertmanagers:

  - path_prefix: /admin/alertmanager

    kubernetes_sd_configs:

      - role: pod

    tls_config:

      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

    relabel_configs:

    - source_labels: [__meta_kubernetes_pod_label_name]

      regex: alertmanager

      action: keep

    - source_labels: [__meta_kubernetes_namespace]

      regex: default

      action: keep

    - source_labels: [__meta_kubernetes_pod_container_port_number]

      regex:

      action: drop

Thanks :)

[Richard Moe]

Answering my own mail:)

Autodiscovery of AlertManagers is not supported in the Prom custom resource. It
requires a custom configuration.
https://github.com/coreos/prometheus-operator/blob/master/Documentation/custom-configuration.md