alertmanager HA

[nina guo]

Hi guys,

I'm implementing alertmanager HA.

alertmanager0:

-----------------------------------------

containers:

      - name: alertmanager

        image: quay.io/prometheus/alertmanager:latest

        args:

          - "--config.file=/etc/alertmanager/config.yml"

          - "--storage.path=/alertmanager"

          - "--cluster.listen-address=0.0.0.0:9094" 

-----------------------------------------------

alertmanager1:

--------------------------------------------

containers:

      - name: alertmanager

        image: quay.io/prometheus/alertmanager:latest

        args:

          - "--config.file=/etc/alertmanager/config.yml"

          - "--storage.path=/alertmanager"

          - "--cluster.listen-address=0.0.0.0:9095"

          - "--cluster.peer=internal_ip_of_alertmanager0:9094"        

alertmanager service:

-------------------------------------------------

apiVersion: v1

kind: Service

metadata:

  name: alertmanager

  namespace: monitoring

spec:

  selector:

    app: alertmanager

  type: NodePort

  ports:

    - port: 9093

      targetPort: 9093

      nodePort: 31000

I have a question that if alertmanager0 restarts, then the internal ip will be changed. So I may have to update the deployment file and re-apply again.

May I ask if there is a way to assign a DNS something like this to the internal IP address?

[sayf eddine Hammemi]

Hello, you can make use of Kubernetes Statefulset instead of deployment in this case

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1cabcbca-7905-4337-807e-33b7359685c7n%40googlegroups.com.

[nina guo]

May I ask what's the difference once I change to Statefulset?

[sayf eddine Hammemi]

The statefulset keeps track of the pods (hence the word stateful), they will get assigned the same name and they will keep the same hostname every time they die and come back, 
ie: a statefulset named foobar with two pods will always have two pods named foobar-0 and foobar-1 
combined with a headless service, the pods are reachable at foobar-{0/1}.<service_name>.<namespace>.svc.cluster.local

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fb12fdf7-0bb1-4670-b9d1-312d6ebd481an%40googlegroups.com.

[nina guo]

Thank you, great, I resolved the issue once I change to StatefulSet.

[nina guo]

one more question is whether the alert manager can be accessed by the following way in Prometheus configmap?

alerting:

      alertmanagers:

      - scheme: http

        static_configs:

        - targets: 

          - alertmanager0-0.alertmanager0.monitoring.svc.cluster.local:9093

          - alertmanager1-0.alertmanager1.monitoring.svc.cluster.local:9093