Alert manager cannot be connect

[nina guo]

Hi,

I used the following files to deploy alertmanager, but it cannot be communicated.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager1
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alertmanager1
  template:
    metadata:
      name: alertmanager1
      labels:
        app: alertmanager1
    spec:
      containers:
      - name: alertmanager
        image: prometheus/alertmanager:latest
        args:
          - "--config.file=/etc/alertmanager/config.yml"
          - "--storage.path=/alertmanager"
          - "--web.external-url=http://localhost:9093/alertmanager1/"
          - "--log.level=debug"
        ports:
          - containerPort: 9093
        volumeMounts:
        - name: config-volume
          mountPath: /etc/alertmanager
        imagePullPolicy: Always
      imagePullSecrets:
        - name: regcred
      volumes:
        - name: config-volume
          configMap:
            name: alertmanager-config
---
apiVersion: v1
kind: Service
metadata:
  name: alertmanager1
  namespace: monitoring
spec:
  selector:
    app: alertmanager1
  ports:
    - port: 9093
      targetPort: 9093

alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets: ['alertmanager1.monitoring.svc:9093']

[nina guo]

level=error ts=2022-05-13T13:27:59.353Z caller=notifier.go:527 component=notifier alertmanager=http://alertmanager1:9093/alertmanager/api/v2/alerts count=19 msg="Error sending alert" err="bad response status 404 Not Found"

[nina guo]

alerting:
      alertmanagers:
      - scheme: http

        path_prefix: "/alertmanager/"

        static_configs:
        - targets: ['alertmanager1.monitoring.svc:9093']

[Brian Candler]

In web.external-url, you configured the path prefix to "/alertmanager1/" not "/alertmanager/", this is why you get the 404.

In my configs I omit the trailing slash from both of these.

[nina guo]

Thank you. I'm deploying 2 alertmanagers with 2 separate deployment files. So I have to specify 2 different 
--web.external-url=http://localhost:9093/alertmanager1/

--web.external-url=http://localhost:9093/alertmanager2/

And then configure as below, am I right?

alerting:
      alertmanagers:
      - scheme: http

        path_prefix: "/alertmanager1/"

        static_configs:
        - targets: ['alertmanager1.monitoring.svc:9093']

alerting:
      alertmanagers:
      - scheme: http

        path_prefix: "/alertmanager1/"

        static_configs:
        - targets: ['alertmanager1.monitoring.svc:9093']

Why I configured web external url is because I setup ingress in our environment, In order to access alertmanager, I have to specify web external url.

[Brian Candler]

On Monday, 16 May 2022 at 09:05:28 UTC+1 ninag...@gmail.com wrote:

Thank you. I'm deploying 2 alertmanagers with 2 separate deployment files. So I have to specify 2 different 
--web.external-url=http://localhost:9093/alertmanager1/

--web.external-url=http://localhost:9093/alertmanager2/

Only if you are running two alertmanagers on the same host.  Why do you want to do that?

 

And then configure as below, am I right?

alerting:
      alertmanagers:
      - scheme: http

        path_prefix: "/alertmanager1/"

        static_configs:
        - targets: ['alertmanager1.monitoring.svc:9093']

alerting:
      alertmanagers:
      - scheme: http

        path_prefix: "/alertmanager1/"

        static_configs:
        - targets: ['alertmanager1.monitoring.svc:9093']

No, unless those two "alerting" sections are present in two different prometheus servers.

If you want a single prometheus to send all alerts to both alertmanagers, then you'd have a single "alerting" section with two targets. Unfortunately the "path_prefix" is common to all targets, so you would also have to configure alert_relabel_configs to set the __alerts_path__ label separately for each target.

It's possible, but I don't understand what you're actually trying to achieve here, and therefore I suspect there's a much better and simpler way.

[nina guo]

Thank for your reply.

on Prometheus Pod 1:

alerting:
      alertmanagers:
      - scheme: http

        path_prefix: "/alertmanager1/"

        static_configs:
        - targets: ['alertmanager1.monitoring.svc:9093']

On Prometheus Pod 2:

alerting:
      alertmanagers:
      - scheme: http

        path_prefix: "/alertmanager1/"

        static_configs:
        - targets: ['alertmanager1.monitoring.svc:9093']

I just want to implement when one alertmanager is failed, there is another redundancy one is working well. 

So may I ask what's the difference between totally 2 different deployments and 1 deployment with 2 replicas?

[Brian Candler]

On Tuesday, 17 May 2022 at 02:18:58 UTC+1 ninag...@gmail.com wrote:

I just want to implement when one alertmanager is failed, there is another redundancy one is working well. 

What you want is alertmanager clustering.

Create two separate alertmanagers in separate hosts (or pods). They can use the same path prefix '/' or '/alertmanager/' as appropriate, since they have separate hostnames (or service names).

Configure prometheus to send alerts to both alertmanagers, by listing them both in the 'targets' section:

    alerting:
      alertmanagers:

        - static_configs:
          - targets: ['alertmanager1.monitoring.svc:9093', 'alertmanager2.monitoring.svc:9093']

 

Configure the two alertmanagers to gossip to each other with the --cluster.* flags. For more information, please read the documentation:

https://prometheus.io/docs/alerting/latest/alertmanager/#high-availability

https://github.com/prometheus/alertmanager#high-availability

> So may I ask what's the difference between totally 2 different deployments and 1 deployment with 2 replicas?

That's nothing to do with prometheus or alertmanager. Those are only kubernetes concepts.