mTLS between AlertManager and Prometheus

[Tamas Neumer]

Hi!

I'm currently working on putting the monitoring stack (kube-prometheus) into an Istio-mesh. Later the goal would have been to enforce mTLS between all components.

My first attempt was to secure the connection between AlertManager and Prometheus. I have added the Istio sidecar (as per https://istio.io/latest/docs/ops/integrations/prometheus/#tls-settings) and mounted the certificates to the Prometheus pod.

Then I found this repo (https://github.com/uvw/kube-prometheus-stack-istio) and tried the following:

1) Configured Prometheus to use tls certificates and https when sending alerts to alertmanager.

spec:

   alerting:

     alertmanagers:

     - apiVersion: v2

     name: alertmanager-main

     namespace: monitoring

     pathPrefix: /

     port: web

     scheme: https

     tlsConfig:

       caFile: /etc/prom-certs/root-cert.pem

       certFile: /etc/prom-certs/cert-chain.pem

       insecureSkipVerify: true

       keyFile: /etc/prom-certs/key.pem

In this case prometheus logs something like

level=error ts=2021-09-01T10:22:39.820Z caller=notifier.go:527 component=notifier alertmanager=https://172.17.0.8:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"https://172.17.0.8:9093/api/v2/alerts\": http: server gave HTTP response to HTTPS client"

Then I noted that the author of the repo sets the "appProtocol" field to "http" on the K8s Service to ix the Istio protocol selection.

In this case I saw the following logs:

level=error ts=2021-09-01T10:22:51.633Z caller=notifier.go:527 component=notifier alertmanager=https://172.17.0.11:9093/api/v2/alerts count=3 msg="Error sending alert" err="Post \"https://172.17.0.11:9093/api/v2/alerts\": EOF"

Has anyone had any luck w.r.t. establishing mTLS between Prometehus and Alertmanager?

[Julien Pivotto]

Hello,

It seems that istio might only support HTTP/2, which has been disabled
in Prometheus because of several issues in the golang HTTP/2 stack.

We are willing to re-introduce this in the future for all users, and
have introduce a hidden environment variable to re-enable. This is a
temporary solution, that will be hopefully removed in the future:

If you set the env variable PROMETHEUS_COMMON_ENABLE_HTTP2 (to any value)
you can see if that fixes your issue.

> <https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection>
> .

>
> In this case I saw the following logs:
>
> level=error ts=2021-09-01T10:22:51.633Z caller=notifier.go:527
> component=notifier alertmanager=https://172.17.0.11:9093/api/v2/alerts
> count=3 msg="Error sending alert" err="Post
> \"https://172.17.0.11:9093/api/v2/alerts\": EOF"
>
> Has anyone had any luck w.r.t. establishing mTLS between Prometehus and
> Alertmanager?
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a6e7cff7-7779-4ad7-911e-e139e37f3337n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Tamas Neumer]

Hi!

I guess I will follow up on this and use the env-var in the mean time. https://github.com/prometheus/prometheus/issues/9068

Thank you!