Prometheus Authentication

208 views
Skip to first unread message

Siradj Eddine Fisli

unread,
Jan 29, 2024, 5:45:12 AM1/29/24
to Prometheus Users
I have two prometheus instances , one is in agent mode remote writing metrics to the second one, i want to add authentication mechanism, also i am using kube-prometheus-stack. is there any solution ? 
also prometheus is accessible via https, i configured that using cert-manager and letsencrypt.

Brian Candler

unread,
Jan 29, 2024, 6:24:32 AM1/29/24
to Prometheus Users
Using --web.config-file you can make Prometheus require HTTP Basic Authentication (basic_auth_users) or TLS client certificate authentication (client_auth_type, client_ca_file, client_allowed_sans).

If you want this to happen only for certain endpoints like remote_write, then you'll need to bind prometheus to 127.0.0.1 and run a reverse proxy in front of it with whatever authorization policy you want.

Siradj Eddine Fisli

unread,
Jan 29, 2024, 7:35:00 AM1/29/24
to Prometheus Users
Actually i am using ingress-nginx to expose prometheus endpoint , shall i use nginx controller auth ? but what should i pass as argument to remote_write ?

Brian Candler

unread,
Jan 29, 2024, 9:10:29 AM1/29/24
to Prometheus Users
See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

There are settings for "authorization", "basic_auth" and "tls_config" that can be used to enable authentication to the remote_write endpoint.

Siradj Eddine Fisli

unread,
Jan 29, 2024, 9:13:44 AM1/29/24
to Brian Candler, Prometheus Users
---
$ htpasswd -c auth prometheus

$ kubectl create secret generic basic-auth --from-file=auth

apiVersion: v1
data:
  auth: XXXXXXXXXXXXX==
kind: Secret
metadata:  
  name: basic-auth
  namespace: monitoring
type: Opaque

----
ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: basic-auth

agentprom side:

    remoteWrite:
      - url: https://endpoint.com
        basicAuth:
          username:
            name: basic-auth
            key: auth

now when i enter the endpoint using browser it asks for the username and password, but i see this in prometheus-agent logs:

url=https://endpoint.com msg="non-recoverable error" count=417 exemplarCount=0 err="server returned HTTP status 401 Unauthorized: <html>"

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/49b22ab5-97ba-4fcb-9229-837d15c3f80en%40googlegroups.com.

Siradj Eddine Fisli

unread,
Jan 29, 2024, 9:24:48 AM1/29/24
to Brian Candler, Prometheus Users
now i'am getting this : 
failed to get basic auth password: unable to get secret \"\": resource name may not be empty"

Siradj Eddine Fisli

unread,
Jan 29, 2024, 9:31:27 AM1/29/24
to Brian Candler, Prometheus Users
Note that I followed these steps : https://kubernetes.github.io/ingress-nginx/examples/auth/basic/ ,
So that the secret has a key (data.auth), otherwise the ingress-controller returns a 503.

Le lun. 29 janv. 2024 à 15:10, 'Brian Candler' via Prometheus Users <promethe...@googlegroups.com> a écrit :
--

Siradj Eddine Fisli

unread,
Jan 29, 2024, 2:44:35 PM1/29/24
to Brian Candler, Prometheus Users
Solution:
- Created new secret with username and password cause the one created for the ingress must have data.auth otherwise it ll return error 503.    

remoteWrite:
      - url: https://endpoint/api/v1/write
        basicAuth:
          username:
            name: <secret-name>
            key: username
          password:
            name: <secret-name>
            key: password
Reply all
Reply to author
Forward
0 new messages